Senior Databricks Data Architect

Senior Databricks Data Architect

Important Notes

• Client: Ameren
• Role: Senior Databricks Data Architect
• Location: St. Louis, MO
• Work Arrangement: Hybrid — onsite in St. Louis required
• Must-Have: Databricks, Python, PySpark, and AWS or Azure
• Level: Seasoned / Senior Architect-level candidate
• Candidate should be comfortable working onsite in St. Louis, MO.

Job Summary

Ameren is looking for a seasoned Databricks Data Engineer / Architect to design and build scalable cloud data platforms and ETL/ELT pipelines. The ideal candidate will have strong hands-on expertise with Databricks, PySpark, Python, SQL, Delta Lake, Terraform, and cloud platforms such as Azure or AWS.

The role combines data engineering, cloud architecture, infrastructure automation, CI/CD, performance optimization, and data governance.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Python, PySpark, and Databricks.
• Build and maintain Databricks notebooks, workflows/jobs, Delta Lake tables, Unity Catalog, and Delta Live Tables.
• Implement Medallion Architecture using Bronze, Silver, and Gold layers.
• Optimize Spark/Databricks workloads for performance, scalability, reliability, and cost.
• Apply Spark optimization techniques including:
• Partitioning
• Data skew handling
• Caching
• Adaptive Query Execution (AQE)
• Cluster/resource optimization
• Develop complex and optimized SQL queries for transformation, validation, and analytics.
• Use Terraform to provision and manage Databricks and cloud infrastructure.
• Automate infrastructure including Databricks workspaces, clusters, jobs, storage, IAM, networking, and permissions.
• Develop and maintain CI/CD pipelines using Jenkins and GitHub.
• Manage Git branching, pull requests, code reviews, automated testing, and deployments.
• Integrate data from databases, APIs, files, streaming sources, and other enterprise systems.
• Implement data quality, governance, lineage, security, and compliance controls.
• Work with Delta Lake capabilities including ACID transactions, schema evolution, time travel, and access controls.
• Monitor and troubleshoot data pipelines and implement appropriate alerting and observability.
• Optimize cloud costs through autoscaling, scheduling, efficient cluster configurations, and spot instances where appropriate.
• Collaborate with engineering and architecture teams in an Agile environment.
• Participate in code reviews and establish data engineering best practices.

Must-Have Requirements

• Strong hands-on Databricks experience.
• Advanced Python skills.
• Strong PySpark experience with distributed data processing.
• Advanced SQL, including complex queries, window functions, and query optimization.
• Strong experience with Delta Lake and Lakehouse architecture.
• Experience with Databricks:
• Clusters
• Notebooks
• Workflows/Jobs
• Unity Catalog
• Delta Live Tables
• Strong AWS or Azure cloud experience.
• Hands-on Terraform / Infrastructure as Code experience.
• Experience with GitHub, branching, pull requests, and code reviews.
• Strong understanding of CI/CD, preferably with Jenkins.
• Experience with Spark/Databricks performance tuning.
• Strong understanding of data modeling and big-data architecture.
• Ability to work hybrid onsite in St. Louis, MO.

Preferred Cloud Skills

Azure

• Azure Data Lake Storage (ADLS)
• Azure Data Factory
• Azure Synapse
• Azure Key Vault
• Azure IAM/security

AWS

• Amazon S3
• AWS Glue
• Amazon EMR
• IAM
• AWS Lambda