Senior Data Engineer -Databricks

Position Summary
We are seeking Senior Data Engineers with deep hands-on Databricks experience to build and maintain modern data platforms supporting complex enterprise and government programs.
This is a hands-on engineering role. You will design and implement data pipelines and lakehouse solutions, contribute to architectural decisions, review your peers' code, and work directly with clients on technical solutions. You'll partner with architects and the practice lead on standards and direction, and mentor less experienced engineers on the team.
Key Responsibilities Databricks Engineering

• Build scalable, secure data solutions on the Databricks Lakehouse Platform using Apache Spark, Delta Lake, and cloud-native services.
• Implement batch, streaming, ETL, ELT, analytics, and ML-enabled data pipelines.
• Apply medallion architecture patterns across ingestion, transformation, orchestration, and consumption layers.
• Implement data governance in Unity Catalog lineage, access controls, auditing, metadata, and secure sharing following established standards.
• Configure clusters, serverless compute, and workloads for performance, reliability, and cost efficiency.
• Develop with PySpark, Spark SQL, Python, Delta Live Tables, Structured Streaming, Auto Loader, MLflow, and Databricks Workflows.
• Integrate Databricks with Azure, AWS, or Google Cloud services.
• Contribute to migrations and modernization from legacy databases, data warehouses, Hadoop, and traditional ETL platforms.
• Follow and help improve engineering practices for source control, automated testing, CI/CD, infrastructure as code, monitoring, and production support.
• Participate in architecture reviews, code reviews, technical assessments, and root-cause analysis.
• Troubleshoot production issues and tune underperforming Spark workloads and queries.

Collaboration and Technical Contribution

• Mentor junior and mid-level engineers through pairing, code review, and knowledge sharing.
• Translate business, functional, and security requirements into technical designs and working code.
• Contribute estimates and technical input to project plans, and flag risks and dependencies early.
• Document designs, pipelines, and operational runbooks.
• Build reusable components, templates, and accelerators the wider team can adopt.
• Participate in client meetings as a technical contributor; support proposals, solution estimates, and demonstrations as needed.
• Communicate status, blockers, and technical trade-offs clearly to project leads and stakeholders.

Required Qualifications

• Bachelor's degree in computer science, information technology, data engineering, engineering, or a related discipline or equivalent practical experience.
• At least 6 years in data engineering, data architecture, or analytics engineering.
• At least 3 years of hands-on experience designing and implementing solutions on Databricks.
• Strong working experience with:
• Databricks Lakehouse Platform
• Apache Spark and PySpark
• Spark SQL and advanced SQL development
• Delta Lake and medallion architecture
• Unity Catalog and data governance concepts
• ETL and ELT pipeline design
• Batch and real-time data processing
• Data modeling and data warehousing
• Python-based data engineering
• Databricks Workflows, Jobs, and cluster management
• Experience deploying Databricks solutions on Azure, AWS, or Google Cloud.
• Experience with CI/CD, Git-based development, automated testing, and infrastructure as code.
• Demonstrated ability to optimize Spark workloads, cluster configurations, query performance, and cloud costs.
• Clear written and verbal communication, with the ability to explain technical concepts to non-technical audiences.
• Authorized to work in the United States or Canada.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional, or Databricks Certified Machine Learning Professional.
• Azure, AWS, or Google Cloud associate- or professional-level certification.
• Experience in consulting, professional services, systems integration, or managed services.
• Experience supporting federal, state, or local government clients.
• Familiarity with federal security, privacy, governance, and compliance requirements.
• Experience with Azure Data Factory, ADLS, Synapse, AWS Glue, S3, Snowflake, dbt, Kafka, Airflow, or Terraform.
• Experience with MLflow, MLOps, generative AI, Databricks Mosaic AI, or vector search.
• Experience working on distributed or fully remote engineering teams.

What Success Looks Like

• Pipelines and solutions that are scalable, secure, reliable, and well documented.
• Committed work delivered on schedule, with risks raised before they become escalations.
• Code and designs that raise the quality bar for the team.
• Efficient use of Databricks consumption and cloud infrastructure.

Adoption of shared architectures and reusable components across engagements