Senior Databricks Engineer

Job Title: Databricks Engineer

Employment Type: Full Time

Location: Vienna, VA (Onsite)

Job Description

We are seeking an experienced Senior Databricks Engineer to design, develop, and optimize enterprise-scale data platforms on Databricks. The ideal candidate will have strong expertise in building scalable data pipelines, cloud data engineering, and modern Lakehouse architectures. This role requires working closely with data architects, analytics teams, and business stakeholders to deliver high-performance data solutions.

Required Skills

• 7+ years of experience in Data Engineering.
• 3+ years of hands-on experience with Databricks.
• Strong experience with Apache Spark (PySpark/Scala/Spark SQL).
• Experience building ETL/ELT pipelines using Databricks.
• Strong knowledge of Delta Lake, Unity Catalog, and Lakehouse architecture.
• Experience with Azure Databricks (preferred) or AWS Databricks.
• Strong SQL programming skills.
• Experience with Python and/or Scala.
• Experience with cloud storage technologies (ADLS, S3, Azure Blob).
• Knowledge of CI/CD, Git, Azure DevOps, or Jenkins.
• Experience with workflow orchestration tools such as Airflow or Databricks Workflows.
• Understanding of data modeling, performance tuning, and optimization.
• Experience with streaming technologies is a plus (Kafka, Structured Streaming).
• Strong communication and stakeholder management skills.

Preferred Qualifications

• Azure Data Engineer or Databricks certifications.
• Experience with Snowflake, Azure Synapse, or Microsoft Fabric.
• Experience implementing data governance and security best practices.
• Financial Services or Federal domain experience is a plus.

Responsibilities

• Design and develop scalable data pipelines using Databricks.
• Build and optimize Spark applications for large-scale data processing.
• Develop ETL/ELT workflows using PySpark and SQL.
• Implement Delta Lake best practices for data quality and performance.
• Collaborate with architects and business teams to define data solutions.
• Optimize Databricks clusters, jobs, and notebooks.
• Implement CI/CD pipelines and DevOps best practices.
• Monitor production workloads and troubleshoot performance issues.
• Ensure data security, governance, and compliance standards are met.
• Document technical solutions and mentor junior engineers.