Senior Data Engineer (Databricks)

Senior Data Engineer (Databricks)  

12 Months contract – Extension   / Fulltime Job

Location:   Vienna, VA 22182

Interview : ASAP

Job description:

Job Summary

• We are seeking a highly skilled Senior Data Engineer with 10+ years of experience in designing and building scalable data platforms using Databricks, Apache Spark, and cloud technologies.
• The ideal candidate will have strong expertise in data engineering, ETL development, distributed data processing, and modern data architectures, along with exposure to Machine Learning and Generative AI solutions.

Required Skills

• 10+ years of experience in Data Engineering and Big Data technologies
• Hands-on experience with Databricks
• Strong expertise in Apache Spark (PySpark/Spark SQL)
• Strong programming experience in Python and SQL
• Experience with Scala (preferred)
• Experience designing and implementing enterprise-scale data architectures
• Expertise in building scalable ETL/ELT data pipelines
• Experience with Delta Lake, Unity Catalog, and Databricks Workflows
• Experience with cloud platforms (AWS, Azure, or Google Cloud Platform)
• Knowledge of data warehousing concepts and dimensional modeling
• Experience with CI/CD, Git, and Agile methodologies
• Strong analytical, problem-solving, and communication skills
• Preferred Skills
• Experience with Machine Learning pipelines
• Exposure to Generative AI and Databricks Mosaic AI
• Experience with MLflow
• Experience modernizing legacy data platforms
• Client-facing consulting experience

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build ETL/ELT solutions to ingest, transform, and process large-scale structured and unstructured data.
• Develop high-performance data solutions using Python, SQL, and Spark.
• Implement Delta Lake architecture for reliable and scalable data processing.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with data architects, data scientists, and business stakeholders to deliver data-driven solutions.
• Design and maintain enterprise data models and cloud-based data architectures.
• Support data migration and modernization initiatives to the Databricks platform.
• Implement data quality, governance, security, and monitoring best practices.
• Develop reusable frameworks and automation for data engineering workflows.
• Integrate Machine Learning and AI capabilities into enterprise data platforms.
• Troubleshoot production issues and optimize existing data pipelines.
• Participate in architecture discussions and recommend best practices for scalable data solutions.
• Work within Agile teams and contribute to sprint planning, code reviews, and continuous improvement.