Databricks Data Engineer :: Hybrid :: W2 only

Senior Databricks Engineer
Location: Boston, MA – Hybrid (locals preferred)
Experience: 10+Years
Position Overview
We are seeking an experienced Senior Databricks Engineer with strong hands-on expertise in Scala, Apache Spark, SQL, AWS, and data pipeline engineering. The ideal candidate will have experience designing, developing, and optimizing scalable data ingestion and ETL pipelines, preferably using Databricks.
The candidate must be comfortable with hands-on coding, data engineering problem-solving, and developing reliable, high-performance data processing solutions. Candidates must also be prepared for a coding exercise involving PySpark/Scala and SQL.

Key Responsibilities
·       Design, develop, and maintain scalable data ingestion pipelines, ETL workflows, and data processing jobs using Databricks and Apache Spark.
·       Develop and optimize Spark applications using Scala and PySpark.
·       Build robust data pipelines to ingest, transform, validate, and process structured and semi-structured data.
·       Write complex SQL queries for data transformation, validation, reconciliation, and performance optimization.
·       Configure and manage Databricks workspaces, clusters, jobs, and workflow scheduling.
·       Develop, schedule, monitor, and troubleshoot Databricks jobs and workflows.
·       Optimize Spark jobs and data pipelines for performance, scalability, reliability, and cost efficiency.
·       Integrate Databricks-based data solutions with AWS services and cloud-based data platforms.
·       Implement data quality checks, error handling, logging, and monitoring for production pipelines.
·       Troubleshoot pipeline failures, performance bottlenecks, and data processing issues.
·       Collaborate with data architects, analysts, engineers, and other stakeholders to translate business requirements into technical solutions.
·       Follow established coding standards, engineering best practices, and data governance requirements.
·       Support production deployments, maintenance, and continuous improvement of data engineering solutions.

Required Skills and Qualifications
·       9 years of overall experience in Scala/Spark data engineering and data pipeline design.
·       Strong hands-on experience with Scala and Apache Spark programming.
·       Strong SQL skills, including complex queries, joins, aggregations, and query optimization.
·       Hands-on experience designing and developing data ingestion pipelines, ETL pipelines, and data processing jobs.
·       Experience with AWS services and cloud-based data engineering solutions.
·       Strong understanding of distributed data processing, data transformations, and pipeline architecture.
·       Experience troubleshooting and optimizing Spark applications and data workflows.
·       Strong analytical, problem-solving, and communication skills.
·       Ability to participate in technical coding assessments and explain implementation decisions.

Preferred Skills
·       Hands-on experience with Databricks.
·       Experience configuring and managing Databricks workspaces and clusters.
·       Experience with Databricks job scheduling and workflow management.
·       Experience developing and maintaining production-grade ETL and ingestion pipelines in Databricks.
·       Familiarity with data quality, pipeline monitoring, and production support practices.