Back to Jobs
Technology

Senior Databricks Developer (AWS)

Decision Six Inc.Malvern, PA🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationMalvern, PA, United States
SQLGitApache SparkAWSETLAirflowCloudFormationDatabricksGitHub ActionsJenkinsKafkaPythonRedshiftTerraformUnityAzureCI/CDAzure DevOpsDelta LakeIAM

Job description

Senior Databricks Developer (AWS)

We are seeking a Senior Databricks Developer with 7+ years of Data Engineering experience and deep expertise in Databricks, AWS, PySpark, Python, and SQL. This is a highly technical, hands-on engineering role within a data engineering team focused primarily on building, enhancing, and supporting enterprise data pipelines. The ideal candidate will have extensive experience developing cloud-native data solutions in AWS, implementing Delta Lake and Medallion architectures, and delivering scalable, production-grade data pipelines using Databricks. This is not a consulting role; we are looking for a developer who enjoys building and optimizing data platforms and solving complex data engineering challenges.

Responsibilities

  • Design, develop, and maintain large-scale ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
  • Build scalable and highly reliable data ingestion, transformation, and orchestration frameworks within AWS.
  • Develop and support Delta Lake implementations and Medallion Architecture (Bronze, Silver, Gold) data pipelines.
  • Create and optimize batch and streaming pipelines that process large volumes of structured and unstructured data.
  • Leverage AWS services including S3, Glue, Lambda, EMR, IAM, EventBridge, CloudWatch, Kinesis, and Redshift to support end-to-end data workflows.
  • Optimize Spark jobs, Delta tables, and Databricks workloads for performance, scalability, and cost efficiency.
  • Partner with Data Engineers, Architects, Analytics teams, and business stakeholders to deliver high-quality data solutions.
  • Implement data quality controls, monitoring, alerting, and operational support processes.
  • Troubleshoot and resolve complex production issues across Databricks and AWS environments.
  • Contribute to engineering best practices, code reviews, automation, and platform modernization initiatives.

Required Qualifications

  • 7+ years of Data Engineering experience building enterprise-scale data platforms and pipelines.
  • 5+ years of hands-on Databricks development experience in production environments.
  • Strong expertise in Python, PySpark, and SQL with a focus on pipeline development.
  • Extensive experience developing and supporting data pipelines within AWS.
  • Strong knowledge of Delta Lake, Lakehouse Architecture, and Medallion Architecture (Bronze, Silver, Gold).
  • Hands-on experience with AWS services such as S3, Glue, Lambda, EMR, Redshift, Kinesis, IAM, and CloudWatch.
  • Deep understanding of Spark architecture, distributed data processing, performance tuning, partitioning, and optimization.
  • Experience building highly scalable batch and streaming data processing solutions.
  • Strong understanding of data modeling, data warehousing concepts, and modern data architectures.
  • Experience supporting production environments, monitoring data pipelines, and troubleshooting complex data issues.

Preferred Qualifications

  • Experience with Databricks Workflows, Delta Live Tables (DLT), Unity Catalog, and Structured Streaming.
  • Experience with Kafka, Airflow, or cloud-native orchestration frameworks.
  • Experience implementing CI/CD pipelines using GitHub Actions, Jenkins, or Azure DevOps.
  • Infrastructure as Code experience utilizing Terraform or CloudFormation.
  • Databricks Certified Data Engineer Associate or Professional certification.
  • Experience working within large enterprise AWS environments supporting mission-critical data platforms.

Key Skills

Databricks, AWS, PySpark, Python, SQL, Delta Lake, Medallion Architecture, Lakehouse Architecture, ETL/ELT Pipelines, Spark Optimization, Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, S3, Glue, Lambda, EMR, Kinesis, Redshift, CloudWatch, Kafka, Airflow, GitHub, CI/CD, Terraform.

Will your resume get past this employer's screening? Check your resume against this job, free

Similar jobs