AWS Data Engineer with Databricks - W2

Job Title: AWS Data Engineer Databricks- W2

Location: Malvern, PA

We are looking for an experienced AWS Data Engineer with strong hands-on experience in Databricks, PySpark, and AWS data services to support a large-scale enterprise data engineering project.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS.
• Build and optimize data engineering solutions using Databricks and PySpark.
• Develop ETL/ELT pipelines for batch and near-real-time data processing.
• Work with AWS services such as S3, Glue, Lambda, Redshift, EMR, and CloudWatch.
• Develop data transformations using Python, SQL, and PySpark.
• Work with Databricks Delta Lake and implement efficient data storage and processing solutions.
• Optimize Spark jobs, Databricks notebooks, and data pipelines for performance and scalability.
• Implement data quality, validation, monitoring, and error-handling processes.
• Collaborate with data architects, analysts, application teams, and business stakeholders.
• Participate in data migration, modernization, and cloud transformation initiatives.
• Follow enterprise security, governance, and development standards.

Required Skills

• 8+ years of experience in Data Engineering.
• Strong hands-on experience with AWS data services.
• Strong Databricks experience in an enterprise environment.
• Strong PySpark / Apache Spark experience.
• Strong programming experience with Python.
• Advanced SQL skills.
• Experience building and supporting ETL/ELT data pipelines.
• Strong experience with AWS S3, Glue, Lambda, Redshift, and/or EMR.
• Hands-on experience with Delta Lake and Databricks data engineering.
• Experience with data warehousing and dimensional data modeling.
• Experience with Git and CI/CD processes.

Preferred Qualifications

• Experience with Databricks Workflows, Unity Catalog, and Delta Live Tables (DLT).
• Experience with AWS Lakehouse architecture.
• Experience with Terraform or infrastructure-as-code.
• Experience with Airflow or other workflow orchestration tools.
• Experience with streaming technologies such as Kafka/Kinesis.
• Financial services or banking domain experience is a plus.
• Experience working in large enterprise environments.