Data Engineer

Job title:- Data Engineer (Preferred Ex JPMC candidates)

Location: Wilmington, DE (Onsite)

Experience – 8+ years

JD:-

We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, PySpark, Python, and SQL. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data-processing solutions in a cloud-based data environment.

The candidate should be a hands-on Data Engineer who has personally built and supported Databricks solutions, rather than someone with only theoretical exposure. They should be able to explain real examples of PySpark transformations, ETL/ELT pipelines, Delta Lake implementations, performance optimization, data-quality controls, and production troubleshooting.

Key Skills:- Databricks | PySpark | Python | SQL | ETL/ELT | Delta Lake | Spark | Cloud | Data Engineering.

Responsibilities

·                     Design and develop scalable ETL/ELT data pipelines using Databricks.

·                     Develop data-processing and transformation solutions using PySpark and Python.

·                     Write complex SQL queries for data transformation, validation, and analysis.

·                     Build and maintain solutions using Delta Lake and Lakehouse architecture.

·                     Integrate data from multiple sources, including databases, APIs, files, and cloud storage.

·                     Optimize Spark jobs, Databricks workloads, and SQL queries for performance and cost efficiency.

·                     Implement data-quality, validation, reconciliation, monitoring, and error-handling mechanisms.

·                     Troubleshoot production data issues and perform root-cause analysis.

·                     Collaborate with business analysts, architects, and application teams to translate business requirements into technical solutions.

·                     Follow engineering best practices for testing, source control, CI/CD, documentation, and production deployment.

Required Skills & Experience

·                     5–8 years of overall Data Engineering experience.

·                     Strong hands-on experience with Databricks.

·                     Strong hands-on development experience using PySpark / Apache Spark.

·                     Strong Python programming skills and advanced SQL skills.

·                     Experience developing production-grade ETL/ELT pipelines.

·                     Experience with Delta Lake and Lakehouse concepts.

·                     Experience processing large-volume and complex datasets.

·                     Good understanding of data modeling, data quality, and data governance principles.

·                     Strong troubleshooting and performance-tuning skills.

·                     Experience with at least one cloud platform: Microsoft Azure, AWS, or Google Cloud.

·                     Experience with Git and CI/CD practices.

Good to Have Skills

·                     Databricks Unity Catalog experience.

·                     Databricks Workflows / Jobs and cluster optimization experience.

·                     Experience with Azure Data Factory or Apache Airflow.

·                     Experience with Kafka or Spark Structured Streaming.

·                     Knowledge of cloud-native storage and data services.

·                     Databricks or cloud certification.

·                     Experience in banking or financial-services data environments.