Data Engineer with Databricks

NDA Partners is seeking a Technical Solutions Engineer to lead the implementation, optimization, and ongoing support of healthcare IT solutions for clients. This is a client-facing technical role that blends hands-on data analysis (SQL, Python, Unix) with healthcare domain knowledge (HL7, FHIR, HIPAA) and strong communication. You’ll be the primary technical point of contact for client implementations — gathering requirements, configuring and deploying solutions, training end users, and resolving escalations.

Job Description

We are seeking an experienced Data Engineer with strong hands-on expertise in Databricks, PySpark, Python, and SQL. The role is responsible for designing, developing, and optimizing scalable data pipelines and data-processing solutions in a cloud-based data environment.

This is a hands-on position. The ideal candidate has personally built and supported Databricks solutions, not just theoretical exposure, and can walk through real examples of PySpark transformations, ETL/ELT pipelines, Delta Lake implementations, performance optimization, data-quality controls, and production troubleshooting.

Key Skills: Databricks | PySpark | Python | SQL | ETL/ELT | Delta Lake | Spark | Cloud | Data Engineering

Responsibilities

• Design and develop scalable ETL/ELT data pipelines using Databricks
• Develop data-processing and transformation solutions using PySpark and Python
• Write complex SQL queries for data transformation, validation, and analysis
• Build and maintain solutions using Delta Lake and Lakehouse architecture
• Integrate data from multiple sources, including databases, APIs, files, and cloud storage
• Optimize Spark jobs, Databricks workloads, and SQL queries for performance and cost efficiency
• Implement data-quality, validation, reconciliation, monitoring, and error-handling mechanisms
• Troubleshoot production data issues and perform root-cause analysis
• Collaborate with business analysts, architects, and application teams to translate business requirements into technical solutions
• Follow engineering best practices for testing, source control, CI/CD, documentation, and production deployment

Required Skills & Experience

• 5–8 years of overall Data Engineering experience
• Strong hands-on experience with Databricks
• Strong hands-on development experience with PySpark / Apache Spark
• Strong Python programming skills and advanced SQL skills
• Experience developing production-grade ETL/ELT pipelines
• Experience with Delta Lake and Lakehouse concepts
• Experience processing large-volume and complex datasets
• Solid understanding of data modeling, data quality, and data governance principles
• Strong troubleshooting and performance-tuning skills
• Experience with at least one cloud platform: Microsoft Azure, AWS, or Google Cloud
• Experience with Git and CI/CD practices

Nice to Have

• Databricks Unity Catalog experience
• Databricks Workflows / Jobs and cluster optimization experience
• Experience with Azure Data Factory or Apache Airflow
• Experience with Kafka or Spark Structured Streaming
• Knowledge of cloud-native storage and data services
• Databricks or cloud certification
• Experience in banking or financial-services data environments