Databricks Engineering

Location: Southern California Experience: Deep expertise in data engineering and large-scale data integration Job Type: Permanent
### Top Skills & Technologies Databricks | Delta Lake | Spark | SQL | ELT | ETL | Lakeflow Connect | Data Quality
### Job Summary Thinking Minds' client is seeking a lead data engineer to design, develop, and optimize scalable ingestion and processing solutions using Databricks native capabilities. This role combines hands-on engineering with architectural ownership, team mentoring, and stakeholder collaboration.

You will modernize enterprise data integration, deliver analytics-ready datasets, and strengthen the reliability, performance, and governance of cloud-based data platforms.
### Ideal Candidate Should Have
Deep practical expertise in distributed data processing, enterprise migration, and modern data pipeline architecture. A strong engineering mindset focused on automation, scalability, operational reliability, and measurable data quality. The ability to mentor engineers, explain architectural trade-offs, and translate business requirements into maintainable technical solutions.

Confidence collaborating across Product, Engineering, Analytics, Architecture, and Business teams.
###

Key Responsibilities
Design enterprise ingestion frameworks using Databricks Native Data Loader, Lakeflow Connect, and other Databricks-native capabilities. Build and optimize batch and streaming pipelines integrating Oracle, MySQL, SQL Server, APIs, and other enterprise sources. Develop robust ELT and ETL workflows with Spark, Delta Lake, and cloud-native technologies, covering extraction, transformation, validation, delivery, and monitoring. Transform raw and unstructured data into reliable datasets that satisfy functional and non-functional business requirements.

Lead architecture decisions across ingestion, storage, processing, and governance; improve infrastructure efficiency and platform performance. Implement Data Quality controls, investigate production issues through root cause analysis, and recommend corrective actions. Write unit and integration tests, apply Test-Driven Development (TDD) practices, and maintain technical documentation.

Mentor junior engineers, conduct architecture and code reviews, and promote DevOps, CI/CD, and Agile practices, including sprint planning and estimation.
### Required Qualifications
Hands-on experience building scalable data solutions with Databricks, Spark, and Delta Lake. Strong SQL skills and practical understanding of relational database integration and enterprise data migration. Experience implementing data validation, monitoring, testing, and production reliability practices.

Ability to design distributed processing solutions and lead technical delivery with business and technology stakeholders.
###

Preferred Qualifications
Experience improving reusable ingestion frameworks, automated deployment workflows, and enterprise data governance.
Familiarity with preparing trusted enterprise datasets for advanced analytics and Generative AI applications.
Thinking Minds' client offers a collaborative engineering environment, opportunities to guide platform modernization, and scope to develop technical leadership while delivering enterprise data solutions.