Lead Data Engineer

Role: Lead Data Engineer

Locations: San Francisco, CA (Hybrid Onsite)

Duration: 12+ Months Contract

Overview:

•             We are seeking a highly skilled Lead Data Engineer to design develop

and optimize scalable data ingestion and processing solutions using Databricks native capabilities

•             The ideal candidate will possess deep expertise in data engineering

largescale data integration and cloudbased data platforms with handson experience in Databricks Native Data Loader Lakeflow Connect Delta Lake and enterprise data migration from relational databases such as Oracle MySQL and SQL Server

•             This role will lead the design and implementation of modern data

pipelines mentor engineering teams and collaborate closely with business and technology stakeholders to deliver reliable scalable and highperforming data solutions

Key Responsibilities:

Databricks Data Engineering Data Ingestion:

•             Design and implement enterprisescale data ingestion frameworks using

Databricks Native Data Loader Lakeflow Connect and other Databricksnative capabilities

•             Build and optimize batch and streaming data pipelines for ingesting

data from Oracle MySQL SQL Server APIs and other enterprise data sources

•             Develop robust ELTETL pipelines leveraging Databricks Spark Delta Lake

and cloudnative technologies

•             Implement endtoend data movement transformation validation and

monitoring solutions

•             Lead architectural decisions related to data ingestion storage

processing and governance

Data Engineering:

•             Create and maintain optimal data pipeline architecture

•             Build scalable data pipelines that transform raw and unstructured data

into analyticsready datasets

•             Assemble large complex datasets that meet both functional and

nonfunctional business requirements

•             Identify design and implement process improvements including

automation infrastructure optimization and scalability enhancements

•             Develop infrastructure for extraction transformation and delivery of

data from diverse data sources using distributed processing technologies

•             Build and maintain new integrations required for optimal data

extraction transformation and loading ETLELT

•             Implement processes to validate data and monitor data quality ensuring

production data is accurate and reliable

•             Perform root cause analysis on data issues and recommend corrective

actions

•             Write unit and integration tests adopt TestDriven Development TDD

practices and maintain technical documentation

•             Ensure platform reliability performance optimization and adherence to

enterprise standards

Leadership Collaboration:

•             Provide technical leadership and mentoring to junior engineers

•             Collaborate with Product Engineering Analytics Architecture and

Business stakeholders to understand requirements and deliver data solutions

•             Drive best practices in coding DevOps CICD testing and documentation

•             Participate in architecture reviews code reviews and technical design

discussions

•             Support Agile development processes and contribute to sprint planning

and estimation activities