Senior Data Engineer

Job Description

We are seeking a highly experienced Senior Data Engineer with 10+ years of experience in data engineering and strong hands-on expertise in Databricks, PySpark, Python, SQL, and large-scale data processing. The ideal candidate will have extensive experience designing, developing, and optimizing enterprise-level data pipelines, implementing complex ETL/ELT processes, and building scalable data engineering solutions.

The candidate must possess strong technical expertise in Apache Spark, distributed computing, data transformation, and modern data lakehouse architectures. The ability to develop production-ready data pipelines, troubleshoot complex data processing challenges, and optimize performance across large datasets is essential.

This is an onsite contract position based in Irvine, California.

Key Responsibilities

• Design, develop, and maintain scalable, high-performance data pipelines using Databricks, PySpark, Python, and SQL.

• Build, implement, and optimize complex ETL/ELT workflows to process large volumes of structured and unstructured data.

• Develop advanced data transformations and processing logic using PySpark and Apache Spark.

• Design and implement scalable data lakehouse architectures using Databricks and Delta Lake.

• Optimize Spark jobs, cluster configurations, partitioning, caching, and query execution for improved performance.

• Develop data ingestion frameworks to integrate data from multiple sources, including relational databases, APIs, files, and cloud storage.

• Implement incremental data processing, Change Data Capture (CDC), and efficient data-loading strategies.

• Develop and manage automated workflows using Databricks Workflows and other orchestration tools.

• Implement data quality checks, validation frameworks, error handling, and monitoring mechanisms.

• Work with Delta Tables, schema evolution, ACID transactions, and data versioning.

• Collaborate with data architects, business stakeholders, data scientists, and engineering teams to deliver enterprise-grade data solutions.

• Troubleshoot complex production issues and ensure the reliability, scalability, and availability of data pipelines.

• Implement data security, governance, access controls, and engineering best practices.

• Participate in technical design discussions, code reviews, and performance optimization initiatives.

Required Technical Skills

• Databricks: Extensive hands-on experience developing and managing enterprise data engineering solutions.

• PySpark: Strong expertise in distributed data processing, complex transformations, Spark DataFrames, and performance tuning.

• Python: Advanced programming skills for data engineering, automation, and pipeline development.

• SQL: Strong proficiency in complex queries, joins, aggregations, window functions, and query optimization.

• Apache Spark: Deep understanding of Spark architecture, execution plans, partitioning, shuffling, caching, and optimization techniques.

• Delta Lake: Hands-on experience with Delta Tables, ACID transactions, schema evolution, and lakehouse architecture.

• ETL/ELT: Extensive experience designing and developing production-grade data pipelines.

• Cloud Platforms: Strong experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud Platform.

• Data Modeling: Experience with data warehousing, dimensional modeling, and large-scale data architecture.

• Workflow Orchestration: Experience scheduling, monitoring, and managing complex data workflows.

• Version Control: Experience with Git and collaborative software development practices.

Preferred Qualifications

• Experience with Azure Databricks, Azure Data Lake Storage (ADLS), Azure Data Factory, or Microsoft Fabric.

• Experience with AWS services such as Amazon S3, AWS Glue, or other cloud-based data processing services.

• Experience with Spark Structured Streaming and Kafka.

• Strong understanding of incremental loads, Change Data Capture (CDC), and Slowly Changing Dimensions (SCD).

• Experience with CI/CD pipelines, automated testing, and deployment processes.

• Knowledge of data governance, metadata management, and enterprise data security.

• Experience supporting and optimizing mission-critical data pipelines in production environments.

Candidate Requirements

• 10+ years of overall experience in data engineering or closely related data development roles.

• Strong, hands-on experience with Databricks and PySpark.

• Proven experience building and maintaining complex, production-grade data pipelines.

• Strong knowledge of Python, SQL, Apache Spark, and cloud-based data platforms.

• Excellent analytical, troubleshooting, and problem-solving skills.

• Ability to work independently and collaborate effectively with cross-functional teams.

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related technical field, or equivalent practical experience.

Key Skills

Databricks | PySpark | Python | SQL | Apache Spark | Delta Lake | Delta Tables | ETL/ELT | Data Pipelines | Data Engineering | Azure Databricks | AWS | Google Cloud Platform | Data Lakehouse | Data Warehousing | Spark Optimization | Cloud Data Engineering