Role Overview

Key Responsibilities

• Develop, test, and maintain ETL and data engineering jobs that bring data into Databricks and support downstream analytics and reporting.
• Work with business and technical stakeholders to understand requirements and translate them into scalable, production-ready data pipelines.
• Build and maintain pipelines across AWS, Databricks, and Snowflake environments.
• Enhance existing legacy data engineering jobs as part of broader modernization and migration efforts.
• Create and maintain technical documentation, technical specifications, test plans, and operational support materials.
• Perform testing and quality assurance to validate data accuracy, pipeline reliability, and readiness for production deployment.
• Use Terraform to provision, configure, and manage cloud infrastructure supporting data engineering workloads.
• Set up and maintain CI/CD pipelines using Concourse, GitHub Actions, or similar tooling to enable smooth, repeatable deployments.
• Troubleshoot pipeline issues, resolve defects quickly, and help ensure data jobs run reliably within expected service levels.
• Provide post-delivery support, including issue investigation, defect resolution, and operational handoff.
• Identify and mitigate risks, including single points of failure, reliability gaps, and operational dependencies.
• Collaborate with engineers, analysts, and reporting teams to share knowledge, strengthen best practices, and improve delivery quality.

Project Scope

Required Skills and Experience

• Hands-on experience developing ETL, ELT, or data engineering pipelines in production environments.
• Strong experience with Databricks, Spark-based processing, and modern data pipeline development.
• Experience working with AWS cloud services and Snowflake data platforms.
• Proficiency with infrastructure-as-code practices, preferably Terraform.
• Experience implementing CI/CD practices using Concourse, GitHub Actions, or comparable deployment automation tools.
• Ability to write clear technical specifications, documentation, and support materials.
• Strong testing, quality assurance, troubleshooting, and operational support skills.
• Ability to work collaboratively with cross-functional teams and translate requirements into working technical solutions.

Preferred Qualifications

• Experience supporting analytics or BI migrations, especially from Looker to Tableau.
• Experience developing data assets for Tableau Cloud reporting and analytics use cases.
• Familiarity with supply, yield, inventory, or advertising-related data domains.
• Experience modernizing or enhancing legacy data engineering jobs.
• Strong understanding of operational reliability, risk mitigation, and single-point-of-failure reduction.