Data Engineer - Databricks

Position Summary:

We are seeking a skilled and driven Data Engineer with 3-5 years of hands-on expertise in data engineering and cloud data warehousing. In this role, you will build, optimize, and maintain our enterprise Databricks Lakehouse platform, which serves as the foundational engine for our data and AI initiatives. You will translate technical requirements into robust data pipelines, implement modern engineering standards, and support data governance, CI/CD, and cost-optimization frameworks. 

Key Responsibilities:

Pipeline Development & Optimization

Build and maintain robust, enterprise-grade data pipelines on the Databricks Lakehouse platform to meet business intelligence and AI/ML demands.

Develop reusable frameworks and apply performance optimization techniques to improve cloud compute efficiency and reduce runtime costs.

Write and refine clean, efficient code for complex data processing using Python, SQL, Scala, or PySpark. 

Data Engineering & Governance

Execute engineering best practices across the entire data lifecycle, spanning ingestion, transformation, and workflow orchestration.

Implement end-to-end data pipelines utilizing modern patterns such as Delta Live Tables (DLT) and Apache Spark.

Apply data governance frameworks, access controls, and data protection strategies locally using Unity Catalog. 

Operations & Collaboration

Support automated CI/CD practices and robust monitoring infrastructure for seamless, reliable deployment of data solutions.

Collaborate closely with senior architects, analysts, and developers in an environment focused on engineering excellence.

Work alongside cross-functional engineering units, InfoSec teams, and product owners to deliver high-quality data products.

Required Qualifications:

Education: Bachelor’s degree in computer science, Data Engineering, Information Systems, or a related technical field.

Experience: 3–5 years of dedicated experience in data engineering, with a strong focus on building cloud-based data solutions. 

Technical Skills:

Proficiency with the Databricks ecosystem (Delta Lake, Unity Catalog, Delta Live Tables).

Strong hands-on experience with Apache Spark (PySpark/Scala) and advanced SQL.

Experience with cloud platforms (AWS, Azure, or Google Cloud Platform) and orchestration tools (Airflow, ADF, or Databricks Workflows).

Familiarity with Version Control (Git) and CI/CD tools (GitHub Actions, Azure DevOps, or Jenkins).