Back to Jobs
Recently postedTechnology

Principal Data Engineer

CGT StaffingPittsburgh, PA🇺🇸United States

Quick overview

SeniorityLeader
Work modeHybrid
LocationPittsburgh, PA, United States
SQLScalaApache SparkMLflowAirflowApacheData PipelineDatabricksKafkaPowerShellPythonUnityAzureCI/CDAzure DevOpsAzure Data FactoryDelta Lake

Job description

About the Opportunity

We are seeking an experienced Data Engineering Manager to lead and support the development of modern, scalable data platforms. This individual will play a key role in designing, building, and optimizing enterprise-level data solutions using Microsoft Azure and Azure Databricks. The ideal candidate will bring strong hands-on data engineering experience, particularly with Azure Databricks, PySpark, Python, SQL, and Lakehouse Architecture. This role is suited for a technical leader who enjoys remaining involved in development while guiding and mentoring data engineering teams. We are also open to experienced Lead Data Engineers, Principal Data Engineers, and Databricks Technical Leads who have strong technical expertise and leadership capabilities.

Key Responsibilities

  • Lead the design, development, and implementation of scalable enterprise data platforms using Azure Databricks.
  • Build, maintain, and optimize batch and streaming data pipelines using Azure Data Factory and other Azure data services.
  • Design and implement Lakehouse Architecture, Delta Lake, and Medallion Architecture.
  • Develop and optimize data processing solutions using PySpark, Python, SQL, and Apache Spark.
  • Improve data pipeline performance, scalability, reliability, and efficiency.
  • Implement data governance, security, and quality standards across enterprise data platforms.
  • Provide technical leadership, mentorship, and guidance to data engineering team members.
  • Collaborate with cross-functional teams to deliver reliable data solutions that support business and analytics needs.

Must-Have Qualifications

  • 5–7+ years of experience in data engineering, data architecture, or related technical roles.
  • 2–4 years of hands-on experience working with Azure Databricks.
  • Strong experience with PySpark, Python, SQL, and Apache Spark.
  • Hands-on experience with Delta Lake, Lakehouse Architecture, and Medallion Architecture.
  • Experience building and managing batch and streaming data pipelines using Azure Data Factory and Azure data services.
  • Strong understanding of data governance, pipeline optimization, and Azure cloud infrastructure.
  • Experience designing and implementing production-level data engineering solutions.

Nice-to-Have Qualifications

  • Experience managing, leading, or mentoring data engineering teams.
  • Knowledge of Scala and PowerShell.
  • Experience with Unity Catalog, Kafka, MLflow, and Apache Airflow.
  • Familiarity with Azure DevOps and CI/CD practices.
  • Relevant certifications, including Azure Data Engineer, Databricks Data Engineer, or Azure Solutions Architect.

Will your resume get past this employer's screening? Check your resume against this job, free

Similar jobs