Principal Data Engineer

Principal Data Engineer

We are seeking an experienced Principal Data Engineer to lead the design, development, and implementation of scalable cloud-based data engineering and analytics solutions. The ideal candidate will have strong expertise in Microsoft Azure, Databricks, data lake architecture, data governance, and large-scale data processing. This role requires a combination of hands-on technical expertise, architecture capabilities, and technical leadership to drive data engineering initiatives, optimize performance, and ensure data quality and governance across enterprise environments.

Responsibilities

• Architect, design, and implement scalable Azure-based data lake solutions.
• Lead the development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
• Establish technical roadmaps for data engineering capabilities using Qlik Replicate, Databricks, and Unity Catalog.
• Optimize Databricks resources and workloads for high-volume data processing and cost efficiency.
• Implement data governance, lineage, metadata management, and data quality standards using Unity Catalog.
• Develop and optimize scalable ETL pipelines and data workflows.
• Lead data migration initiatives from on-premises environments to cloud platforms and between cloud environments.
• Monitor data pipeline performance and troubleshoot production issues.
• Ensure compliance, security, and data governance best practices.
• Collaborate with business and technical stakeholders to align data engineering initiatives with organizational objectives.
• Provide technical leadership, mentor engineering teams, and facilitate knowledge-sharing sessions.

Requirements

<>Required Technical Skills

• Strong expertise in Microsoft Azure data services.
• Hands-on experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Databricks, Azure Stream Analytics, and Azure Synapse Analytics.
• Advanced proficiency in SQL and programming languages such as Python, Scala, and Spark SQL.
• Strong experience with Databricks Delta Live Tables and Structured Streaming.
• Expertise in Databricks Unity Catalog, data governance, data lineage, and metadata management.
• Experience with Qlik Replicate.
• Strong understanding of data lake architecture and enterprise data engineering.
• Experience designing and optimizing large-scale ETL pipelines.
• Expertise in cloud data migration and performance optimization.

<>Preferred Qualifications

• Bachelor's or Master's degree in IT, Computer Science, or a related discipline.
• 15–17 years of overall IT experience.
• Proven experience delivering enterprise-scale cloud data analytics solutions.
• Azure Data Engineering or Databricks certifications are preferred.
• Experience with Hadoop, Kafka, and Apache Spark.
• Familiarity with Power BI or Qlik Sense.
• Knowledge of AI/ML, NLP, and Generative AI concepts.
• Experience working with Agile and Kanban methodologies.

<>Leadership & Communication Skills

• Strong technical leadership and team management capabilities.
• Excellent communication and stakeholder management skills.
• Ability to translate complex technical challenges into business solutions.
• Strong ownership, accountability, and problem-solving abilities.
• Experience identifying technical risks and implementing mitigation strategies.