Lead Data Engineer

Role: Lead Data Engineer
Location: Houston, TX - Onsite

Mode: Contract

Experience Required: 15+ yr

Job Description:

Job Summary: We are seeking a visionary Principal Data Engineer to lead the delivery of innovative data analytics solutions, with a focus on technical excellence, governance, and operational performance in a high-volume data environment. This role involves strategic and technical leadership in building Azure-based data lake solutions, driving maturity in data management, and engaging with cross-functional teams to leverage data for organizational success.

Key Responsibilities:

Architect, design, and implement scalable Azure-based data lake solutions.

Establish roadmaps for capability-building across technologies such as Qlik Replicate, Databricks Delta Live Tables, and Unity Catalog.

Provide technical leadership, ensuring adherence to best practices and quality standards.

Lead the development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.

Optimize Databricks resources for high-volume data processing without exceeding budget.

Oversee the use of Databricks Unity Catalog for data governance, lineage, and metadata management.

Develop and enforce robust data governance policies, ensuring compliance, security, and data quality.

Monitor data pipeline performance and proactively resolve production issues.

Collaborate with cross-functional teams to align project objectives with business goals.

Translate technical challenges into actionable insights for executive teams.

Mentor and develop a high-performing team skilled in Azure, Databricks, and data streaming technologies.

Facilitate knowledge transfer sessions to empower internal teams with cutting-edge methodologies.

Education and Experience:

Bachelor’s or Master’s degree in IT or a related discipline.

15-17 years of experience in the IT industry with expertise in delivering cost-effective cloud-based data analytics solutions.

Proven track record in building data lake architectures and optimizing performance in big data environments.

Certification in Azure Data Engineering or Databricks (preferred).

Technical Skills:

Strong knowledge of Azure technologies, including ADF, ADLS, Databricks, Stream Analytics, Delta Live Tables, and Synapse Analytics.

Advanced proficiency in SQL and programming languages such as Python, Scala, or Spark-SQL.

Experience in designing and developing scalable ETL pipelines and optimizing data workflows.

Deep understanding of data migration from on-premises to cloud, cloud to cloud.

Leadership and Soft Skills:

Proven ability to lead diverse teams and manage end-to-end analytics projects.

Exceptional communication and collaboration skills for engaging executives and stakeholders.

Strong sense of ownership, accountability, adaptability, and commitment to results.

Expertise in identifying risks and implementing effective mitigation strategies.

Nice to Have:

Familiarity with big data tools like Hadoop, Kafka, and Spark.

Experience with visualization tools such as PowerBI or QlikSense.

Knowledge of AI/ML, NLP, and GenAI concepts.

Familiarity with AGILE and KANBAN methodologies.