Lead Data Engineer

Data Engineering Technical Lead

Domain

Energy / Gas / Oil & Gas / Utilities / Commodity Trading / ETRM

Job Overview

We are looking for an experienced Data Engineering Technical Lead with strong hands-on experience in modern cloud data engineering technologies and a background in the Energy, Gas, Oil & Gas, Utilities, Commodity Trading, or ETRM domain.

The ideal candidate will have real-world experience delivering data engineering solutions for actual client or enterprise projects, along with strong stakeholder management, problem-solving, ownership, and communication skills.

Candidates should be able to demonstrate hands-on project experience rather than experience limited to personal learning, training, or certifications.

Key Technical Skills

• Microsoft Azure
• Azure Data Factory (ADF)
• SQL
• Databricks
• PySpark
• Delta Live Tables (DLT)
• Snowflake
• ETL / ELT
• Data pipelines and data transformation
• Data integration and data processing
• Cloud data engineering
• Data warehousing and data lake/lakehouse environments
• AI / Generative AI (GenAI)
• Agentic AI

Data Engineering Responsibilities

• Lead the design, development, and implementation of scalable data engineering solutions.
• Develop and maintain data pipelines using Azure, ADF, Databricks, PySpark, and SQL.
• Build and optimize data transformation and processing workflows.
• Work with Delta Live Tables (DLT) and modern Databricks-based data platforms.
• Develop solutions integrating data from multiple enterprise and source systems.
• Work with Snowflake and other cloud data platforms where applicable.
• Troubleshoot complex data engineering and pipeline issues.
• Ensure data quality, reliability, scalability, and performance.
• Participate in architecture and technical design discussions.
• Identify opportunities to leverage AI, GenAI, and Agentic AI within data engineering and business processes.

Energy / ETRM Domain Experience

Candidates should preferably have experience in the Gas, Energy, Oil & Gas, Utilities, Commodity Trading, or ETRM domain.

Relevant experience may include:

• Energy Trading & Risk Management (ETRM)
• Gas trading and energy markets
• Physical and financial commodity trading
• Trade lifecycle management
• Positions and exposure
• Pricing and market data
• Risk management
• Settlements
• Scheduling
• Energy data management and analytics
• Integration of ETRM systems with enterprise data platforms

Experience working with ETRM platforms such as Endur, RightAngle, Allegro, TriplePoint, or similar platforms is highly preferred.

Leadership & Stakeholder Management

• Strong stakeholder management experience.
• Ability to work directly with business, technical, and client stakeholders.
• Strong communication and presentation skills.
• Ability to understand business requirements and translate them into technical solutions.
• Strong problem-solving and analytical skills.
• Demonstrated ownership and accountability for deliverables.
• Ability to lead technical discussions and drive projects toward successful completion.
• Ability to work independently while collaborating effectively with cross-functional teams.

Real-World Experience Requirement

Candidates must be able to demonstrate actual project/client experience with the technologies and responsibilities listed above.

Experience should be based on:

• Production or enterprise projects
• Client engagements
• Actual implementation work
• Hands-on development and technical leadership

Personal learning, self-study, tutorials, or certifications alone will not be considered sufficient experience.

Resume Requirements

The resume should clearly reflect all relevant hands-on project and client experience.

Candidates with significant experience in Databricks and Snowflake should ensure that this experience is explicitly included in the resume, even if it is not currently listed.

The resume should clearly identify:

• Client/project name or industry where appropriate
• Technologies used
• Candidate's specific responsibilities
• Data engineering solutions developed
• Databricks / Snowflake / Azure experience
• PySpark / SQL / ADF / DLT experience
• AI / GenAI / Agentic AI experience, if applicable
• ETRM / Energy / Gas domain experience
• Business problems solved
• Stakeholders managed
• Technical leadership and ownership responsibilities
•  

Preferred Candidate Profile

The ideal candidate will have a combination of:

Data Engineering + Azure + Databricks + Snowflake + PySpark + SQL + ETRM/Energy Domain + Technical Leadership + Strong Communication & Stakeholder Management.

Candidates who possess strong Databricks/Snowflake experience combined with Energy/ETRM domain knowledge will be particularly valuable for this position.