Lead Data Engineer

We are seeking an experienced Lead Data Engineer with strong expertise in the Oil & Gas industry, specifically in liquids and gas systems, to lead the design, development, and delivery of modern data products that enable operational excellence, engineering analytics, and business decision-making.

The ideal candidate combines deep domain knowledge with exceptional software engineering skills. This role is responsible for integrating data from diverse operational and enterprise systems, designing scalable data pipelines, and building high-quality data products that support engineering, operations, commercial, and analytics teams.

As the technical lead, you will establish engineering best practices, mentor other developers, and collaborate closely with business stakeholders, engineers, data scientists, and solution architects to transform complex operational data into trusted, consumable assets.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines that ingest data from multiple operational and enterprise systems.
• Build modern, reusable data products that provide clean, governed, and analytics-ready datasets.
• Integrate structured and semi-structured data from a variety of sources including:
  • SCADA systems
  • Historians (OSI PI, Canary, etc.)
  • Pipeline management systems
  • Measurement systems
  • ERP platforms
  • Asset management systems
  • Laboratory systems
  • IoT and sensor data
  • APIs, flat files, and cloud storage
• Develop robust data models that support operational reporting, engineering analysis, forecasting, optimization, and advanced analytics.
• Write high-quality, maintainable, and well-tested Python and SQL code following software engineering best practices.
• Design orchestration workflows and automated data pipelines using modern data engineering frameworks.
• Optimize pipeline performance, reliability, scalability, and monitoring.
• Collaborate with business users to translate operational requirements into scalable technical solutions.
• Partner with data scientists and analytics teams to deliver production-ready datasets for machine learning and AI initiatives.
• Define data quality standards, validation rules, lineage, and governance practices.
• Mentor and provide technical leadership to other data engineers.
• Participate in architecture decisions and establish data engineering standards across the organization.

Required Qualifications

• Bachelor''''s degree in Computer Science, Engineering, Information Systems, or related technical field.
• 8+ years of professional data engineering or software engineering experience.
• 3+ years leading technical teams or serving as a senior technical contributor.
• Demonstrated experience within the Oil & Gas industry, with strong knowledge of liquids and natural gas operations, pipeline systems, gathering systems, production, transportation, or midstream operations.
• Strong understanding of operational and engineering data used in oil and gas environments.

Required Technical Skills

Programming

• Expert-level Python
• Advanced SQL
• Object-oriented programming
• REST API integration
• Git version control

Data Engineering

• ETL/ELT pipeline development
• Data modeling (dimensional and normalized)
• Data warehousing
• Data lake architectures
• Data transformation frameworks
• Workflow orchestration
• Data quality and validation
• Metadata management
• Data governance principles

Cloud & Platforms

• Microsoft Azure
• AWS
• Google Cloud Platform

Experience with technologies such as:

• Databricks
• Snowflake
• Microsoft Fabric
• Azure Data Factory
• Delta Lake

Databases

• SQL Server
• PostgreSQL
• Oracle
• Azure SQL
• Snowflake
• NoSQL databases (preferred)

Preferred Oil & Gas Domain Experience

Candidates should have experience working with data related to:

• Liquids pipeline operations
• Natural gas transportation
• Midstream operations
• Pipeline integrity
• Measurement and metering
• Flow and pressure monitoring
• Compression systems
• Storage facilities
• Scheduling and nominations
• Production accounting
• Asset performance
• Regulatory reporting
• Operational KPIs

Experience integrating data from systems such as SCADA, PI Historian, measurement systems, GIS, ERP, and engineering applications is highly desirable.