Senior Data Engineer – Forecasting Platforms & Operations

Position Overview

We are seeking a Senior Data Engineer to support the development, integration, reliability, and modernization of data pipelines and applications within an energy forecasting platform.

This role combines advanced data engineering with hands-on production application support. The primary focus is designing, building, and maintaining scalable ETL/ELT pipelines, data integrations, databases, APIs, and automated workflows supporting load, wind, and solar forecasting. The role will also provide secondary operational support for cloud-hosted applications and services responsible for generating and delivering forecasts to business customers.

The ideal candidate has strong expertise in Python, SQL, data integration, relational databases, and cloud environments, along with the ability to troubleshoot production applications, trace dependencies across systems, and resolve operational issues. Experience supporting forecasting, analytics, or machine-learning data pipelines is highly desirable.

Key Responsibilities

Data Engineering and Integration

• Design, develop, maintain, and optimize ETL/ELT pipelines for forecasting, weather, telemetry, market, asset, and operational data.

• Build automated data ingestion and transformation workflows integrating APIs, databases, flat files, SFTP, cloud storage, and external data providers.

• Develop scalable, reusable data processing frameworks using Python and SQL.

• Support relational database environments, particularly PostgreSQL, for forecasting applications and analytical workflows.

• Implement data validation, quality checks, reconciliation, source-to-target validation, and failure-recovery mechanisms.

• Improve pipeline monitoring, logging, restartability, and error handling.

• Integrate new wind, solar, energy storage, and other generation assets into existing data pipelines.

• Collaborate with data scientists to deliver reliable, timely, and well-structured data for model development, training, forecasting, and performance analysis.

• Modernize legacy workflows, standardize data processing, and reduce manual and hard-coded processes.

Forecasting Platform and Production Support

• Provide secondary technical support for production load, wind, and solar forecasting applications and associated data-delivery workflows.

• Troubleshoot issues across application, data, database, cloud infrastructure, integration, and delivery layers.

• Support AWS-hosted workloads and services, including Amazon S3, AWS Lambda, Amazon ECS, Amazon EC2, and Amazon CloudWatch.

• Assist with application deployments, release validation, configuration changes, operational readiness, and post-deployment monitoring.

• Support scheduled jobs, automated forecasting processes, reporting services, and customer-facing forecast delivery.

• Maintain SFTP integrations, secure file transfers, SSH keys, directory configurations, and customer data-exchange processes.

• Participate in incident response, root-cause analysis, change management, service-ticket resolution, and production support.

• Provide backup technical coverage for critical forecasting platform functions when primary support resources are unavailable.

Reliability, Automation, and Documentation

• Automate repetitive operational tasks to improve reliability and reduce manual effort.

• Implement monitoring and alerting for data pipelines, scheduled jobs, data feeds, and production applications.

• Document system dependencies, troubleshooting procedures, recovery steps, support processes, and operational handoffs.

• Participate in cross-training and knowledge sharing to improve technical coverage and operational resilience.

• Continuously improve production support processes, data engineering standards, and platform reliability.

Required Qualifications

• Strong experience designing, developing, and supporting enterprise ETL/ELT pipelines or data integration solutions.

• Advanced SQL skills and experience with relational databases such as PostgreSQL, Oracle, or SQL Server.

• Proficiency in Python for data processing, automation, integration, and troubleshooting.

• Experience integrating data from APIs, databases, flat files, cloud storage, SFTP, and other enterprise sources.

• Experience supporting production data pipelines, scheduled jobs, and failed processing workflows.

• Experience working with AWS or comparable cloud platforms.

• Strong understanding of application and data troubleshooting, logging, monitoring, root-cause analysis, and production support.

• Experience with Git-based version control and modern software development practices.

Preferred Qualifications

• Hands-on experience with AWS services such as S3, Lambda, ECS, EC2, CloudWatch, and Step Functions.

• Experience with PostgreSQL/Aurora, Amazon Redshift, Databricks, Apache Spark, or PySpark.

• Familiarity with CI/CD pipelines, application deployments, and cloud-hosted production applications.

• Experience with SFTP administration, SSH key management, secure data exchange, and automated file transfers.

• Experience developing data-quality checks, reconciliation processes, monitoring frameworks, and automated alerts.

• Familiarity with energy markets, utility operations, renewable energy generation, SCADA/telemetry, weather data, or load, wind, and solar forecasting.

• Experience supporting machine-learning, analytics, or forecasting data pipelines.

• Experience working in Agile environments using tools such as Jira and Confluence.

Key Success Factors

• Reliable and scalable data pipelines that support forecasting operations.

• Effective troubleshooting and timely resolution of production data and application issues.

• Improved automation, monitoring, and data quality.

• Strong documentation and cross-functional collaboration.

• Increased operational resilience and dependable forecast data delivery.