Back to Jobs
Recently postedFull timeTechnology

Data Engineer III

IPAtlanta🇺🇸United States

Quick overview

Salary$83k - $121k/yr
SeniorityMid Senior
Employment typeFull Time
Work modeHybrid
LocationAtlanta, United States
SQLGitApache SparkCompliancePythonCI/CDDelta Lake

Job description

Position Title:

Data Engineer III

Pay Rate:

$83,000 - $121,000

Multiple factors, including Individual experience, skills and abilities will determine where an employee is ultimately placed in the pay range.

Category/Shift:

Salaried Full-Time

Physical Location:

75 5th Street NW
Atlanta, GA 30308
United States

The Job You Will Perform:

The Data Engineer is an important member of the Cost Innovation Team, which is a strategic, innovative, and cross-functional team that is focused on creating value for the company through the advanced analytics of manufacturing data. This position offers a flexible working arrangement provided that the home office is within a typical commuting distance to the A2C office.

Core Responsibilities

Analytics & Data Science Enablement

  • Partner with analysts, data scientists, engineers, and business stakeholders to translate functional requirements into scalable technical data solutions.

  • Deliver self-service, analytics-ready datasets that empower downstream teams to consume data without deep knowledge of underlying source systems.

  • Enable industrial analytics use cases by providing trusted, contextualized time-series, asset, event, and operational data for analytics platforms and applications.

  • Eliminate redundant data-preparation and contextualization efforts across disparate analytics and application teams.

  • Help establish standard data definitions, reusable data products, and consistent business context across enterprise and industrial analytics environments.

  • Support downstream analytics workflows, including advanced analytics, data science, visualization, and industrial process analytics.

Data Ingestion & Integration

  • Design, build, and maintain robust pipelines to ingest data from relational databases, operational systems, industrial historians, APIs, cloud platforms, and flat files.

  • Establish reliable and secure connections between source systems, cloud data platforms, and industrial data platforms.

  • Write optimized SQL queries and extraction logic for high-performance data retrieval.

  • Implement flexible ingestion patterns including batch, incremental, streaming, and replication-based approaches based on business latency requirements.

  • Integrate high-volume time-series data and industrial metadata while maintaining relationships between assets, measurements, events, and other operational entities.

  • Support ingestion and integration patterns used by platforms such as Cognite Data Fusion (CDF), Seeq, and Microsoft Fabric.

Data Storage & Lakehouse Engineering

  • Design and manage scalable storage structures across raw (bronze), intermediate (silver), and curated (gold) layers.

  • Leverage Spark-native formats and engines such as Delta Lake to support scalable analytics workloads.

  • Design curated data structures that support both enterprise analytics and industrial data consumption.

  • Enforce data retention, archiving, and purging policies to manage data lifecycles.

  • Monitor and optimize storage, compute costs, and cloud resource utilization.

Data Transformation

  • Develop Python and SQL-based transformations to clean, standardize, enrich, and unify data.

  • Develop, tune, and proactively optimize scalable PySpark applications to efficiently process high-volume datasets.

  • Build modular, reusable transformation components rather than siloed, application-specific logic.

  • Resolve complex data-quality anomalies including schema drift, missing values, duplicates, timestamp alignment, and inconsistent identifiers.

  • Engineer efficient incremental processing patterns to handle high-volume, frequently changing datasets.

  • Transform raw operational and historian data into analytics-ready time series, calculations, conditions, events, and contextualized datasets for downstream consumption.

  • Enable reusable data transformations that can support platforms such as Seeq, Cognite, Fabric, and custom analytics applications.

Semantic Modeling & Industrial Data Contextualization

  • Translate complex source schemas into intuitive, business-aligned semantic entities, asset hierarchies, and industrial data models.

  • Build and maintain relationships between assets, equipment, time series, events, files, and other operational data.

  • Standardize naming conventions, identifiers, metadata, and relationship mappings across source systems.

  • Perform data contextualization to connect operational data with relevant equipment, process, and business context.

  • Support asset-centric and knowledge-graph-style data models commonly used in industrial data platforms such as Cognite.

Pipeline Orchestration & Operations

  • Architect end-to-end workflows that seamlessly coordinate ingestion, transformation, contextualization, validation, and publication.

  • Orchestrate data movement between source systems, lakehouse environments, industrial data platforms, and analytics applications.

  • Implement resilient error handling including retries, automated recovery, checkpointing, and alerting.

  • Monitor pipeline health, SLA compliance, data freshness, and data drift using centralized observability tools.

  • Design pipelines capable of supporting both traditional enterprise data and high-frequency industrial time-series workloads.

Data Governance, Security & DevOps

  • Enforce data security and privacy standards, including row- and column-level security, RBAC (Role-Based Access Control), and masking of sensitive data.

  • Apply consistent access-control and governance patterns across enterprise data platforms and industrial analytics environments.

  • Embed automated validation checks as needed to enforce data quality.

  • Maintain Git-based development workflows using branching, code reviews, and pull requests.

  • Implement automated CI/CD pipelines for code testing, data validation, configuration management, and repeatable environment deployments.

  • Support deployment and lifecycle management of data pipelines, transformation logic, semantic models, and analytics configurations across development, test, and production environments.

  • Maintain comprehensive documentation of data architectures, pipeline schemas, source-to-target mappings, contextual relationships, and lineage maps.

Preferred Experience

  • Bachelor’s Degree in Computer Science, Software Engineering, Data Science, Engineering, or related field + 2 years of relevant experience; equivalent Master’s degree accepted; non-relevant bachelor’s degree requires 5+ years of equivalent hands-on experience.

  • Experience working with:

    • SQL and Python for data engineering and transformation.

    • PySpark and Microsoft Fabric for lakehouse transformations.

Haystack's AI agent can fill in and submit this application for you. How AI auto apply works

Similar jobs