Haystack
← Back to Jobs
Remote
Technology

Big Data Engineer

Digitive LLCRichmond, VA🇺🇸United StatesPosted 13 Aug 2026

Quick Overview

Work Type
Remote
Level
Mid Senior

Job Description

 
Position :      Big Data Engineer – Azure / Databricks / OT Data
Location:     Primarily Remote – candidate must be based on the east coast and willing to travel periodically to Parsippany, NJ and/or Richmond, VA
Travel:           Approximately once per month initially; expected to decrease as the individual ramps up. Travel expenses covered.
Duration:      6-Month Contract-to-Hire
Interview Process: 2 video
Permanent Candidates: Candidates interested only in direct-hire opportunities may also be considered
 
Project Overview
Client is building out a Digital Transformation and Data Engineering team focused on creating an enterprise Unified Data Platform / Unified Data Layer (UDL).
Today, critical business and operational data resides across numerous systems, including SAP/ERP, CRM, SaaS applications, cloud platforms, plant Operational Technology (OT), Historian systems, and other manufacturing platforms. Operational reporting is similarly fragmented across these environments.
The goal is to bring this data together into a trusted, governed, scalable Azure data platform that can serve as a unified source for enterprise reporting, Power BI, analytics, AI/ML, automation, finance, supply chain, logistics, HSE, commercial operations, and plant operations.
The organization is a Microsoft/Azure shop and is currently implementing Azure Databricks, with Azure Data Factory serving as a core integration/orchestration platform. Snowflake may potentially be introduced in the future.
 
What They're Looking For
Client is seeking a hands-on Big Data Engineer who can build production-grade pipelines connecting traditional enterprise applications with manufacturing and OT environments.
This person must be technically strong but also self-sufficient and highly communicative. The engineer will work directly with data teams, IT, plant/controls teams, business stakeholders, and other departments to understand source systems and bring their data into the unified platform.
The ideal candidate has worked in continuous or batch manufacturing or an adjacent industrial environment such as chemicals, pharmaceuticals, CPG, oil & gas, energy, or utilities.
 
Key Responsibilities
  • Design and build scalable batch, CDC, near-real-time, and streaming data pipelines.
  • Build pipelines using Azure Databricks, Azure Data Factory, PySpark/Spark, SQL, ADLS/OneLake, and related Azure services.
  • Integrate data from ERP, CRM, SaaS, SAP S/4HANA/DataSphere, APIs, manufacturing applications, and OT systems.
  • Ingest and contextualize plant/industrial data from Historian, SCADA, PLC, IoT/IIoT, telemetry, and time-series environments.
  • Work with industrial connectivity/protocol concepts such as OPC UA and MQTT, along with secure OT-to-IT integration patterns.
  • Develop landing → curated → semantic/consumption layers within the Unified Data Platform.
  • Implement data contracts, schema/versioning, SCD handling, partitioning, caching, clustering, and performance optimization.
  • Build dimensional and semantic models supporting Power BI datasets, APIs, analytics, AI/ML applications, and agents.
  • Partner with plant controls/OT teams around signal quality, security, change control, network boundaries, and downtime windows.
  • Implement data-quality checks for freshness, completeness, schema changes, drift, and validation.
  • Establish monitoring, lineage, alerting, troubleshooting procedures, and production runbooks.
  • Implement RBAC, Key Vault/secrets management, data classifications, retention, governance, and MDM standards.
  • Automate testing and deployment through Git-based CI/CD and structured dev/test/prod environments.
  • Monitor and optimize Databricks/Azure performance and cloud costs.
  • Create documentation, data dictionaries, technical specifications, runbooks, and knowledge-transfer materials.
 
Required Qualifications
  • 5+ years of production data engineering experience building enterprise-scale data pipelines.
  • Strong hands-on experience with Azure Databricks.
  • Strong hands-on experience with Azure Data Factory (ADF).
  • Advanced PySpark/Spark, Python, and SQL skills.
  • Experience with Delta Lake/Lakehouse or similar modern data architectures.
  • Experience developing batch, CDC, and/or streaming pipelines.
  • Strong understanding of Spark Structured Streaming and data-processing performance optimization.
  • Experience integrating multiple enterprise source systems through APIs, databases, files, events, and streaming technologies.
  • Experience with Azure technologies such as ADLS, Synapse/Fabric, Azure SQL, Event Hubs, Key Vault, Azure DevOps, or equivalent services.
  • Experience implementing data quality, testing, monitoring, observability, governance, and CI/CD.
  • Strong communication skills with the ability to independently work with technical and nontechnical stakeholders.
 
Critical OT / Manufacturing Requirement
Candidates need meaningful experience with OT, IoT, industrial streaming, or time-series data. 
 
 

Skills

SQL
Snowflake
Azure
Databricks
Git
IoT
Power BI
Python
Vault

Similar jobs

Browse related jobs

Apply now