Quick overview
Job description
We are a high-growth company transforming how businesses operate by integrating AI, IoT, and cloud-native services into scalable, real-time platforms. As a Platform Data Engineer, you’ll play a critical role in building and maintaining the data infrastructure that powers our products, services, and insights.
You’ll join a multidisciplinary team focused on ingesting, processing, and managing massive streams of sensor and operational data across a wide array of devices—from drones and robots to industrial systems and smart environments.
Responsibilities
- Design, build, and maintain scalable, reliable, and high-throughput data ingestion pipelines for structured and semi-structured data.
- Implement robust and secure data lake and SQL-based storage architectures optimized for performance and cost.
- Develop and maintain internal tools and frameworks for data ingestion using Python, Golang, and SQL.
- Build and maintain inference pipelines to support model serving and near real time prediction workflows
- Develop and maintain internal tools and deployment processes to support ML development workflow , including experimentation, testing, deployment and monitoring.
- Collaborate cross-functionally with Cloud, Edge, Product, and AI teams to define data contracts, schemas, and retention policies.
- Use AWS cloud infrastructure (including Argo Workflows, S3, Lambda, Glue, Kinesis, Athena, and RDS) to support end-to-end data workflows.
- Employ Infrastructure-as-Code (IaC) practices using Terraform to manage data platform infrastructure.
- Monitor data pipelines for quality, latency, and failures using tools such as CloudWatch, SumoLogic, or DataDog.
- Build and maintain ingestion pipelines that land curated sensor and operational data into Delta tables on Databricks to support ML experimentation and model training.
- Continuously optimize storage, partitioning, and query performance across large-scale datasets.
- Participate in architecture reviews and ensure the platform adheres to security, compliance, and best practice standards.
Skills and Qualifications
- 5+ years of professional experience in software engineering or data engineering.
- Strong programming skills in Python and Golang.
- Deep understanding of SQL and modern data lake architectures (e.g., using Parquet, Iceberg, or Delta Lake).
- Hands-on experience with AWS services including but not limited to: S3, Lambda, Glue, Kinesis, Athena, and RDS.
- Proficiency with Terraform for automating infrastructure deployment and management.
- Hands-on experience with Databricks for ML workflows: MLflow experiment tracking and model registry, Model Serving, and building training datasets in Delta Lake (PySpark/Spark SQL).
- Experience working with real-time or batch data ingestion at scale, and designing fault-tolerant ETL/ELT pipelines.
- Familiarity with DVC for dataset versioning and management.
- Familiarity with event-driven architectures and messaging systems like Kafka or Kinesis.
- Strong debugging and optimization skills across cloud, network, and application layers.
- Excellent collaboration, communication, and documentation skills.
Bonus Points
- Experience working with time-series or IoT sensor data at industrial scale.
- Exposure to gRPC and protobuf-based data contracts.
- Experience supporting ML pipelines and feature stores.
- Working knowledge of Kubernetes concepts.
- Databricks Machine Learning certification (Professional)
- Prior startup experience and/or comfort working in fast-paced, iterative environments.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- EH

Senior Data Platform EngineerRecently posted Auto Apply
Ellipsis Health · San Francisco - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
GCPSQLGit+22Technology$150k - $170k/yr - TT
Lead Platform EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 17 Nov 2026 🇺🇸
FastAPIMicroservicesGit+10Technology$220k - $300k/yr - FA

DevOps Engineer Level 3Recently posted Auto Apply
Fantom Corporation · Fort Meade · On-site · Posted 17 Nov 2026 🇺🇸
DockerGitAnsible+12TechnologySalary not listed - HA

Site Reliability Engineer, TeleradiologyRecently posted Auto Apply
Harrison.ai · Remote · Remote · Posted 16 Nov 2026 🇺🇸
GCPGrafanaRadiology+1TechnologySalary not listed - GA

Senior DevOps EngineerRecently posted Auto Apply
Gallup · Omaha Riverfront · Hybrid · Posted 16 Nov 2026 🇺🇸
DockerMySQLGit+12Technology$135k - $170k/yr - GA

DevOps EngineerRecently posted Auto Apply
Gallup · Omaha Riverfront · Hybrid · Posted 16 Nov 2026 🇺🇸
DockerMySQLGit+12Technology$90k - $135k/yr - EA

DevOpsStorage Engineer - FederalRecently posted Auto Apply
ELEVI Associates · Chantilly · Hybrid · Posted 17 Nov 2026 🇺🇸
GitAWSAnsible+18$185k - $240k/yr - EA

DevOps Engineer - FederalRecently posted Auto Apply
ELEVI Associates · Chantilly · Hybrid · Posted 17 Nov 2026 🇺🇸
DockerEncryptionOAuth+6Technology$155k - $200k/yr - ST
Applied Science Platform EngineerRecently posted Auto Apply
Standinsurance · San Francisco · On-site · Posted 17 Nov 2026 🇺🇸
RecruitingETLArticulate+6TechnologySalary not listed - SI

Staff Platform Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · Remote - No Preference · Remote · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerDynamoDB+37TechnologySalary not listed - SI

Staff Platform Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · NYC - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerDynamoDB+37TechnologySalary not listed - SI

Staff Platform Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · Bay Area - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerDynamoDB+37TechnologySalary not listed
