DataOps Engineer

Job Description

We are looking for a DataOps Engineer who can build, automate, monitor, and maintain data pipelines and cloud data platforms. The ideal candidate should have strong experience with AWS, Python, PySpark, SQL, Terraform, and modern DataOps practices.

Key Responsibilities

• Build and maintain automated ETL/ELT data pipelines.
• Automate data processing, scheduling, deployments, and testing.
• Monitor data pipelines and platforms to ensure high availability and reliability.
• Set up logging, alerts, health checks, and performance monitoring.
• Troubleshoot production issues and perform Root Cause Analysis (RCA).
• Manage deployments, releases, rollbacks, and performance tuning.
• Work with data engineering and BI teams to improve data quality and reliability.
• Work with third-party/managed service providers and ensure SLA requirements are met.
• Create reusable data engineering frameworks and automation solutions.
• Use Infrastructure as Code (IaC) to manage cloud infrastructure.
• Monitor and optimize AWS cloud costs (FinOps).
• Handle and process complex XML data.

Required Skills

• Strong hands-on experience with AWS.
• Experience with EC2, EKS, S3, Glue, Lambda, and IAM.
• Strong knowledge of Python, PySpark, and SQL.
• Experience with Terraform and AWS CloudFormation.
• Knowledge of ETL/ELT, CI/CD, orchestration, and automation.
• Experience with DataOps and Medallion Architecture.
• Strong understanding of data monitoring, observability, logging, and alerting.
• Experience with data pipeline testing and deployment.
• Good understanding of AWS security and access management.
• Knowledge of cloud cost optimization / FinOps.
• Experience handling XML data.

Nice to Have

• Experience with AI-assisted coding tools such as Codex.
• Experience working with managed service providers.
• Strong communication and stakeholder management skills.