Data Automation Engineer

In this role, you will:

• Design and implement scalable data automation workflows using AWS services, with integration to selected Azure data platforms where required.
• Develop ETL/ELT processes to ingest, transform, and move data across Amazon DynamoDB, SQL Server hosted on AWS, Azure SQL, and other enterprise data sources.
• Design, develop, and support batch and near-real-time ingestion pipelines using Apache Spark and technologies such as Kafka or Flume, and collaborate with the search engineering team to integrate those pipelines with the existing Apache Solr platform.
• Evaluate and apply Generative AI services and frameworks, such as Amazon Bedrock, Azure OpenAI, Hugging Face, and LangChain, to:
• Prototype and evaluate selected GenAI-assisted capabilities, such as metadata enrichment, data-quality analysis, structured data extraction, anomaly identification, and natural-language access to enterprise data. Recommend suitable use cases for future implementation.
• Develop scalable data-processing solutions using Amazon EMR and containerized deployment environments such as AWS Fargate or Kubernetes.
• Integrate Amazon Connect customer-interaction data into analytical data stores for operational reporting and analytics.
• Apply source-control, build, containerization, and CI/CD practices using tools such as GitHub, Azure DevOps, Jenkins, and Docker.
• Implement data solutions in accordance with established security and compliance controls, including identity and access management, KMS encryption, VPC isolation, role-based access control, and firewall policies.
• Support Agile DevOps processes with sprint-based delivery of pipeline and AI-enabled features.

Required Qualifications:

• Bachelor’s degree in Computer Science or a related field and 5+ years of experience in data engineering, data automation, or a related discipline.
• Candidates must be able to independently design, develop, test, and troubleshoot Python- and SQL-based data pipelines in AWS environments, including integrations with Azure services where required, and clearly explain their personal contribution to production implementations.
• Strong hands-on experience with Apache Spark and working knowledge of at least one streaming or ingestion technology, such as Apache Kafka or Apache Flume.
• Hands-on experience with multiple AWS data and integration services, including several of the following: Amazon S3, AWS Glue, AWS Lambda, Amazon EMR, AWS Step Functions, and at least one AWS database service.
• Practical experience integrating at least one LLM platform or model service, such as Amazon Bedrock, Azure OpenAI Service, or an open-source model, into a Python-based workflow.
• Experience integrating REST APIs and external services into Python-based data pipelines and automated workflows.
• Experience using Jira and one or more source-control, build, or CI/CD platforms, such as GitHub, Azure DevOps, or Jenkins.
• Strong troubleshooting and performance-optimization skills across SQL, Spark, batch pipelines, and near-real-time ingestion workflows.
• Experience supporting production data platforms, including SLA monitoring, incident resolution, root-cause analysis, data reconciliation, performance troubleshooting, vulnerability remediation, and recurring maintenance.
• Good communication and presentation skills.

Preferred Qualifications:

• Relevant certifications, such as AWS Certified Data Engineer – Associate, AWS Certified Machine Learning – Specialty, Microsoft Certified: Azure AI Engineer Associate, or Databricks Certified Data Engineer.
• Familiarity with retrieval-augmented generation pipelines, embeddings, and vector-search technologies such as Apache Solr, Amazon OpenSearch Service, pgvector, or similar platforms.
• Experience with multi-cloud data integration (AWS and Azure).
• Experience with Docker and Kubernetes for containerized deployment, scalable data processing, and orchestration.
• Experience operationalizing Generative AI workflows, including prompt and model configuration, evaluation, observability, monitoring, and lifecycle management.
• Knowledge of data lineage/governance tools (Purview, Unity Catalog, AWS Glue Catalog).
• Familiarity with infrastructure-as-code tools, such as Terraform, AWS CloudFormation, or Bicep, for automated deployments.
• Experience with compliance frameworks (FedRAMP, PCI-DSS, HIPAA).