Data Engineer with Databricks & AWS

Job Title: Data Engineer with Databricks & AWS

Location: Santa Clara, CA Onsite Need only Locals
12+ Months Contract

Position Summary

We are seeking a Senior Data Engineer with strong experience designing, developing, and supporting scalable data pipelines using Databricks and AWS. The engineer will support business-critical data solutions across supply chain, capacity planning, and global sourcing.

The ideal candidate will have hands-on experience with S3 ingestion, Databricks, Delta/Iceberg tables, ETL frameworks, data quality, SQL/Athena validation, CI/CD, production troubleshooting, and data-layer architecture.

Key Responsibilities

• Design, develop, and support scalable data pipelines in Databricks and AWS.
• Build and maintain data ingestion pipelines using Amazon S3.
• Onboard new data feeds and perform schema validation and data quality checks.
• Develop and manage Bronze, Silver, and Gold data layers.
• Design and maintain Delta Lake / Apache Iceberg tables.
• Enhance ETL frameworks and improve pipeline reliability and scalability.
• Perform data validation using SQL and Amazon Athena.
• Troubleshoot production data pipeline issues and ensure timely resolution.
• Monitor production pipelines and datasets using Splunk.
• Implement and support CI/CD deployments for data engineering workflows.
• Support migration of legacy Hive workloads to unified ODP platforms.
• Maintain business-critical reporting datasets and ensure data accuracy and availability.
• Collaborate with business stakeholders, data engineering teams, and technical teams to understand requirements and deliver reliable data solutions.

Required Skills

• Strong hands-on experience as a Senior Data Engineer / Data Pipeline Engineer.
• Expertise in Databricks and AWS.
• Strong experience developing scalable ETL/data pipelines.
• Experience with Amazon S3 and cloud-based data ingestion.
• Experience with Delta Lake and/or Apache Iceberg.
• Strong SQL skills and experience with Amazon Athena.
• Experience with Bronze/Silver/Gold data architecture.
• Experience with schema validation and data quality frameworks.
• Production troubleshooting and incident resolution experience.
• Experience with CI/CD and deployment automation.
• Experience with Splunk or similar monitoring/logging tools.
• Experience working with business-critical datasets and reporting pipelines.

Preferred / Nice-to-Have

• Experience with Hive-to-lakehouse/platform migrations.
• Experience supporting supply chain, capacity planning, or global sourcing data use cases.
• Experience with unified ODP/data platform environments.
• Strong stakeholder management and communication skills.

Thanks

Sri Vardhan Chilakamukku
Infobahn SoftWorld Inc.