AWS Data Engineer

# Job Description: AWS Data & Solution Engineer

**Location:** New York City, NY
**Work Mode:** Hybrid 2 3 days onsite per week
**Experience:** 10+ Years

## Job Summary

We are seeking a Senior AWS Data & Solution Engineer with strong hands-on expertise in Python, PySpark, and AWS cloud services to design, develop, and optimize enterprise-scale data solutions for a Fortune 500 client. The role focuses on building an enterprise data lake on AWS, developing scalable data pipelines, implementing event-driven integrations, and delivering secure REST APIs. The ideal candidate will have experience with Apache Iceberg, Kafka, AWS Glue, and agentic AI workflows, along with the technical leadership skills needed to drive complex engineering initiatives.

## Key Responsibilities

- Design, develop, unit test, and maintain scalable data processing applications using Python, PySpark, and Apache Spark.
- Build data pipelines integrating relational databases such as Oracle, NoSQL databases such as DynamoDB and MongoDB, and data sources including Amazon S3 and HDFS.
- Develop AWS Lambda functions using Python libraries such as awswrangler, pandas, boto3, json, and requests.
- Build and optimize ETL pipelines using AWS Glue, AWS Glue Data Catalog, and Amazon EMR.
- Implement event-driven data integrations using Apache Kafka, Amazon MSK, and Confluent libraries.
- Use Apache Iceberg to manage schema evolution, change data capture (CDC), ACID-compliant data merges, and snapshot management.
- Design, develop, and secure RESTful APIs using FastAPI or Flask, including OAuth2/JWT authentication and Swagger/OpenAPI specifications.
- Implement workflow orchestration using AWS Step Functions and Amazon EventBridge.
- Optimize Spark workloads, SQL queries, data access patterns, and storage formats such as Avro, Parquet, ORC, JSON, and XML.
- Configure AWS Glue crawlers and catalog tables, collections, and objects from Oracle, MongoDB, and Amazon S3.
- Containerize applications using Docker and deploy them to Kubernetes-based environments, including Amazon EKS.
- Build and leverage agentic AI workflows and use AI-assisted development tools such as GitHub Copilot or GPT-based coding assistants.
- Monitor, troubleshoot, and resolve application and pipeline failures using Amazon CloudWatch and Datadog.
- Diagnose production issues, perform root-cause analysis, and implement proactive performance and reliability improvements.
- Participate in code reviews, release management, production deployments, and Agile/Scrum ceremonies.
- Prepare technical documentation, deployment guides, runbooks, and user support materials.
- Coordinate with onsite and offshore teams, communicate technical designs, and mentor engineers on complex development scenarios.

## Required Technical Skills

- **Experience:** 10+ years in software engineering, data engineering, or related technical roles.
- **Programming:** Strong hands-on expertise in Python, PySpark, pandas, boto3, and requests.
- **AWS Services:** Amazon S3, AWS Glue ETL, Glue Data Catalog, AWS Lambda, Amazon EventBridge, AWS Step Functions, Amazon Athena, and Amazon EMR.
- **Data Engineering:** Enterprise data lakes, ETL/ELT pipelines, distributed processing, data integration, and pipeline optimization.
- **Messaging & Streaming:** Hands-on experience with Apache Kafka, Amazon MSK, and event-driven architectures.
- **Data Formats:** Practical experience with Avro, Parquet, ORC, JSON, and XML, including compression and performance optimization.
- **Databases & SQL:** Strong SQL skills, including complex queries, with experience in relational and NoSQL databases.
- **API Development:** Experience building REST APIs using FastAPI or Flask, including OAuth2/JWT security and Swagger/OpenAPI.
- **Apache Iceberg:** Experience with Iceberg tables, CDC merges, schema evolution, and snapshots.
- **AI-Assisted Development:** Hands-on experience building agentic AI workflows and using Copilot/GPT-based tools for development.
- **Containers:** Experience with Docker and Kubernetes, preferably Amazon EKS.
- **Monitoring:** Experience with CloudWatch, Datadog, troubleshooting, and production support.
- **Certification:** AWS certification in Solutions Architecture, Data Engineering, or Data Analytics is required.

## Preferred Qualifications

- Experience with Snowflake, Amazon Redshift, and DynamoDB.
- Experience designing enterprise-scale cloud data platforms for large organizations.
- Strong understanding of application reliability, security, scalability, and cost optimization.
- Ability to translate business requirements into technical designs and deliver production-ready solutions.
- Experience leading technical discussions, coordinating distributed teams, conducting code reviews, and mentoring engineers.
- Strong communication, analytical, problem-solving, and stakeholder-management skills.