AWS Data Engineer

Job Summary: We are seeking an experienced AWS Data Engineer with strong expertise in AWS Cloud, data engineering, ETL processes, database systems, and real-time and offline analytic processing. The ideal candidate will have hands-on experience with AWS data services, Spark, Python, PySpark, SQL, CI/CD, and test automation, along with the ability to translate business requirements into scalable technical solutions.

Key Responsibilities: Write ETL (Extract/Transform/Load) processes, design database systems, and develop tools for real-time and offline analytic processing. Troubleshoot software and processes to ensure data consistency and integrity. Integrate data from a variety of sources to help business partners generate insights and make informed decisions. Build AWS cloud architecture and support AWS services and technologies, including ECS, Glue, S3, Glue Crawler, Redshift, Step Functions, IAM, CloudWatch, CloudFormation, Lambda, Secrets Manager, and SageMaker.

Translate business specifications into design specifications and code. Write programs, ad hoc queries, and reports while ensuring code is well structured, sufficiently documented, maintainable, and reusable. Partner with internal clients to gain an understanding of business functions and informational needs. Develop working knowledge of tools, technologies, applications, and databases within specific business areas and company-wide systems. Participate in all phases of solution development. Explain technical considerations during project meetings, including meetings with business clients.

Build Spark data processing applications using Python and PySpark. Develop and optimize complex SQL queries and data solutions. Apply test automation and test-driven development practices. Work with CI/CD pipeline tools such as GitHub. Test code thoroughly to ensure accuracy and intended functionality. Review completed solutions with clients to ensure adequate understanding and alignment with requirements. Provide data analysis guidance as required. Provide tool and data support to business users and fellow team members.

Required Qualifications: Strong experience with AWS Cloud. Hands-on experience with AWS services including ECS, Glue, S3, Glue Crawler, Redshift, Step Functions, IAM, CloudWatch, CloudFormation, Lambda, Secrets Manager, and SageMaker. Expert experience building Spark data processing applications using Python and PySpark. Expert-level SQL development experience. Experience developing and supporting ETL processes and data integration solutions. Experience with database design and real-time and offline analytic processing. Experience troubleshooting data consistency and integrity issues.

Experience with test automation and test-driven development practices. Experience with CI/CD pipeline tools such as GitHub. Strong programming, analytical, troubleshooting, and problem-solving skills. Ability to translate business specifications into technical designs and code. Ability to communicate technical concepts effectively with business and technical stakeholders.

Education: Bachelors Degree