AWS Data Engineer

Job Duties:

Must have below:

• 10+ Years Experience
• Great Communicator/Client Facing/ Attention to detail
• Individual Contributor and ability to work as a team.
• 100% Hands on in the mentioned skills

Programming Skills:

Python and Data Pipelines:

• Advanced Proficiency in Python concepts like Code Structures, Modules, Packages, Class, SubClass, Inheritance, Multi-Threading and Functional Programming.
• Experience in developing reusable Python packages for internal or public usage
• Ability to write automating ETL processes and scheduling jobs like Airflow DAG.
• Ability to track job pipeline runs to reprocess error records
• Ability to orchestration different pipelines to run in sequence or parallel.
• Troubleshoot data pipeline errors and fix issues
• Export or Import data to/from various formats like CSV, JSON, XML etc preferably from S3 or other cloud storage.
• Experience in using AI IDE tool

SQL:

• Advanced SQL skills, including complex joins, CTE''s and subqueries
• Experience in optimizing SQL queries for performance and optimization in data warehouse technologies preferably Snowflake

Testing and documentation:

• Proficiency in Python unit, integration, and system test.
• Proficiency in implementing DBT tests for data validation and quality checks

Code Generation:

• Experience in generating code using configurations using python and jinja templates
• Version control:
• Experience in GitHub, including implementing CI/CD process from scratch

AWS Expertise:

• Data Storage solutions:
• In depth understanding of AWS S3 for data storage, ECS, IAM including best practices for organization and security

Cloud Security:

• Knowledge of AWS security best practices, including IAM roles, encryption standard, secure coding guidelines, DBT profiles access configurations and more.

Data Integration (nice to have):

• Experience with AWS lambda for serverless data processing tasks

Workflow Orchestration (nice to have):

• Proficiency in using Apache Airflow on AWS to design ,schedule and monitor complex data flows
• Ability to integrate Airflow with AWS services and DBT models such as triggering a DBT model or EMR or reading from s3 writing to redshift or snowflake.
• Experience in API integrations to Salesforce, SFTP, SharePoint, One Drive, and others.

DBT Core/ Cloud Proficiency (nice to have):

• Experience in creating complex DBT models including full refresh, incremental models, snapshots, and documentation.
• Ability to write and maintain DBT macros for reusable code.
• Experience in creating custom DBT macros using jinja and Python allowing for reusable components within DBT models.

Monitoring and Logging (nice to have):

• Familiarity with AWS cloud watch, Data Dog for monitoring the pipelines and setting up alerts for workflow failures.