Data Engineer

• Design, develop, and maintain scalable batch data pipelines for transaction data processing and integration.
• Build, enhance, and support data pipelines using Python, Spark, AWS Glue, and Databricks.
• Develop complex SQL queries to extract, transform, validate, and analyze data.
• Work with Snowflake for data storage, transformation, and analytical processing.
• Use Apache Airflow for workflow orchestration, scheduling, and monitoring of data pipelines.
• Troubleshoot pipeline failures, investigate data quality issues, and optimize performance.
• Support the integration of one client's data into current client's existing data ecosystem.
• Collaborate with engineers across multiple initiatives within a small, agile team structure.
• For API-focused initiatives, develop and maintain APIs to serve data to downstream applications and systems.
• Follow established engineering standards, data governance practices, and best practices for reliable data processing.

• Data Engineering: 5–7 years of hands-on experience building and supporting data pipelines.
• Programming: Strong Python programming skills.
• Big Data: Hands-on experience with Apache Spark.
• AWS: Experience with AWS Glue and related data engineering services.
• SQL: Strong SQL skills, including complex queries, joins, aggregations, and data analysis.
• Databricks: Experience developing and processing data using Databricks.
• Snowflake: Experience working with Snowflake for data processing and warehousing.
• Orchestration: Experience with Apache Airflow for scheduling and managing data workflows.
• Data Processing: Experience with batch pipelines, ETL/ELT processes, data transformation, and pipeline support.