Sr Data Engineer – Data Ingestion Framework (Python, DBT and orchestration technologies such as Dagster or Control-M)

Title – Sr Data Engineer – Data Ingestion Framework

Location-Onsite @ Boston, MA

Duration –Contract

Experience: 10–12 Years

(1 st and 2 nd Page of the resume should show and highlighted the key technologies as below, specifically in the projects).

Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.

Job Description: Sr Data Engineer – Data Ingestion Framework

2 positions 1 need to be local to Boston, MA and one can be remote.

Position Overview

We are looking for an experienced Lead Data Engineer with strong expertise in Python and DBT to design and develop a scalable, reusable, and technology-agnostic data ingestion framework.

The role involves replacing existing ETL capabilities with an open-source technology stack leveraging Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.

The ideal candidate should have hands-on experience building enterprise-grade data ingestion frameworks, developing reusable connectors, implementing metadata-driven pipelines, and integrating heterogeneous source systems with modern cloud data platforms.

Key Responsibilities

1. Data Ingestion Framework Development

• Develop a reusable, scalable, and technology-agnostic data ingestion framework using Python.
• Design modular and extensible components to support multiple source systems.
• Develop metadata-driven and configuration-based ingestion capabilities.
• Build reusable connectors and adapters for different source technologies.
• Implement full and incremental data ingestion patterns, including CDC where applicable.
• Implement robust error handling, logging, auditing, retry, and reprocessing mechanisms.

2. Data Integration & Lakehouse Architecture

• Develop ingestion pipelines to integrate data from heterogeneous source systems into AWS S3 and Snowflake.
• Implement data ingestion from SQL Server, Oracle, PostgreSQL, REST APIs, SFTP, AWS S3, and other source systems.
• Design and implement data lakehouse ingestion patterns using AWS S3.
• Work with file formats such as CSV, JSON, Parquet, and other structured/semi-structured formats.
• Implement schema handling, data validation, partitioning, and data movement strategies.

3. Python & dbt Engineering

• Develop production-grade Python applications and reusable ingestion libraries.
• Build and maintain dbt models for data transformation and integration.
• Implement dbt testing, documentation, incremental models, and reusable macros.
• Optimize Python-based ingestion processes and dbt transformations for performance and scalability.
• Follow engineering best practices for modularity, maintainability, and code quality.

4. Workflow Orchestration

• Implement and manage workflow orchestration using Control-M and Dagester.
• Develop reusable workflows for scheduling, dependency management, retries, and pipeline monitoring.
• Integrate orchestration workflows with Python ingestion components and Snowflake.

5. Testing, Deployment & Production Support

• Develop unit tests and integration tests for ingestion framework components.
• Support migration and deployment across DEV, SIT, UAT, and PROD environments.
• Collaborate with QA teams to perform functional, integration, regression, and performance testing.
• Troubleshoot ingestion failures, data discrepancies, and performance bottlenecks.
• Support production deployment, stabilization, documentation, and knowledge transfer.

6. Technical Leadership & Collaboration

• Work closely with the onshore solution architect, customer stakeholders, and offshore engineering teams.
• Provide technical guidance, code reviews, and engineering best practices.
• Participate in architecture discussions, technical design reviews, and sprint planning.
• Mentor developers and ensure adherence to established engineering standards.

Required Technical Skills

• Python: Expert-level hands-on experience in Python development, including object-oriented programming, reusable libraries, APIs, error handling, and performance optimization.
• DBT: Strong hands-on expertise in DBT Core, including models, macros, tests, incremental processing, and deployment practices.
• Data Engineering: Extensive experience building enterprise-scale ETL/ELT pipelines and reusable data ingestion frameworks.
• Snowflake: Strong experience with Snowflake architecture, data loading, stages, Snowpipe, SQL, and performance optimization.
• AWS: Hands-on experience with AWS S3, IAM, and cloud-based data integration patterns.
• Data Integration: Experience integrating relational databases, REST APIs, SFTP, and cloud-based storage systems.
• SQL: Advanced SQL development and query optimization skills.
• CI/CD: Experience with Git-based version control, automated builds, testing, and deployment pipelines.

Good-to-Have Skills

• Hands-on experience with Dagster for data orchestration and asset-based pipeline development.
• Experience with utilizing Control-M JOB scheduler.
• Experience with Apache Iceberg and open table formats.
• Experience implementing metadata-driven ingestion frameworks.
• Experience migrating ETL workloads from SnapLogic or similar integration platforms.
• Experience with Docker, Terraform, and infrastructure automation.
• Experience implementing data quality, observability, and monitoring solutions.
• Experience working in an Agile environment utilizing SCRUM

Qualifications

• 10–12 years of overall experience in data engineering and data integration.
• Strong architectural and hands-on engineering capabilities.
• Experience working on enterprise data modernization initiatives.
• Excellent communication and stakeholder management skills.
• Ability to work independently in a distributed onshore/offshore delivery model.

Mandatory Skills

1. Expert-level Python development.
2. Strong hands-on dbt experience.
3. Enterprise data ingestion framework development.
4. Snowflake and AWS S3 experience.
5. Strong SQL and data integration experience.

Preferred: Dagster, Control-M, Apache Iceberg, and SnapLogic migration experience.

Regards,

Sai Srikar

Email: