Data engineer

Job Title: Senior Data Engineer (Python, SQL, Apache Flink)

Location: Columbus, OH (Hybrid)

Duration: Long Term

Job Description

We are seeking an experienced *Senior Data Engineer* with strong hands-on expertise in *Python, SQL, Apache Flink, and Apache Kafka* to design, develop, optimize, and maintain scalable data engineering solutions.

The ideal candidate should have extensive experience building real-time and batch data pipelines, developing complex SQL transformations, writing production-quality Python code, and implementing distributed stream-processing solutions using Flink and Kafka.

Key Responsibilities

·        Design, develop, and maintain scalable ETL/ELT data pipelines using Python and SQL for data ingestion, transformation, validation, and processing.

·        Write clean, efficient, reusable, and production-ready Python code and complex SQL queries to support enterprise data engineering solutions.

·        Develop Python-based data integration frameworks to extract, process, and transform data from REST APIs, relational databases, files, and streaming platforms

·        Utilize Python libraries such as Pandas, NumPy, PyArrow, and SQLAlchemy for data manipulation, processing, and database integration.

·        Develop and optimize advanced SQL queries using CTEs, window functions, joins, subqueries, aggregations, and analytical functions.

·        Design and maintain relational database schemas, data models, views, stored procedures, and reusable data transformation frameworks.

·        Implement data cleansing, deduplication, enrichment, normalization, and complex business transformations using Python and SQL.

·        Develop reusable Python modules, utility functions, SQL scripts, and automation frameworks to improve data processing efficiency.

·        Implement incremental data processing, Change Data Capture (CDC), and data synchronization using Python and SQL.

·        Optimize Python applications and SQL queries for large-scale datasets, focusing on execution performance, memory efficiency, scalability, and resource utilization

·        Analyze SQL execution plans and improve database performance through indexing, partitioning, query optimization, and efficient data access patterns.

·        Develop automated data quality checks, validation rules, reconciliation processes, and monitoring solutions using Python and SQL.

·        Implement robust exception handling, logging, retry mechanisms, and error recovery to ensure reliable data pipeline execution.

·        Build and integrate REST APIs and external data services using Python to support enterprise data ingestion and processing.

·        Develop unit tests, integration tests, and automated testing frameworks using PyTest and SQL-based validation techniques.

·        Design and implement SQL transformations for historical data processing, data migrations, reporting, and analytical workloads.

·        Troubleshoot and resolve Python application errors, SQL performance bottlenecks, data discrepancies, and production pipeline failures.

·        Collaborate with data architects, database developers, and engineering teams to develop scalable, reliable, and high-performance data solutions.