Data Engineer

Job Title: Data Engineer

Location: Pittsburgh, PA, Cleveland, OH, or Dallas, TX – (5 days Onsite) - Local to any of these locations

Job Type: Permanent Full Time

Required qualifications to be successful in this role:

• 5+ years of experience in data engineering and big data processing
• Strong expertise in Apache Spark (Spark Core, Spark SQL) and PySpark for large-scale batch processing
• Experience working with structured and semi-structured data, including complex transformations and performance tuning
• Proficiency in data ingestion and integration from sources like Oracle, SQL Server, Hive, HDFS, and S3; transform data into ‘curated data models''
• Experience writing data to Hive tables, Data Lakes (Iceberg), and downstream reporting systems
• Strong knowledge of SQL and data modeling concepts
• Hands-on experience with Apache Airflow for workflow orchestration (DAG design, scheduling expectations, monitoring)
• Proficiency in shell scripting for job automation, file validation, dependency handling, and logging. Trigger Spark Jobs, perform file checks and validation; Archive & purge data; mange job dependency, logging & error handling
• Strong understanding of batch processing and batch job scheduling frameworks
• Experience migrating from CA7/Control-M  Airflow (daily, hourly, weekly schedules) CI/CD for data pipelines
• Experience ensuring data quality, reliability, and compliance in regulated environments
• Good communication and documentation skills