Data Engineer - AWS & PySpark

Data Engineer AWS & PySpark

Location: Dallas, TX or Richmond, VA
Client: Confidential

Job Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in AWS and PySpark. The ideal candidate must have prior professional experience working with Banking domain and be capable of developing scalable data pipelines and data processing solutions in a cloud environment.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and AWS.
• Develop and optimize large-scale data processing and ETL workflows.
• Work with AWS cloud services to build reliable and high-performance data solutions.
• Perform data transformation, cleansing, validation, and integration.
• Optimize PySpark jobs for performance, scalability, and cost efficiency.
• Collaborate with data architects, analysts, developers, and business stakeholders.
• Troubleshoot data pipeline issues and ensure data quality and reliability.
• Follow engineering best practices for code development, testing, deployment, and documentation.

Required Skills

• Strong hands-on experience with PySpark.
• Strong experience with AWS cloud services.
• Experience developing ETL/data pipelines and processing large datasets.
• Strong Python and SQL skills.
• Experience with data transformation, integration, and data quality.
• Mandatory: Prior Capital One project/client experience.
• Strong communication and problem-solving skills.

Preferred

• Experience with AWS data services such as S3, Glue, EMR, Lambda, Redshift, or similar.
• Experience with distributed data processing and cloud-based data platforms.
• Financial services/banking domain experience.