Back to Jobs
Technology

PySpark Developer (Cloudera, Iceberg, Airflow & AWS)

Integrated Technology Strategies, Inc.United States🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationUnited States
SQLGitApache SparkAWSETLAirflowApacheKafkaCI/CDIAM

Job description

Job Title: Senior PySpark Developer
Location: Remote – US
Duration: 12 Months
Positions: 1
Experience: 10–12+ Years
Level: Senior / Lead Developer

Job Description:

Seeking a Senior PySpark Developer with strong hands-on experience in Cloudera CDP, Apache Iceberg, Airflow, and AWS to build and optimize large-scale data engineering solutions.

Mandatory Skills:

  • 10+ years of overall IT experience with 6+ years in Data Engineering.

  • Strong hands-on experience with PySpark and Apache Spark.

  • Cloudera Data Platform (CDP).

  • Apache Iceberg and Lakehouse architecture.

  • Apache Airflow for workflow orchestration.

  • Strong SQL, ETL/ELT, and data modeling experience.

  • Large-scale data processing and pipeline development.

  • Data partitioning, performance tuning, and query optimization.

  • Git, CI/CD, and DevOps practices.

Good to Have:

  • AWS services: S3, Glue, Lambda, EMR, ECS/EKS, RDS, IAM, and CloudWatch.

  • Kafka and event-driven architecture.

  • Data quality, metadata, and data governance.

  • Healthcare, Medicaid, Claims Processing, or Insurance domain experience.

  • AWS or Cloudera certifications.

Key Responsibilities:

  • Design and develop scalable PySpark data pipelines.

  • Build and optimize ETL/ELT workflows using Cloudera CDP and Airflow.

  • Implement Apache Iceberg-based data lakehouse solutions.

  • Optimize distributed data processing and application performance.

  • Support cloud migration, modernization, and production deployments.

  • Collaborate with architects and engineering teams on enterprise data solutions.

Similar jobs