PySpark Developer

Important skills:

Strong development skills and significant experience in Data Lakes, PySpark, Pandas, and AWS pipelines.



Qualifications

10+ years hands-on Python development experience for big data application.

Extensive working experience in implementing scalable and efficient data processing pipelines using big data technologies, such as Hadoop/EMR, Spark, Hive.

Strong development experience in AWS platforms/services such as Lambda, Eventbridge, Step Functions, Redshift, S3, Glue.

Extensive hands-on development experience in making API calls, SQL queries, data manipulation/transformation, analysis and visualization.

Strong experience in data analysis/mining, hands-on experience on analyzing large datasets and extracting meaningful insights.

Solid understanding and hand-on implementation experience in CI/CD pipeline. Familiar with tools such as Gitlab/Terraform.

Critical thinking and strong analytical skills to drive solutions, alternative plans, and conclusions

Collaborates effectively and forms positive partnerships with team members in groups and levels across a matrixed organization

Possess strong analytical and problem-solving skills



Skills

Required 10+ years of Python development experience for big data application

Required 5+ years of development experience in AWS platforms/services such as EMR, Lambda, Eventbridge, Step Functions, Redshift, S3, Glue

Required 5+ years of development experience in making API calls, SQL queries, Spark, data manipulation/transformation, analysis and visualization

Hands on experience with CI/CD pipeline tools such as Gitlab/Terraform