Sr OpenShift API Developer - Big Data

Job Description:

• Platform Management: Deploy, configure, and maintain containerized Spark Applications on OpenShift or Google Cloud
• Data Pipeline Development: Design and implement large-scale data processing workflows using Apache Spark and Apache Airflow
  Optimization: Tune Spark jobs for performance, leveraging OpenShift's resource management capabilities(e.g., Kuberetes orchestration, auto-scaling).
  Integration: Integrate spark with other data sources (e.g., Kafka, s3, cloud storage) and sinks (e.g., databases, data lakes)
  CICD Implementation: Build and maintain CI/CD pipelines for deploying Spark application in OpenShift using tools like GitHub actions, Sonar, Harness.
• Monitoring & Troubleshooting: Monitor cluster health, Spark job performance, and resource utilization using OpenShift tools(e.g., Prometheus, Grafana) and resolve issues proactively Security: Ensure compliance with security standards, Implementing role-based access control(RBAC) and encryption for data in transit and at rest.
• Collaboration: Work with cross-functional teams to define requirements, Architect solutions, and support production deployments.

Technical Skills:

• Proficiency in Spark frameworks(Python/PySpark, Scala, or Java)
• Handson experience with micro services architecture solutions (eg, API design, development, and testing)

• Proficiency in creating and maintaining conda environments and dependencies
• Working knowledge with Docker and Kubernetes solutions (e.g., pods, deployments, services and images)
• Knowledge of distributed systems, cloud platforms (AWS, GP, Azure), and data storage solutions (e.g., S3, HDFS)
• Programing: Strong coding skills in Python, Scala, or Java; experience with shell scripting is a plus.
• Tools: Experience with Git Actions, Helm, Harness, and CI/CD tools.
  Problem-Solving: Ability to debug complex issues across distributed systems and optimize resource usage.