Haystack
← Back to Jobs
Technology

Senior Kubernetes Engineer

Baanyan Software Services, Inc.New York, NY🇺🇸United StatesPosted 4 Aug 2026

Quick Overview

Work Type
Hybrid
Level
Mid Senior

Job Description

Overview
We are seeking an expert Kubernetes engineer to configure, deploy, and operate mission-critical Amazon EKS infrastructure supporting petabyte-scale data processing workloads. This role requires deep technical expertise in Kubernetes internals, large-scale cluster management, and Apache Spark optimization to support thousands of concurrent jobs processing large volumes of Bigdata.
Core Responsibilities
  • Apache Spark optimization to support thousands of concurrent jobs processing large volumes of Bigdata. (candidates are working in smaller environments)
  • Active contributions to open-source Kubernetes, Karpenter, or Spark projects
  • Configure persistent volume claims and provision container-native storage using Amazon EBS CSI driver for block storage and Amazon EFS CSI driver for shared file access
  • Design, deploy, and maintain production-grade Amazon EKS clusters architected for petabyte-scale data processing with high availability and fault tolerance
  • Operate in air-gapped private VPC environments without internet access, managing secure package repositories and container registries
  • Debug and resolve complex distributed systems issues across EKS, Karpenter, and Spark, including scheduling bottlenecks, node scaling delays, and cascading failures under heavy load
  • Implement Karpenter consolidation and disruption policies balancing cost optimization with job resiliency
  • Manage spot and on-demand instance strategies with robust node interruption handling
  • Define and enforce ResourceQuotas, LimitRanges, and PriorityClasses to ensure fair resource distribution and prevent resource starvation
  • Configure persistent volume claims and provision container-native storage using Amazon EBS CSI driver for block storage and Amazon EFS CSI driver for shared file access
  • Optimize storage configurations for resiliency, high-throughput data processing workloads
  • Implement logging, alerting, and anomaly detection to identify provisioning failures, executor loss, and throughput degradation before broader system impact
  • Design fault-tolerant architectures with retry strategies, checkpointing, and graceful degradation patterns minimizing re-computation on failure
  • Develop and manage configs in a private VPC without internet access
Preferred Qualifications
  • 10+ years of overall infrastructure, DevOps, SRE, or cloud engineering experience
  • Kubernetes certification (CKA, CKAD, or CKS) & AWS certifications
  • Active contributions to open-source Kubernetes, Karpenter, or Spark projects
  • Familiarity with FinOps practices and cost optimization at scale
  • Background in data engineering or analytics platforms
 

Skills

AWS
Apache
Apache Spark
Kubernetes

Similar jobs

Browse related jobs

Apply now