Kafka Administrator - Location: Local to NYC only

Principal Consultant – Kafka Administrator

Location: Local to NYC only

Contract 12 months

We are looking for a Principal Consultant – Kafka Administrator based in New York City with 5+ years of hands-on Kafka/Confluent administration experience.

Experience: 5+ years Kafka Administration

Must Have:

• Strong Kafka cluster administration, upgrades, troubleshooting & performance tuning
• Confluent Kafka, KRaft/ZooKeeper, Kafka Connect, Schema Registry & MirrorMaker
• Linux (RHEL/Ubuntu) + Bash/Shell scripting
• PrometheGrafana monitoring & observability
• Kafka HA/DR, resiliency & multi-DC experience
• OpenShift/Kubernetes, Helm & Docker
• Ansible/Terraform and automation/IaC
• Kafka security, ACLs, replication, topics & partitions

Responsibilities

•           Design, implement, and maintain highly available Kafka clusters in production environments.

•           Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.

•           Administer security protocols and access controls to protect the platform.

•           Monitor cluster health and performance using our established monitoring stack (Prometheus, Grafana)

•           Troubleshoot and resolve cluster-related incidents such as broker outages, replication lag.

•           Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.

•           Assist with planning and managing cluster resources to support future growth.

•           Develop Kafka topics, partitions, ACLs, schema registry, and optimize throughput and reliability.

•           Manage a tune of Kafka Connect, Kafka Streams, and Mirror Maker as needed for data pipelines.

•           Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL, Ubuntu).

•           Automate Kafka provisioning, configuration, and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g., Ansible, Terraform).

•           Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).

•           Build custom Kafka tooling and dashboards for producers and consumers, monitor lag and health metrics.

•           Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.

•           Perform root cause analysis and resolution of platform-level incidents and outages.

•           Develop and maintain Linux shell scripts for automation of deployments, monitoring, and maintenance tasks.