Senior Kafka / Confluent Streaming Engineer

We are seeking a Senior Kafka / Confluent Streaming Engineer to design, implement, and operate highly scalable, secure, and resilient event-streaming platforms. The ideal candidate will have advanced expertise in Apache Kafka and Confluent Platform, with strong experience in Kafka architecture, schema management, Kafka Connect, ksqlDB, security, multi-region replication, and streaming platform operations.

Key Responsibilities

• Design and implement Kafka-based event streaming solutions, including topic architecture, partitioning, consumer groups, ordering, retention, replay, and scalability strategies.
• Build and operate Confluent Platform components, including Kafka, Schema Registry, Kafka Connect, ksqlDB, and Control Center.
• Manage Schema Registry with Avro, JSON Schema, and Protobuf, including schema compatibility, evolution, versioning, and governance.
• Develop and manage Kafka Connect source and sink connectors, including performance tuning, error handling, retries, and Dead Letter Queue (DLQ) patterns.
• Use ksqlDB for real-time streaming transformations, aggregations, filtering, and joins where appropriate.
• Monitor Kafka environments using Confluent Control Center, focusing on cluster health, throughput, partition performance, and consumer lag.
• Implement RBAC and enterprise security across Kafka, Kafka Connect, Schema Registry, and ksqlDB, following least-privilege principles.
• Integrate Kafka security with enterprise identity and networking standards, including encryption, authentication, authorization, and BYOK where applicable.
• Design resilient multi-cluster and multi-region Kafka architectures using Confluent Cluster Linking for replication, disaster recovery, hybrid environments, and cloud migration.
• Implement Tiered Storage strategies to optimize Kafka retention, performance, and infrastructure costs by offloading older data to object storage.
• Support Confluent Cloud environments and leverage managed Kafka, connectors, Schema Registry, ksqlDB, and cloud-native scaling capabilities.
• Collaborate on advanced stream processing technologies such as Confluent Cloud for Apache Flink, where applicable.
• Establish operational excellence through SLIs/SLOs, monitoring, alerting, runbooks, capacity planning, and incident response.
• Automate Kafka platform provisioning and configuration using Infrastructure as Code (IaC) and implement CI/CD for connectors, schemas, and streaming artifacts.
• Standardize onboarding processes for producer and consumer teams.
• Define and enforce platform standards covering Kafka naming conventions, event contracts, schema/versioning strategies, testing, and reference architectures.
• Mentor engineers and provide technical leadership on Kafka platform architecture, engineering practices, and production operations.

Required Skills

• Apache Kafka – Advanced
• Confluent Platform – Advanced
• Kafka architecture, partitioning, replication, retention, replay, consumer groups, and ordering
• Schema Registry – Avro, JSON Schema, Protobuf
• Kafka Connect – Source/Sink connectors, tuning, error handling, DLQ
• ksqlDB – Streaming transformations, aggregations, and joins
• Confluent Control Center – Monitoring and consumer-lag analysis
• Kafka RBAC, security, governance, and enterprise identity integration
• Confluent Cluster Linking / multi-region Kafka
• Kafka Tiered Storage
• Confluent Cloud
• Apache Flink / Confluent Cloud for Apache Flink – preferred
• Infrastructure as Code and CI/CD automation
• Production Kafka operations, SRE practices, SLOs/SLIs, and incident management

Cloud & Data Platform Experience

• Databricks – Advanced
• Google Cloud Platform (Google Cloud Platform) – Intermediate
• Microsoft Azure – Intermediate
• Experience integrating Kafka with cloud-native data platforms, Databricks, object storage, and enterprise applications.

Preferred Qualifications

• Experience designing enterprise-scale Kafka platforms and reference architectures.
• Strong understanding of event-driven architecture and event-driven microservices.
• Experience with hybrid cloud and cloud migration initiatives.
• Strong troubleshooting and performance-tuning skills across Kafka brokers, producers, consumers, and connectors.
• Experience establishing Kafka platform governance and mentoring engineering teams.
• Excellent communication and collaboration skills with application, data engineering, cloud, security, and infrastructure teams.