Software Engineer- Python/Kafka/ Databricks

Software Engineer  – Python / Kafka / Databricks

Location: San Francisco, CA – Hybrid ( Need Locals)
Duration: 6–12+ Months
Interview: Video
Domain: Beauty / Retail / Marketing Technology

Job Summary

We are looking for a Software Engineer to join a marketing technology engineering team focused on customer retention and personalized customer engagement.

The team builds highly scalable applications that process customer behavioral and transactional events in real time. Customer events flow through Apache Kafka, are processed using Databricks, Apache Spark and Scala, and are persisted and queried using Delta Lake.

The ideal candidate will have strong backend and distributed systems fundamentals, hands-on experience with Python, Kafka, microservices, SQL, Docker/Kubernetes, and familiarity with Databricks/Spark/Scala. Experience building modern React-based frontends and AI-assisted development workflows is highly desirable.

Key Responsibilities
Design, develop and maintain scalable Python-based microservices.
Build event-driven applications using Apache Kafka.
Develop Kafka consumers/producers and implement message validation, error handling and retry mechanisms.
Process high-volume event streams using Databricks, Spark and Scala/PySpark.
Work with Delta Lake and large-scale distributed data processing.
Develop complex SQL queries, aggregations and data transformations across multiple datasets/tables.
Build ingestion pipelines that consume Kafka events and persist validated data into appropriate data stores.
Design error-handling mechanisms, including error/dead-letter topics for invalid Kafka messages.
Develop containerized applications using Docker and deploy/manage services in Kubernetes environments.
Build APIs and supporting frontend components using React.js or similar modern JavaScript frameworks.
Troubleshoot distributed systems, concurrency issues, asynchronous processing and production failures.
Participate in architecture and design discussions around scalability, reliability and fault tolerance.
Work with Kafka consumer groups, partitioning, message keys, offsets and scaling strategies.
Analyze and resolve Kafka issues involving message size, consumer configuration, throughput and availability.
Use AI-assisted development tools such as Claude Code to accelerate application development and prototyping.
Collaborate with product, marketing and engineering teams to build customer-retention solutions.
Required Skills

Backend / Programming

Strong hands-on experience with Python
Strong understanding of microservices architecture
Experience with REST APIs
Strong object-oriented and software engineering fundamentals
Understanding of asynchronous programming and concurrency
Experience troubleshooting distributed systems

Kafka

Strong hands-on experience with Apache Kafka
Kafka producers and consumers
Consumer groups
Partitions and partition keys
Offsets and message ordering
Error handling / retry mechanisms
Dead-letter or error topics
Kafka scalability and availability
Kafka message/payload size considerations

Data / Big Data

Databricks
Apache Spark
Scala and/or PySpark
Delta Lake
Strong SQL
Aggregations and joins across multiple tables
Large-scale data processing

Cloud / DevOps

Docker
Kubernetes / K8s
CI/CD
Cloud data-storage concepts such as Blob/Object Storage
Monitoring and troubleshooting distributed applications

Frontend

React.js / React JS
JavaScript / TypeScript
Ability to build basic UI applications for viewing and interacting with backend data
Preferred Skills
Experience with Braze or other customer engagement/marketing platforms
Experience with email marketing or push notification systems
Experience in retail, e-commerce, beauty or consumer marketing
Experience with event-driven architecture
Experience with CQRS
Experience with cloud-native architectures
Experience with AI-assisted coding tools such as Claude Code, Cursor or GitHub Copilot
Experience with modern development tools such as uv for Python dependency/build management

What We Are Looking For

The ideal candidate should be able to:

Understand the fundamentals behind Kafka, distributed systems and microservices, rather than simply knowing terminology.
Explain how they would handle Kafka failure, message duplication, ordering, large payloads and consumer scaling.
Write clean production-quality code in Python or another modern programming language.
Understand Python fundamentals, including GIL, mutable default arguments, concurrency and async programming.
Write complex SQL queries involving joins, grouping and aggregations.
Understand Databricks/Spark processing and Delta Lake.
Debug concurrency problems such as race conditions and deadlocks.
Use AI coding assistants effectively while still understanding and validating the generated code.
Quickly prototype and deliver applications using modern development workflows

Regards