Senior Apache Spark Developer

Required Qualifications

• 5+ years of hands-on development experience with Apache Spark.


• Strong expertise in:




• Spark SQL


• Structured Streaming


• DataFrames/Datasets


• Performance tuning and optimization




• Experience integrating Spark with Apache Kafka.


• Strong development skills in:




• Java


• Python




• Experience designing and supporting large-scale distributed systems.


• Solid understanding of software engineering principles, object-oriented design, and testing practices.


• Experience working with relational and distributed data platforms.


• Bachelor s Degree

Key Responsibilities

• Design, develop, and optimize Spark-based applications for large-scale data processing.


• Build streaming and batch data solutions leveraging Apache Kafka and Apache Spark.


• Develop reusable frameworks and components supporting enterprise data engineering initiatives.


• Support migration efforts from on-premises data platforms to Databricks and AWS.


• Collaborate with architects and platform engineers to implement cloud-native data solutions.


• Optimize Spark jobs for performance, scalability, reliability, and operational efficiency.


• Participate in system design, code reviews, testing, and production support activities.


• Implement CI/CD, monitoring, and observability practices for distributed data applications.


• Troubleshoot and resolve complex issues across Spark, Kafka, Databricks, and cloud environments and with other enterprise messaging systems.
  

  Preferred Qualifications

  
  
  
  
  • Databricks experience (SQL).
  
  
  • AWS experience, including services such as:
  
  
  
  
  • S3
  
  
  • EMR
  
  
  • Glue
  
  
  • Lambda
  
  
  • ECS/EKS
  
  
  
  
  • Experience supporting large-scale cloud migration initiatives.
  
  
  • Experience with lakehouse architectures and modern data platform patterns.
  
  
  • Experience handling multi-terabyte or petabyte-scale datasets.
  
  
  • Financial services or regulated industry experience.