Spark / Scala Engineer

·         Candidate Must have 10+ years of experience.

·         Must have Healthcare Domain experience.

·         Design, develop, and optimize scalable batch and real-time data pipelines using Scala, Apache Spark, and Spark Structured Streaming for high-volume data processing.

·         Build robust data engineering solutions using Spark SQL, DataFrames, Datasets, and integrate data with Snowflake, Apache Kafka, and Apache NiFi.

·         Optimize Spark applications through partitioning, caching, shuffle optimization, broadcast joins, and resource tuning to improve performance and scalability.

·         Develop clean, maintainable, and testable Scala applications using functional programming principles, including immutability, higher-order functions, pattern matching, and collections.

·         Design and manage Snowflake data warehouse solutions, including schemas, stages, streams, tasks, and file formats, with knowledge of Snowpark for Scala as a plus.