Lead Scala Data Engineer

We are seeking an experienced Lead Scala Data Engineer to serve as the primary technical owner of an enterprise data-processing framework supporting Medicaid Encounter Processing and enterprise data ingestion.

\n\n

This is a hands-on technical ownership position responsible for maintaining and enhancing a mission-critical production application built with Scala, Apache Spark, Hive, Drools, and Cloudera Data Platform (CDP). The ideal candidate will have extensive experience developing distributed data-processing applications and supporting them in production.

\n\n

Key Responsibilities

\n

\n
• Serve as the primary technical owner of the Scala/Spark application framework supporting Medicaid Encounter Processing.
\n
• Maintain and enhance production applications developed using Scala, Spark, Hive, and Drools.
\n
• Own Drools business-rule implementation and ongoing rule maintenance.
\n
• Support enterprise ingestion and processing of provider, member, reference, eligibility, and encounter data.
\n
• Own Spark and Hive batch-processing workflows.
\n
• Troubleshoot production issues, identify root causes, and resolve application defects.
\n
• Implement business, regulatory, and application changes.
\n
• Manage production releases, version control, and deployment coordination.
\n
• Perform Spark performance tuning and optimization.
\n
• Monitor and provide basic operational support for the Cloudera Data Platform (CDP).
\n
• Maintain technical documentation, operational procedures, and knowledge-transfer materials.
\n
• Coordinate with infrastructure, cloud operations, QA, business, and other technical teams.
\n

\n\n

Required Qualifications

\n

\n
• 7+ years of experience developing enterprise-scale distributed data-processing applications.
\n
• Strong hands-on Scala development experience, preferably 4–6+ years.
\n
• 4–6+ years of Apache Spark and Hive development experience.
\n
• Significant experience developing and maintaining applications on Cloudera Data Platform (CDP) 7.x or equivalent enterprise Hadoop environments.
\n
• Hands-on experience implementing business rules using the Drools Rules Engine, preferably 4–6 years.
\n
• Strong SQL development and query-optimization skills.
\n
• Experience supporting Linux-based production environments.
\n
• Strong experience troubleshooting distributed Spark applications in production.
\n
• Experience with Git and modern version-control practices.
\n
• Demonstrated experience taking technical ownership of production applications, including incidents, defects, enhancements, releases, deployments, and performance issues.
\n

\n\n

Preferred Qualifications

\n

\n
• Medicaid or healthcare industry experience.
\n
• Experience with Medicaid Encounter Processing.
\n
• Experience with Cloudera Manager, HDFS, and YARN.
\n
• Experience integrating with IBM DataStage.
\n
• Familiarity with AWS infrastructure supporting Cloudera.
\n
• Experience working in Agile environments using Jira and Confluence.
\n

\n\n

Ideal Candidate

\n

We are specifically looking for a Scala/Spark Data Engineer with strong Cloudera/CDP experience, rather than a Cloudera Administrator. The successful candidate should be capable of independently owning a production application across the complete Scala + Spark + Hive + Drools + Cloudera technology stack.

\n\n

Job Title: Lead Scala Data Engineer – Spark / Cloudera / Medicaid

\n

Experience Level: Senior/Lead

\n

Primary Skills: Scala, Spark, Hive, Drools, Cloudera CDP, SQL, Linux

\n

Industry Experience: Medicaid/Healthcare preferred