Databricks Engineer

RESPONSIBILITIES

Kforce has a client that is seeking a Databricks Engineer in Purchase, NY.

Overview

We are seeking a skilled Mid-Level Databricks Engineer to join our data engineering team. The ideal candidate will play a key role in the migration of Hadoop workloads to the Databricks Lakehouse platform, contributing to the development, optimization, and monitoring of scalable and repeatable data solutions. You will work closely with cross-functional teams to ensure successful migration, enablement of advanced Databricks features, and ongoing performance improvements.

Key Responsibilities

• Identify and categorize existing Hadoop jobs (ETL, batch, streaming) and data sources to support migration planning
• Participate in selecting scalable and repeatable migration use cases for Minimum Viable Product (MVP) initiatives
• Provision and configure Databricks workspaces, ensuring Lakehouse architecture and federation capabilities are enabled
• Execute pilot migrations of representative Hadoop workloads using tools such as Databricks Migration Accelerator or partner solutions
• Validate and monitor post-migration performance, cost efficiency, and data integrity
• Enable and test advanced Databricks features (e.g., Liquid Clustering, Lakehouse AI monitoring, Serverless warehouse, Unity Catalog) to optimize data layout and pipeline health
• Engage with data engineering, analytics, and governance teams to gather feedback and document learnings, blockers, and feature gaps
• Track and report on key success metrics such as migration time, query latency, cost savings, and feature adoption
• Contribute to the development of a phased roadmap for full-scale migration and advanced feature rollout

REQUIREMENTS

• Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent professional experience
• 5-8 years of experience in data engineering, with hands-on experience in Databricks and Hadoop ecosystems
• Proficiency in Spark (Scala or PySpark), ETL pipeline development, and data migration practices
• Experience with Lakehouse architecture, Delta Lake, and Databricks advanced features (e.g., Lakehouse Federation, Liquid Clustering, Unity Catalog) is a strong plus
• Solid understanding of cloud platforms (Azure) and data governance concepts
• Strong analytical and problem-solving skills with attention to detail
• Ability to work collaboratively in cross-functional teams and communicate effectively with both technical and non-technical stakeholders

Preferred

• Experience with Databricks Migration Accelerator or similar migration tools
• Familiarity with synthetic data generation and pipeline health monitoring
• Understanding of performance tuning, query optimization, and cost management on cloud data platforms
• Ability to document migration patterns and contribute to best practices for broader adoption

The pay range is the lowest to highest compensation we reasonably in good faith believe we would pay at posting for this role. We may ultimately pay more or less than this range. Employee pay is based on factors like relevant education, qualifications, certifications, experience, skills, seniority, location, performance, union contract and business needs. This range may be modified in the future.

We offer comprehensive benefits including medical/dental/vision insurance, HSA, FSA, 401(k), and life, disability & ADD insurance to eligible employees. Salaried personnel receive paid time off. Hourly employees are not eligible for paid time off unless required by law. Hourly employees on a Service Contract Act project are eligible for paid sick leave.

Note: Pay is not considered compensation until it is earned, vested and determinable. The amount and availability of any compensation remains in Kforce's sole discretion unless and until paid and may be modified in its discretion consistent with the law.

This job is not eligible for bonuses, incentives or commissions.

Kforce is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.

By clicking ?Apply Today? you agree to receive calls, AI-generated calls, text messages or emails from Kforce and its affiliates, and service providers. Note that if you choose to communicate with Kforce via text messaging the frequency may vary, and message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You will always have the right to cease communicating via text by using key words such as STOP.