Databricks Architect - Tarrytown, NY [Hybrid]

Job Details:

Role Summary

We are looking for a strong Databricks Architect to lead enterprise-scale Lakehouse architecture and implementation on AWS. The role involves designing scalable data solutions, modernizing existing data platforms, establishing governance standards, and leading architecture discussions with customers.

Key Responsibilities

• Lead enterprise-scale Databricks Lakehouse architecture and implementation on AWS.

• Drive migration of EMR, NiFi, Redshift, and legacy ETL workloads to Databricks.

• Evaluate newer Databricks capabilities, including Serverless, Lakehouse Federation, Genie, Databricks Apps, Lakebase, and AI/ML.

• Architect scalable batch, streaming, CDC, and real-time data solutions.

• Design reusable frameworks using Delta Lake, PySpark, Auto Loader, DLT, and Databricks Workflows.

• Define governance, security, lineage, and data-quality standards using Unity Catalog.

• Establish best practices for performance, scalability, and cost optimization.

• Lead customer workshops, architecture reviews, and solution assessments.

• Define target-state architecture, migration roadmaps, and reusable patterns.

Required Skills & Qualifications

• 15+ years of overall Data & Analytics experience.

• 4+ years of hands-on Databricks experience as specified in the JD.

• Strong experience with Databricks, Delta Lake, Unity Catalog, DLT, and Auto Loader.

• Proficiency in PySpark, Spark SQL, Python, and Structured Streaming.

• AWS experience with S3, IAM, Glue, Redshift, EMR, and Lambda.

• Experience with Airflow, CDC, and enterprise data migration.

• Knowledge of data governance, security, and observability.

• Strong customer-facing, solution architecture, and technical leadership skills.

Preferred Qualifications

• Healthcare or pharmaceutical domain experience is preferred but not mandatory.

• Experience with enterprise-scale data platform modernization and migration is beneficial.