Back to Jobs
Recently postedOther

Databricks Architect

Maven CompaniesTarrytown, NY🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeOn Site
LocationTarrytown, NY, United States
SQLGitApache SparkAWSETLAirflowApacheComplianceDatabricksEMRJenkinsPythonRedshiftUnityCI/CDDelta LakeIAM

Job description

Position: Databricks Architect

Location: Tarrytown, NY (Onsite)

Job Type: Long Term Contract

  • Evaluate and drive adoption of new Databricks capabilities, including Lakebase, Databricks Apps, Lakehouse Federation, Serverless, Genie and AI/ML capabilities.
  • Conduct architecture reviews, define reusable patterns and provide technical governance across projects.
  • Partner with business, engineering, and architecture teams to define target-state architectures and modernization roadmaps.

Key Responsibilities & Skills:

  • Lead enterprise-scale data platform modernization initiatives, driving migration from AWS EMR, Apache NiFi, and legacy ETL frameworks to the Databricks Lakehouse Platform.
  • Architect and implement scalable Lakehouse solutions using Databricks, Delta Lake, Unity Catalog, and Databricks Workflows.
  • Design and govern end-to-end data pipelines for batch, streaming, CDC, and real-time data integration workloads.
  • Provide architecture leadership for large-scale AWS-based data ecosystems leveraging S3, IAM, Redshift, Glue Catalog, Airflow, and Databricks.
  • Develop enterprise data architecture standards covering data modelling, metadata management, lineage, governance, security, and compliance.
  • Drive adoption of Databricks best practices including Delta Live Tables (DLT), Auto Loader, Unity Catalog, Serverless Compute, Lakehouse Federation, and advanced optimization techniques.
  • Lead replatforming and migration programmes involving PySpark applications, Airflow DAGs, EMR workloads, Redshift integrations, and NiFi pipelines.
  • Partner closely with business stakeholders, enterprise architects, product owners, and data science teams to define technology roadmaps and target architectures.
  • Establish governance frameworks using Unity Catalog, data quality controls, observability, monitoring, and operational excellence practices
  • Provide technical leadership, mentoring, architecture reviews, design governance, and solution sign-offs across multiple delivery teams.
  • Lead architecture workshops, executive presentations, solution assessments, and technology evaluations.

Mandatory Technical Skills:

  • Databricks Lakehouse Platform
  • Delta Lake, Delta Live Tables (DLT)
  • Unity Catalog
  • Databricks Workflows
  • Auto Loader
  • PySpark, Spark SQL, Python
  • AWS (S3, IAM, Glue, Redshift, EMR, Lambda)
  • Apache Airflow
  • CDC & Data Migration Frameworks
  • Data Governance & Security
  • CI/CD, GitHub, Jenkins

Similar jobs