Databricks Platform Architect

Job Title: Databricks Platform Architect

Work Location: New York, NY (Remote for now)

Contract duration: Long term contract

Visa: /Only

Job Details: 

Key Responsibilities

Platform Engineering & Architecture

• Design, implement, and maintain Databricks platform architecture to support enterprise-scale data workloads
• Establish best practices for workspace setup, cluster management, and job orchestration
• Optimize platform performance, scalability, and cost efficiency
• Define and enforce standards for data engineering and analytics workloads within Databricks

Developer Enablement

• Support development teams by providing guidance, documentation, and reusable frameworks
• Enable developers with CI/CD pipelines, automation, and deployment standards
• Troubleshoot and resolve platform-related issues impacting development teams
• Provide technical mentorship and onboarding support to engineers using Databricks

Governance, Security & Compliance

• Implement and manage data governance, access controls, and security models (Unity Catalog, RBAC, etc.)
• Ensure compliance with enterprise data policies and regulatory requirements
• Monitor and audit platform usage, access patterns, and data lineage

Integration & Ecosystem Management

• Integrate Databricks with cloud platforms (AWS/Azure/Google Cloud Platform), data lakes, and enterprise systems
• Manage connectivity with tools such as Power BI, APIs, data ingestion frameworks, and orchestration tools
• Support ingestion and processing frameworks (e.g., batch, streaming, Delta Lake)

Operational Excellence

• Monitor platform health, performance, and usage metrics
• Implement automation for provisioning, monitoring, and scaling
• Drive continuous improvements in platform reliability and developer experience

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
• 5+ years of experience in data engineering or platform engineering roles
• 3+ years of hands-on experience with Databricks (architecture, administration, or platform engineering)
• Strong understanding of distributed data processing (Spark)
• Experience with cloud platforms (Azure preferred, or AWS/Google Cloud Platform)
• Knowledge of data lake architecture, Delta Lake, and modern data platforms
• Experience implementing security, governance, and access control frameworks
• Familiarity with CI/CD pipelines, DevOps practices, and infrastructure as code (Terraform, etc.)