Databricks Solution Architect SME

Title: Databricks Solution Architect SME
Location: Remote,
Duration: 12+ Months

Key Responsibilities:
Architecture & Platform Design
•    Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework
•    Define reference architectures for batch, streaming, analytics, and ML workloads
•    Select and standardize cluster, compute, and workspace architectures
•    Design multi-workspace strategies (dev/test/prod, shared vs. isolated)
•    Ensure architectures meet scalability, availability, and performance requirements
Well-Architected Framework Alignment

Apply Databricks best practices across all pillars, including:
•    Security & Governance (Unity Catalog, IAM, data access controls)
•    Reliability & Resilience (job retries, checkpointing, failure isolation)
•    Performance Efficiency (cluster sizing, autoscaling, caching)
•    Cost Optimization (compute policies, workload separation, monitoring)
•    Operational Excellence (monitoring, automation, CI/CD, runbooks)

Implementation & Engineering
•    Lead Databricks workspace, cluster, and Unity Catalog implementations
•    Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming
•    Build and optimize ETL/ELT pipelines using Spark and SQL
•    Integrate Databricks with cloud services (S3/ADLSS, IAM, Key Vault, networking)
•    Establish CI/CD pipelines for notebooks, jobs, and infrastructure

Security, Governance & Compliance
•    Implement Unity Catalog for centralized governance
•    Define data classification, lineage, and audit strategies
•    Enforce least-privilege access and secure networking patterns
•    Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)

Operations & Optimization
•    Monitor platform health, performance, and cost
•    Troubleshoot production issues across jobs, clusters, and data pipelines
•    Perform workload tuning and cost-performance optimization
•    Define SLOs, alerts, and operational metrics

Collaboration & Leadership
•    Partner with Data Engineering, Analytics, ML, Platform, and Security teams
•    Translate business requirements into technical architectures
•    Provide architectural guidance and technical mentorship
•    Communicate risks, tradeoffs, and recommendations to leadership

Required Qualifications:
Experience
•    7+ years in data engineering, analytics, or platform architecture
•    3–5+ years hands-on Databricks experience in production environments
•    Proven experience applying the Databricks Well-Architected Framework
•    Experience designing cloud-native lakehouse architectures
•    Experience supporting mission-critical data platforms

Technical Skills
•    Databricks Lakehouse Platform
•    Apache Spark (PySpark / Scala / Spark SQL)
•    Delta Lake, Delta Live Tables, Structured Streaming
•    Unity Catalog (governance, lineage, access controls)
•    Cloud platforms: AWS, Azure, or Google Cloud Platform
•    Infrastructure as Code (Terraform strongly preferred)
•    CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)
•    Data formats and protocols (Parquet, JSON, Avro)

Certifications Required:
•    Databricks Certified Data Engineer Professional
•    Databricks Certified Professional Architect (or equivalent advanced certification)

Preferred / Additional Certifications
•    AWS Certified Solutions Architect (Associate or Professional)
•    Azure Solutions Architect Expert
•    Google Professional Data Engineer
•    Databricks Machine Learning Professional
•    Snowflake or other cloud data platform certifications

Soft Skills
•    Strong architectural decision-making and documentation skills
•    Excellent communication with technical and non-technical stakeholders
•    Ability to lead design reviews and architecture governance forums
•    Strong troubleshooting and performance-tuning mindset
Nice-to-Have Experience
•    MLflow and MLOps architectures
•    Real-time analytics and streaming pipelines
•    Multi-region or cross-account data architectures
•    Consulting or MSP delivery experience