Databricks Architect

• We are seeking a highly skilled Databricks Engineer with AWS expertise to support Data Services portfolio
• The role will be responsible for designing developing optimizing and supporting enterprise scale data platforms data pipelines and analytics solutions leveraging Databricks Lakehouse AWS Cloud and modern data engineering practices
• The candidate will work closely with data architects’ application teams business stakeholders and platform engineering teams to deliver scalable secure and high-performance data solutions

Key Responsibilities

• Design develop and maintain scalable data pipelines using Databricks PySpark Spark SQL and Delta Lake
• Build and support enterprise data lake and Lakehouse solutions on AWS
• Develop batch and Realtime ingestion frameworks from multiple source systems
• Implement data transformation enrichment cleansing and validation processes
• Optimize Spark workloads for performance scalability and cost efficiency
• Design and implement data models supporting reporting analytics and downstream applications
• Integrate Databricks with AWS services including S3 Glue Lambda Redshift EventBridge and IAM
• Support data migration and modernization initiatives from traditional data warehouses to Databricks Lakehouse
• Implement CICD pipelines and Infrastructure as Code IaC using Terraform and Gitbased repositories
• Collaborate with security teams to ensure compliance with client governance risk and data security standards
• Troubleshoot production incidents and provide L3 operational support
• Participate in architecture reviews code reviews and technical governance activities

Mandatory Technical Skills

• Databricks
• Databricks Lakehouse Architecture
• Databricks Workflows
• Delta Lake
• Unity Catalog
• Databricks SQL
• Auto Loader
• Structured Streaming
• Databricks Performance Tuning
• Medallion Architecture
• AWS
• Amazon S3
• AWS Glue
• AWS Lambda
• AWS IAM
• Amazon Redshift
• AWS CloudWatch
• SNS SQS
• EventBridge
• KMS
• Secrets Manager
• Data Engineering
• PySpark
• Apache Spark
• Python
• SQL
• Data Warehousing Concepts
• ETLELT Design
• Data Modeling
• Data Quality Frameworks
• Batch RealTime Processing

Preferred Skills

• Snowflake
• Microsoft Fabric
• Google Cloud Platform BigQuery
• Kafka
• Airflow
• dbt
• Terraform
• Docker
• Kubernetes OpenShift
• Informatica
• Collibra
• Data Vault Modeling
• Dimensional Data Modeling
• Banking Domain Experience
• Strong experience within Banking and Financial Services
• Understanding of enterprise data governance regulatory reporting risk compliance and audit requirements
• Experience working in large global organizations with stringent security and operational standards

Qualifications

• Bachelors or Masters degree in Computer Science Engineering Information Systems or related discipline
• Databricks Certified Data Engineer Preferred
• AWS Certified Data Analytics or AWS Solution Architect Preferred