|
Job Description
|
- Architect and manage a secure, scalable, and highly available Databricks Lakehouse Platform on AWS.
- Design data platforms leveraging AWS S3, Delta Lake, Unity Catalog, and Databricks Workspaces for enterprise analytics and AI.
- Define networking architecture including VPC, PrivateLink, IAM roles, security groups, and encryption standards.
- Establish multi-environment strategies (Dev, Test, UAT, Prod) with automated provisioning through Terraform and Infrastructure as Code (IaC).
- Implement enterprise-wide data governance, lineage, metadata management, and compliance controls using Unity Catalog.
- Design high-performance data ingestion and processing architectures supporting batch, streaming, and real-time analytics workloads.
- Enable AI and GenAI capabilities through Mosaic AI, Vector Search, Model Serving, AI/BI Dashboards, and Genie Spaces.
- Define CI/CD, monitoring, logging, and observability frameworks using GitHub, Jenkins, CloudWatch, and Databricks Workflows.
- Optimize platform performance, reliability, scalability, and cloud costs through FinOps and workload optimization practices.
- Provide architectural leadership and best practices for Lakehouse modernization, advanced analytics, AI governance, and enterprise data platform adoption.
- Core Technologies: Databricks, AWS S3, Delta Lake, Unity Catalog, Spark/PySpark, Terraform, GitHub, CI/CD, CloudWatch, Mosaic AI, Genie, Vector Search, MLflow, Kafka, Airflow, Kubernetes.
Certification :
- DataBricks Platform Administrator /Databricks Certified Data Engineer/
- AWS Certification (DevOps/Solution Architect)
- Terraform
Deliverables:
-Process Flows
-Mentor and Knowledge transfer to client project team members
-Participate as primary, co and/or contributing author on any and all project deliverables associated with their assigned areas of responsibility
-Participate in data conversion and data maintenance
-Provide best practice and industry specific solutions
-Advise on and provide alternative (out of the box) solutions
-Provide thought leadership as well as hands on technical configuration/development as needed.
-Participate as a team member of the functional team
-Perform other duties as assigned.
|
S.No
|
Contractor Qualifying Questions
|
Mandatory
|
|
1
|
Do you have experience as a Databricks on AWS Platform Engineer with Platform Architecture expertise?
|
Yes – Answer of YES is mandatory to qualify
|
|
2
|
Are you local to one of the listed locations?
|
Yes – Answer of YES is mandatory to qualify
|
|
3
|
Do you have a Databricks Platform Administrator or Databricks Certified Data Engineer certification?
|
Yes – Answer of YES is mandatory to qualify
|
|
4
|
Do you have an AWS certification in DevOps or Solutions Architecture?
|
Yes – Answer of YES is mandatory to qualify
|
|
5
|
Do you have experience providing architectural leadership and best practices for Lakehouse modernization, advanced analytics, AI governance, and enterprise data platform adoption?
|
Yes – Answer of YES is mandatory to qualify
|
|
6
|
Do you have hands-on experience with Databricks and AWS S3?
|
Yes – Answer of YES is mandatory to qualify
|
|
7
|
Do you have hands-on experience with Delta Lake?
|
Yes – Answer of YES is mandatory to qualify
|
|
8
|
Do you have hands-on experience with Unity Catalog, Spark/PySpark, Terraform, GitHub, CI/CD, CloudWatch, Mosaic AI, Genie, Vector Search, MLflow, Kafka, Airflow, and Kubernetes?
|
Yes – Answer of YES is mandatory to qualify
|
|
|
Top Skills & Years of Experience
|
|
S.No
|
Custom Skill Requirements
|
|
1
|
Databricks on AWS Platform Engineering and Platform Architecture experience.
|
|
2
|
Architect and manage secure, scalable, and highly available Databricks Lakehouse platforms on AWS.
|
|
3
|
Design enterprise data platforms leveraging AWS S3, Delta Lake, Unity Catalog, and Databricks Workspaces for analytics and AI.
|
|
4
|
Define networking architecture including VPC, PrivateLink, IAM roles, security groups, and encryption standards.
|
|
5
|
Establish multi-environment strategies across Dev, Test, UAT, and Production using Terraform and Infrastructure as Code (IaC).
|
|
6
|
Implement enterprise data governance, lineage, metadata management, and compliance controls using Unity Catalog.
|
|
7
|
Design high-performance data ingestion and processing architectures supporting batch, streaming, and real-time analytics workloads.
|
|
8
|
Enable AI and GenAI capabilities through Mosaic AI, Vector Search, Model Serving, AI/BI Dashboards, and Genie Spaces.
|
|
9
|
Define CI/CD, monitoring, logging, and observability frameworks using GitHub, Jenkins, CloudWatch, and Databricks Workflows.
|
|
10
|
Optimize platform performance, reliability, scalability, and cloud costs through FinOps and workload optimization practices.
|
|
11
|
Provide architectural leadership for Lakehouse modernization, advanced analytics, AI governance, and enterprise data platform adoption.
|
|
