Azure Databrick Admin

Scope of Work – Azure Databricks Platform Administration

The responsibilities for the Azure Databricks platform administration should include, but are not limited to:
• Platform Setup & Configuration
o Provisioning and configuring Azure Databricks workspaces
o Managing clusters, pools, and job configurations
o Integration with Azure services (ADLS, Snowflake, Monte Carlo, Solidatus, Atlan, Key Vault, Kafka, etc.)

• User Access & Security Management
o Managing role-based access control (RBAC) and permissions
o Integration with Azure Active Directory (AAD)
o Implementing security best practices, including data encryption and secrets management

• Cluster & Job Management
o Monitoring cluster performance, uptime, and cost optimization
o Troubleshooting failed jobs and workflows
o Scheduling and managing automated jobs

• Performance Optimization
o Tuning Spark jobs and clusters for optimal performance
o Managing autoscaling and resource utilization
o Identifying and resolving performance bottlenecks

• Monitoring & Incident Management
o Setting up monitoring, alerting, and logging (Azure Monitor, Log Analytics)
o Proactive issue detection and resolution
o Incident response and root cause analysis

• Cost Management & Optimization
o Tracking usage and optimizing costs across environments
o Providing recommendations for resource efficiency

• CI/CD & Release Management
o Supporting deployment pipelines for Databricks notebooks and code
o Version control integration (e.g., Git)
o Managing environment promotion (Dev, Test, Prod)

• Backup, Recovery & Compliance
o Ensuring backup strategies and disaster recovery readiness
o Supporting compliance requirements and audit activities

• Documentation & Support
o Maintaining operational runbooks and documentation
o Providing periodic reporting and status updates
o Offering L2/L3 support for platform-related issues