Quick overview
Job description
Senior Infrastructure Engineer – Terraform & Databricks
San Francisco, CA
Type : Contract
Need candidates who can work onsite from Day 1 (3 days a week : Tuesday, Wednesday and Thursday – 3 days work from office) – NO REMOTE
Highly preferred local candidates
Experience: 09+ years
About the role
Building an enterprise Databricks Lakehouse on AWS, and all of it is managed as code. In this role you’ll design, build and run the Terraform/Terragrunt foundation behind our Databricks workspaces, Unity Catalog, compute and security controls.
You’ll work closely with Cloud Infrastructure/SRE, Security Engineering and Data Engineering. Your goal is to take our platform from sandbox to production-ready multi-environment setup that is secure, well governed and cost-efficient.
What you will do
- Design, build and maintain reusable Terraform modules for Databricks account-level (MWS) and workspace-level resources. This covers workspaces, Unity Catalog (metastores, catalogs, schemas, volumes, grants), cluster policies, instance pools, SQL warehouses, jobs, DLT pipelines and service principals.
- Use Terragrunt to manage environments consistently across sandbox, nonprod, preprod and prod, with minimal configuration drift.
- Provision and secure the supporting AWS infrastructure: customer-managed VPCs, PrivateLink/VPC endpoints, security groups, NACLs, IAM roles and policies, S3, KMS customer-managed keys (CMK) and Secrets Manager.
- Lead security hardening. That means fixing findings from security architecture reviews and cloud security posture management (CSPM) tools such as Wiz, enforcing PrivateLink-only access, IP access lists, CMK encryption and secret/OAuth token rotation.
- Set up identify and access controls, including Okta SSO and SCIM sync, role-based (RBAC) and tag-based (ABAC) access in Unity Catalog, and least-privilege service principals.
- Build and own CI/CD for infrastructure. This includes GitHub Actions plan/apply workflows with approvals, pre-commit hooks, Terratest, and security scanning with Checkov or tfsec.
- Set up monitoring and alerting with CloudWatch, Grafana and Splunk log forwarding. That covers cluster and job health, token expiry and cost anomalies.
- Drive FinOps practices: cluster policy guardrails, spot usage, moving work-loads from all-purpose to job clusters, enforcing tagging, and analysing DBU usage.
- Enable integrations with Dagster, dbt Cloud and BI tools such as Tableau, Power BI and Looker via SQL warehouses.
- Write and maintain architecture docs, ADRs, runbooks and module references, and enforce naming and tagging standards.
- Databricks – Hands-on build data pipelines using Pyspark
- Used Agentic IDE to build data pipelines
- Understands Medallion architecture
- Understands ML flow – Model inferencing
- Have domain expertise or understanding of Credit decisioning or Risk or Servicing (collections)
Skills Required
- 7+ years in infrastructure, platform or DevOps engineering with atleast 3 years of hands-on Terraform in production. Terragrunt experience is strongly preferred.
- 2+ years running Databricks on AWS, including workspace provisioning, Unity Catalog, cluster policies and the Databricks Terraform provider.
- Deep AWS knowledge; VPC networking, PrivateLink, IAM, KMS, S3, Secrets Manager and CloudWatch.
- Experience with CI/CD for Infrastructure as code (GitHub Actions, Jenkins or similar) and with GitOps workflows.
- Strong grasp of cloud security and compliance: encryption, network isolation, least privilege and audit logging. Experience in regulated environments such as financial services is a plus.
- Scripting skills in Python and/or Bash. Go is a plus for Terratest.
- Comfortable with Git, Code reviews and writing clear technical documentation.
Nice to have
- Databricks certifications (Data Engineer Associate/Professional or Platform Administrator), or HashiCorp Terraform Associate.
- Experience with Dela Live Tables, Vector Search, Model Serving or MLops Infrastructure.
- Familiarilty with Dagster, Dbt, Spark and Delta Lake.
- Experience with Wiz, Checkov, tfsec or similar policy-as-code and CSPM tools.
- Background in FinOps or cloud cost optimization.
- Experience with SOC 2 or similar compliance frameworks
Similar jobs
- ML
Senior QA Automation EngineerRecently posted
Masterapp Labs · Glen Allen, VA · On-site · Posted 16 Nov 2026 🇺🇸
JiraTechnologySalary not listed - MI
Workday Financials ConsultantRecently posted
Mindlance · New York, NY · Hybrid · Posted 16 Nov 2026 🇺🇸
WorkdayBudgetingERP+6TechnologySalary not listed - TC

Systems AdministrationRecently posted
Tri-Force Consulting Services Inc · New York, NY · On-site · Posted 16 Nov 2026 🇺🇸
Account ManagementActive DirectoryCompliance+3TechnologySalary not listed - SI

Storage Engineer | Storage & Backup | Experienced HireRecently posted
Susquehanna International Group, LLP · Bala Cynwyd, PA · Hybrid · Posted 16 Nov 2026 🇺🇸
AnsibleBashGrafana+3TechnologySalary not listed - AS

Python Kafka DeveloperRecently posted
Amaze Systems Inc · Farmington, WA · On-site · Posted 16 Nov 2026 🇺🇸
DockerMicroservicesSQL+19TechnologySalary not listed - CS

Senior Java DeveloperRecently posted
Cynet Systems · San Antonio, TX · Hybrid · Posted 16 Nov 2026 🇺🇸
DockerSpringSpring Boot+17Technology$45 - $50/hr - RI

Senior Backend Python DeveloperRecently posted
Rivago infotech inc · Charlotte, NC · Hybrid · Posted 16 Nov 2026 🇺🇸
DjangoFlaskMicroservices+18TechnologySalary not listed - BE

Workday Financial Solution ArchitectRecently posted
BESTPEGASUS LLC · New York, NY · On-site · Posted 16 Nov 2026 🇺🇸
Power BITechnologySalary not listed - IN

M365 SharePoint Online EngineerRecently posted
InterSources Inc. · Blythewood, SC · Hybrid · Posted 16 Nov 2026 🇺🇸
Microsoft OfficeSharePointWindows ServerTechnologySalary not listed - GL
Senior Software Engineer – Security Automation & AnalyticsRecently posted
Geopaq Logic · Seattle, WA · Hybrid · Posted 16 Nov 2026 🇺🇸
SQLGitLooker+9TechnologySalary not listed - SE
SAP R2R Lead - W2 CONTRACTRecently posted
SunRay Enterprise Inc · United States · On-site · Posted 16 Nov 2026 🇺🇸
Stakeholder ManagementManagement AccountingReconciliation+2TechnologySalary not listed - TM

Mission Systems Engineer, LeadRecently posted
The MITRE Corporation · McLean, VA · On-site · Posted 16 Nov 2026 🇺🇸
MATLABSQLPythonTechnology$146.4k - $183k/yr