Python Backend Engineer (Cloud Infrastructure)

JOB Title: Python Backend Engineer (Cloud Infrastructure)

Experience: 5+ Years

Type: Contract

Location: Remote, US (Onshore)

Duration: 6 Months

Working Hours: EST

Interview Hours: EST

Job Summary

We are seeking an experienced Senior Python Engineer - Databricks & Data Migration with strong hands-on expertise in Databricks, Terraform (Infrastructure as Code), and large-scale database/data platform migrations, along with familiarity with AWS cloud services.

The ideal candidate should have proven experience managing Databricks environments, automating infrastructure deployments, executing complex data migrations, and optimizing data pipelines.

Healthcare data experience is mandatory, including experience working with clinical, claims, patient, provider, or other healthcare datasets.

This is a hands-on engineering role requiring strong technical ownership, troubleshooting capabilities, and the ability to independently drive infrastructure and migration initiatives end to end.

Key Responsibilities

Design, configure, deploy, and manage Databricks environments, including clusters, workflows, security, and performance optimization.

Build and automate infrastructure deployments using Terraform and Infrastructure as Code (IaC).

Lead end-to-end database and data platform migrations, including schema migration, ETL modernization, data validation, and performance tuning.

Develop, optimize, and troubleshoot large-scale data pipelines using Databricks, PySpark, Python, and SQL.

Implement scalable data ingestion, transformation, and migration workflows.

Work with cloud infrastructure and services, with familiarity with AWS services such as S3, IAM, EC2, Glue, and RDS.

Optimize Databricks workloads for scalability, reliability, and cost efficiency.

Ensure data quality, security, and HIPAA compliance when handling healthcare datasets.

Troubleshoot infrastructure, pipeline failures, data inconsistencies, and migration-related issues.

Independently own technical deliverables while collaborating with engineering teams.

Required Skills

1. Databricks Architecture & Administration – Mandatory

Strong hands-on experience with Databricks architecture, administration, and production deployments.

Experience with clusters, workflows, Delta Lake, Unity Catalog, access controls, and performance optimization.

Ability to troubleshoot and optimize Databricks environments.

2. Terraform & Infrastructure as Code – Mandatory

Strong experience with Terraform and IaC for infrastructure provisioning and automation.

Experience managing infrastructure configurations, reusable modules, and deployment automation.

Understanding of cloud infrastructure architecture and security.

3. Database & Data Platform Migrations – Mandatory

Proven experience executing large-scale database or data platform migrations.

Hands-on expertise in schema migration, ETL/ELT modernization, data reconciliation, validation, and performance tuning.

Experience migrating legacy or on-premises data platforms to modern cloud-based architectures.

4. Data Engineering – Mandatory

Strong proficiency in SQL, PySpark, Python, Apache Spark, and ETL/ELT pipelines.

Experience building and optimizing scalable data processing workflows.

Strong troubleshooting and performance optimization capabilities.

5. Healthcare Data Experience – Mandatory

Hands-on experience working with healthcare datasets, including clinical, claims, patient, provider, or EHR/EMR data.

Understanding of HIPAA, PHI handling, healthcare data security, and compliance.

Experience with HL7/FHIR healthcare interoperability standards is preferred.

6. AWS Cloud Familiarity – Required

Working knowledge of AWS cloud infrastructure and services.

Familiarity with S3, IAM, EC2, Glue, RDS, and cloud networking concepts.

Understanding of cloud-based data storage, security, and integration patterns.

Deep AWS architecture or administration expertise is not mandatory.

Preferred Qualifications

Databricks or Terraform certifications.

Experience working with Databricks on AWS, Azure, or Google Cloud Platform.

Familiarity with AWS DMS, AWS Glue, or other migration tools.

Experience with CI/CD pipelines and GitHub Actions.

Experience handling large-scale healthcare data migrations.

Strong communication, problem-solving, and independent technical ownership.

Critical Hiring Priorities

Databricks Architecture & Administration – Mandatory

Terraform / Infrastructure as Code – Mandatory

Large-Scale Database & Data Platform Migrations – Mandatory

Healthcare Data Experience – Mandatory

SQL, PySpark, ETL/ELT & Performance Optimization – Mandatory

AWS Cloud Familiarity – Required (Deep Expertise Not Mandatory)

Independent Ownership & Troubleshooting – Mandatory

Important Note for Recruitment:

We are prioritizing candidates with strong production experience in Databricks, Terraform, and large-scale data migrations, along with familiarity with AWS.

Candidates with Databricks experience on Azure or Google Cloud Platform can also be considered, provided they have strong Terraform and migration experience and a working understanding of AWS.

Profiles primarily focused on Python development, AI/ML, or LLM applications without substantial Databricks infrastructure and data migration experience will not be considered a strong fit.