Data Engineer (Databricks, Azure, Kafka)

Job Title: Data Engineer (Databricks, Azure, Kafka)

Location: Remote

Job Type: W2 / C2C / 1099

Job Summary:
We are looking for an experienced Data Engineer with 9+ years of experience to design, build, and maintain data pipelines using Databricks, Spark, and Azure cloud services. The ideal candidate has hands-on experience with modern data engineering tools, cloud automation, and event-driven architectures.

Job Responsibilities:

• Design, develop, and optimize scalable data pipelines using Databricks and Spark (PySpark)
• Build and maintain ETL/ELT workflows integrating with Snowflake and other data platforms
• Develop and manage Azure-based data solutions using ADF, ADLS, Azure Functions, and Key Vault
• Implement CI/CD pipelines and infrastructure automation using GitHub Actions, Terraform, and DevOps best practices
• Build and maintain event-driven architectures using Kafka, including CDC and streaming integrations
• Work with AI/GenAI initiatives involving LLM fine-tuning, RAG, and prompt engineering
• Monitor, troubleshoot, and optimize data pipeline performance and reliability
• Collaborate with data architects, analysts, and business stakeholders to translate requirements into technical solutions
• Ensure data quality, security, and governance standards are met across pipelines
• Orchestrate and schedule workflows using Airflow

Required Skills:

• 8+ years of experience in Data Engineering
• Data Engineering: Databricks, Spark (PySpark), Python/Scala, SQL, Snowflake
• Azure: Azure Cloud Services, ADF, ADLS, Azure Functions, Key Vault, DevOps
• AI/GenAI: LLM Fine-Tuning, RAG, Prompt Engineering
• Automation & DevOps: Airflow, GitHub, GitHub Actions, CI/CD, Terraform, Infrastructure Automation
• Kafka, Event-Driven Architecture, CDC/Streaming Integrations

Preferred Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or related field
• Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect) is a plus
• Experience working in large-scale enterprise or regulated industry environments (finance, healthcare, insurance)
• Strong communication skills with ability to work directly with client stakeholders