Start searching
Back to Jobs
Recently postedTechnology

Data Engineer

MW Partners LLCSan Jose, CA🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationSan Jose, CA, United States
SQLGitApache SparkETLNLPAirflowDatabricksGitHub ActionsPythonTerraformUnityVaultGenerative AIAzureCI/CDAzure DevOpsAzure Data FactoryDelta LakeIAM

Job description

Responsibilities and duties:

Data Engineering

  • Build and maintain ETL/ELT pipelines for structured and unstructured data sources.
  • Develop data-processing workloads using Azure Databricks, Python/PySpark, and SQL.
  • Optimize Databricks jobs, compute, and data layouts for performance and cost.
  • Work with Delta Lake and Unity Catalog for data storage, governance, and access control.
  • Implement data quality checks, monitoring, error handling, and recovery mechanisms.

Cloud & Platform Engineering

  • Build, operate, and optimize Azure services including App Service, Functions, Blob/ADLS storage, Azure SQL DB for Postgres, Key Vault, Container Registry, and Azure Monitor.
  • Experience with cloud AI services including vector search and embedding pipelines.
  • Improve platform performance, scalability, reliability, security, and cost efficiency.
  • Implement secure identity and access management using Azure managed identities, Key Vault, and secrets managed via scopes.
  • Support development, staging, and production environments.

Automation & Operations

  • Improve platform observability through logging, monitoring, dashboards, and alerting.
  • Automate infrastructure and deployments using Infrastructure as Code (IaC) and CI/CD.
  • Troubleshoot issues across cloud services, storage, databases, networking, and data pipelines.
  • Support production incidents and root-cause analysis.
  • Collaborate with application, AI/ML, and data engineering teams.

Requirements:

  • 3–5 years of experience in data engineering, cloud engineering, or a related role.
  • Hands-on experience with Microsoft Azure, Databricks, APIs, and object storage.
  • Strong skills in Python/PySpark and SQL.
  • Experience building and operating production data ETL/ELT pipelines.
  • Experience with relational and semi-structured/unstructured data, database technologies. 
  • Experience with Spark, Delta Lake, and lakehouse architectures.
  • Working knowledge of Databricks Unity Catalog and data-access controls.
  • Understanding of cloud security concepts such as RBAC, managed identities, service principals, and Key Vault.
  • Experience with Git, CI/CD, monitoring, troubleshooting, and performance optimization.

Nice to Have:

  • Experience with Terraform or ARM templates.
  • Experience with Azure DevOps or GitHub Actions.
  • Familiarity with Azure networking, including VNets and private endpoints.
  • Experience with Azure Data Factory, Databricks Workflows, or Airflow.
  • Familiarity with data platforms supporting AI/ML, NLP, analytics, or GenAI applications.

Similar jobs

Apply