Data Engineer

Software Guidance & Assistance, Inc., (SGA), is searching for a Data Engineer for a contract assignment with one of our premier investment management clients in San Jose, CA. This is a hybrid role with 2-3 days per week in office.
We are looking for a Cloud & Data Engineer to help build, optimize, and operate our Voice of Customer (VoC) AI Platform. The platform ingests structured and unstructured data from multiple enterprise sources with varying degrees of signal-to-noise ratio and is primarily built on Microsoft Azure, with data processing and ETL/ELT workloads powered by Azure Databricks. This is a hands-on role spanning data engineering, cloud infrastructure, and platform reliability.
Responsibilities:

• Data Engineering
  • Build and maintain ETL/ELT pipelines for structured and unstructured data sources.
  • Develop data-processing workloads using Azure Databricks, Python/PySpark, and SQL.
  • Optimize Databricks jobs, compute, and data layouts for performance and cost.
  • Work with Delta Lake and Unity Catalog for data storage, governance, and access control.
  • Implement data quality checks, monitoring, error handling, and recovery mechanisms.
• Cloud & Platform Engineering
  • Build, operate, and optimize Azure services including App Service, Functions, Blob/ADLS storage, Azure SQL DB for Postgres, Key Vault, Container Registry and Azure Monitor.
  • Experience with cloud AI services including vector search and embedding pipelines.
  • Improve platform performance, scalability, reliability, security, and cost efficiency.
  • Implement secure identity, and access management using Azure managed identities, Key Vault, secrets managed via scopes.
  • Support development, staging, and production environments.
• Automation & Operations
  • Improve platform observability through logging, monitoring, dashboards, and alerting.
  • Automate infrastructure and deployments using Infrastructure as Code (IaC) and CI/CD.
  • Troubleshoot issues across cloud services, storage, databases, networking, and data pipelines.
  • Support production incidents and root-cause analysis.
  • Collaborate with application, AI/ML, and data engineering teams.

Required Skills:

• 3-5 years of experience in data engineering, cloud engineering, or a related role.
• Hands-on experience with Microsoft Azure, Databricks, APIs, and object storage.
• Strong skills in Python/PySpark, SQL.
• Experience building and operating production data ETL/ELT pipelines.
• Experience with relational and semi-structured/unstructured data, database technologies.
• Experience with Spark, Delta Lake, and lakehouse architectures.
• Working knowledge of Databricks Unity Catalog and data-access controls.
• Understanding of cloud security concepts such as RBAC, managed identities, service principals, and Key Vault.
• Experience with Git, CI/CD, monitoring, troubleshooting, and performance optimization.

Preferred Skills:

• Experience with Terraform or ARM templates.
• Experience with Azure DevOps or GitHub Actions.
• Familiarity with Azure networking, including VNets and private endpoints.
• Experience with Azure Data Factory, Databricks Workflows, or Airflow.
• Familiarity with data platforms supporting AI/ML, NLP, analytics, or GenAI applications.

SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at .
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company to request an accommodation or assistance regarding our policy.