Sr. Data Engineer

Join a dynamic data engineering team supporting large-scale analytics and cloud data solutions for a hybrid onsite role in San Francisco, CA. This position focuses on building reliable, high-performance pipelines and modern data platform capabilities using Databricks, Azure Data Factory, and DevOps practices.
Responsibilities

• Design, build, and maintain scalable data pipelines and analytics solutions for big data environments.
• Develop and optimize workflows that support efficient, reliable data processing and delivery.
• Work closely with cross-functional partners to translate data needs into actionable technical solutions.
• Support data platform operations with a focus on quality, availability, performance, and stability.
• Apply cloud and DevOps practices to automate data workflows and improve delivery efficiency.
• Troubleshoot issues across data pipelines and environments to help ensure smooth day-to-day operations.
• Contribute to data governance, security, and compliance practices within the data ecosystem.
• Share knowledge and help mentor others while promoting collaborative ways of working.

Skills

• 8+ years of experience in data engineering with a strong big data background.
• Hands-on experience with Databricks and Azure Data Factory.
• Experience with DevOps tools and practices related to data engineering and CI/CD.
• Strong working knowledge of Hadoop, Spark, Kafka, or similar distributed data technologies.
• Proficiency in Python, Scala, or Java.
• Solid understanding of SQL and NoSQL databases.
• Experience building ETL processes, data pipelines, and real-time or batch data solutions.
• Strong problem-solving skills and the ability to communicate effectively with technical and business stakeholders.

Preferred Skills

• Experience with cloud data platforms such as Azure, AWS, or Google Cloud Platform.
• Knowledge of Flink or other streaming/data processing frameworks.
• Familiarity with data modeling and distributed computing concepts.
• Background supporting machine learning pipelines or AI-enabled data solutions.
• Experience in Agile environments and with team leadership or mentoring.
• Knowledge of California emissions-related data or compliance work is a plus.

Horizontal Talent is committed to fostering an inclusive workplace where diverse perspectives are valued and everyone has the opportunity to thrive. We welcome candidates of all backgrounds and experiences who are excited to contribute to a collaborative, respectful, and equitable environment.

By applying for this position, you acknowledge and agree that Horizontal Talent may contact you regarding your application using automated technology, including phone calls, SMS/text messages, or email, which may be delivered by our virtual AI recruiter, Alex.