Big Data Engineer

Seeking a Big Data Engineer to develop, maintain, and optimize scalable data solutions using Databricks, Apache Spark, Azure Data Factory, Delta Lake, and Azure Data Lake Storage. The consultant will build ETL/ELT pipelines, optimize Spark workloads, automate deployments, and maintain data quality, governance, and security.

Required Experience

• 4+ years with ETL/ELT, Python, R, SQL, Azure, CI/CD, DevOps, data modeling, data lakes, and data warehouses.
• 3+ years with Databricks, Apache Spark, Spark SQL, Delta Lake, Azure Data Factory, and Unity Catalog.
• Strong understanding of Spark architecture, RDDs, DataFrames, clusters, notebooks, and jobs.
• Experience implementing data validation, metadata management, lineage, cataloging, encryption, access controls, and auditing.
• Strong troubleshooting, Agile delivery, and cross-functional collaboration skills.

Preferred

• MLflow, Scikit-learn, or TensorFlow experience.
• Databricks Certified Associate Developer for Apache Spark.
• Microsoft Azure Data Engineer Associate certification.

Important Requirements

Selected candidates must pick up equipment in Austin on the first day and may be required to attend onsite meetings with minimal notice.