Databricks Azure Consultant

Role: Databricks Azure

Location: Houston, TX - Hybrid 2-3 days

Duration: 6 months

JD:

Role Summary

• We are seeking an experienced Databricks Consultant to design, develop, and implement scalable data solutions on the Databricks platform.
• The ideal candidate will have strong expertise in data engineering, cloud platforms, Spark, and modern data architectures, with the ability to work closely with business and technical stakeholders.

Key Responsibilities

• Design and develop data pipelines using Databricks, Apache Spark, and PySpark.
• Build and maintain ETL/ELT processes for large-scale data ingestion and transformation.
• Implement data lake and lakehouse architectures on Databricks.
• Integrate data from multiple sources, including databases, APIs, and cloud storage.
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
• Collaborate with business users, architects, and developers to understand requirements and deliver solutions.
• Develop data models and support reporting, analytics, and AI/ML initiatives.
• Implement data quality, governance, security, and monitoring standards.
• Support production deployments, troubleshooting, and performance tuning.
• Prepare technical documentation and provide knowledge transfer to project teams.

Required Skills

• Strong hands-on experience with Databricks.
• Expertise in Apache Spark, PySpark, and SQL.
• Experience with cloud platforms such as Azure, AWS, or Google Cloud Platform.
• Knowledge of Delta Lake, Unity Catalog, and Databricks Workflows.
• Experience with data warehousing and data lake concepts.
• Strong understanding of ETL/ELT frameworks and data modeling.
• Familiarity with CI/CD, Git, and DevOps practices.
• Excellent problem-solving and communication skills.

Preferred Skills

• Experience with Azure Data Factory (ADF), Azure Synapse, or AWS Glue.
• Knowledge of Machine Learning and MLOps on Databricks.
• Experience with streaming technologies such as Kafka and Structured Streaming.
• Databricks or Cloud Platform certifications.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
• 5+ years of experience in Data Engineering/Big Data.
• 2+ years of hands-on Databricks implementation experience.

Nice to Have

• Databricks Certified Data Engineer Associate/Professional.
• Experience in Utilities, Energy, Banking, Healthcare, or Retail domains.
• Experience with Agile delivery methodologies.
• Primary Technologies: Databricks, PySpark, Spark SQL, Delta Lake, Azure/AWS/Google Cloud Platform, ADF, Kafka, Git, CI/CD.

Experience: 7+ Years