Senior Databricks Engineer

Who we are:

ConnectedX is focused on Digital Transformation and Product Engineering Services, enabling clients achieve their business, operating and technology needs for the digital age. Our unique industry-based, consultative approach helps clients implement digital transformation initiatives. Headquartered in Dallas, Texas, U.S. ConnectedX is a preferred partner for leading Fortune 1000 enterprises and is consistently admired by the clients and employees across the globe.

What we are looking for:

Job Title: Senior Databricks Genie Engineer

Experience: 10+ Years
Location: Plano,Texas
Employment Type: Contract

We are seeking a highly experienced Databricks Genie Engineer with 10+ years of data engineering experience and deep expertise in the Databricks Lakehouse platform. The ideal candidate will design, build, and optimize scalable data pipelines, implement AI-powered data interactions using Databricks Genie, and drive enterprise-grade data solutions across cloud environments.

You will play a key role in architecting modern data platforms, enabling self-service analytics through natural language querying (Genie), and ensuring high performance, security, and cost optimization across large-scale workloads.

Key Responsibilities

Databricks & Lakehouse Engineering

• Design and implement scalable data pipelines using Databricks (PySpark, SQL).
• Architect and manage Delta Lake tables (Bronze, Silver, Gold layers).
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement Unity Catalog for data governance and access control.
• Manage cluster configuration, autoscaling, and job orchestration.

Databricks Genie Implementation

• Configure and optimize Databricks Genie for natural language-to-SQL analytics.
• Build semantic models and curated datasets to support Genie use cases.
• Improve AI-driven query accuracy and optimize prompt engineering.
• Collaborate with business users to enable self-service analytics.
• Troubleshoot and tune Genie-generated SQL for performance and correctness.

Data Architecture & ELT

• Design modern ELT architectures leveraging Delta Lake.
• Implement incremental processing (CDC, streaming with Structured Streaming).
• Develop data quality frameworks and validation pipelines.
• Handle schema evolution and medallion architecture best practices.

Cloud & Integration

• Work across AWS / Azure / Google Cloud Platform cloud platforms.
• Integrate with S3 / ADLS / GCS object storage.
• Implement CI/CD pipelines for Databricks deployments.
• Manage secrets, tokens, and secure connectivity.
• Integrate BI tools (Power BI, Tableau) and AI/ML workflows.

Required Qualifications

• 10+ years of experience in Data Engineering.
• 5+ years of hands-on experience with Databricks.
• Strong expertise in PySpark and Spark SQL.
• Deep understanding of Delta Lake and Lakehouse architecture.
• Experience implementing Databricks Genie or AI-driven analytics solutions.
• Strong experience with cloud platforms (AWS/Azure/Google Cloud Platform).
• Expertise in building scalable ELT pipelines.
• Solid understanding of data governance and security frameworks.
• Experience with CI/CD (Azure DevOps, GitHub Actions, Terraform, etc.).
• Strong troubleshooting and debugging skills.

Preferred Qualifications

• Experience with MLflow and MLOps.
• Experience with streaming frameworks (Kafka, Event Hub).
• Certification in Databricks (Databricks Data Engineer Professional).
• Experience with data cataloging and metadata management tools.
• Experience with large enterprise/global data platforms.

Key Skills

• Databricks Genie
• PySpark
• Delta Lake
• Spark Optimization
• Unity Catalog
• Lakehouse Architecture
• CI/CD
• Cloud Data Engineering
• AI-driven Analytics
• Data Governance

Soft Skills

• Strong stakeholder communication
• Solution-oriented mindset
• Ability to lead technical initiatives
• Mentorship experience
• Strong documentation skills