Start searching
Back to Jobs
Recently postedTechnology

Lead Data Engineer - Databricks

Wise Skulls Corp.United States🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationUnited States
Stakeholder ManagementSQLApache SparkAWSETLLookerSnowflakeTableauAirflowApacheDatabricksGoogle CloudHadoopHiveKafkaPythonAzurePower BICI/CD

Job description

Responsibilities
  • ⁠⁠⁠⁠⁠⁠⁠Data Engineering & Platform Development
  • Design, develop, and maintain scalable data pipelines and data products supporting analytics, reporting, AI, and operational use cases.
  • Build and optimize ETL/ELT frameworks for large-scale data ingestion, transformation, validation, and consumption.
  • Develop and manage cloud-native data platforms leveraging Snowflake, AWS, Apache Airflow and modern data architectures.
  • Create scalable data models, data marts, semantic layers, and curated datasets that support enterprise analytics initiatives.
  • Optimize SQL workloads, transformation logic, and query performance to improve scalability and cost efficiency.
  • Establish reusable engineering frameworks, accelerators, and best practices to improve delivery consistency across projects.
  • Ensure high standards of data quality, reliability, governance, and observability throughout the data lifecycle.
  • Develop and maintain Snowflake-based data ecosystems, leveraging advanced features for performance optimization and data sharing.
  • Build and orchestrate data workflows using Airflow and other workflow scheduling platforms.
  • Collaborate directly with client stakeholders to gather requirements, define roadmaps, and develop scalable technical solutions.
  • Present solution designs, technical recommendations, and project updates to both technical and business audiences.
  • Prepare and maintain comprehensive project documentation, technical specifications, architecture diagrams, and operational runbooks.
Qualifications
⁠⁠⁠⁠⁠⁠⁠Required Qualifications
  • 6+ years of experience in Data Engineering, Big Data Engineering, or Cloud Data Platform development.
  • Bachelor's or Master's degree in Computer Science, Engineering, Analytics, Mathematics, Information Systems, or related disciplines.
  • Strong hands-on expertise in SQL, Python, and PySpark.
  • Extensive experience working with Snowflake, Databricks, or similar cloud-native data platforms.
  • Proven experience building and supporting large-scale ETL/ELT data pipelines.
  • Strong understanding of data warehousing concepts, dimensional modeling, and modern Lakehouse architectures.
  • Experience implementing Medallion Architecture and enterprise-grade data modeling practices.
  • Hands-on experience with workflow orchestration tools such as Apache Airflow or equivalent scheduling frameworks.
  • Experience working with cloud ecosystems including AWS, Azure, or Google Cloud Platform.
  • Strong knowledge of performance tuning, optimization, monitoring, and operational support for data platforms.
  • Demonstrated experience leading engineering teams and coordinating with client and internal stakeholders.
  • Excellent analytical, problem-solving, communication, and stakeholder management skills.
  • Ability to work independently and lead complex initiatives in fast-paced consulting environments.
Preferred Qualifications
  • Experience with streaming and real-time data processing frameworks.
  • Familiarity with DataOps, CI/CD, Infrastructure as Code, and DevOps practices.
  • Experience with data governance, data quality frameworks, and metadata management.
  • Exposure to AI/ML data pipelines and feature engineering workflows.
  • Experience with visualization tools such as Tableau, Power BI, or Looker.
  • Hands-on experience with Big Data technologies including Spark, Hadoop, Hive, HBase, Kafka, or related platforms.
  • Consulting or client-facing delivery experience in enterprise-scale environments.

Similar jobs

Apply