Back to Jobs
RemoteTechnology

Iceberg DBA / Lakehouse Operations Engineer_remote in dallas,TX

Floga technologiesDallas, TX🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeRemote
LocationDallas, TX, United States
SQLShellApache SparkAWSApacheHivePythonAzure

Job description

Hello,

Good morning,

I'm Rahul from Floga Technologies. We are a growing IT services organization delivering scalable technology solutions and are expanding our team with qualified talent.

We have an immediate opening for a Iceberg DBA / Lakehouse Operations Engineer Experience with strong hands-on experience to support a long-term engagement with our client. Must collaborate effectively with cross-functional teams to deliver high-quality solutions.

Job Title: Iceberg DBA / Lakehouse Operations Engineer

Location: Remote (Based on the business needs candidates needs to be flexible to work from office, Dallas, TX)

Experience: 10+ years

Duration: Long Term

Job Summary

We are seeking a highly skilled Iceberg DBA / Lakehouse Operations Engineer to manage the reliability, performance, and operational integrity of the Apache Iceberg data layer supporting enterprise analytics and business-critical applications.

The role involves managing a large-scale, multi-engine Lakehouse environment supporting workloads across Spark, Hive, and Impala. The engineer will play a key role in enterprise data modernization initiatives, particularly migrations from Hive and Teradata to Apache Iceberg.

The ideal candidate will have strong hands-on experience with Iceberg table operations, metadata management, data modeling, and query performance optimization in cloud-based environments.

This role is critical to maintaining data accuracy, availability, and performance, as any degradation can directly impact downstream reporting, analytics, and business-critical decision-making.

Key Responsibilities

1. Iceberg Data Layer Ownership & Operations

  • Own day-to-day operations and administration of Apache Iceberg tables supporting multiple enterprise applications.
  • Ensure data reliability, consistency, and availability across Lakehouse workloads.
  • Maintain operational integrity of datasets at multi-terabyte to petabyte scale.
  • Monitor table health, metadata growth, and storage utilization.
  • Ensure consistent data access and operational stability across environments.

2. Advanced Table Management & Optimization

  • Perform advanced Iceberg table maintenance and optimization activities, including:
    • Minor and major compaction.
    • Small file mitigation and optimization.
    • Snapshot expiration and metadata compaction.
    • Orphan file cleanup and storage management.
  • Optimize data layout and query performance through file sizing, distribution, and partitioning strategies.
  • Implement partition evolution and partition pruning techniques.
  • Apply clustering and data ordering strategies, such as Z-ordering or equivalent approaches.
  • Monitor and optimize table maintenance operations to improve query performance and storage efficiency.

3. Data Modeling & Lakehouse Architecture

  • Support and enforce data modeling best practices, including:
    • Third Normal Form (3NF) for source-aligned datasets.
    • Medallion architecture (Bronze, Silver, and Gold) for curated data pipelines.
  • Ensure Iceberg table designs align with ingestion patterns, downstream consumption, and performance requirements.
  • Balance data integrity, normalization, and analytical query performance.
  • Collaborate with data engineering teams to ensure consistent implementation of layered data architecture across applications.

4. Multi-Engine Query Performance & Consistency

  • Ensure consistent and optimized query performance across:
    • Apache Spark / Spark SQL (CDE).
    • Hive.
    • Impala (CDW).
  • Troubleshoot query performance bottlenecks, metadata inconsistencies, and inefficient execution plans.
  • Analyze query execution plans, scan patterns, and data access behavior.
  • Ensure consistent table metadata and query results across multiple processing engines.

5. Hive & Teradata Modernization Support

  • Support enterprise data modernization initiatives involving Hive-to-Iceberg and Teradata-to-Iceberg migrations.
  • Assist with schema alignment, data type mapping, and table structure conversion.
  • Perform data validation, reconciliation, and accuracy checks between source and target datasets.
  • Troubleshoot migration-related issues and ensure post-migration stability and performance.
  • Collaborate with data engineering teams to optimize migrated workloads.

6. Metadata & Data Lifecycle Management

  • Manage Iceberg metadata to ensure scalability, consistency, and performance.
  • Maintain consistent table states and metadata across multiple query engines.
  • Implement data retention and archival policies.
  • Manage snapshot lifecycle operations, expiration, and cleanup.
  • Optimize time-travel capabilities and metadata maintenance.
  • Monitor metadata growth and prevent performance degradation.

7. Production Support & Incident Management

  • Provide L2/L3 operational support for production issues across Iceberg-based Lakehouse workloads.
  • Participate in on-call rotations to support critical data platforms.
  • Respond to and resolve P1/P2 incidents within defined SLAs.
  • Troubleshoot data inconsistencies, reporting discrepancies, query failures, and performance degradation.
  • Perform Root Cause Analysis (RCA) and implement permanent corrective and preventive actions.
  • Collaborate with platform, infrastructure, and application teams during incident triage and resolution.

8. Security & Data Governance

  • Support fine-grained data access controls using Apache Ranger policies and Role-Based Access Control (RBAC).
  • Ensure data validation, reconciliation, and accuracy between source systems and Iceberg datasets.
  • Maintain secure and compliant data access across applications and processing environments.
  • Collaborate with security and governance teams to enforce enterprise data access policies.

Required Skills & Qualifications

  • Strong hands-on experience with Apache Iceberg and/or Hive-based data lake environments.
  • Strong understanding of Iceberg table architecture, metadata management, and table maintenance operations.
  • Experience with table-level optimization, compaction, snapshot management, and performance tuning.
  • Understanding of data modeling concepts, including normalization and 3NF.
  • Knowledge of modern Lakehouse architecture and Medallion architecture (Bronze/Silver/Gold).
  • Hands-on experience with Spark SQL, Hive, and Impala.
  • Working knowledge of NiFi and Trino.
  • Strong understanding of partitioning strategies, partition evolution, and partition pruning.
  • Experience with columnar file formats such as Parquet and ORC.
  • Understanding of distributed query processing and large-scale data management.
  • Experience supporting multi-terabyte to petabyte-scale data platforms.

Preferred Skills

  • Experience with Hive-to-Iceberg and/or Teradata-to-Iceberg migrations.
  • Experience with Cloudera CDP, particularly CDE and CDW.
  • Familiarity with cloud platforms such as AWS and Azure.
  • Experience with Apache Ranger, data governance, and access management.
  • Scripting and automation experience using Python and Shell.
  • Experience supporting enterprise-scale Lakehouse modernization initiatives.

Thanks & Regards,

Rahul H | Floga Technologies.

Sr. Talent Acquisition Specialist
E: rahul

Similar jobs