L3 – Senior Iceberg DBA / Lakehouse Operations Engineer

Role: L3 – Senior Iceberg DBA / Lakehouse Operations Engineer

Location: Remote

Type: Contract

• 10+ years of experience in Big Data / Data Engineering / DBA / Data Operations roles
• Minimum 2+ years of hands-on experience with Apache Iceberg in production environments
• 6+ years of experience working with Cloudera ecosystem (CDP Ecosystem)

Strong expertise in:

• Iceberg table optimization (compaction, metadata management, partition evolution)
• Multi-engine performance tuning (Spark, Hive, Impala)
• Troubleshooting complex data and query performance issues

Proven experience handling:

• P1/P2 production incidents
• Large-scale environments (TB/PB scale)
• Data migration initiatives (Hive/Teradata → Iceberg)
• Lead enforcement of data modeling and Lakehouse standards across applications
• Guide teams on:
• Medallion architecture implementation
• Balancing normalization vs performance
• Review and resolve complex data modeling and performance trade-offs
• Ensure consistency of data structures across domains and workloads
• Mentor and guide L2 resources in operational best practices and troubleshooting

Required Skills:

• Strong hands-on experience with Apache Iceberg and/or Hive-based data lakes
• Understanding of data modeling concepts (normal forms) and modern Lakehouse patterns (Medallion architecture)
• Expertise in:
• Table-level optimization and performance tuning
• Large-scale data management (TB/PB scale)
• Experience with:
• Spark SQL, Hive, Impala, NiFI, Trino
• Strong understanding of:
• Partitioning strategies
• File formats (Parquet/ORC)
• Distributed query processing

Preferred Skills:

• Experience with:
• Hive-to-Iceberg or Teradata-to-Iceberg migration
• Cloudera CDP (CDE/CDW)
• Familiarity with:
• Cloud platforms (AWS, Azure)
• Scripting/automation (Python, Shell)

What You’ll Work On:

• Enterprise-scale Iceberg Lakehouse platform supporting multiple applications
• Large-scale data modernization initiatives
• Performance optimization and stability of mission-critical analytical workloads

 

Why This Role Matters:

• Ensures data correctness and performance for downstream analytics and business-critical reporting
• Enables successful modernization from legacy platforms to Iceberg
• Maintains high availability and reliability of the enterprise data layer

Job Summary:

• We are seeking a highly skilled Iceberg DBA / Lakehouse Operations Engineer to own the reliability, performance, and operational integrity of the Iceberg data layer powering enterprise analytics and business-critical applications.
• This role operates in a large-scale, multi-engine Lakehouse environment, supporting workloads across Spark, Hive, and Impala, and plays a key role in enterprise data modernization initiatives (Hive and Teradata → Iceberg).
• The ideal candidate brings deep expertise in Iceberg table operations, metadata management, and query performance optimization, ensuring consistent, high-performance data access across platforms in a cloud-based environment.
• This role is critical to ensuring data accuracy and performance—any degradation directly impacts downstream reporting, analytics, and business-critical decision-making.

Key Responsibilities:

• Iceberg Data Layer Ownership & Operations
• Own day-to-day operations of Apache Iceberg tables supporting multiple enterprise applications
• Ensure data reliability, consistency, and availability across all Lakehouse workloads
• Maintain operational integrity for datasets at multi-terabyte to petabyte scale

Advanced Table Management & Optimization

• Execute advanced Iceberg table maintenance and optimization strategies:
• Compaction (minor/major) and small file mitigation
• Snapshot expiration and metadata compaction to control metadata growth
• Orphan file cleanup (vacuum) to maintain storage efficiency
• Optimize data layout and performance through:
• File size tuning and distribution strategies
• Partition evolution and pruning optimization
• Clustering and ordering techniques (e.g., Z-ordering or similar patterns)

Data Modeling Standards & Lakehouse Design Alignment

• Normalized data structures (3NF) for source-aligned datasets
• Medallion architecture (Bronze / Silver / Gold layers) for curated data flows
• Ensure Iceberg table design aligns with:
• Data ingestion patterns (raw vs curated layers)
• Downstream consumption and performance requirements
• Assist in structuring datasets to balance:
• Data integrity and normalization
• Query performance and analytical efficiency
• Work with data engineering teams to ensure consistent implementation of layered data architecture across multiple applications

Multi-Engine Query Performance & Consistency

Ensure consistent and performant query behavior across:

• Spark (CDE)
• Hive / Impala (CDW)
• Troubleshoot and resolve:
• Query performance bottlenecks
• Metadata inconsistencies across engines
• Inefficient execution plans and scan patterns

Hive & Teradata Modernization Support

Play a key role in enterprise data platform modernization (Hive and Teradata → Iceberg)

Support:

• Schema alignment and data type mapping
• Data validation and reconciliation
• Troubleshoot migration-related issues and ensure post-migration stability and performance

Metadata & Data Lifecycle Management

Manage Iceberg metadata to ensure:

• Efficient scaling and performance
• Consistent table state across engines
• Execute lifecycle operations:
• Data retention and archival policies
• Snapshot lifecycle management and cleanup
• Time-travel optimization and maintenance

Production Support, Incident Resolution & On-Call

• Provide L2/L3 support for data-related production issues across Iceberg-based Lakehouse workloads
• Participate in on-call rotation to support critical data platforms and ensure timely response to incidents
• Respond to and resolve P1/P2 production incidents within defined SLAs, minimizing impact to downstream applications and reporting

Troubleshoot:

• Data inconsistencies and reporting discrepancies
• Query failures and performance degradation
• Perform root cause analysis (RCA) and implement preventive measures to avoid recurring issues
• Collaborate with platform and application teams during incident triage and resolution
• Security & Data Governance Support
• Support fine-grained access control using:
• Ranger policies and RBAC
• Own and ensure data validation, reconciliation, and accuracy between source and Iceberg datasets
• Ensure secure and compliant access to data across applications