DE&A - Senior Data Engineer (CDAI)

Core Skills & Experience

• Strong hands-on experience designing, building, and optimizing enterprise-scale data pipelines, data architecture, and data products for performance, scalability, and efficient AI agent consumption.

• Deep expertise in Starburst/Trino and distributed SQL, including query optimization, materialized views (MVs), data modeling, storage design, and tuning data products for low-latency, high-concurrency AI agent access.

• Strong experience optimizing data storage, partitioning, indexing, file formats, and materialization strategies to improve query performance, data freshness, and cost efficiency.

• Hands-on development experience with Apache Spark using Scala and/or Python/PySpark, including migrating complex, SQL-intensive transformations to distributed processing frameworks to improve data curation speed and efficiency.

• Strong ability to evaluate and redesign data processing workflows, selecting the appropriate technologies and execution patterns across SQL, Spark, and other processing frameworks to optimize performance.

• Experience with RDBMS technologies and modern lakehouse architecture, preferably Apache Iceberg, including table design, partitioning, compaction, and data lifecycle optimization.

• Understanding of integration patterns across NiFi, Kafka, Spark, RDBMS, and lakehouse platforms for efficient batch and real-time data processing.

• Strong software engineering practices, including CI/CD, automated testing, source control, infrastructure automation, observability, and production support.

• Experience with data governance, metadata management, lineage, data quality, and secure data access in enterprise environments.