Senior Data Engineer – Cloud, Big Data & Data Platforms

Job Title: Senior Data Engineer – Cloud, Big Data & Data Platforms
Location: Seattle, WA (Onsite)

Duration: Long term

Experience: 5+ Years

Job Summary

We are looking for a Senior Data Engineer with 4+ years of experience in designing, developing, optimizing, and maintaining scalable data platforms and pipelines.

The ideal candidate should have strong hands-on experience with Python, PySpark, Apache Spark, SQL, Airflow, AWS/Google Cloud Platform, Data Lakehouse, ETL/ELT, and Data Warehousing.

The candidate should be comfortable working with both batch and streaming data processing, cloud-based data platforms, large-scale transformations, data migration, data quality, and production pipeline optimization. Experience with Databricks, Delta Lake, Kafka/Flink, Snowflake, BigQuery, Trino, Iceberg, or similar technologies will be an added advantage.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, Spark SQL, and SQL.
• Build high-performance batch and streaming data processing solutions using Apache Spark and technologies such as Kafka/Flink.
• Develop cloud-based data solutions across AWS and/or Google Cloud Platform.
• Work with AWS services including S3, EMR, Glue, Athena, Redshift, Lambda, and CloudWatch.
• Work with Google Cloud Platform services including Dataproc, GCS, BigQuery, and Cloud SQL.
• Develop and manage workflows using Apache Airflow, including DAG development, scheduling, monitoring, retries, and failure handling.
• Design and implement Medallion/Lakehouse architectures using Bronze, Silver, and Gold data layers.
• Work with Databricks, Delta Lake, Iceberg, Parquet, Trino, or equivalent data lakehouse technologies.
• Develop incremental processing, CDC, MERGE/UPSERT, schema evolution, and SCD Type I/II implementations.
• Design dimensional models involving fact and dimension tables, Star Schema, and Snowflake Schema.
• Build reliable data ingestion frameworks from databases, APIs, files, SaaS applications, and streaming sources.
• Optimize Spark workloads using partitioning, partition pruning, broadcast joins, AQE, caching, Z-Ordering, and shuffle optimization.
• Troubleshoot and optimize production pipelines for performance, scalability, SLA adherence, and cloud cost efficiency.
• Implement automated data quality, validation, reconciliation, monitoring, and alerting frameworks.
• Participate in cloud/platform migrations and perform data validation and production-readiness testing.
• Develop CI/CD and deployment workflows using Git, Jenkins, GitHub Actions, Docker, and Kubernetes.
• Collaborate with analytics, backend, product, and business teams to deliver reliable data products.
• Ensure data security, governance, lineage, compliance, and operational reliability across data platforms.