Senior AI Data Engineer

We are seeking an experienced Senior AI Data Engineer / AI Data Engineering Lead with 11+ years of IT experience and strong expertise across Data Engineering, Generative AI, LLMs, Machine Learning, Google Cloud Platform Cloud, and enterprise data platforms.

The ideal candidate will have extensive experience designing and implementing large-scale data solutions on Google Cloud Platform, along with hands-on experience developing GenAI/RAG solutions, AI-enabled data pipelines, real-time streaming architectures, and modern data platforms.

The candidate should also have experience with AutoSys workload automation and AutoSys migration projects, particularly migrating existing on-premises AutoSys environments/workloads to cloud-based platforms, including Google Cloud Platform, while maintaining scheduling dependencies, job workflows, SLAs and production operations.

Strong experience in the Banking and Financial Services domain is highly preferred, with an understanding of banking data, transactions, risk, compliance, regulatory requirements and enterprise security.

Key Responsibilities

• Lead the design and development of enterprise-scale AI and Data Engineering platforms on Google Cloud Platform.
• Design highly scalable batch and real-time data pipelines using Dataflow, Apache Beam, Pub/Sub, Kafka, Spark and PySpark.
• Develop complex ETL/ELT pipelines using Python, SQL, PySpark and Apache Spark.
• Design and optimize BigQuery data warehouses, data lakes and lakehouse architectures.
• Build data pipelines supporting Generative AI, LLM and Machine Learning applications.
• Design and implement enterprise RAG (Retrieval-Augmented Generation) solutions.
• Build ingestion pipelines for structured and unstructured enterprise data.
• Implement embeddings, vector search, semantic search and vector databases.
• Integrate Google Gemini and Vertex AI into enterprise AI and data solutions.
• Develop AI agents and agentic workflows using enterprise data and APIs.
• Build and maintain real-time streaming solutions using Kafka, Pub/Sub and Dataflow.
• Implement data quality, governance, security, lineage and monitoring frameworks.
• Lead AutoSys workload automation migration initiatives involving legacy/on-premises environments.
• Analyze existing AutoSys job schedules, calendars, dependencies, boxes, conditions and workflows.
• Develop migration strategies for moving AutoSys workloads from on-premises infrastructure to cloud environments, preferably Google Cloud Platform.
• Assess existing AutoSys jobs and identify opportunities for modernization and optimization.
• Migrate complex enterprise batch-processing workflows while maintaining business-critical SLAs.
• Convert or integrate AutoSys scheduling workflows with modern cloud orchestration technologies such as Cloud Composer / Apache Airflow, Cloud Workflows or equivalent platforms.
• Design scheduling and dependency-management solutions for Google Cloud Platform-based data pipelines.
• Troubleshoot production scheduling failures, job dependencies, alerts and batch-processing issues.
• Validate migrated workloads through parallel testing, reconciliation and production cutover.
• Develop documentation covering AutoSys architecture, migration strategy, job inventory, dependencies, scheduling and operational procedures.
• Work with infrastructure, application, Data Engineering and cloud teams during migration and modernization activities.