Python/PySpark Data Engineer

Role Summary

We are looking for a strong Python/PySpark Data Engineer to build scalable data ingestion and integration pipelines across enterprise data sources and metadata platforms. The engineer will work with metadata catalogs, databases, APIs, and batch/Realtime integration patterns while contributing to enterprise data management and data warehousing initiatives.

Key Responsibilities

•           Develop robust data engineering solutions using Python, PySpark and Apache Spark.

•           Consume and integrate metadata/data from Ab Initio Enterprise Meta>Environment (EME/EMH), Oracle databases, and ServiceNow APIs.

•           Perform source-to-target mapping, data transformation, validation, and reconciliation.

•           Build and maintain batch data ingestion and integration pipelines.

•           Work with data warehouse concepts, data models, metadata management, data lineage and data quality.

•           Integrate technical and business metadata from multiple enterprise sources.

•           Understand existing/custom data engineering frameworks and quickly develop solutions within them.

•           Troubleshoot performance, data quality, and integration issues across large datasets.

Required Skills

•           Strong hands-on expertise in Python-based data engineering.

•           Strong experience with PySpark / Apache Spark and large-scale data processing.

•           Experience working with Oracle/SQL and relational databases.

•           Experience consuming REST APIs and integrating API-based data sources.

•           Good understanding of ETL/ELT, source-to-target mapping, data warehousing and dimensional/relational data modeling.

•           Understanding of metadata management, business/technical metadata, lineage and data governance concepts.

•           Experience building production-grade batch ingestion/integration pipelines.

•           Ability to understand and work with custom enterprise data frameworks.

Good to Have

•           Experience with real-time/streaming integration, such as Kafka or Spark Structured Streaming.

•           Exposure to Google Cloud Platform (Google Cloud Platform) and its data services.

•           Experience with enterprise metadata/catalog solutions, particularly Ab Initio metadata environments.

•           Experience integrating with ServiceNow.

Ideal Profile: A hands-on engineer who is particularly strong in Python + PySpark, understands enterprise data and metadata ecosystems, and can quickly learn existing frameworks rather than relying only on off-the-shelf ETL tools