Google Cloud Platform (GCP) - Data Engineer

A Bachelor's or Higher Degree is the minimum entry required for the position

The GCP Data Engineer is responsible for designing, building, deploying, and maintaining end-to-end data pipelines. This includes data ingestion, transformation, analysis, storage, and loading of data to support business use cases.

Core Responsibilities

• Build and maintain data pipelines on Google Cloud Platform.
• Create and manage Apache Airflow DAGs using various operators.
• Implement batch and real-time data integration solutions.
• Design and optimize BigQuery-based data warehouse solutions.
• Handle API-based and streaming data ingestion.
• Develop data security and privacy controls.
• Collaborate with development and testing teams to deliver business solutions.
• Create and maintain technical documentation.
• Participate in Agile/Scrum teams.

Required GCP Skills

• BigQuery
• Cloud Composer (Airflow)
• Dataflow
• Pub/Sub
• Cloud Functions
• IAM (Identity and Access Management)

Technical Skills

• Advanced SQL, particularly in BigQuery
• Python programming
• Data Warehousing concepts
• Data Modeling
• API integrations

DevOps / CI-CD Skills

Knowledge of:

• GitHub
• Terraform

The Google Cloud Platform Data Engineer is responsible to build and to deploy end to end data pipelines w.r.t. the use cases provided. This includes but not limited to data ingestion, data analysis, data transformation and data load. Roles and Responsibilities: Develop and maintain data pipelines and data storage solutions using GCP. Experience in Google Cloud Platform including but not limited to BigQuery, Cloud Storage, Cloud Composer, Dataflow, Pub/Sub, Cloud Functions and IAM. Developing DAGs in Apache Airflow using various Operators. Implementing data integration solutions using GCP services. Maintain ETLs operating on a variety of sources. Hands On with advanced SQL on BiqQuery Data Warehousing and ability to execute queries quickly. Knowledge in CI/CD is desirable. Proficiency in Python programming and Unix scripting. Knowledgeable in CICD processes GitHub, cloud build, terraform etc. Development and implementation of data security and privacy measures to protect sensitive data. Handling real time data and API based ingestion. Knowledge of data warehousing and data modeling concepts. Collaborate within the development and the testing teams to implement solutions that meet business requirements. Write and maintain technical documentation wherever necessary. Strong analytical and problem solving skills. Good communication and collaboration skills. Experience working in Agile/Scrum environments.