Start searching
Back to Jobs
Recently postedTechnology

Data Engineer Ab initio

Futuremindz llcNew York, NY🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeOn Site
LocationNew York, NY, United States
DockerOracleSQLScalaShellETLJavaKafkaPythonKubernetesCI/CDLinux

Job description

Job Title: - Data Engineer Ab initio - Ab Initio,
Location: Berkley Heights, NJ
Duration - Longterm
Role Overview:
  • We are seeking a highly skilled resource to design and implement high-performance, event-driven data
  • pipelines, ensuring low-latency data processing and high availability system for the large credit card
  • processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE, EME, Conduct>It) to
  • build stateful services that ingest, filter, and transform data from sources like Kafka or message queues,
  • pushing updates to dashboards or downstream databases in near-real-time.
Key Responsibilities:
  • Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and
  • aggregations.
  • Implement event-driven data pipelines using Kafka, MQ, and file streams.
  • Ensure the resilience of continuous flows, including checkpointing and recovery, to guarantee "exactly-
  • once" processing.
  • Apply advanced Ab Initio components (e.g., Reformat, Rollup, Join, Partition) to ensure low-latency
  • performance.
  • Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting data issues
  • Develop ETL pipelines for batch and real-time data ingestion and transformation.
  • Implement and ensure data validation, data security, integrity, and compliance across big data
  • platforms.
  • Monitor and troubleshoot performance issues in large-scale clusters.
  • Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions.
  • Automate workflows and improve operational efficiency using scripting and orchestration tools.
Required Skills & Qualifications:
  • Deep understanding of credit card process system
  • Deep knowledge of GDE (Graphical Development Environment), EME (Enterprise Meta>Environment),
  • Conduct>It, and Continuous Flows.
  • Understanding of Kafka, message queues, and real-time stateful services.
  • Proficiency in Unix/Linux shell scripting, SQL, and database technologies (e.g., Oracle, Teradata).
  • Experience in Java , Scala, Python or Kafka is plus.
  • Familiarity with Linux/Unix environments and shell scripting.
  • Understanding of data security, governance, and compliance standards.
  • Experience with cloud-based big data platforms
  • Exposure to containerization (Docker, Kubernetes) for big data workloads.
  • Knowledge of CI/CD pipelines for data engineering projects.
Behavioral Skills:
  • Good Communication skills
  • 5 days Work from Office at Berkley Heights, NJ
  • Team Player
  • Ability to work in a changing environment
  • Strong problem solving and analytical skills
  • Ability to work independently or within a team
  • Manage day-to-day challenges and communicate developmental risks with the technical team

Similar jobs

Apply