Senior Data Engineer

Role : Senior Data Engineer - BFSI  Cloud & Payments Platform

Location : Charlotte, NC (Hybrid)

Role Summary

We are seeking an experienced Senior Data  Engineer to design, develop, and optimize large-scale real-time data platforms supporting transaction tracing, reconciliation, observability, analytics, and graph-enabled investigations for Banking and Financial Services (BFSI) clients. The role requires expertise in streaming architectures, event-driven systems, distributed data processing, transaction lifecycle management, and modern cloud data platforms.

Key Responsibilities

Real-Time Data Engineering & Platform Development

• Design and develop real-time and batch data ingestion pipelines.
• Build scalable data processing frameworks using Kafka, Spark Streaming, and PySpark.
• Develop cloud-native and distributed data processing solutions.
• Create reusable data engineering components, frameworks, and accelerators.
• Design highly available, fault-tolerant, and performance-optimized data  pipelines.
• Establish data engineering standards, patterns, and best practices.

BFSI Data Processing & Transaction Traceability

Develop and support data solutions across:

• Payments Platforms (ACH, RTP, FedNow, SWIFT, ISO 20022)
• ATM Transaction Processing

Key responsibilities include:

• Design transaction tracing and correlation frameworks.
• Create canonical transaction and event processing models.
• Correlate transactions using STAN, RRN, timestamps, terminal IDs, and account references.
• Build reconciliation and exception management workflows.
• Enable end-to-end transaction lifecycle visibility and auditability.
• Perform log analysis and transaction mapping.

Data Quality, Reliability & Observability

• Implement reconciliation, validation, and data quality frameworks.
• Establish end-to-end lineage tracking across all processing stages.
• Detect missing transactions, duplicate events, processing anomalies, and broken lineage chains.
• Build monitoring, alerting, and operational observability solutions.
• Define data quality metrics, SLAs, and operational dashboards.
• Support root-cause analysis and transaction recovery investigations.

Security, Governance & Compliance

• Ensure adherence to banking security and regulatory requirements.
• Implement secure data processing and access control mechanisms.
• Implement data encryption, masking, and secure data handling practices.
• Support audit, lineage, traceability, and regulatory reporting requirements.
• Ensure compliance with enterprise data governance standards.

Cloud Engineering & DevSecOps

• Build and deploy cloud-native data engineering solutions.
• Implement CI/CD pipelines for data platform deployments.
• Support containerized workloads and Kubernetes-based data  platforms.
• Enable DevSecOps practices across data engineering solutions.

Data, Analytics & Knowledge Graph Engineering

• Design Operational Data platform  Data pipelines.
• Develop real-time analytics and event-streaming pipelines.
• Design data preparation and transformation pipelines for Neo4j and graph databases.
• Implement entity extraction and relationship mapping frameworks.
• Build graph-ready datasets supporting transaction tracing and investigation.
• Support graph analytics, impact analysis, and root-cause investigations.
• Enable AI/ML and GenAI initiatives through trusted and governed data assets.

Leadership & Stakeholder Management

• Collaborate with solution architects, business teams, operations, payments, fraud, and technology stakeholders.
• Participate in architecture reviews, technical governance, and design discussions.
• Mentor junior and mid-level data engineers.
• Support estimation, planning, and technical solution discussions.

Required Qualifications

• Bachelor''s or Master''s degree in Computer Science, Engineering, Information Technology, or related field.
• 8+ years of experience in Data Engineering and distributed data processing.
• Strong hands-on expertise with:
• Apache Kafka
• Apache Spark/PySpark
• Python
• SQL
• NOSQL DB
• CDC Frameworks
• Event-Driven Architectures
• Graph Databases
• Experience building large-scale streaming and batch processing pipelines.
• Strong understanding of distributed systems and real-time data processing.
• Experience with Operational Data platforms
• Knowledge of transaction processing systems, event correlation, and log analytics.
• Strong troubleshooting, performance tuning, and optimization skills.

Preferred Qualifications

• Banking and Financial Services domain experience.
• Hands-on experience with Payments and Transaction Processing platforms.
• Knowledge of ISO 8583, ACH, RTP, SWIFT, and ISO 20022 standards.
• Experience with Neo4j and Graph Database technologies.
• Familiarity with Knowledge Graph and Graph Analytics solutions.
• Experience supporting observability and transaction monitoring platforms.
• Exposure to AI/ML and GenAI-enabled analytics solutions.
• Cloud certifications in Azure, AWS, or Google Cloud Platform preferred.

Success Measures

• High-performance real-time data processing pipelines.
• Accurate transaction correlation and traceability across systems.
• Reliable transaction lineage and audit tracking.
• High-quality graph-ready datasets supporting investigations and analytics.
• Improved observability and operational monitoring.
• Reduced transaction processing failures and reconciliation gaps.
• Secure, scalable, and compliant data engineering solutions.
• Successful delivery of enterprise-grade data  platforms supporting banking and payments modernization.