Sr AWS Data Engineer (with ETL, Python, AWS Glue, Lambda, Node.js & Microservices exp)
Quick overview
Job description
- We are looking for a Senior AWS Engineer to extend our event-driven, serverless financial platform.The platform combines Python-based AWS Glue ETL pipelines with Node.js Lambda microservices.
- A candidate who has built both Python Glue pipelines and Node.js Lambda services in the same production environment would be the strongest fit.
- AWS Glue ETL
- Microservices - Nodejs or Python
- Deployment pipeline - Jenkins, Cloud formation, SAM
- ETL - Glue Job, Step Functions
- API Gateway - Lamda / DynamoDB
Design, build, and maintain independently deployable microservices (single-purpose AWS Lambda functions) within an event-driven architecture
Develop and maintain Node.js (ES6+/async-await) Lambda microservices for file intake, validation, transformation, submission and acknowledgement handling of transaction files (JSON, XLSX, CSV)
Build event-driven Node.js services triggered by S3 event notifications, EventBridge, SQS and API Gateway, with clear contracts between services
Use the AWS SDK for Node.js (v3 preferred) to integrate with core AWS services such as S3, DynamoDB, SNS/SES, Secrets Manager and Step Functions.
Manage dependencies, Lambda layers, bundling and cold start/memory/timeout tuning for Node.js functions
Define clean service boundaries and contracts (event payloads, schemas, S3 formats) enabling independent development and deployment
Maintain and extend AWS Glue ETL jobs (PySpark + native Oracle JDBC) that ingest, transform, and reconcile daily transaction history and position snapshot files
Build and troubleshoot Step Functions workflows (ASL state machines) orchestrating multi-stage pipelines: file validation, ingestion, reconciliation, alerting
Design and maintain Lambda functions (Python and Node.js) for lightweight orchestration, file validation, DynamoDB writes, and SNS/SES alerting
Write and optimize Oracle SQL (native JDBC and Lambda-proxied) against a complex financial schema
Manage DynamoDB data models for transaction history, reconciliation status, and fund-level summaries, including GSIs and TTL-based retention
Implement robust error handling and transactional integrity (per-record commit/rollback, idempotency, duplicate-detection) for financial data where silent failures are unacceptable
Apply retries, dead-letter queues and idempotency keys (for example, a file ID checked with DynamoDB conditional writes) so S3 events delivered more than once never cause duplicate processing
Own unit, integration, and smoke test suites (pytest/unittest, Jest/Mocha, moto and aws-sdk-client-mock for AWS mocking) and maintain/improve code coverage
Support CI/CD pipelines (Jenkins, AWS SAM/CloudFormation), including build, test, lint and packaging of Node.js services
Build and maintain observability: structured CloudWatch logging/metrics, Grafana dashboards, and metric registries for pipeline health
Participate in security reviews (SOC2-aligned findings, SQL injection guards, least-privilege IAM, KMS encryption) and remediate findings, including Node.js dependency vulnerability management (npm audit)
Produce and maintain technical documentation (architecture references, runbooks, deployment guides, changelogs)
Required Skills & Experience:
Python (3.12+) strong proficiency, including testable, well-structured ETL code
Node.js (current LTS) has strong proficiency, including ES6+, async/await, and JavaScript or TypeScript for AWS Lambda microservices
Experience designing and operating microservices architectures: single-purpose services, loose coupling, independent deployment, event contracts and idempotency
AWS Glue (PySpark, Glue 4.0, DynamicFrames) and AWS Lambda (Python and Node.js runtimes)
AWS Step Functions (Amazon States Language) for workflow orchestration
AWS SAM / CloudFormation infrastructure as code, multi-environment deployment
Oracle SQL complex joins, DML across transactional schemas, stored procedure like code patterns
DynamoDB key design, GSIs, conditional writes, pagination
S3 - event-driven architectures (EventBridge notifications), data lake patterns (Parquet)
Strong testing discipline: pytest/unittest, Jest/Mocha, mocking AWS services (moto, aws-sdk-client-mock), stubbing Spark/Glue runtime dependencies for offline unit tests
Experience with financial/transactional data where correctness, auditability, and idempotency are critical
CI/CD experience, ideally Jenkins, with multi-environment promotion gates
Familiarity with CloudWatch metrics/logs, Grafana, and building operational dashboards.
Similar jobs
- PG

Data Engineer – Databricks, PySpark, Cloud | Hybrid - 71701Recently posted
PRIMUS Global Services Inc. · Princeton, NJ · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLScalaApache Spark+4TechnologySalary not listed - PH

Data EngineerRecently posted
Photon · Sunnyvale, CA · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLAWSETL+6TechnologySalary not listed - GE

SQL Server DBARecently posted
Genesis10 · Charlotte, NC · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLShellMicrosoft Office+3Technology$53 - $61/hr - NT
Oracle DBA with Security ClearanceRecently posted
NorthHill Technology Resources · Fairfax, VA · Hybrid · Posted 17 Nov 2026 🇺🇸
MariaDBMongoDBOracle+6TechnologySalary not listed - IT

ETL LeadRecently posted
ITSYNTAX INC · Dallas, TX · Hybrid · Posted 16 Nov 2026 🇺🇸
OraclePL/SQLSQL+6Salary not listed - VI

Sr. Data EngineerRecently posted
Visa · Foster City, California · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLScalaApache Spark+14Technology$131.9k - $204k/yr - VI

Staff Data EngineerRecently posted
Visa · Foster City, California · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLScalaApache Spark+13Technology$156.8k - $251.1k/yr - FA

Systems Engineer (Data Flow/NiFi) Level 2Recently posted Auto Apply
Fantom Corporation · Fort Meade · On-site · Posted 17 Nov 2026 🇺🇸
ApacheComplianceComputer Vision+5TechnologySalary not listed - LS

Senior Database EngineerRecently posted
London Stock Exchange Group · St Louis, MO · Hybrid · Posted 17 Nov 2026 🇺🇸
OracleShellAWS+5TechnologySalary not listed - KT

Senior Data EngineerRecently posted
Kforce Technology Staffing · Atlanta, GA · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLETLAirflow+5TechnologySalary not listed - QT

Data Engineer LeadRecently posted
QUANTUM TECHNOLOGIES LLC · United States · Hybrid · Posted 17 Nov 2026 🇺🇸
DockerSQLGit+12Technology$83/hr - TE
Data Engineer - San Antonio, TX, Dallas, TX, Austin, TX, Houston, TXRecently posted
TechniPros, LLC · San Antonio, TX · Hybrid · Posted 16 Nov 2026 🇺🇸
SQLApache SparkETL+2TechnologySalary not listed