AWS S3 Python Migration Platform Engineer

Role Purpose

Lead the design, build, and operation of a secure, scalable migration platform to move object data from OBSv2 to AWS S3. The role focuses on reliable execution, automation, governance, monitoring, and production support for large-scale enterprise data migration.

Key Responsibilities

• Own the end-to-end migration workflow from OBSv2 source buckets to AWS S3 destination buckets.
• Build and operate AWS-based orchestration using Step Functions, Lambda, Fargate, AWS Batch, DynamoDB, SQS, and rclone.
• Support both manifest-based and dynamic migration approaches, including batching, sharding, task sizing, and retry handling.
• Implement governance for bandwidth allocation, queue management, prioritisation, throttling, and job tracking.
• Build observability through CloudWatch dashboards, logs, metrics, alerts, health checks, and progress reporting.
• Support secure multi-account deployment, IAM controls, privacy safeguards, testing, performance tuning, and production cutover.

Expected Deliverables

• Production-ready migration orchestration platform supporting both manifest and no-manifest paths.
• Hub-side governance for bandwidth control, queue processing, throttling, and completion reporting.
• Reusable rclone-based migration runtime with progress reporting and dynamic throttle support.
• DynamoDB-backed job, task, queue, bandwidth, and summary state management.
• CloudWatch dashboards, alarms, logs, runbooks, deployment automation, and operational documentation.

Preferred Experience

• Experience with Dell ECS, OBSv2, or other S3-compatible enterprise object storage platforms.
• Experience delivering enterprise-scale migrations in regulated, production environments.
• Experience with hub-and-spoke operating models across multiple AWS accounts.
• Ability to produce clear technical documentation and leadership-ready reporting.

Mandatory Technical Skills

AWS Serverless

• Hands-on experience with AWS Lambda, Step Functions, SQS, SNS, EventBridge, API Gateway, SSM Parameter Store, and Secrets Manager.

AWS Compute and Containers

• Strong working knowledge of ECS Fargate, AWS Batch, container task definitions, sidecar patterns, resource sizing, and runtime environment configuration.

AWS Storage

• Deep understanding of Amazon S3, S3 Intelligent-Tiering, S3-compatible APIs, object metadata, prefix structures, large object transfers, and lifecycle considerations.

Object Storage Migration

• Experience migrating large object datasets from on-premises or S3-compatible platforms such as Dell ECS/OBSv2 to AWS S3.

Rclone

• Hands-on experience configuring and operating rclone for high-volume transfers, including transfers, checkers, retries, checksum validation, no-traverse behaviour, stats, logs, and bandwidth limiting.

Python Development

• Strong Python skills for Lambda and Fargate services, including boto3, JSON processing, exception handling, logging, dataclasses, streaming files, and API integration.

DynamoDB

• Ability to design and operate DynamoDB tables, keys, indexes, scans, updates, batch writes, Decimal handling, and state-management patterns.

Networking

• Understanding of AWS Direct Connect, Transit Gateway, VPCs, private subnets, routing, S3 Gateway Endpoints, NAT Gateway scoping, and high-throughput network design.

IAM and Security

• Strong knowledge of IAM roles, cross-account access, STS assume-role, API keys, Secrets Manager, least-privilege design, and secure operational access.

Observability

• Experience with CloudWatch metrics, logs, alarms, dashboards, structured logging, operational telemetry, and migration progress reporting.

CI/CD and Deployment

• Experience with GitLab repositories, CI/CD pipelines, environment-specific configuration, infrastructure deployment, and safe teardown procedures.

Know-How Skills and Domain Understanding

• Understand large-scale object storage migration patterns, including manifests, prefix-based partitioning, batching, and validation.
• Balance migration performance across data volume, object count, compute resources, network capacity, and rclone concurrency.
• Design fair bandwidth allocation, throttling, queue handling, retry, and escalation models for production migrations.
• Apply strong operational discipline across monitoring, runbooks, restart procedures, completion checks, and stakeholder reporting.
• Maintain clear technical documentation, architecture notes, input contracts, implementation details, and support procedures.