Senior Azure Data Engineer – Python/PySpark & Financial Market Data

Important Notes

• Client: SMBC (Sumitomo Mitsui Banking Corporation)
• Locations: Charlotte, NC (Hybrid)
• Duration: 12 months, with possible extension
• Experience: 10+ years
• Interview: Mostly online
• Industry: Financial Services / Capital Markets
• Key Focus: Azure Data Engineering, Python/PySpark, Databricks, and financial market data

Job Summary

SMBC is undergoing a digital transformation across its Americas Division and is developing a strategic data platform for SMBC Capital Markets and Nikko Securities Group.

The Data Engineer will join the Data Strategy team to design and develop cloud-based data solutions on Microsoft Azure. Initial responsibilities will focus on building a reference data system for securities and pricing data, with future expansion into additional data domains.

The ideal candidate will have strong hands-on experience with Python, PySpark, Azure Data Services, Databricks, SQL, APIs, and financial market data.

Key Responsibilities

• Design and develop scalable data platform solutions using Azure Cloud.
• Build and maintain data pipelines using Azure Data Factory and Databricks.
• Develop data processing solutions using Python and PySpark.
• Support the implementation of reference data systems for securities and pricing data.
• Work with cross-functional teams across SMBC Capital Markets and Nikko Securities.
• Develop and integrate APIs using FastAPI or similar Python frameworks.
• Implement ETL/ELT workflows for structured and unstructured data.
• Work with Azure databases, data lakes, and related cloud services.
• Support API Gateway management and Azure Functions.
• Develop solutions using Azure Data Lake Storage Gen2.
• Follow internal development standards and best practices.
• Participate in DevOps, CI/CD, and software delivery activities.
• Support future expansion into additional financial data domains.

Must-Have Requirements

• 10+ years of relevant Data Engineering experience.
• Strong hands-on experience with Python and PySpark.
• Proven experience designing and developing solutions on Microsoft Azure.
• Experience with the following Azure services:
  • Azure Data Factory
  • Azure Data Lake Storage Gen2
  • Azure Databases
  • Microsoft Fabric / Azure Data Fabric
  • Azure Functions
  • API Gateway Management
• Strong experience with Azure Databricks.
• Advanced SQL skills with relational or NoSQL databases.
• Experience developing APIs using:
  • FastAPI
  • Similar Python API frameworks
• Good understanding of ETL and ELT processes.
• Familiarity with DevOps lifecycle and CI/CD practices.
• Experience with Git, Jenkins, or similar DevOps tools.

Preferred Requirements

• Experience in the financial services or capital markets industry.
• Knowledge of:
  • Financial instruments
  • Asset classes
  • Securities data
  • Pricing data
  • Market data
  • Reference data systems
• Experience working with broker-dealer or swap-dealer environments.
• Experience collaborating with distributed teams across the US and India.
• Experience developing enterprise-scale strategic data platforms.