Why This Role Stands Out
This role offers a fantastic opportunity to spearhead a significant platform migration, sharpening your skills in Databricks, Snowflake, and Azure cloud technologies. You'll thrive here if you have strong Scala/Spark and SQL expertise, enjoying hands-on development and CI/CD implementation within a reputable company. Embrace this chance to make a tangible impact on critical financial reporting systems.
Quick Overview
Seniority
Mid Senior
Work mode
On Site
Location
New York, NY, United States
Posted
Yesterday
SQLScalaETLSnowflakeTDDAirflowAzureBDDDatabricksGitHub ActionsPythonReconciliationRegulatory ReportingUnity
Job Description
Position: Databricks Specialist
Location: NYC, NY (Onsite)
Duration; 12 Months
We're migrating complex on-prem regulatory reporting pipelines from a legacy ETL + Autosys + SQL + Teradata stack to a modern Databricks + Snowflake platform on Azure. The role is hands-on: design, implement, test, and reconcile production pipelines feeding regulatory reports under strict parity requirements.
Must-have
- Scala / Spark production experience writing Spark applications in Scala (not just notebooks); comfortable with the DataFrame API, joins, window functions, partitioning, and performance tuning
- Databricks Serverless compute, Unity Catalog, Asset Bundles, Databricks CLI
- SQL fluency confortable writing, analyzing and extracting requirements from complex SQL scripts
- Snowflake schema design, performance, Spark-Snowflake connector
- Azure ADLS, networking basics, secrets/identity (Entra ID / managed identities)
- Orchestration Airflow (DAG authoring, sensors, retries, SLAs)
- CI/CD Artifactory, GitHub Actions pipelines: build, sharded test matrices, artifact promotion through dev QA UAT prod
- Testing Experience in TDD, writing unit tests (ScalaTest, AnyFlatSpec) and BDD (Concordion or equivalent)
- Data quality & reconciliation building automated parity checks against legacy outputs, drift detection, row-level reconciliation tooling
- Large-scale migrations proven track record migrating legacy ETL (Autosys/Informatica/etc.) to cloud data platforms, including dependency mapping and cutover planning
- Modern data engineering practices medallion architecture (Bronze/Silver/Gold), idempotent pipelines, schema evolution, lineage, observability
Nice-to-have
- Financial services / regulatory reporting domain
- Python (Databricks utilities, tooling)
- Spec-driven development workflows (specs plans tasks implementation)
- Gradle (composite builds) and JVM tooling
Similar jobs
- UC
Experienced Federal Law Enforcement Officer – Up to $60K Incentive
NewU.S. Customs and Border Protection
Hardin, MT🇺🇸$63.4k - $127.5k/yrHybridYesterday - KT
Robotic Controls Technician
NewKforce Technology Staffing
Austin, TX🇺🇸HybridYesterdayRoboticsTCP/IPLESS - SG
Epic Training Manager
NewSage Group
Raleigh, NC🇺🇸HybridYesterdayCurriculum DevelopmentEHR - V-
100% Remote - AI Applications Engineer
NewV-CENTRIX-US LLC
United States🇺🇸RemoteYesterdaySQLAzure.NET+6 - VI
AI Platform Architect
NewVega Intellisoft Inc.
Manor, TX🇺🇸On-siteYesterdayNode.jsAWSSnowflake+4 - AS
Ethernet Expert
NewAmaze Systems Inc
United States🇺🇸HybridYesterdayTCP/IPAnsiblePuppet