Databricks Data Engineer / Business Analyst - Open Contracting Data Standard (OCDS)

Position Summary

Scicom Infrastructure Services is seeking a Data Engineer / Business Analyst to support a large-scale Databricks implementation. This hybrid role will bridge business, contracting, data-governance, and engineering teams to define business requirements, construct data contracts, map source data, and develop reliable Databricks data pipelines.

The successful candidate must have hands-on experience with the Open Contracting Data Standard (OCDS) and understand how contracting and procurement information is structured across planning, tender, award, contract, and implementation stages. OCDS provides a standardized model for publishing and analyzing data throughout the public-contracting process and uses defined schemas, codelists, releases, records, and packages.

This individual will work closely with business stakeholders, procurement subject-matter experts, data architects, Databricks engineers, governance teams, and program leadership to translate complex business and contracting requirements into enforceable technical specifications and production-ready data products.

Key Responsibilities

Data Contracts and Business Analysis

·         Lead requirements-gathering sessions with procurement, contracting, program, analytics, governance, and technical stakeholders.

·         Define, construct, document, and maintain data contracts between data producers and consumers.

·         Establish data-contract requirements covering:

o    Dataset purpose and ownership

o    Source and target systems

o    Schemas, fields, and data types

o    Required and optional attributes

o    Business definitions and transformation rules

o    Data-quality expectations

o    Validation and reconciliation rules

o    Refresh frequency and delivery schedules

o    Versioning and schema-evolution requirements

o    Security classifications and access controls

o    Service-level expectations

o    Issue ownership and change-management procedures

·         Translate business requirements into user stories, acceptance criteria, process flows, data mappings, interface specifications, and technical requirements.

·         Conduct source-system analysis, data profiling, gap assessments, and source-to-target mapping.

·         Identify differences between existing procurement data and required OCDS structures.

·         Facilitate agreement among data owners, producers, consumers, architects, and governance teams.

·         Maintain traceability from business requirements through data models, engineering implementation, testing, and acceptance.

·         Evaluate requested changes for downstream effects on data products, reports, integrations, and analytical use cases.

·         Support backlog refinement, sprint planning, demonstrations, testing, and stakeholder acceptance.

OCDS Responsibilities

·         Apply the Open Contracting Data Standard to procurement and public-contracting datasets.

·         Map source-system data to appropriate OCDS fields and structures.

·         Work with data across the contracting lifecycle, including:

o    Planning

o    Tender

o    Award

o    Contract

o    Implementation

·         Develop and maintain mappings for OCDS releases, records, release packages, record packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related contracting elements.

·         Interpret and apply OCDS schemas, codelists, validation rules, and implementation guidance.

·         Determine whether standard OCDS fields meet project requirements or whether documented extensions are necessary.

·         Support the construction of complete contracting records from multiple transactional releases.

·         Establish rules for handling amendments, updates, cancellations, corrections, and historical changes.

·         Validate transformed data against applicable OCDS JSON schemas.

·         Identify missing, invalid, inconsistent, or nonconforming procurement data and work with stakeholders to resolve deficiencies.

·         Document assumptions, business rules, mappings, extensions, and exceptions.

·         Support the publication, exchange, analysis, or internal use of standardized contracting data.

Databricks Data Engineering

·         Design, develop, test, and maintain data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL.

·         Build ingestion and transformation pipelines for structured and semi-structured procurement data.

·         Process JSON, CSV, XML, Parquet, relational database, API, and file-based data sources.

·         Implement Bronze, Silver, and Gold data layers using medallion architecture.

·         Build normalized, dimensional, analytical, and OCDS-aligned data models.

·         Use Delta Lake capabilities for schema enforcement, schema evolution, versioning, auditability, and reliable processing.

·         Develop reusable frameworks for mapping source procurement data into OCDS-compatible outputs.

·         Implement batch and incremental ingestion patterns.

·         Use Databricks Workflows, notebooks, jobs, Auto Loader, Delta Live Tables or Lakeflow capabilities, as appropriate.

·         Develop REST API integrations for source ingestion and downstream data delivery.

·         Implement automated data-quality, reconciliation, completeness, and conformity checks.

·         Support Unity Catalog implementation for metadata, ownership, lineage, access control, and data discovery.

·         Optimize Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts.

·         Participate in code reviews, automated testing, CI/CD, deployment, monitoring, and production support.

·         Investigate pipeline failures, data discrepancies, and performance issues.

Data Quality and Governance

·         Define measurable quality rules for accuracy, completeness, validity, timeliness, consistency, and uniqueness.

·         Develop validation controls for required OCDS fields, identifiers, dates, amounts, currencies, organizations, classifications, and contracting relationships.

·         Create dashboards or reports that show data-contract compliance and data-quality results.

·         Establish processes for detecting and managing schema drift.

·         Document data lineage from original procurement systems through Databricks transformations and downstream products.

·         Work with governance teams to assign data owners, stewards, classifications, retention requirements, and access policies.

·         Ensure sensitive procurement and supplier information is handled according to security and privacy requirements.

·         Support auditability through documented rules, version-controlled mappings, validation results, and change histories.

Required Qualifications

·         Bachelor’s degree in computer science, information systems, data analytics, business analysis, engineering, public administration, supply-chain management, or a related field.

·         At least five years of combined data-engineering, data-analysis, business-analysis, or data-integration experience.

·         Hands-on experience implementing or working with the Open Contracting Data Standard.

·         Demonstrated experience constructing, documenting, negotiating, or maintaining data contracts.

·         Experience mapping procurement or contracting data to OCDS schemas.

·         Strong knowledge of OCDS releases, records, schemas, codelists, identifiers, contracting stages, and validation practices.

·         At least three years of hands-on experience with Databricks.

·         Strong experience with:

o    Apache Spark

o    PySpark

o    Python

o    SQL

o    Delta Lake

o    ETL and ELT pipelines

o    Data modeling

o    JSON and JSON Schema

o    REST APIs

o    Data-quality validation

o    Source-to-target mapping

·         Experience gathering requirements and translating them into implementable engineering specifications.

·         Ability to communicate effectively with technical and nontechnical stakeholders.

·         Experience writing user stories, acceptance criteria, business rules, data dictionaries, interface specifications, and process documentation.

·         Strong analytical, troubleshooting, facilitation, and documentation skills.

·         Experience working within Agile delivery teams.

Preferred Qualifications

·         Experience with government procurement, public-sector contracting, grants, acquisition, supplier, or financial data.

·         Experience implementing OCDS extensions or tailoring OCDS for specific organizational requirements.

·         Familiarity with procurement classifications, organizational identifiers, tender processes, awards, amendments, milestones, transactions, and contract implementation.

·         Databricks Certified Data Engineer Associate or Professional certification.

·         Experience with Unity Catalog, Delta Live Tables, Lakeflow, Databricks Workflows, or Structured Streaming.

·         Experience with Microsoft Azure, AWS, or Google Cloud.

·         Experience with Azure Data Factory, ADLS Gen2, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Power BI.

·         Experience using JSON Schema validation tools and automated data-quality frameworks.

·         Knowledge of metadata management, master-data management, data lineage, and reference-data governance.

·         Experience with CI/CD, Git, Azure DevOps, GitHub Actions, Jenkins, or Terraform.

·         Experience supporting large federal, state, local-government, or regulated-enterprise data programs.

·         Familiarity with federal acquisition, procurement, reporting, transparency, or open-data requirements.

Core Competencies

·         Ability to operate equally well in technical engineering and business-analysis discussions.

·         Strong understanding of how data contracts create accountability between data producers and consumers.

·         Ability to convert complex procurement processes into clear data structures and transformation rules.

·         Attention to detail when interpreting schemas, codelists, business definitions, and validation requirements.

·         Strong stakeholder-facilitation and conflict-resolution skills.

·         Ability to identify gaps and ambiguities before they become engineering defects.

·         Commitment to documentation, traceability, governance, and data quality.

·         Ability to work effectively within a large, multidisciplinary Databricks team.