Data Engineer

Job Summary We are seeking a Data Engineer with expertise in Azure Databricks to support the modernization of enterprise data platforms. The ideal candidate will have experience migrating Oracle PL/SQL-based ETL processes to Databricks, developing scalable cloud-based data pipelines, and ensuring data quality, reliability, and performance.

Key Responsibilities Gather business and technical requirements from stakeholders. Analyze Oracle SQL and PL/SQL ETL processes for migration. Migrate Oracle ETL workflows to Azure Databricks using Spark SQL and PySpark. Design, develop, test, deploy, and maintain scalable data pipelines. Create technical design documents and data mappings. Validate migrated data with business users and SMEs. Implement incremental data processing using Delta Lake. Convert complex Oracle PL/SQL logic into Databricks pipelines. Perform data validation and reconciliation between source and target systems.

Monitor, troubleshoot, and enhance production ETL pipelines. Optimize pipeline performance and resolve production issues. Participate in source control, CI/CD, and release management activities. Manage assigned projects and deliver high-quality solutions.

Required Qualifications Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field. 46 years of experience in ETL, database development, or data engineering. 4+ years of advanced SQL development and query optimization. 3+ years of Oracle PL/SQL development experience. 3+ years of Azure Databricks development experience. Experience migrating legacy ETL processes to cloud-based data platforms. Strong experience with Spark SQL, PySpark, Delta Lake, and Azure Data Lake Storage (ADLS). Experience with Git, source control, and CI/CD processes.

Strong analytical and problem-solving skills. Excellent communication and collaboration skills.

Preferred Qualifications Experience in healthcare, biomedical research, or clinical data environments. Experience converting Oracle PL/SQL packages, procedures, and functions into Databricks pipelines. Experience with Azure Data Factory or Databricks Workflows. Knowledge of dimensional data modeling and enterprise data warehousing. Experience with Python. Experience with Power BI or Tableau. Familiarity with AI or machine learning-enabled data engineering solutions.

Education: Bachelors Degree