Back to Jobs
Technology

Data Engineer

Y & L Consulting Inc.United States🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationUnited States
SQLScalaGitApache SparkETLTableauApacheDatabricksPythonAzurePower BICI/CDAzure DevOpsAzure Data FactoryDelta Lake

Job description

VENDORS OR C_2_C CANDIDATES WILL NOT BE CONSIDERED!!!

Data Engineer
Key Responsibilities
Design, develop, and maintain scalable ETL/ELT pipelines using Informatica, Azure Data Factory and Databricks.
Process and transform structured and unstructured data from various sources into a centralized data lake or warehouse.
Leverage Azure Data Lake Storage and Databricks Lakehouse to build efficient data storage and analytics solutions.
Ensure seamless integration with Azure services such as Azure Event Hubs, Azure Stream Analytics, and Azure Functions.
Optimize data pipelines for performance, scalability, and cost-effectiveness.
Implement partitioning, indexing, and caching strategies to handle large-scale datasets.
Key Skills :
5 years of experience as a data engineer or in a similar role.
Hands-on expertise with Informatica, Azure Data Factory, Azure Synapse, and Azure Data Lake.
Proficiency in Databricks for data transformation, analytics, and pipeline orchestration.
Strong programming skills in Python, Scala, or SQL.

Experience with distributed data processing frameworks (e.g., Apache Spark).
Familiarity with CI/CD pipelines and version control systems (e.g., Azure DevOps, Git).
Experience with Delta Lake and Databricks.
Knowledge of real-time data ingestion and processing.
Exposure to BI tools such as Power BI or Tableau.
Strong problem-solving and debugging capabilities.
Excellent communication and collaboration skills.
Knowledge of data modeling techniques and schema designs for analytics.
Preferred Azure / Databricks certifications.
\

Similar jobs