Start searching
Back to Jobs
Recently postedTechnology

Data Engineer

AIT Global, Inc.Columbus, IN🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationColumbus, IN, United States
FastAPIFlaskSQLScalaApache SparkNLPSnowflakeDatabricksDeep LearningHiveIoTJavaKafkaLLMPythonGenerative AIRESTAzureNode.jsReactMachine LearningAgile

Job description

Job Title: Data Engineer
Location: Columbus, IN
This role requires strong expertise in Python, PySpark, Databricks, and Machine Learning/AI, along with hands-on experience in React, Node.js, Azure Web Applications, and OpenAI API integration. The engineer will design, build, and deploy scalable data quality and observability solutions - including AI/ML-powered validation
Deliverables:
  • Analyzing complex business systems, industry requirements, and/or data regulations
  • Processing and managing large data sets
  • Design and development for a Big Data platform using open source and third-party tools 4. Data Quality Engineering & Automation
  • Design and develop data quality profiling pipelines using PySpark, Python, and Databricks to ensure data accuracy and reliability.
  • Build automated validation, cleansing, and anomaly detection frameworks leveraging ML/AI models.
  • Integrate OpenAI APIs for intelligent rule generation, data summarization, and NLP-based insights.
  • Full-Stack Development (exp in Web & API)
  • Develop responsive React-based user interfaces for self-service data quality monitoring and visualization.
  • Build and maintain RESTful APIs using Node.js and Python (Flask/FastAPI) to serve ML models and DQ metrics.
  • Host and manage applications using Azure Web Apps, ensuring secure API connectivity and scalable architecture.
  • Cloud and Data Integration
  • Work with Azure SQL Database, Azure Storage, and Databricks for seamless data access and validation.
  • Integrate data pipelines with APIs, Dataverse, Snowflake, or other cloud data sources. 15. Support Azure Application Insights and monitoring integration for performance tracking. 16. ML/AI Enablement
  • Contribute to ML projects focused on data quality improvement and predictive monitoring.
  • Apply techniques like anomaly detection, clustering, or deep learning to identify data issues proactively.
  • Use OpenAI / LLM APIs to automate root cause analysis, data profiling, and rule suggestions. 20. Collaboration and Documentation
  • Partner with Data engineers, Architect team, Cloud support team and business stakeholders to translate requirements into technical solutions.
  • Document DQ frameworks, ML workflows, APIs, and integration processes for reusability and transparency.
  • SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka or equivalent college coursework
  • SQL query language
  • Clustered compute cloud-based implementation experience
  • Developing applications requiring large file movement for a Cloud-based environment and other data extraction tools and methods from a variety of sources
  • Build analytical solutions 28. Experience with IoT technology, Experience in Agile software development
  • Manage Offshore delivery team from India, work assignment review, weekly report review with client

Similar jobs

Apply