Back to Jobs
Recently postedTechnology

Sr AI Engineer

ISOFTChicago, IL🇺🇸United States

Quick overview

Work modeOn-site
LocationChicago, IL, United States
SQLGitApache SparkAWSETLMLOpsMLflowScikit-learnApacheDatabricksDeep LearningGoogle CloudHugging FaceKafkaLLMPyTorchPythonTensorFlowTerraformUnityGenerative AIAzureMachine LearningCI/CDDelta LakeLangChainRAG

Job description

Role: Sr AI Engineer

Location:Onsite. Chicago, IL

Type: C2C

Send profiles to

Job Description:

We are seeking a Senior AI Engineer with deep, hands-on expertise in Databricks to design, build, and deploy production-grade AI and machine learning solutions at scale. You will lead the end-to-end lifecycle of data-intensive AI applications—from building resilient, large-scale data ingestion and transformation pipelines to training, fine-tuning, and operationalizing ML and Generative AI models in a unified lakehouse environment.

Key Responsibilities:

  • Large-Scale Data Engineering & Transformation: Architect and optimize high-throughput batch and streaming ETL/ELT pipelines using Apache Spark, PySpark, Delta Lake, and Databricks Lakehouse architectures to process multi-terabyte datasets.

  • AI & Machine Learning Model Development: Design, train, evaluate, and fine-tune predictive ML models, deep learning architectures, and Generative AI/LLM workflows (e.g., RAG pipelines, agentic systems) tailored to business-critical use cases.

  • MLOps & Production Deployment: Operationalize models at scale using MLflow, Databricks Model Serving, and CI/CD best practices, ensuring automated retraining, drift monitoring, and low-latency inference.

  • Data Governance & Feature Engineering: Build reusable feature pipelines using Databricks Feature Store and enforce data quality, lineage, and access controls via Unity Catalog.

  • Performance & Cost Optimization: Tune Spark jobs, cluster configurations, Photon engine workloads, and Delta table partitioning/Z-ordering to maximize compute efficiency and minimize cloud infrastructure costs.

  • Cross-Functional Technical Leadership: Partner with data scientists, software engineers, and product managers to translate complex business requirements into scalable technical architectures, while mentoring junior engineers on coding and MLOps standards.

Required Qualifications:

  • Experience: 5+ years of hands-on experience in AI/ML engineering, data science, or big data engineering, with at least 3+ years building production systems on Databricks.

  • Databricks Ecosystem Mastery: Proven expertise in PySpark, Spark SQL, Delta Lake, MLflow, Unity Catalog, and Databricks Workflows.

  • Programming & Software Engineering: Strong proficiency in Python and SQL, with solid software engineering fundamentals (Git, modular design, unit testing, CI/CD).

  • AI/ML Frameworks: Deep familiarity with modern ML and deep learning libraries (e.g., PyTorch, TensorFlow, Scikit-learn, Hugging Face, LangChain).

  • Cloud Infrastructure: Hands-on experience deploying data and AI workloads on major cloud platforms (AWS, Azure, or Google Cloud Platform).

Preferred Qualifications:

  • Experience building and deploying Generative AI / LLM applications using Databricks Mosaic AI, Vector Search, and Retrieval-Augmented Generation (RAG).

  • Familiarity with infrastructure-as-code tools (Terraform, Databricks Asset Bundles) for automated environment provisioning.

  • Experience with real-time streaming technologies (Apache Kafka, Spark Structured Streaming).

  • Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or a related quantitative field.

Will your resume get past this employer's screening? Check your resume against this job, free

Similar jobs