Data Engineer

Data Engineer

Denver, Colorado

6 Months Contract

Face-to-Face Interview Mandatory in Denver, CO

NO REMOTE | NO FAKE PROFILES

Job Description

The Data Engineer IV designs, builds, and maintains ETL pipelines and data infrastructure that feed the IIA Data Lake, anomaly detection models, and AI agents. This role focuses on constructing robust, scalable data pipelines using Spark/Scala, ensuring data quality and availability across a growing portfolio of network data sources, and enabling downstream consumers such as data scientists, agents, and dashboards to access reliable, well-structured data.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines using Apache Spark (Scala) to ingest, transform, and load network data into the IIA Data Lake.
• Onboard new data sources including network telemetry, syslogs, SNMP traps, device configuration data, and ticketing systems.
• Build ingestion pipelines from raw source data to query-ready formats.
• Implement monitoring and alerting solutions to ensure data pipeline reliability and performance.
• Develop and manage deployment pipelines to facilitate continuous integration and delivery of data engineering solutions.
• Manage and optimize data storage solutions including distributed file systems, relational databases, flat files, and external sources accessed through APIs.
• Implement data quality checks, validation rules, and automated testing to ensure pipeline reliability and data integrity.
• Optimize pipeline performance for large-scale data processing involving billions of events per day across batch and mini-batch processing patterns.
• Manage and evolve data schemas, partitioning strategies, and storage formats to support efficient querying and downstream consumption.
• Support data backfills and recovery when upstream issues or schema changes require reprocessing.
• Collaborate across teams to ensure data solutions align with production architectures and business requirements.
• Work with data scientists and agent developers to understand data requirements and deliver datasets supporting anomaly detection models and AI agent workflows.
• Provide technical guidance on data engineering best practices and methodologies.
• Document and communicate data engineering processes and standards to business intelligence, data, and analytics professionals.
• Continuously evaluate and improve data engineering tools and approaches to enhance performance and efficiency.
• Perform other duties as required.

Required Qualifications

• Expertise in Scala preferred, or Java, with proficiency in Python.
• Strong experience with Apache Spark for distributed data processing.
• Proficiency in building and maintaining ETL pipelines at scale.
• Experience with AWS S3, Glue, Athena, and EMR.
• Strong understanding of relational databases and SQL.
• Knowledge of data architecture, data warehousing, partitioning strategies, and columnar storage formats such as Parquet.
• Experience implementing data quality checks and validation frameworks.
• Experience with workflow orchestration tools; Airflow preferred.
• Proficiency with Linux-based operating systems and shell scripting.
• Experience with Git-based version control and collaborative development workflows.
• Demonstrated ability and willingness to continually expand technical skills and learn from and teach others.

Preferred Qualifications

• Experience with streaming or mini-batch data processing, including Spark Streaming or Structured Streaming.
• Experience with Apache Kafka or similar messaging/streaming platforms.
• Experience with NoSQL databases.
• Experience in the telecommunications industry or other large-scale network operations environments.

Candidate Requirements

• Local Denver, CO candidates preferred.
• H1B candidates are acceptable.
• Face-to-face interview is mandatory in Denver, CO.
• Remote candidates are not acceptable.
• Candidates must be able to work onsite in Denver, Colorado.
• No fake profiles.