Data Engineer - Cloud Data & Cybersecurity Analytics

Data Engineer Cloud Data & Cybersecurity Analytics

Location: Charlotte, NC 100% Onsite
Work Arrangement: Onsite, Monday Friday

Job Summary

We are seeking an experienced Data Engineer to develop, manage, and optimize cloud-based data solutions supporting cybersecurity analytics. The ideal candidate will have strong hands-on experience with AWS Glue, Apache Kafka, Terraform, Snowflake, Data Warehousing, Data Lakes, SQL, Python, and Spark.

This role will work closely with cross-functional teams to build secure, scalable, and high-performance data pipelines and platforms for processing cybersecurity-related data.

Key Responsibilities

• Develop and maintain AWS Glue ETL pipelines to process, transform, and integrate cybersecurity-related data.
• Design, implement, and optimize Data Warehouse and Data Lake architectures.
• Work with AWS S3 and Snowflake for large-scale data storage and analytics.
• Develop and manage Terraform Infrastructure-as-Code (IaC) deployments.
• Build and optimize Snowflake data models, queries, and data processing workflows.
• Perform Snowflake performance tuning and optimization for large datasets.
• Develop and manage APIs for data ingestion and data exchange between front-end applications, backend systems, and external platforms.
• Work with RESTful APIs, JSON, OAuth, API keys, and other authentication mechanisms.
• Develop, maintain, and troubleshoot SSIS packages.
• Monitor production data pipelines and batch jobs.
• Troubleshoot ETL failures and data processing issues.
• Work with Apache Kafka for data ingestion and ETL processing.
• Implement data transformation and processing solutions using Python, SQL, and Spark.
• Support data governance, security controls, encryption, access management, and compliance requirements.
• Participate in code reviews, security audits, architecture discussions, and technical design sessions.
• Identify opportunities to improve data pipeline reliability, scalability, performance, and operational efficiency.

Required Qualifications

• 5+ years of experience in cloud-based data engineering and big data processing.
• Strong hands-on experience with AWS Glue for ETL development and job orchestration.
• Strong expertise with Terraform and cloud Infrastructure-as-Code automation.
• Strong understanding of Data Warehouse and Data Lake architectures.
• Hands-on experience with AWS S3 and Snowflake.
• Strong Snowflake experience, including:
  • Data storage and modeling
  • SQL development
  • Query optimization
  • Performance tuning
  • Large-scale data processing
• Experience developing and managing API-based data ingestion from front-end applications and external systems.
• Hands-on experience creating and debugging SSIS packages.
• Strong knowledge of REST APIs, JSON, OAuth, API keys, and authentication mechanisms.
• Strong programming and data transformation skills using Python, SQL, and Spark.
• Hands-on experience with Apache Kafka, particularly in data ingestion and ETL processes.
• Understanding of cloud data security best practices, including encryption, access controls, and secure data handling.
• Strong troubleshooting and problem-solving skills.

Preferred Qualifications

• Experience working with cybersecurity or security analytics data.
• Experience with enterprise-scale cloud data platforms.
• Knowledge of data governance and compliance frameworks.
• Experience with CI/CD and automated deployment pipelines.
• Experience optimizing high-volume streaming and batch data pipelines.

eye