Start searching
Back to Jobs
Recently postedTechnology

data Engineer

Cynosure Technologies LLCNew York, NY🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeOn Site
LocationNew York, NY, United States
DockerAWSAirflowApacheGoogle CloudHadoopHiveJavaPigPythonAzureKubernetesMachine Learning

Job description

title : Data Engineer

location : NY/Chicago/Dallas (Onsite)

Responsibilities:
  • Design, develop, and maintain batch processing pipelines for large-scale data transformation and analysis using technologies like Apache Hadoop, Pig, Hive, Spark, or similar frameworks.
  • Experience with Airflow and/or Cloud Composer for workflow management and orchestration
  • Work with data scientists and product managers to develop and deploy data pipelines for data warehousing and data lake use cases
  • Collaborate with other engineers to integrate batch processing systems with our data streaming and real-time analytics infrastructure
  • Ensure data quality, data security, and data privacy across all batch processing systems
  • Develop and implement batch processing jobs using programming languages like Java or Python. (Experience with both is a plus)
  • Work with public cloud providers such as Google Cloud Platform (Google Cloud Platform), Amazon Web Services (AWS), or Microsoft Azure
  • Experience with Dataproc is a strong preference
  • Optimize batch pipelines for performance and resource efficiency.
  • Monitor and troubleshoot batch jobs to ensure successful execution and data quality.
  • Implement DevOps principles and practices to ensure efficient and reliable deployment of batch processing systems
  • Knowledge of containerization technologies such as Docker and Kubernetes
Requirements:
  • Bachelor's degree in Computer Science or a related field
  • 5+ years of experience in software engineering, with a focus on batch processing and analytics
  • Strong programming skills in Java or Python
  • Experience with a public cloud provider, with a focus on Google Cloud Platform
  • Strong experience in batch processing systems such as Hadoop, Pig, Hive, and Spark
  • Experience with Dataproc is a plus
  • Knowledge of DevOps principles and practices
  • Understanding of containerization technologies such as Docker and Kubernetes
  • Excellent problem-solving skills, with the ability to work independently and collaboratively
  • Excellent communication skills, with the ability to explain complex technical concepts to non-technical stakeholders
  • Experience with machine learning and data science technologies is a plus
  • Certification in a public cloud provider or a relevant technology is a plus

Similar jobs

Apply