Start searching
Back to Jobs
Technology

Bigdata /Hadoop/Spark Admin

Envision Technology SolutionsBerkeley Heights, NJ🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeOn Site
LocationBerkeley Heights, NJ, United States
ScalaApache SparkEncryptionSplunkApacheBashGrafanaHadoopHiveKafkaPrometheusPythonKubernetesLinux

Job description

Role: Apache Spark / Big Data Administrator

Location: Berkley Heights, NJ Onsite

Job Description

  • 10+ years solid experience with deploying, configuring, monitoring, and troubleshooting Apache Spark clusters and associated big data frameworks (such as Hadoop, Kafka, or cloud-based data platforms) to ensure high availability and performance.
  • Focuses on cluster health, performance tuning, and production data pipelines.
  • Technical Experience: Deep hands-on experience with Apache Spark architecture (Driver, Workers, Executors, RDDs/DataFrames) and cluster managers (YARN, Kubernetes, or Mesos).
  • OS & Scripting: Strong proficiency in Linux/Unix administration and scripting languages (Bash, Python, or Scala).
  • Ecosystem Knowledge: Familiarity with big data and streaming tools like Hadoop HDFS, Kafka, Hive, or monitoring suites (Splunk, Prometheus, Grafana).
  • Debugging Skills: Strong analytical capabilities to inspect logs, thread dumps, and metrics under heavy production loads
  • Excellent coordination with stakeholders and communication skills

Must have experience with below responsibilities

  • Cluster Management: Install, configure, and maintain Apache Spark clusters and node infrastructure in production and staging environments.
  • Performance & Monitoring: Monitor cluster health, resource utilization (CPU, memory), and execution logs using tools like Spark UI, Splunk, or Elastic. Identify bottlenecks and optimize job execution times.
  • Troubleshooting: Diagnose and resolve out-of-memory errors, executor failures, shuffle issues, and node crashes in distributed workloads.
  • Pipeline & Integration Support: Support data engineers and developers in managing Spark applications, streaming jobs, and integrations with message brokers like Kafka.
  • Security & Access Control: Implement security principles, role-based access control, authentication, and data encryption across the cluster.
  • Backup & Disaster Recovery: Execute backup, recovery, and failover plans for Spark applications, configurations, and critical data stores.

Patching & Upgrades: Apply software patches, driver updates, and version upgrades with minimal disruption to ongoing production jobs

Similar jobs

Apply