Big Data Platform Engineer

Title: Big Data Platform Engineer
Location : Jersey City, NJ Day one Onsite

s and s

Description:

Seeking an experienced Spark, HDFS, and Kafka Administrator/Engineer to manage, support, and optimize enterprise-grade big data platforms. Deep hands-on expertise with Apache Spark, HDFS, and Kafka is required and will be the primary focus of this role. Experience administering these technologies within a Cloudera environment is preferred but secondary to strong expertise in the underlying technologies

Main areas of responsibilities:

· Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads. Support Spark job performance, resource management, capacity planning, and production operations.

· HDFS Administration: Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting. Strong understanding of HDFS architecture and integration with distributed compute platforms is required.

· Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers. Support performance, capacity, availability, security, and production operations.

· Monitoring and Performance Tuning: Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and relevant platform management tools. Proactively identify and resolve performance bottlenecks across Spark, HDFS, and Kafka.

· Security and Compliance: Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.

· Automation and Scripting: Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.

· Troubleshooting and Support: Provide expert-level troubleshooting and technical support for Spark, HDFS, Kafka, and related big data technologies, including participation in an on-call rotation.

· Platform Administration: Support installations, patching, upgrades, migrations, and platform lifecycle activities while minimizing downtime. Cloudera Hadoop/CDP administration will be part of the role but is secondary to expertise in Spark, HDFS, and Kafka.

Requirements:

· Excellent problem-solving, analytical, and communication skills.

· Minimum 3 years of hands-on production experience administering and supporting Apache Spark, HDFS, and Kafka.

· Strong, demonstrable expertise in Spark administration, configuration, performance tuning, and troubleshooting.

· Strong, demonstrable expertise in HDFS administration, storage management, security, performance, and troubleshooting.

· Strong, demonstrable expertise in Kafka administration, topic management, producer/consumer architecture, performance, security, and troubleshooting.

· Experience with Linux operating systems and command-line tools.

· Experience with enterprise monitoring and operational tooling such as Grafana and Splunk.

· Experience with Cloudera Hadoop/CDP and Cloudera Manager is preferred but is not the primary skill requirement.

· Cloudera certifications or equivalent big data certifications are desirable but not required.

· Solid understanding of Kerberos, SSL/TLS, IAM/LDAP, and security concepts for distributed data platforms.