Job Description

Job Description:

Strong Core Python & Engineering Foundations: Writing clean, generic, reusable Python code using Object-Oriented Programming (OOP), classes, and functions. They must understand code architecture (e.g., how values/references are passed).
Python Packaging & Deployment: Hands-on experience building Python packages, using wheel files, and publishing/serving them via PyPI.
Automated Testing: Routine experience writing Unit Tests and Integration Tests in an automated CI/CD setup.
CI/CD Tools: Strong familiarity with GitHub Actions for automating builds and deployments.
PySpark Core Knowledge: They don''t need to be a Databricks wizard, but they must know the engine under the hood. They need to understand:
Optimizing PySpark code when things run slow.
How partitioning and ordering work.
How to speed up heavy data transfers (e.g., tuning JDBC connection parameters to pull huge tables).
System Security & Access Controls: Understanding basic security concepts, access-based controls, and how to securely control data access across systems.

Lead Data Engineer

Quick Overview