Haystack
← Back to Jobs
Full time
Engineering

Prompt Engineer

BlackCube LabsSydney, New South Wales🇦🇺AustraliaPosted 8 Aug 2026

Why This Role Stands Out

In this hybrid Prompt Engineer role at BlackCube Labs, you will drive innovation by designing and optimizing advanced prompt frameworks for cutting-edge LLM applications, with significant opportunities for skill development and leadership. If you thrive on tackling complex challenges, possess strong analytical and programming skills, and enjoy collaborating with talented teams to shape the future of AI, this position offers a dynamic and rewarding career path. Apply today to make a tangible impact in a forward-thinking engineering environment.

Quick Overview

Work Type
Hybrid
Schedule
Full Time
Level
Mid Senior

Job Description

Key Responsibilities
  • Design,develop, and maintain prompt frameworks, including system prompts,few-shot examples, role-based prompts, and reasoning workflows forproduction-grade LLM applications.
  • Build andmanage automated evaluation frameworks to measure model performance,accuracy, latency, and regression across releases.
  • Conductstructured A/B testing across prompt variations, model versions, andconfiguration settings to optimize task-specific outcomes.
  • Convertproduct requirements and edge-case scenarios into effective promptinstructions, personas, constraints, and guardrails.
  • Partnerwith ML engineers and product teams to determine when prompt engineeringis sufficient versus when fine-tuning, RAG, or other AI architectures arerequired.
  • Create andmaintain a centralized prompt repository with version control, documentation,and performance benchmarks for organizational reuse.
  • Leadred-teaming and adversarial testing exercises to identify jailbreak risks,hallucinations, and model vulnerabilities.
  • Defineevaluation criteria, annotation guidelines, and quality standards toensure consistency, safety, and reliability of AI-generated outputs.
  • Mentorengineers and stakeholders on prompt engineering best practices,evaluation methodologies, and the capabilities and limitations of modernLLMs.
  • Presentprompt strategies, benchmark results, and trade-off analyses to product,engineering, and leadership teams.
  • Applyadvanced prompting techniques, including chain-of-thought, zero-shot,few-shot, and role-based prompting.
  • Driveprompt testing, evaluation, benchmarking, and continuous optimizationefforts.
  • Improve AIresponse quality through systematic assessment, tuning, and refinement.
  • Managecontext handling and prompt orchestration for complex AI workflows.
Technical Skills
  • Strongprogramming and scripting skills in one or more modern programminglanguages(C#, Python, Javascript).
  • Experiencebuilding automation, evaluation pipelines, APIs, or AI-poweredapplications using enterprise-grade development practices.
  • Hands-onexperience with LLM platforms, prompt engineering, model evaluation, andAI application development.
  • Familiaritywith prompt orchestration frameworks, vector databases, RAG architectures,and AI agent workflows.
  • Understandingof data analysis, experimentation, benchmarking, and performanceoptimization.
  • Experiencewith version control systems, CI/CD pipelines, and cloud platforms.
  • Strongknowledge of REST APIs, JSON, and system integration patterns.
  • Ability tocollaborate effectively with software engineers, data scientists, andproduct teams to deliver production-ready AI solutions.
Experience Requirements
  • 4-7 yearsof combined experience in NLP, AI/ML products, software development,technical writing, or related fields.
  • At least 2years of direct, hands on prompt engineering experience with productionLLM applications.
  • Proventrack record of owning and managing prompt systems end-to-end, from designand implementation through monitoring and optimization in production.

Similar jobs

Apply now