Haystack
← Back to Jobs
Full time
Engineering
MO

Prompt Engineer

MoonfireSydney, New South Wales🇦🇺AustraliaPosted 8 Aug 2026

Why This Role Stands Out

This hybrid Prompt Engineer role at Moonfire offers a fantastic opportunity to shape the future of LLM applications through innovative prompt design and rigorous evaluation, empowering you to significantly impact AI development. You'll thrive here if you possess a strong analytical mindset, a passion for AI, and a desire to collaborate with a dynamic team to push the boundaries of what's possible. Apply today to join a forward-thinking company and elevate your skills in a rapidly evolving field!

Quick Overview

Seniority
Mid Senior
Employment type
Full Time
Work mode
Hybrid
Location
Sydney, New South Wales, Australia

Job Description

Key Responsibilities

  • Design, develop, and maintain prompt frameworks, including system prompts, few-shot examples, role-based prompts, and reasoning workflows for production-grade LLM applications.

  • Build and manage automated evaluation frameworks to measure model performance, accuracy, latency, and regression across releases.

  • Conduct structured A/B testing across prompt variations, model versions, and configuration settings to optimize task-specific outcomes.

  • Convert product requirements and edge-case scenarios into effective prompt instructions, personas, constraints, and guardrails.

  • Partner with ML engineers and product teams to determine when prompt engineering is sufficient versus when fine-tuning, RAG, or other AI architectures are required.

  • Create and maintain a centralized prompt repository with version control, documentation, and performance benchmarks for organizational reuse.

  • Lead red-teaming and adversarial testing exercises to identify jailbreak risks, hallucinations, and model vulnerabilities.

  • Define evaluation criteria, annotation guidelines, and quality standards to ensure consistency, safety, and reliability of AI-generated outputs.

  • Mentor engineers and stakeholders on prompt engineering best practices, evaluation methodologies, and the capabilities and limitations of modern LLMs.

  • Present prompt strategies, benchmark results, and trade-off analyses to product, engineering, and leadership teams.

  • Apply advanced prompting techniques, including chain-of-thought, zero-shot, few-shot, and role-based prompting.

  • Drive prompt testing, evaluation, benchmarking, and continuous optimization efforts.

  • Improve AI response quality through systematic assessment, tuning, and refinement.

  • Manage context handling and prompt orchestration for complex AI workflows.


Technical Skills

  • Strong programming and scripting skills in one or more modern programming languages(C#, Python, Javascript).

  • Experience building automation, evaluation pipelines, APIs, or AI-powered applications using enterprise-grade development practices.

  • Hands-on experience with LLM platforms, prompt engineering, model evaluation, and AI application development.

  • Familiarity with prompt orchestration frameworks, vector databases, RAG architectures, and AI agent workflows.

  • Understanding of data analysis, experimentation, benchmarking, and performance optimization.

  • Experience with version control systems, CI/CD pipelines, and cloud platforms.

  • Strong knowledge of REST APIs, JSON, and system integration patterns.

  • Ability to collaborate effectively with software engineers, data scientists, and product teams to deliver production-ready AI solutions.


Experience Requirements

  • 4-7 years of combined experience in NLP, AI/ML products, software development, technical writing, or related fields.

  • At least 2 years of direct, hands-on prompt engineering experience with production LLM applications.

  • Proven track record of owning and managing prompt systems end-to-end, from design and implementation through monitoring and optimization in production.

Similar jobs

Browse related jobs

Apply now