Lead Prompt Engineer for Production-Grade LLMs

Key Responsibilities

• Design,develop, and maintain prompt frameworks, including system prompts,few-shot examples, role-based prompts, and reasoning workflows forproduction-grade LLM applications.
  
• Build andmanage automated evaluation frameworks to measure model performance,accuracy, latency, and regression across releases.
  
• Conductstructured A/B testing across prompt variations, model versions, andconfiguration settings to optimize task-specific outcomes.
  
• Convertproduct requirements and edge-case scenarios into effective promptinstructions, personas, constraints, and guardrails.
  
• Partnerwith ML engineers and product teams to determine when prompt engineeringis sufficient versus when fine-tuning, RAG, or other AI architectures arerequired.
  
• Create andmaintain a centralized prompt repository with version control, documentation,and performance benchmarks for organizational reuse.
  
• Leadred-teaming and adversarial testing exercises to identify jailbreak risks,hallucinations, and model vulnerabilities.
  
• Defineevaluation criteria, annotation guidelines, and quality standards toensure consistency, safety, and reliability of AI-generated outputs.
  
• Mentorengineers and stakeholders on prompt engineering best practices,evaluation methodologies, and the capabilities and limitations of modernLLMs.
  
• Presentprompt strategies, benchmark results, and trade-off analyses to product,engineering, and leadership teams.
  
• Applyadvanced prompting techniques, including chain-of-thought, zero-shot,few-shot, and role-based prompting.
  
• Driveprompt testing, evaluation, benchmarking, and continuous optimizationefforts.
  
• Improve AIresponse quality through systematic assessment, tuning, and refinement.
  
• Managecontext handling and prompt orchestration for complex AI workflows.
  

Technical Skills

• Strongprogramming and scripting skills in one or more modern programminglanguages(C#, Python, Javascript).
  
• Experiencebuilding automation, evaluation pipelines, APIs, or AI-poweredapplications using enterprise-grade development practices.
  
• Hands-onexperience with LLM platforms, prompt engineering, model evaluation, andAI application development.
  
• Familiaritywith prompt orchestration frameworks, vector databases, RAG architectures,and AI agent workflows.
  
• Understandingof data analysis, experimentation, benchmarking, and performanceoptimization.
  
• Experiencewith version control systems, CI/CD pipelines, and cloud platforms.
  
• Strongknowledge of REST APIs, JSON, and system integration patterns.
  
• Ability tocollaborate effectively with software engineers, data scientists, andproduct teams to deliver production-ready AI solutions.
  

Experience Requirements

• 4-7 yearsof combined experience in NLP, AI/ML products, software development,technical writing, or related fields.
  
• At least 2years of direct, hands-on prompt engineering experience with productionLLM applications.
  
• Proventrack record of owning and managing prompt systems end-to-end, from designand implementation through monitoring and optimization in production.