Hiring QA Gen AI Automation Engineer for Jersey City, NJ

Hiring QA Gen AI Automation Engineer for Jersey City, NJ

Role: QA Gen AI Automation Engineer
Location: Jersey City, NJ
Duration: Long Term

Key Responsibilities:
Automation Testing:
Develop and maintain automated regression tests using tools like Selenium WebDriver/Playwright and pytest.
Conduct API testing using tools such as Postman.
Generative AI Testing:
Design and execute test cases for Generative AI models and applications.
Evaluate the performance and quality of Generative AI outputs, ensuring they meet functional requirements and quality standards.
Validate AI systems such as Chatbots, RAG applications, and LLM-based workflows.
Perform LLM evaluation using frameworks such as DeepEval and RAGAS.
Programming & Scripting:
Write and maintain automated test scripts in Python.
Work with XML, JSON, and other scripting languages to create efficient testing environments for AI systems.
Testing Tools & Frameworks:
Utilize JMeter for performance testing and load testing.
Implement and manage Continuous Integration (CI) pipelines using tools such as Jenkins and Bamboo.
Database & SQL:
Hands on experience with Panda Data frames
Execute SQL queries for database testing (inserts, updates, joins, etc.).
Test non-GUI applications (e.g., SQL, flat files, XML).
Work with Vector Databases such as OpenSearch or similar vector DB technologies.
Agile & DevOps:
Collaborate within an Agile (Scrum) framework for efficient testing and development.
Utilize Atlassian JIRA and Confluence for project tracking and documentation.
Product Evaluation & Troubleshooting:
Identify and troubleshoot product issues, ensuring quality and stability.
Evaluate product performance and provide feedback to development teams.
Web Application Testing:
Conduct testing for web applications and web services (REST, SOAP).
Required Skills & Experience:
Programming:
Proficient in Python for writing test scripts and automating processes.
Automation Testing:
Experience with Selenium WebDriver/Playwright and pytest for automation testing.
Hands-on experience in API testing using tools like Postman.
Generative AI & AI Testing:
Understanding of AI systems and AI Testing concepts.
Knowledge of Large Language Models (LLMs), RAG, and Chatbot AI systems.
Experience designing and testing Generative AI systems and applications.
Knowledge of LLM evaluation techniques and frameworks such as DeepEval and RAGAS.
Ability to assess the output quality, coherence, and accuracy of Generative AI models.
Database & Search Technologies:
Strong understanding of SQL for testing databases (including joins, inserts, and updates).
Understanding of Vector Databases such as OpenSearch or similar vector DB technologies.
Testing Tools:
Experience with JMeter for performance and load testing.
Familiarity with CI/CD tools like Jenkins or Bamboo.
Agile Methodology:
Experience working in an Agile (Scrum) environment.
Familiar with JIRA and Confluence for issue tracking and project management.
Web & API Testing:
Experience testing web applications and web services.
Product Performance:
Capable of evaluating and troubleshooting performance-related issues.
Nice to Have:
Agentic AI & Observability:
Understanding of Agents, MCP, and Agentic AI systems.
Familiarity with observability tools such as Langfuse.
Cloud Experience:
Familiarity with AWS services such as AWS Bedrock for cloud-based AI environments.
Development Experience:
A background in software development is a plus.
Industry Knowledge:
Understanding of the Property and Casualty (P&C) insurance industry is beneficial.