The ideal candidate will have strong hands-on experience with Python, APIs, Generative AI, LLMs, and Agentic Systems, along with a strong understanding of AI evaluation and benchmarking. You will work closely with engineering and product teams to assess agent performance, identify gaps, design evaluation frameworks, and drive improvements in overall agent quality and user outcomes. Key Responsibilities • Test and evaluate LLM-powered Agentic Systems across different business scenarios and user journeys. • Develop and implement evaluation frameworks, datasets, and benchmarks for AI agents. • Define and measure key agent performance metrics, including: • Tool-calling accuracy and success rate • Response accuracy and relevance • Reliability and consistency • Task completion rate • Multi-turn conversation performance • Overall system effectiveness • Design evaluation datasets, including questions, expected answers, test cases, and multi-turn conversational scenarios. • Benchmark agent performance across different models, prompts, tools, workflows, and system configurations. • Analyze agent failures and identify root causes and areas for improvement. • Evaluate the agent's ability to correctly understand user intent, reason through tasks, select appropriate tools, and generate accurate responses. • Work with large volumes of business documentation, product information, and data to quickly understand complex business domains. • Develop Python-based scripts and tools for automated evaluation, testing, data processing, and analysis. • Work with APIs and system integrations to test agent interactions with external tools and services. • Collaborate with AI/ML engineers, software engineers, product managers, and domain experts to continuously improve the agent. • Monitor evaluation results and provide data-driven recommendations to improve agent quality and user experience. • Contribute to building scalable processes for continuous evaluation and regression testing of Agentic AI systems. Required Skills & Experience Technical Skills • Strong programming experience in Python. • Solid understanding of REST APIs, API integrations, and system interactions. • Strong understanding of Generative AI and Large Language Models (LLMs). • Deep understanding of Agentic AI systems, including: • AI agents and workflows • Tool/function calling • Prompting and context management • Multi-turn conversations • Agent orchestration • Retrieval and external knowledge integration • Experience with AI/LLM evaluation and benchmarking. • Experience designing datasets and test scenarios for AI systems. • Strong analytical skills with the ability to work with evaluation data and identify patterns, failures, and performance gaps.
Submit your application directly to the hiring team.