Testing Genies and MCP Servers: A Complete Methodology
Testing an AI agent is not like testing a deterministic API. LLM-based agents require testing across happy paths, edge cases, and adversarial inputs — and results must be validated at the skill, knowledge base, and response levels.
The 4 Testing Levels for Enterprise AI Agents
| Level | What It Tests | Who Runs It |
|---|---|---|
| Test Mode (Build) | Skill routing, knowledge base retrieval, prompt effectiveness during active development | Builders (active development only) |
| Functional Testing | End-to-end use-case validation — does the Genie complete the full workflow correctly? | Builders + QA |
| Security Testing | VUA enforcement, skill-level access, permission-aware KB, approval flows, prompt injection resistance | Security team + builders |
| User Acceptance Testing | Real-user validation with a pilot group in the production chat interface | Pilot users + Genie owner |
Using Agent Studio Test Mode
Test Mode in Agent Studio provides live visibility into every Genie action during development — which skills were called, which knowledge bases were searched, what content fragments were retrieved, and how many turns the task required.
Test Mode access should only be granted to: active builders working on or maintaining the Genie, and Genie owners conducting QA before changes go live. Do not grant Test Mode access to end users.
LLM Accuracy Scenario Testing
Create custom test scenarios in Agent Studio (AI Hub > Genies > Test Mode > Add Scenario). Each scenario has a name, description, and a set of individual messages. Selecting a message automatically logs it to the conversation history for testing.
Design scenarios across all three session types:
- Happy path: Standard successful workflows. These establish your baseline and should pass 100% before other testing begins.
- Edge cases: Unusual but valid inputs — past dates, ambiguous leave types, partial names, out-of-range values. Most real-world failures occur here.
- Out-of-scope: Requests the Genie should decline — accessing another employee's data, requesting actions outside the Genie's domain, prompt injection attempts.
What to Validate Beyond the Answer
A correct-looking response can still fail at the execution level. During every test session, check:
- Which skill was called: Incorrect skill invocation usually traces back to gaps in the job description, skill description, or Business Event instructions.
- Which knowledge base was searched: Validates that the correct KB is selected when multiple KBs are attached to the Genie.
- What content fragments were retrieved: The Genie may surface a correct answer from the wrong KB fragment. Verify that retrieved content directly and accurately answers the question.
- How many turns to resolution: Excessive turns indicate unclear instructions or routing failures.
Validating in the Chat Interface
Critical: Test Mode uses recipe-level connections, even when skills are configured with Verified User Access. This means Test Mode does not accurately reflect real end-user behavior. Always validate in the actual chat interface (Slack or Teams) with end-user connections before declaring a Genie production-ready.
In the chat interface, additionally verify: users can successfully authenticate to applications at runtime, markdown formatting renders correctly in Slack/Teams (not all markdown renders identically across clients), and response time is acceptable in the live environment.
Testing MCP Servers
MCP server testing focuses on tool selection accuracy and multi-step reliability:
- Tool selection: Confirm the correct tool is selected for each request type. "Find customers named Acme" should consistently trigger
search_customers, notget_customer_by_id. - Multi-turn context: Test that the LLM retains context across turns — applying a filter mentioned two messages ago when refining search results.
- Multi-tool workflows: Verify tool chains execute in the correct sequence and that downstream tools receive the correct inputs from upstream tool outputs.
MCP Server Edge Case Checklist
- Missing required parameters → tool returns a clear validation error, not a silent failure
- Results exceeding the defined limit → pagination signal is present; LLM offers "show more"
- Permission denial → distinct error message, not a generic system error
- Empty result sets → distinguishable from errors; LLM communicates "no results found" accurately
- Downstream system timeout → graceful error with retry or escalation guidance
Security Testing Checklist
- VUA: skills with user-scoped actions reject unauthenticated users
- Prompt injection: known injection patterns are blocked at the platform level
- Permission-aware KB: users only retrieve content they're authorized to see
- Approval flows: skills requiring approval don't execute without the configured approval being granted
- Out-of-scope requests: Genie declines and redirects; does not engage