Improving Your AI Agent's Performance

Once your AI Agent is live, you can improve its performance over time by paying attention to how it handles real conversations. This article covers what to look for and what actions to take.

Monitor Your Inbox

Your most valuable feedback comes from real conversations. Make it a habit to review AI conversations in your Inbox, especially in the first few weeks after going live or after making knowledge base changes.

What to look for:

Conversations where the AI escalated. What was the customer asking? Could you have prevented the escalation with better knowledge?
Conversations where the source badges show the wrong content being cited. Is outdated or irrelevant content confusing the agent?
Conversations where the tone feels off. Do you need to adjust your tone sliders or add brand phrases?
Conversations where the AI gave a correct but incomplete answer. Would a Fact or Verified Answer fill the gap?

Act on Escalation Patterns

Escalations are the clearest signal for improvement. Each escalation type points to a specific action:

You're Seeing	Action
Repeated Knowledge Gap escalations for the same topic	Add a Verified Answer or Knowledge Source covering that topic
Knowledge Gap escalations for topics you've already covered	Check if the content is stale, too vague, or buried in a long document. Break it into a focused Fact or Verified Answer.
Customer Request escalations (asking for a human)	Review recent conversations to see if customers are asking for a human because the AI isn't helpful, or simply because they prefer human contact. The former is fixable; the latter is normal.
High volume of Refund Request or Product Complaint escalations	These will always escalate, but if the volume is high, consider adding more detail to your return policy Knowledge Source so the AI provides clear instructions before handing off.

Strengthen Your Knowledge Base

The most impactful improvement you can make is keeping your knowledge base accurate and complete.

Weekly review (5 minutes):

Check Content Health for failed sources or sources that need attention
Look for pending Shopify Pages or Guidelines that need approval or dismissal
Archive any temporary Facts that have expired (promotions, seasonal info)

Monthly review (15 minutes):

Read through a sample of recent AI conversations. Are there patterns in what the AI handles well vs. where it struggles?
Check if your Verified Answers still reflect current policies and pricing
Review your top Knowledge Sources. Are they still up to date?
Look at your guidelines. Are any no longer relevant? Are there new behaviors you want to enforce?

Fine-Tune Your Persona

If the AI's tone doesn't feel right in real conversations, small adjustments make a noticeable difference:

Responses feel too stiff? Lower the Formality slider by 10-15 points.
Responses feel too casual? Raise Formality. Consider removing informal brand phrases.
Responses feel cold when customers have problems? Raise the Empathy slider.
Responses feel over the top? Lower Enthusiasm. Check if you have too many exclamation-heavy brand phrases.

After adjusting, test in the Playground with a few common customer questions to confirm the change feels right.

Optimize Your Verified Answers

Verified Answers are your highest-confidence responses. Optimize them over time:

Check usage counts. Low-usage Verified Answers might need better variants. The question and variants may not match how your customers actually phrase things.
Add variants from real conversations. When you see a customer ask a question in a way your Verified Answer didn't match, add that phrasing as a variant.
Review match confidence levels. If a Verified Answer is triggering on the wrong questions, tighten the confidence from Broad to Confident or Exact. If it's not triggering when it should, loosen it.
Use the Playground to test. After adding variants or adjusting confidence, test with a few phrasings to confirm the behavior.

Review Your Skills

Make sure the skills you have enabled are working for your store:

Track Order enabled but customers rarely ask about orders? It's fine to leave it on. There's no cost to having it available.
Cancel Order enabled but you're getting cancellations you'd rather review first? Disable it and let those requests escalate to your team instead.
Upsell Recommendations enabled but recommendations feel generic? Add upsell bundles to give the AI specific products to suggest.

Use the Playground After Changes

This comes up in every article because it's the single most important habit. Every time you:

Add or update a Knowledge Source, Fact, or Verified Answer
Change a guideline
Adjust tone sliders or persona settings
Enable or disable a skill

Open the Playground, test the relevant scenario, and confirm it works as expected. It takes 30 seconds and prevents surprises in real customer conversations.

What "Good" Looks Like

There's no single metric that tells you your AI Agent is performing well, but here are signs you're on the right track:

Knowledge Gap escalations are rare. The AI has answers for most of what customers ask.
Conversations resolve without override. You're reading conversations in the Inbox but rarely need to take over.
Source badges show the right content. The AI is citing your current, relevant Knowledge Sources and Facts.
Tone feels natural. Conversations read like they were written by a knowledgeable team member, not a chatbot.
Customers reply positively. "Thanks!", "That's helpful", "Perfect" after AI responses are good signals.