Testing Your AI Agent in the Playground
The Playground lets you have a test conversation with your AI Agent before it talks to real customers. Every change you make to your persona, skills, or knowledge base can be tested here.
Go to AI Agent → Playground to start testing.
How the Playground Works
The Playground runs your message through the same process your live agent uses. It reads your knowledge base, applies your persona and tone settings, uses your enabled skills, and follows your guidelines. The only difference is that it doesn't send real SMS messages or take real actions on your Shopify store. Skill actions like cancelling an order or opting out a subscriber are simulated, not executed.
Starting a Test Conversation
Open the Playground tab
Type a message in the input field and send it
The AI responds just like it would to a real customer
For basic testing (persona, tone, knowledge base questions), you can start typing right away. But some features require a test scenario to work properly:
Selecting a Subscriber Search for and select a real subscriber from your store. The agent will have access to their order history and profile, just like in a real conversation. You need to select a subscriber to test:
Track Order: The agent needs a real subscriber to look up their orders
Cancel Order: The agent needs a real subscriber to find their unfulfilled orders (the order won't actually be cancelled in the Playground)
Initiate Return: Same as above, the agent needs real order data to work with
Selecting a Flow or Campaign Choose a specific flow or campaign as the conversation context. The first message from that flow or campaign loads into the conversation automatically. You need to select a flow or campaign to test:
Discount-related questions: The agent gets discount information from your Flows and Campaigns. If you ask about discounts without a test scenario selected, the Playground will prompt you to set one. This is how it works in production too: the agent only offers discounts that are configured in the flow or campaign the customer is in.
What to Look For in Responses
Each AI response in the Playground includes attribution details that help you understand how the agent arrived at its answer:
Source Badges Below each response, you'll see badges showing which knowledge was used:
Knowledge Source citations (which documents were referenced)
Facts that were applied
Verified Answers that matched
This tells you whether the agent is drawing from the right content. If it's citing an outdated source or missing a fact you just added, you know what to fix.
Skill Indicators If the agent used a skill (searched products, checked inventory, etc.), you'll see what was called and whether it succeeded.
Processing Time Each response shows how long it took, so you can get a sense of the response speed customers will experience.
Giving Feedback
Each AI response has thumbs up and thumbs down buttons. Use these to flag responses:
Thumbs up: The response was good. Helps you track which parts of your setup are working well.
Thumbs down: The response was wrong, off-tone, or unhelpful. Add an optional comment explaining what was wrong.
Quick Actions: Turning Responses Into Knowledge
If the Playground produces a great response, you can save it directly to your knowledge base:
Add as Verified Answer: Takes the customer's question and the AI's response and creates a new Verified Answer. Useful when the agent nails an answer and you want to guarantee that same response every time.
Add as Fact: Saves a piece of information from the response as a new Fact. Useful when the agent says something accurate that you want to make sure it always knows.
These quick actions save you from switching tabs and re-typing content.
What to Test
After setting up your persona:
Send a casual greeting ("hey!") and check that the tone matches your brand
Ask a question and see if the sign-offs and brand phrases appear naturally
Check that the emoji usage matches your frequency setting
After adding or changing knowledge:
Ask the specific question your new content should answer
Try a few variations of the question to test matching
Check that the source badges show the right content being used
If you added a Verified Answer with Bypass LLM, confirm the exact wording comes back
After enabling a skill:
Select a subscriber, then ask about their order (Track Order)
Ask "what do you have in stock?" or ask about a specific product (Search Products, Check Inventory)
Select a flow that has a discount, then ask about deals (Upsell Recommendations, discount handling)
After adding guidelines:
Test the specific scenario your guideline addresses
For example, if you added "always escalate allergy complaints," try: "I had a reaction to your moisturizer"
Check that the agent follows the rule
Edge cases:
Ask something the agent shouldn't know the answer to. Does it escalate or give a generic response?
Send a very short message ("ok", "thx"). Does it handle it gracefully?
Ask about a competitor. Does it stay on-brand?
Conversation Persistence
Your Playground conversation is saved in your browser session. You can navigate to other tabs and come back without losing the thread. However, if you make changes to your persona, skills, or knowledge base, the conversation clears automatically so you're testing against the latest configuration.
Tips
Test after every change. It takes 30 seconds to send a test message. Don't wait until you've made 10 changes and then try to figure out which one caused a problem.
Test like a real customer. Customers don't type perfectly formatted questions. Try shorthand, typos, and casual language: "wheres my order", "u got this in blue?", "return policy?"
Use the subscriber selector for realistic tests. If you select a subscriber with a recent order, you can test "Where's my order?" with real data instead of a generic response.
Select a flow to test discounts. The agent only knows about discounts configured in a flow or campaign. Without that context, it can't offer a deal.
Pay attention to what the agent doesn't know. If the agent gives a vague answer or escalates when it shouldn't need to, that's a gap in your knowledge base. Use the quick actions to fill it immediately.
Don't just test the happy path. Try questions the agent might struggle with. It's better to find gaps in the Playground than in a real customer conversation.
💡Tip:
Still have questions?
Please feel free to reach out to our wonderful Support team at support@txtcart.ai or via Live Chat.