AI Responses

DuffyBot uses Anthropic's Claude AI to generate support responses. This page explains how responses are generated, how the confidence system works, and how to configure the AI's behavior.

How AI Responses Work

When a user sends a message in a ticket, here is what happens behind the scenes:

  1. Message received: DuffyBot detects a new message in a ticket channel
  2. FAQ check: The message is first checked against your FAQ entries. If there is an exact match, a pre-written response is sent instantly (no AI cost)
  3. Knowledge base search: If no FAQ matches, DuffyBot searches your knowledge base for relevant content
  4. AI generation: The relevant KB content, conversation history, and your settings are sent to the AI, which generates a natural response
  5. Confidence check: The AI evaluates how confident it is in the answer. If confidence is below your threshold, it escalates to human staff instead
  6. Response sent: The response is posted in the ticket channel

Confidence Scoring

Every AI response includes a confidence score that indicates how sure the AI is about its answer. The confidence system has three levels:

  • High confidence: The AI found clear, relevant information in your knowledge base and is sure about the answer. Response is sent normally.
  • Medium confidence: The AI found some relevant information but is not entirely sure. Response is sent with a note that a staff member may follow up.
  • Low confidence: The AI could not find enough information to give a good answer. Instead of guessing, it escalates to human staff.

Confidence Threshold

You can configure the confidence threshold in Settings > Behavior. This controls how confident the AI needs to be before responding:

  • Lower threshold: The AI responds to more questions, but some answers may be less accurate
  • Higher threshold: The AI only responds when very confident, which means more escalations to humans but higher accuracy
Tip: Start with the default threshold and adjust based on what you see. If the AI is giving too many wrong answers, raise the threshold. If it is escalating too often when it could answer, lower it.

Escalation to Humans

DuffyBot escalates to human staff in these situations:

  • The AI's confidence is below the configured threshold
  • The user explicitly asks to speak with a human
  • The message is a complaint or emotional situation the AI determines needs a human touch
  • The ticket involves actions the AI cannot perform (account changes, refunds, etc.)

When escalation happens, DuffyBot:

  1. Posts an escalation notice in the ticket
  2. Mentions configured staff roles so they get a notification
  3. Changes the ticket status to "escalated"
  4. Stops responding with AI (staff handle it from there)

Anti-Hallucination Guardrails

DuffyBot includes multiple safeguards to prevent the AI from making things up:

  • KB-only responses: The AI is instructed to only answer based on knowledge base content. It will not invent information.
  • Source verification: The AI must find matching KB content before answering. No matches means escalation.
  • "I don't know" responses: When uncertain, the AI is trained to say it does not know rather than guess.
  • Confidence scoring: Low-confidence answers are automatically escalated.
  • Prompt injection resistance: User inputs are sanitized before being sent to the AI, preventing manipulation attempts.
Info: No AI system is perfect. Staff should review AI responses periodically, especially for complex or sensitive topics. Use the Corrections feature in the dashboard to flag and fix wrong answers.

Conversation Style

You can set the AI's conversation style in Settings > Behavior. Options include:

  • Professional: Formal, business-appropriate tone. Good for SaaS products, enterprise support.
  • Friendly: Warm and approachable. Good for communities, gaming servers.
  • Casual: Relaxed and conversational. Good for small communities.

Response Length

Control how detailed AI responses are:

  • Concise: Short, to-the-point answers
  • Balanced: Moderate detail with clear explanations
  • Detailed: Thorough responses with step-by-step instructions where applicable

Perspective Mode

DuffyBot can respond as either:

  • First person: "We can help you with that" (speaks as your team)
  • Third person: "The team can help you with that" (speaks about your team)

First person is the default and works best for most servers. It makes the AI feel like part of your team.

Debug Mode

Staff can enable debug mode to see detailed information about each AI response:

  • Confidence score and reasoning
  • Detected intent (question, complaint, request, etc.)
  • Number of KB results found
  • AI cost (input and output tokens)
  • KB hints (what the AI thinks is missing from the knowledge base)

Debug info is shown in spoiler blocks, so it is only visible to staff who click to expand it. Users never see debug information.

Toggle debug mode with /debug on or /debug off in any ticket channel.

FAQs vs AI Responses

FAQs and AI responses serve different purposes:

  • FAQs are checked first and provide instant, pre-written responses. They are free (no AI cost) and always accurate. Use them for your most common, straightforward questions.
  • AI responses handle everything else. They are flexible and can answer questions the FAQ list does not cover, but they cost tokens and can occasionally be wrong.
Tip: A good strategy is to create FAQs for your top 10-20 most common questions, then let the AI handle the rest. This saves cost and improves response quality for frequent issues.
TicketsDashboard Guide