Skip to main content

Reflection System

Overview​

The reflection system validates task completion after the agent provides a response. Located in lib/agent/ReactGraph.ts.

Architecture​

Agent Response → Reflection Node → {completed: boolean, feedback: string}
                                           ↓
                              If false → feedback added as message → agent retries
                              If true  → task complete

Key Components​

  • Reflection Prompt: Lines 733-744 in ReactGraph.ts
  • State Update: Line 797 sets reflectionFeedback: { feedback, completed }
  • Feedback Loop: Lines 406-413 add feedback as message on next iteration if completed: false

Known Issue: Hallucination​

Problem: Reflection model used its own knowledge to judge extracted data instead of trusting the agent.

Example:

  • Agent correctly extracted "391,810" for Iceland population from Wikipedia
  • Reflection rejected: "does not match the population given on the page"
  • Result: Infinite retry loop → timeout

Root Cause: Model evaluated factual correctness, not task completion.

Fix​

Updated reflection prompt to evaluate task completion, not factual accuracy:

IMPORTANT: You are evaluating whether the agent COMPLETED the task, not whether the answer is factually correct.
- If the user asked for data from a webpage and the agent provided a specific value, trust that the agent extracted it correctly from the page.
- Do NOT reject responses based on your own knowledge of what the answer "should be" - the webpage may have different/updated data.
- Focus on: Did the agent attempt the task and provide a response in the requested format?

Debugging​

Log marker for reflection output:

🤖 Added reflection feedback as message:

Note: reflectionFeedback.feedback in ChatPage state shows empty BEFORE reflection runs. Check logs for actual feedback content.