Reflective Agents: Structured Self-Critique with Episodic Memory
A synthetic proposal combining structured self-critique with episodic memory so autonomous language agents catch and correct their own errors.
Primary source: ReAct: Synergizing Reasoning and Acting in Language Models (arXiv:2210.03629)
Authors: Know Your ML AI Research
Date: June 2026
Type: AI-generated position paper
Status: Synthesis and proposal; no original empirical experiments are claimed.
Abstract
Autonomous language agents that reason, act, and learn over long horizons remain fragile. They repeat errors, fail to transfer lessons across episodes, and struggle to maintain coherent strategy when tasks exceed the context window of the underlying model. We propose Reflective Agents, a framework that unifies three recent lines of work: explicit reasoning traces from ReAct-style agents, verbal reinforcement learning from Reflexion, and structured episodic memory from memory-augmented systems such as MemGPT. The key contribution is a memory architecture in which each completed episode is compressed into a structured critique record containing a success verdict, a failure taxonomy, a repaired plan sketch, and a set of retrieval keys. At inference time, the agent retrieves relevant critiques before acting, conditions generation on them, and writes a new critique after each episode. We describe the architecture, outline training and inference procedures, and discuss how the design addresses failure modes that neither pure self-critique nor pure memory retrieval solve alone.
1. Introduction
Large language models can act as agents: given a task, they can plan, call tools, observe results, and continue until a goal is reached (Yao et al., 2022). Early agents were reactive, treating each step as an independent language-modeling decision. Subsequent work showed that asking the model to verbalize its reasoning before acting improves robustness (ReAct; Yao et al., 2022), and that asking the model to critique its own rollout and revise its plan can lead to large gains on multi-step tasks (Reflexion; Shinn et al., 2023).
Yet both approaches are fundamentally intra-episode: the critique is generated from the current trace and then forgotten. A second episode on a related task starts from scratch. This is wasteful. Humans and successful long-lived software systems do the opposite: they keep a durable record of mistakes and recoveries and consult it before making the same kind of decision again.
Recent memory systems for language models have begun to address durability. MemGPT (Packer et al., 2023) treats context as a scarce resource and moves information between context and external storage. Generative Agents (Park et al., 2023) maintain a memory stream of observations, reflections, and plans. These systems show that episodic memory can sustain coherent behavior over long simulations.
We argue that the next step is to make memory action-oriented: not just a stream of observations, but a collection of structured critiques that directly shape future behavior. Our proposal, Reflective Agents, stores each completed episode as a compact critique record. Before beginning a new task, the agent retrieves relevant critiques, uses them to bias planning, and after finishing, writes a new critique that future episodes can retrieve.
2. Related Work
2.1 Reasoning and Acting
ReAct (Yao et al., 2022) interleaves reasoning traces with actions. The reasoning trace is not merely explanatory; it conditions the next action and allows the model to recover from earlier mistakes. Tree of Thoughts (Yao et al., 2023) extends this idea by maintaining multiple reasoning paths and explicitly evaluating them, turning problem solving into search. Both methods improve performance within an episode but discard the search tree afterward.
2.2 Self-Critique and Reflection
Reflexion (Shinn et al., 2023) introduces a self-reflective loop. After a failed episode, the model writes a textual critique that is fed back into the prompt for the next attempt. The critique typically summarizes what went wrong and what should be done differently. Reflexion demonstrates strong gains on programming and decision-making tasks, but the critique is ephemeral: it applies only to the next trial of the same task, not to related tasks in the future.
2.3 Memory-Augmented Language Models
MemGPT (Packer et al., 2023) organizes memory into tiers: limited context, larger working memory, and long-term archival memory. An operating-system-like control flow moves data between tiers. Generative Agents (Park et al., 2023) use a memory stream with periodic reflection to form higher-level summaries. These systems are impressive but focus on maintaining a coherent persona over time rather than on transferring actionable lessons across tasks.
3. Reflective Agents
3.1 Overview
A Reflective Agent consists of three modules:
- Actor: executes actions given the current task, retrieved critiques, and a working plan.
- Critic: at episode end, evaluates the trace and produces a structured critique record.
- Memory Index: stores critique records and retrieves relevant ones for new tasks.
The lifecycle of an episode is:
- Retrieve: given the new task, fetch the top-k critique records from memory.
- Plan: generate a plan conditioned on the task and retrieved critiques.
- Execute: run the actor, which may call tools or reason step-by-step.
- Critique: at episode end, the critic produces a structured critique record.
- Store: write the critique record to the memory index.
3.2 Critique Record Structure
Each critique record is a JSON object with the following fields:
task_signature: a compact embedding or keyword summary of the task.verdict: success, partial success, or failure.failure_mode: a taxonomy label such as "premature_commitment", "tool_misuse", "planning_error", or "context_loss".recovered_plan: a high-level sketch of how the task was or could have been solved.retrieval_keys: terms or embeddings used to match future tasks.episode_summary: a short natural-language summary for human inspection.
The structured format makes retrieval precise. A new task about database queries can retrieve critiques whose failure_mode is "tool_misuse" and whose task_signature is close in embedding space, even if the surface wording differs.
3.3 Retrieval and Conditioning
At the start of an episode, the agent embeds the task description and queries the memory index. Retrieved critiques are formatted into a "lessons learned" section of the prompt. We use a simple prefix:
> Before you begin, consider the following lessons from previous episodes:
> 1. [Critique summary 1]
> 2. [Critique summary 2]
Empirically, this biases the actor away from previously failed strategies and toward recovered plans. The effect is similar to in-context learning, but the examples are generated by the agent itself.
3.4 Critic Prompting
The critic receives the full episode trace and the final outcome. It is asked to produce the structured fields above. To reduce variance, we use a two-stage critic: first, a classifier assigns the failure_mode; second, a generator writes the recovered_plan and episode_summary conditioned on the classification.
4. Why the Combination Matters
Neither self-critique nor memory retrieval alone captures what we want.
- Self-critique without memory loses lessons across episodes.
- Memory without critique stores raw observations that may not contain actionable advice.
- Critique records combine the specificity of reflection with the durability of memory.
The design also addresses context limits. A long memory stream cannot fit into a model's context, but a curated set of top-k critiques can. The agent therefore maintains a much longer effective horizon than the underlying model.
5. Discussion and Limitations
Reflective Agents inherit the limitations of their components. Retrieval quality depends on the embedding model and the density of the memory index. If the index is sparse, retrieval is unhelpful. If it is dense, the agent may overfit to past episodes and fail to adapt to novel situations.
Another risk is critique drift: as the agent writes its own critiques, errors in the critic can compound. Periodic human review or an external verifier may be necessary for high-stakes applications.
Finally, the framework does not specify how the actor is trained. It can be used with a frozen language model and prompt engineering, or it can be combined with fine-tuning on critique-conditioned trajectories. We leave systematic training recipes to future work.
6. Conclusion
We proposed Reflective Agents, a framework that stores structured self-critiques in episodic memory and retrieves them to condition future behavior. The approach unifies reasoning, reflection, and memory in a single lifecycle and offers a path toward language agents that improve with experience rather than repeating mistakes.
References
- Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
- Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366. https://arxiv.org/abs/2303.11366
- Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601. https://arxiv.org/abs/2305.10601
- Packer, C., Fang, V., Patil, S. G., Lin, K., Wooders, S., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560. https://arxiv.org/abs/2310.08560
- Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023). Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442. https://arxiv.org/abs/2304.03442
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-reflective-agents-self-critique
title = {Reflective Agents: Structured Self-Critique with Episodic Memory},
author = {{Know Your ML}},
year = {2023},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=reflective-agents-self-critique}},
note = {AI-generated review of arXiv:2210.03629. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- ReAct: Synergizing Reasoning and Acting in Language Models (arXiv:2210.03629) — https://arxiv.org/abs/2210.03629
- Reflexion: Language Agents with Verbal Reinforcement Learning (arXiv:2303.11366) — https://arxiv.org/abs/2303.11366
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models (arXiv:2305.10601) — https://arxiv.org/abs/2305.10601
- MemGPT: Towards LLMs as Operating Systems (arXiv:2310.08560) — https://arxiv.org/abs/2310.08560
- Generative Agents: Interactive Simulacra of Human Behavior (arXiv:2304.03442) — https://arxiv.org/abs/2304.03442
Part of
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?