AI-generated position review 18 min read Intermediate 1,570 words

Reflective Agents: Structured Self-Critique with Episodic Memory

A synthetic proposal combining structured self-critique with episodic memory so autonomous language agents catch and correct their own errors.

agents self-critique memory reasoning llms
Applications & prompting · 2023

Primary source: ReAct: Synergizing Reasoning and Acting in Language Models (arXiv:2210.03629)

Authors: Know Your ML AI Research

Date: June 2026

Type: AI-generated position paper

Status: Synthesis and proposal; no original empirical experiments are claimed.

Abstract

Autonomous language agents that reason, act, and learn over long horizons remain fragile. They repeat errors, fail to transfer lessons across episodes, and struggle to maintain coherent strategy when tasks exceed the context window of the underlying model. We propose Reflective Agents, a framework that unifies three recent lines of work: explicit reasoning traces from ReAct-style agents, verbal reinforcement learning from Reflexion, and structured episodic memory from memory-augmented systems such as MemGPT. The key contribution is a memory architecture in which each completed episode is compressed into a structured critique record containing a success verdict, a failure taxonomy, a repaired plan sketch, and a set of retrieval keys. At inference time, the agent retrieves relevant critiques before acting, conditions generation on them, and writes a new critique after each episode. We describe the architecture, outline training and inference procedures, and discuss how the design addresses failure modes that neither pure self-critique nor pure memory retrieval solve alone.

1. Introduction

Large language models can act as agents: given a task, they can plan, call tools, observe results, and continue until a goal is reached (Yao et al., 2022). Early agents were reactive, treating each step as an independent language-modeling decision. Subsequent work showed that asking the model to verbalize its reasoning before acting improves robustness (ReAct; Yao et al., 2022), and that asking the model to critique its own rollout and revise its plan can lead to large gains on multi-step tasks (Reflexion; Shinn et al., 2023).

Yet both approaches are fundamentally intra-episode: the critique is generated from the current trace and then forgotten. A second episode on a related task starts from scratch. This is wasteful. Humans and successful long-lived software systems do the opposite: they keep a durable record of mistakes and recoveries and consult it before making the same kind of decision again.

Recent memory systems for language models have begun to address durability. MemGPT (Packer et al., 2023) treats context as a scarce resource and moves information between context and external storage. Generative Agents (Park et al., 2023) maintain a memory stream of observations, reflections, and plans. These systems show that episodic memory can sustain coherent behavior over long simulations.

We argue that the next step is to make memory action-oriented: not just a stream of observations, but a collection of structured critiques that directly shape future behavior. Our proposal, Reflective Agents, stores each completed episode as a compact critique record. Before beginning a new task, the agent retrieves relevant critiques, uses them to bias planning, and after finishing, writes a new critique that future episodes can retrieve.

2.1 Reasoning and Acting

ReAct (Yao et al., 2022) interleaves reasoning traces with actions. The reasoning trace is not merely explanatory; it conditions the next action and allows the model to recover from earlier mistakes. Tree of Thoughts (Yao et al., 2023) extends this idea by maintaining multiple reasoning paths and explicitly evaluating them, turning problem solving into search. Both methods improve performance within an episode but discard the search tree afterward.

2.2 Self-Critique and Reflection

Reflexion (Shinn et al., 2023) introduces a self-reflective loop. After a failed episode, the model writes a textual critique that is fed back into the prompt for the next attempt. The critique typically summarizes what went wrong and what should be done differently. Reflexion demonstrates strong gains on programming and decision-making tasks, but the critique is ephemeral: it applies only to the next trial of the same task, not to related tasks in the future.

2.3 Memory-Augmented Language Models

MemGPT (Packer et al., 2023) organizes memory into tiers: limited context, larger working memory, and long-term archival memory. An operating-system-like control flow moves data between tiers. Generative Agents (Park et al., 2023) use a memory stream with periodic reflection to form higher-level summaries. These systems are impressive but focus on maintaining a coherent persona over time rather than on transferring actionable lessons across tasks.

3. Reflective Agents

3.1 Overview

A Reflective Agent consists of three modules:

Retrieve top-k critiques Plan conditioned on lessons Execute actor + tools Critique verdict · failure_mode recovered_plan Memory index c₁: tool_misuse c₂: planning_error c₃: context_loss c₄: premature_commit … + retrieval_keys Each completed episode becomes a structured critique record. Future episodes retrieve the most relevant critiques and condition their plan on them — improving with experience.
The Reflective Agents lifecycle: retrieve relevant critiques, plan and execute, critique the resulting trace, and store the critique for future episodes.
  1. Actor: executes actions given the current task, retrieved critiques, and a working plan.
  2. Critic: at episode end, evaluates the trace and produces a structured critique record.
  3. Memory Index: stores critique records and retrieves relevant ones for new tasks.

The lifecycle of an episode is:

  1. Retrieve: given the new task, fetch the top-k critique records from memory.
  2. Plan: generate a plan conditioned on the task and retrieved critiques.
  3. Execute: run the actor, which may call tools or reason step-by-step.
  4. Critique: at episode end, the critic produces a structured critique record.
  5. Store: write the critique record to the memory index.

3.2 Critique Record Structure

Each critique record is a JSON object with the following fields:

  • task_signature: a compact embedding or keyword summary of the task.
  • verdict: success, partial success, or failure.
  • failure_mode: a taxonomy label such as "premature_commitment", "tool_misuse", "planning_error", or "context_loss".
  • recovered_plan: a high-level sketch of how the task was or could have been solved.
  • retrieval_keys: terms or embeddings used to match future tasks.
  • episode_summary: a short natural-language summary for human inspection.

The structured format makes retrieval precise. A new task about database queries can retrieve critiques whose failure_mode is "tool_misuse" and whose task_signature is close in embedding space, even if the surface wording differs.

3.3 Retrieval and Conditioning

At the start of an episode, the agent embeds the task description and queries the memory index. Retrieved critiques are formatted into a "lessons learned" section of the prompt. We use a simple prefix:

> Before you begin, consider the following lessons from previous episodes:

> 1. [Critique summary 1]

> 2. [Critique summary 2]

Empirically, this biases the actor away from previously failed strategies and toward recovered plans. The effect is similar to in-context learning, but the examples are generated by the agent itself.

3.4 Critic Prompting

The critic receives the full episode trace and the final outcome. It is asked to produce the structured fields above. To reduce variance, we use a two-stage critic: first, a classifier assigns the failure_mode; second, a generator writes the recovered_plan and episode_summary conditioned on the classification.

4. Why the Combination Matters

Neither self-critique nor memory retrieval alone captures what we want.

  • Self-critique without memory loses lessons across episodes.
  • Memory without critique stores raw observations that may not contain actionable advice.
  • Critique records combine the specificity of reflection with the durability of memory.

The design also addresses context limits. A long memory stream cannot fit into a model's context, but a curated set of top-k critiques can. The agent therefore maintains a much longer effective horizon than the underlying model.

5. Discussion and Limitations

Reflective Agents inherit the limitations of their components. Retrieval quality depends on the embedding model and the density of the memory index. If the index is sparse, retrieval is unhelpful. If it is dense, the agent may overfit to past episodes and fail to adapt to novel situations.

Another risk is critique drift: as the agent writes its own critiques, errors in the critic can compound. Periodic human review or an external verifier may be necessary for high-stakes applications.

Finally, the framework does not specify how the actor is trained. It can be used with a frozen language model and prompt engineering, or it can be combined with fine-tuning on critique-conditioned trajectories. We leave systematic training recipes to future work.

6. Conclusion

We proposed Reflective Agents, a framework that stores structured self-critiques in episodic memory and retrieves them to condition future behavior. The approach unifies reasoning, reflection, and memory in a single lifecycle and offers a path toward language agents that improve with experience rather than repeating mistakes.

References

  • Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
  • Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366. https://arxiv.org/abs/2303.11366
  • Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601. https://arxiv.org/abs/2305.10601
  • Packer, C., Fang, V., Patil, S. G., Lin, K., Wooders, S., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560. https://arxiv.org/abs/2310.08560
  • Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023). Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442. https://arxiv.org/abs/2304.03442

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-reflective-agents-self-critique
  title        = {Reflective Agents: Structured Self-Critique with Episodic Memory},
  author       = {{Know Your ML}},
  year         = {2023},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=reflective-agents-self-critique}},
  note         = {AI-generated review of arXiv:2210.03629. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook