A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning

Current surveys lack a coherent and unified starting point for discussing the three paradigms due to two main limitations: 1) Focusing on a specific paradigm or domain: For example, Huang et al. (2024) explore frameworks in the planning paradigm. Hu et al. (2024); Gallotta et al. (2024) examine LLM-based agents in the context of games. In contrast, these paradigms represent general principles and mental models that govern the development and behavior of these agents. Hence, their frameworks should be discussed in a task-agnostic way. To achieve this, we extract universal workflows. Unless otherwise specified, in this survey, the term “framework” refers to a complete workflow for specific low-level tasks, while “workflow” refers to the task-agnostic process derived from one or more frameworks. 2) Lack of a unified basis for comparison: Although Wang et al. (2024) cover all three paradigms, they do not analyze the implementation of algorithmic frameworks in a unified basis. In contrast, we summarize task-agnostic LLM-profiled roles as the foundation for the development of algorithmic frameworks across different paradigms. Notably, Wang et al. (2024) also discuss LLM profiling, but their focus is on personas, which are not relevant to the general roles that underpin these frameworks.

This survey offers the following contributions.

1) Reviewing the breadth of environments the agents are applicable to (§2): Before investigating LLM-based agents, it is essential to define the scope of universality by summarizing the environments in which these agents operate. We categorize the evaluated environments and tasks from the original manuscripts of the reviewed frameworks into two main types: decision-making environments and natural language interaction environments, the latter framed from traditional Natural Language Processing (NLP) tasks for agent-based setups.

2) Summarizing universal LLM-profiled roles and workflows (§3 and §4): Workflow designs under planning and feedback-learning paradigms are often based on complex algorithms, such as depth- or breadth-first search (Yao et al., 2023a) and Monte Carlo Tree Search (Hao et al., 2023) for planning, as well as Reinforcement Learning (RL) for feedback learning (Shinn et al., 2023). The complexity increases when these algorithms are adapted to different environments. We aim to transcend the boundaries of these complex workflows by exploring those that can be universal across all types of environments. Additionally, we also investigate the workflows for tool use. Such exploration is based on three universal LLM-profile roles.

3) Highlighting essential perspectives on LMPRs, workflows, and frameworks (§5 and §6): We compare the implementations of LMPRs across various workflow types, emphasize workflows across different paradigms, and present alternative viewpoints to our formalization to prevent confusion. Additionally, we critically discuss three limitations of current frameworks and systematically identify potential workflow designs in future research.

$\text{glm}_{\text{policy}}$ is designed to generate decisions, which could be an action or a series of actions (plans) for execution in external environments or planning. In contrast to typical RL policy models, which learn to maximize cumulative rewards through trial and error, LLM-profiled policy models, denoted as $\text{glm}_{\text{policy}}$, utilize pre-trained knowledge and commonsense derived from extensive textual data. We distinguish between two types of $\text{glm}_{\text{policy}}$: an actor $\text{glm}_{\text{actor}}$ directly maps a state to an action, whereas a planner $\text{glm}_{\text{planner}}$ generates a sequence of actions from a given state.

$\text{glm}_{\text{eval}}$ provides feedback crucial for different workflows. During planning, it evaluates each step of actions or the resulting states (Hao et al., 2023; Yao et al., 2023a), and during feedback learning, they revise the entire decisions (Shinn et al., 2023; Wang et al., 2023b). Further details are provided in the next subsection.

They predict or describe changes to the environment. Generally, dynamic models form part of a comprehensive world model by predicting the next state $s^{\prime}$ from the current state $s$ and action $a$. While typical RL uses the probability distribution $p(s^{\prime}\mid s,a)$ to model potential next states, LLM-based dynamic models directly predict the next state $s^{\prime}=\text{glm}_{\text{dynamic}}(s,a)$.

In the previous subsection, we categorize the original works proposing Chain-of-Thought (CoT) prompting, including zero-shot CoT (Wei et al., 2022) and CoT with few-shot demonstrations (Kojima et al., 2022), under the base workflow since the original work solves task directly via the base workflow. However, these methods themselves can be generalized to different types of LMPRs and workflows, as shown in Appendix B. Some points of the specific use in agents should be highlighted: 1) For planner profiling, zero-shot CoT implementations often fail to produce long-horizon plans (Wang et al., 2023b). 2) To actor profiling under autonomous tool-use workflows, particularly with reasoning-acting strategies, the tool definitions are required to be included in the prompt; and few-shot demonstrations are important to give a clue of generation formats for LLMs including the indication of when to stop.

Both the base workflow using $\text{glm}_{\text{planner}}$ and search workflows generate a sequence of actions (i.e., a plan). However, they differ fundamentally in how the plans are generated and used. 1) Greedy Generation vs. Exploration: The base workflow leverages $\text{glm}_{\text{planner}}$ to greedily generate a static plan in a single inference step. This approach often struggles with long-horizon plans for complex tasks (Sun et al., 2023). In contrast, search workflows explore multiple potential solutions and support backtracking, allowing for more robust exploration of options. 2) Plan Execution: The plans generated by $\text{glm}_{\text{planner}}$, BFS, DFS, and Beam Search are intended for full execution. However, this can lead to unexecutable actions, especially in the stochastic decision-making environments, if prior actions $(a_{1},...,a_{t-1})$ leads to a state where the next action $a_{t}$ is invalid (e.g., ‘standing in front of the fridge‘ but needing to ‘open the microwave‘). This may cause abrupt interruption in the agent performance. In contrast, in the simulated-absed search via MCTS, only the action at the root node is executed in the actual environment. Subsequent simulated states and actions are discarded, though some implementations may partially retain these states to avoid recomputation. This search process repeats after every action taken, continuously recalculating the best action for the updated state.

The $\text{glm}_{\text{actor}}$ performs one of three types of actions, depending on the workflow: 1) In base and feedback-learning workflows, $\text{glm}_{\text{actor}}$ consistently generates task-relevant actions to be executed within the task environment (executable task actions). 2) In a search workflow, actions generated by $\text{glm}_{\text{actor}}$ are used for planning algorithms rather than direct execution (planning task actions). 3) In autonomous tool-use workflows, $\text{glm}_{\text{actor}}$ is responsible for generating tool-related actions (tool actions).

In feedback-learning workflows, the generation is passed to $\text{glm}_{\text{policy}}$ for learning, while in search workflows, it is used for planning. This fundamental difference (learning vs. planning) leads to the following distinctions: 1) Feedback Use: As shown in Figure 1, for learning, the output is generated for $\text{glm}_{\text{policy}}$ to revise and regenerate the entire decision. In search workflows, the output is used to construct a search tree/graph for action selection and further expansion. 2) Feedback Forms: These components—$\text{glm}_{\text{policy}}$ and serch trees/graphs—require different forms of feedback. $\text{glm}_{\text{policy}}$ processes free-form text, while MCTS utilizes continuous values as rewards, and trees/graphs in BFS, DFS, or beam search rely on discrete values for node selection. To meet these varying needs, distinct task formulations are employed, as summarized in Table 3. a) Generating free-form text: $\text{glm}_{\text{eval}}$ is prompted to reflect on previous states and actions, generating reflective text as part of the $\text{glm}_{\text{policy}}$ prompt in feedback-learning workflows (Shinn et al., 2023; Gou et al., 2024). b) Binary/multiclass classification: $\text{glm}_{\text{eval}}$ is prompted with specific constraints to generate discrete output tokens, typically “no” or “yes.” The tokens are converted into scalar values (e.g., 0/1) for use as rewards in MCTS simulations (Hao et al., 2023), or to guide decision-making during tree traversal (Yao et al., 2023a). c) Binary classification with scalar values: This approach differs from the previous one by employing token logits generated by LLMs as scalar feedback values. For example, the probability of a “yes” response is computed as:

where $l_{\text{``yes''}}$ and $l_{\text{``no''}}$ are the logits for “yes” and “no” tokens, respectively. ⁴⁴4Note that such implementations of $\text{glm}_{\text{eval}}$ are less common due to their inaccessibility of state-of-the-art black-box LLMs. These scalar values can then be used as rewards in MCTS. d) Multi-choice QA: This formulation is used in scenarios that require selecting from multiple choices, such as choosing from top-N possible actions in traversal-based search workflows (Yao et al., 2023a).

ReAct (Yao et al., 2023b) tries to unify the autonomous tool-use workflow and the base workflow via the reasoning-acting strategy. In this scenario, they implicitly unify tools as a part of task environments, and tool actions and task actions are unified to similar formats. However, the two workflows are still distinct: 1) specifying and comparing different tool-use workflows in a fine-grained manner, 2) distinguishing the two can avoid an illusion that it is universally implementable workflow. Specifically, for the latter, the sequence in which reasoning and action outputs alternate is task-dependent. For QA tasks, the generations of reasoning steps and tool actions are fixed, with alternating prompts for thinking and acting. ⁵⁵5Reasoning-Acting Strategy (QA tasks): See an example prompt in Table 7. In contrast, for embodied tasks, the decision whether to proceed with thinking or acting in the next step is autonomously determined by $\text{glm}_{\text{policy}}$ ⁶⁶6Reasoning-Acting Strategy (embodied tasks): See an example prompt in Table 5.

Another future direction is the development of universal tool use. Although tool use is a universally applicable paradigm, current research tends to focus on using tools for specific tasks such as NLIE-QA or specialized purposes—either enabling $\text{glm}_{\text{policy}}$ to retrieve new information or enabling $\text{glm}_{\text{eval}}$ for validation.

The formulation can help develop a more universally used workflow, e.g., Reflexion (Shinn et al., 2023). However, some deliberate design should be reconsidered for future work, such as the use of ground truths as external feedback in Remark 3. Another questionable setting is deliberate multi-step NLIEs, e.g., what if a question is simple enough to be answered without the need to be decomposed into subquestions.

Our systematic review reveals potential directions for devising new workflows by intertwining existing ones. These workflows can be combined within a single paradigm—for instance, integrating different feedback sources or blending validation-based tool use with autonomous tool use. There is even greater potential for combining workflows across paradigms, such as incorporating optional feedback sources into non-validation tool-use workflows. In fact, validation-style workflows (Gou et al., 2024; Jiang et al., 2023) inherently merge elements of both the feedback-learning and tool-use paradigms.