Single LLMs Hit a Wall. Here’s What Comes Next

Over the past few years, the pace of advancement in Large Language Models (LLMs) has been nothing short of phenomenal. With the emergence of models like GPT-4, Claude, and Gemini, expectations ran high that "perhaps a single model could solve everything by itself." However, when you actually try to solve complex business problems in a production environment, you quickly realize that a single LLM call is far from sufficient.

In this article, we will view the technological journey toward AGI as a single spectrum, extending from the simplest form of LLM to Multi-Agent Systems, and structurally analyze why the architecture of each stage had to evolve into the next. Furthermore, by examining the Multi-Agent Scaling Law revealed in recent research, we will discuss its implications for B2B multi-agent platforms such as Agentria.

Figure 1. The Agentic Progression spectrum toward AGI. Moving from left to right, the system becomes more structured, more autonomous, and possesses greater execution capabilities. However, the right side is not unconditionally better; the appropriate position may vary depending on task complexity, reliability, and cost.

Starting with a Single Prompt

This is the most basic structure. When a user input is received, it is passed to the LLM along with prompt engineering techniques such as Chain-of-Thought, and external knowledge is injected through RAG (Retrieval-Augmented Generation) if necessary, before generating the output.

Figure 2. Single Prompting structure. User input is passed to the LLM along with a CoT prompt, and relevant information is retrieved from Long-term Memory via RAG to generate the output.

Even within this structure, the branches of research are rich. The Chain-of-Thought technique published by Wei et al. in 2022 elevated accuracy on the GSM8K math benchmark from 17% to 78% [1]. RAG has effectively become a standard component of production AI systems since Meta's Lewis et al. paper in 2020, and advanced techniques like HyDE, Query Rewriting, and Re-ranking continue to emerge [2]. On the fine-tuning side, parameter-efficient techniques such as LoRA and QLoRA have significantly lowered the barrier to entry.

However, the limitations of this structure are clear. If you ask, "Analyze our company's sales data and propose a strategy for next year," most LLMs will offer generalities like "Data-driven decision making is important. Analyze customer segments and identify market trends." Complex problems cannot be solved with a single LLM call, the physical limitations of the context window (the Lost-in-the-middle phenomenon) still remain, and since LLMs are inherently next-token prediction models, structural thinking—like breaking down complex tasks into sub-tasks and reintegrating them—is fundamentally difficult.

Prompt Chaining: The Emergence of Pipelines

"Then, wouldn't it work if we broke it down into multiple stages?" This natural idea is Prompt Chaining. The output of the first LLM call is connected as the input for the second call, and verification stages are inserted in between. Indeed, many production systems operate in this manner, and it is highly useful for validating hallucinations or refining outputs.

Figure 3. Prompt Chaining structure. A task is broken down into multiple stages, where the output of each LLM call is passed as input to the next stage. Knowledge injection via RAG is possible in between.

Yet, this approach also has structural limitations. Because pipelines only flow in a pre-determined order, it is difficult to respond flexibly to unexpected situations. If one chain produces an unusual output, that error propagates to the next stage, potentially ruining the final result. Moreover, handling dynamic branching, such as "if this condition, go to path A, if that condition, go to path B," is limited. Consequently, there arose a need for a structure capable of more flexible and dynamic decision-making, which naturally led to the concept of Agents.

Single Agent: The Emergence of AI that Acts

An agent does not merely answer questions; it is a system with roles and goals that actually interacts with its environment. In the ReAct (Reasoning + Acting) paradigm, an agent repeats a loop of thinking, acting, observing results, and thinking again [3]. In the Plan & Execute pattern, it first creates a plan and then executes each step sequentially. What is crucial at this stage is that the agent has Long-term Memory, uses external Tools, calls APIs, and can execute code. However, the limitations of a single agent fundamentally stem from the autoregressive nature of LLMs.

Figure 4. Single Agent (ReAct, Plan & Execute) structure. The agent interacts with Long-term Memory, Tools, and the External Environment, performing actual actions beyond simple responses.

Structural Weaknesses Imposed by the Autoregressive Nature of LLMs

An LLM is not a model that completes a sentence all at once. It is an autoregressive model that decomposes the joint probability p(x₁:T) using the chain rule, iteratively calculating the conditional probability p(xₜ|x<ₜ) of the next token based on the prefix up to that point at each step. This structure creates three fundamental problems.

First is Error Amplification. Once an incorrectly generated token enters the prefix, it continues to prompt subsequent generation based on that error. Bengio et al. already pointed out this issue at NeurIPS 2015, which means that early mistakes can quickly amplify [4].

Second is the issue of Self-Rationalization. Even if a verification stage is attached to a single agent, if the structure involves the same model looking at the same context to correct its own answer, it easily becomes self-rationalization rather than independent oversight. According to research published by Huang et al. at ICLR 2024, there are clear limits to the utility of an LLM's intrinsic self-correction [5].

Third is the lack of exploration of alternatives. A single agent follows a single generation trajectory, and by the nature of decoding—which maintains only limited candidates, like greedy search or beam search—it is structurally difficult to undo early choices to explore and compare other hypotheses in parallel [4].

These three limitations naturally demand a transition to a multi-agent structure capable of role separation, independent verification, and parallel exploration.

Multi-Agent System: A Collaborative Group of Specialists

A multi-agent system is a structure where multiple specialized agents collaborate to solve problems. At the top is a main agent (Orchestrator) that distributes tasks to sub-agents through Plan & Execute. Each sub-agent has its own prompt, its own Long-term Memory, and its own Tools. This allows each agent to focus on its role, keeping prompts concise and enhancing expertise.

Figure 5. Multi-Agent structure. The main Agent distributes tasks to sub-agents through Plan & Execute, and each sub-agent maintains its own Long-term Memory, Tools, and External Environment.

Of course, multi-agent systems have their own unique challenges. Representative issues include communication overhead between agents, orchestration complexity, difficulty in debugging, and increased API call costs. However, despite these challenges, recent research experimentally demonstrates that increasing the number of agents leads to substantial performance gains.

$$L(N, D, C) \propto \left(\frac{1}{N^\alpha}, \frac{1}{D^\beta}, \frac{1}{C^\gamma}\right)$$

Figure 6. Neural Scaling Law. System performance (error) improves in a power-law fashion according to the number of model parameters (N), dataset size (D), and compute used in training (C) [6].

If so, can increasing the number of agents and their collaborative structure, rather than scaling up the model itself, yield a similar effect?

A paper by Qian et al. published in ICLR 2025 offers the first systematic answer to this question [7]. The research team built a multi-agent collaboration network called MacNet based on DAG (Directed Acyclic Graph) and conducted experiments by exponentially increasing the number of agents from 2⁰ (=1) to 2⁶ (=64, and over 1,000 in a mesh network).

Figure 7. Multi-Agent Scaling performance in various topologies. As the number of agents (Scale, x-axis) increases, quality (Quality, y-axis) exhibits a logistic growth pattern. The shape of the performance curve varies by topology, and irregular topologies consistently outperformed regular ones [7].

The results were highly impressive. As the number of agents increased, system performance followed a logistic growth pattern. It took the shape of an S-curve, starting with slow growth, passing through a rapid improvement phase, and finally reaching a saturation point. One particularly notable finding was that this collaborative emergence occurred earlier than traditional neural emergence. The research team interpreted this as being due to agent scaling promoting multi-dimensional considerations during interactive reflection and refinement processes [7].

Another important finding is the impact of topology. Irregular topologies consistently outperformed regular ones. A Full Connection approach, which simply links all agents together, offers the highest number of interactions but can actually degrade performance or lower efficiency due to information redundancy and noise. Instead of blindly increasing the number of interactions, building a structure to communicate deeply with appropriate neighboring agents is far more effective for performance enhancement [7].

Meanwhile, in 2024, scaling along a completely different axis from scaling up models was discovered. Instead of making models larger, making an already built model think longer when answering increases its performance. OpenAI's o1 model is a prime example. Instead of outputting an answer immediately, o1 generates a long internal chain-of-thought, explores multiple paths, reviews its own answers, and then delivers the final response. Increasing this "thinking time" consistently improves performance on mathematical reasoning benchmarks such as AIME [8].

This is known as Test-Time Compute Scaling. In contrast to dedicating more compute during training (Pre-training Scaling), it devotes more compute during inference. By analogy, it is like either (1) studying longer to do well on an exam, or (2) being given more time during the exam itself. Amazingly, under certain conditions, giving a smaller model more "exam time" can yield better results than calling a larger model once [8].

Multi-agent systems represent the most sophisticated form of this Test-Time Compute Scaling. While asking the same model to answer multiple times and choosing the best one—such as Best-of-N sampling—is a type of test-time compute, multi-agent systems go a step further. Each agent has different expertise, views the problem from a different perspective, and verifies independently; the reasoning computations generated in this process take the form of structured collaboration rather than simple repetition. The Collaborative Scaling Law introduce earlier [7] experimentally proves that this "structured reasoning" is far more effective than simple repetition.

What This Means for Platform Design

Translating the above findings from the perspective of a B2B multi-agent platform like Agentria yields several design principles:

• Topology is more important than the number of agents. When customers design agents, how they connect them, rather than simply creating a large number of agents, is decisive. It is necessary for the platform to support topology optimization or recommend proven structures.

• We must recognize the saturation point. The fact that the Collaborative Scaling Law follows a logistic pattern means that, past a certain point, adding more agents yields marginal performance improvements. A mechanism is needed to detect this saturation point and provide feedback to the user.

Conclusion

The journey from single prompts to multi-agent systems is not a simple technological evolution. It is a design effort to overcome at the system level the structural limitations stemming from the fundamental nature of LLMs as probabilistic models.

In summary, there are three axes for boosting AI performance: scaling up model parameters (Model Scaling), allowing longer thinking during inference (Test-time Compute Scaling), and scaling up the number of agents and their collaborative structure (Agent Scaling). The Collaborative Scaling Law presented at ICLR 2025 [7] shows that the third axis is not mere intuition but an experimentally verified principle. Moreover, multi-agent systems sit precisely at the intersection of the second and third axes—because they deploy inference-time compute in the form of structured collaboration.

In the next article, we will cover the concrete collaborative strategies (Cooperation, Competition, Coopetition), topologies, communication protocols, and MAS evaluation methodologies required when actually designing multi-agent systems.

References

[1] Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” NeurIPS 2022.

[2] Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS 2020.

[3] Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” ICLR 2023.

[4] Bengio et al., “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks,” NeurIPS 2015.

[5] Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet,” ICLR 2024.

[6] Kaplan et al., “Scaling Laws for Neural Language Models,” OpenAI, 2020.

[7] Qian et al., “Scaling Large Language Model-based Multi-Agent Collaboration,” ICLR 2025.

[8] Snell et al., “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters,” ICLR 2025.