Design Patterns for Compound AI System

• 15 May, 2024
• By: Sudhir
• 4 Comments

Design Patterns for Compound AI Systems

How to build configurable flows and compound AI systems using open source tools.

What are compound AI systems?

Recently, researchers at Berkeley wrote an article — The Shift from Models to Compound AI Systems, where they compiled the progress on LLM apps and stressed on the evolving nature of complex pipelines of components working together to build intelligent systems, rather than a closed singular model. Although, the resulting system might use the same underlying model (like GPT4) but the prompting and context warrants it to be called different components.

Some common deployment patterns of these systems:

• RAG (retrieval and understanding is key) — with access to Thought generation, Reasoning and contextual data, these systems self-reflect and try to understand a user’s query in advanced ways, before responding back with an answer, ideal set up is an Agent Assist system. When combined with user facing models / systems like a Dialogue model, a RAG system could be a part of a Conversational AI or a CoPilot system.

• Multi — Agent Problem Solvers (collaborative role playing is key) — these systems rely on collaborative and automated build up of solutions based on outputs of agents being fed into each other with a well defined role and purpose. Each agent has access to it’s own set of tools and can assume a very specific role while reasoning and planning it’s actions.

• Conversational AI (dialogue is key) automation softwares like customer service agents, which interact with humans within an app / ecosystem and execute repeatable tasks based on inputs and validations from humans. The most important aspect here is conversational memory and dialogue generation with the feeling of having a conversation with a human. The Dialogue Model can have access to an underlying RAG system, or a Multi-Agent problem solver.

• CoPilots (a human in the loop interface is key) — with access to tools, data, reasoning and planning capabilities, and specialized profiles, these systems can independently interact with a human while solving problems with closed solutions. The key differentiator for becoming a CoPilot is understanding of the environment in which the human works. For e.g. MetaGPT Researcher: Search Web and Write Reports, A measured take on Devin, Let’s build something with CrewAI, Autogen Studio.

Components of these systems and how they interact with each other to build complex systems

Compound AI systems are usually deployed using interconnected “modules” which rely on each other to do certain tasks and chain together to execute design patterns.

Here, modules are referred to those individual components in the system which do a well defined task with or without the help of an underlying system like Search Engine, LLM etc. Some common modules include generator, retriever, ranker, classifier etc, traditionally called tasks in NLP. These are conceptual abstractions which are domain specific (e.g. NLP modules might have different abstracted modules than Computer Vision or Recommendation systems, but they might all rely on the same underlying model serving or search provider).

LLM based Autonomous Agents — a key module in compound AI systems : One form of a module is an Autonomous Agent, which can reason and plan in an autonomous manner with the help of an LLM. An autonomous agent might depend on a list of sub-modules to decide how to reason and plan actions while interacting with the environment.

Key capabilities or skills of a Autonomous Agent module:

• Reasoning — logical methods to solve a problem by making observations, generating hypothesis and validating based on data etc.

• Thought — an application of reasoning and generation of a coherent cause-effect relationship.

• Chain of thought — a sequence of thoughts which break a solution into a sequence of logically connected reasonings.

• Planning — decision making skills to do sub-goal formation and build a path from a current state to a future state. This usually benefits from having access to the underlying environment to take action and learn from. Read more about LLM planning here.

• Tools — sub modules which allows an agent to interact with the external world based on instructions from the agent.

• Action — as the name suggests, it is a decisive step which an agent takes in the pursuit of a goal, through a plan. A tool might be invoked to trigger an action. Actions are made in the environment.

• Environment — the external world in which the actions and rewards for the agent originate, like a custom app (like Microsoft Word, Coding Environment) or a game, or a simulation etc.