Glossary

Term

Description

Large Language Model (LLM)

A Large Language Model is a deep learning model for natural language processing. It is trained on massive datasets, often terabytes in size, and has billions or trillions of parameters. An LLM can understand, reason, and generate natural language text, and even audio, images, and video.

Agent

An agent is a computational entity that can perceive its environment and act upon it using executors. It has features such as autonomy, reactivity, social ability, and proactivity. It uses a large language model as its core compute engine to implement a closed-loop workflow of Perception, Planning, and Action.

Token

A token is the basic unit for processing text in a large language model. It can be a word, subword, punctuation mark, or character, depending on the tokenizer used. Both the input and output of an LLM are processed as sequences of tokens. A model's context length, such as 8K or 32K, refers to the maximum number of tokens it supports.

Memory

Memory refers to the mechanism for storing and managing conversation history, user preferences, task status, or external knowledge. It enables the model to have contextual awareness across turns or sessions. Memory can be divided into:

• Short-term Memory: Usually refers to the context of the current conversation, which is the conversation history. It is purged when the session ends.

• Long-term Memory: Persists user information through a vector database, knowledge graph, or structured storage. This information can be retrieved and used in subsequent interactions.

Chain of Thought

Tree of Thoughts extends Chain of Thoughts by exploring multiple reasoning possibilities at each step of a task. It first breaks down a problem into multiple thought steps and generates several ideas at each step, creating a tree-like structure. The search process can be a breadth-first search (BFS) or a depth-first search (DFS).

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation is a model that combines retrieval and generation techniques. It generates answers by referencing information from an external knowledge base. This approach offers strong interpretability and customization. RAG is suitable for many natural language processing tasks, such as Q&A systems, document generation, and intelligent assistants. The benefits of RAG models include versatility, real-time knowledge updates, and more efficient and accurate information services through end-to-end evaluation.

Knowledge Base

A knowledge base is a collection of structured or unstructured data that provides external factual information to a model. It is typically used in a RAG architecture. The content of a knowledge base can come from enterprise documents, FAQs, databases, and web pages. When a user asks a question, the system first retrieves relevant snippets from the knowledge base. It then inputs this information along with the user's query to the LLM to generate a more accurate and traceable answer.

Query Rewriting

Query rewriting is a technique used in information retrieval or RAG systems to semantically optimize or restructure a user's original query. This improves the effectiveness of subsequent retrieval or generation. Common forms include query expansion, query refinement, and historical context fusion.

Intent Recognition

Intent recognition is one of the core tasks of natural language understanding (NLU). It aims to automatically determine the goal or purpose, the intent, behind a user's input statement. In LLM applications, intent recognition can be used to:

• Route user requests to the appropriate business modules, such as skill distribution in a customer service bot.

• Trigger specific tool calls or workflows.

• Improve the retrieval relevance of a RAG system by adjusting the retrieval policy based on the intent.

Observability

Observability is the ability to monitor, trace, and diagnose the internal running state of an LLM application. Because of the black box nature of LLMs, their output can be difficult to debug. Observability tools are needed to record data from key stages, including prompts, model outputs, retrieval results, tool calls, latency, token consumption, and costs.

LangChain

LangChain is a powerful framework designed to help developers build end-to-end applications using language models. It provides a set of tools, components, and interfaces that simplify the process of creating applications powered by LLMs and chat models.

Langfuse

Langfuse is an open source observability platform for LLM applications. It is used to trace, evaluate, and debug applications based on large language models, such as agents and RAG systems.

Dify

Dify is an open source LLM application development platform. It provides a visual interface and an API to support the rapid construction of AI applications based on large language models. It integrates features such as prompt orchestration, knowledge base management, RAG, and agent orchestration. It also supports a variety of mainstream large models.

RagFlow

RagFlow is an open source framework or platform that focuses on enterprise-level RAG applications. It emphasizes deep parsing, semantic text segmentation, and precise retrieval for complex documents, such as PDFs, Word files, and spreadsheets.