Multimodal RLHF annotation

Background information

Documents recently published by OpenAI show that ChatGPT outperforms natural language processing (NLP) models, such as Bidirectional Encoder Representations from Transformers (BERT), in conversational scenarios. Its core optimization mechanism is Reinforcement Learning from Human Feedback (RLHF). This topic analyzes the application of RLHF in ChatGPT and the importance of annotation from the perspective of the training process and goals.

The RLHF training mechanism includes three main stages:

1. Fine-tuning to generate a Supervised Fine-Tuning (SFT) model

   Fine-tuning is based on GPT-3.5. High-quality prompts and their answers are critical. This data is provided by multi-model predictions or by humans, a process that requires highly skilled annotators.

2. Building a reward model

   ChatGPT currently uses a ranking method to evaluate and filter the output of the SFT model. It uses a supervised learning optimization technique based on ordered data or rankings. The manual annotation required at this stage is ranking annotation, which requires precise and large-scale ranked data.

3. Reinforcement learning optimization

   This stage uses proximal policy optimization (PPO) to adjust the model's behavior based on the output of the reward model. This part requires almost no manual annotation.

These three stages form the complete training loop for ChatGPT and are continuously iterated and optimized. Although the demand for manual annotation is less than in pre-training, the quality and quantity of the annotation significantly impact the final model's performance.

Data format examples

CSV and XLSX formats

Demo:

RLHF_data_demo.csv

Manifest format (JSONL format)

{"data":{"topic":"Fruit01","first-question":"Are apples delicious?"}}
{"data":{"topic":"Fruit02","first-question":"Are oranges delicious?"}}

Demo:

RLHF_data_demo.manifest

Configuration description

Subject area (Required)

The Subject area is where you configure the topic field.

Click Current Topic and select a dataset. Then, configure the parameters in the Configuration area on the right. The parameters are described as follows:

Question area (The first-round question is required in automated Q&A mode)

In automated Q&A mode, you can use the Question area to configure the first-round question and the subject. This configuration is not applicable in manual input mode.

Click First-round Question and configure the parameters in the Configuration area on the right. The parameters are described as follows:

Answer area (Optional)

The Answer Area is where you configure the chatbot's text responses to questions.

Select the answer area. Then, configure the parameters in the Configuration area on the right. The parameters are described as follows:

Global questions (Optional)

You can use Global questions to ask about the entire subject as needed.

Select a question to configure its parameters in the Configuration area on the right. The parameters are described as follows:

Global configuration

Configure the user-defined functions (UDFs) for the template in Global configuration.

Click to configure the Q&A bots in the panel on the right. You can configure up to three answer bots. After you select a bot's UDF, add the configuration parameters.

Demo

Multimodal RLHF annotation - Automated Q&A

1. Click to start the Q&A session. The bot automatically begins the session.

2. You can rewrite the initial question and the bot's answer, and then answer the configured questions.

Multimodal RLHF annotation - Input-based Q&A

1. In the dialog box, you can enter text or upload images, videos, or audio files.

2. You can rewrite the bot's answer and answer the questions. You can also click a sent message in the history to modify it and have the bot generate a new answer.