The prompt feedback optimization feature uses the examples you provide to automatically generate an optimized prompt that produces your desired results.
Demonstration
Prompt before optimization | Prompt after optimization |
The initial prompt is: Classify car-related articles into one of six categories: Product analysis, Car dealer sales, Classic nostalgia, Quality complaints, Sales performance, or Other. Output the result in JSON format as | The optimized prompt includes the initial prompt (original classification instructions), added examples (few-shot examples) such as "ID3 sales article → Car dealer sales", and content hints (notes that clarify classification boundaries). |
Incorrect inference result: When given a text about the relationship between a car brand's quality complaints and sales, the model outputs the classification | Correct inference result: The model correctly classifies the input text about the car brand, outputting |
Features
This feature does the following:
Adds your sample data to the prompt.
It automatically evaluates the prompt's results against your Evaluation Data over multiple rounds, reflecting on the outcomes to optimize the prompt and generate content hints.
Compared to automatic prompt optimization, this feature evaluates performance based on the data you provide. This produces higher-quality responses tailored to your use case.
We recommend selecting Qwen-Max as the inference model.
Alibaba Cloud Model Studio automatically selects a portion of your sample data to add to the prompt. We recommend providing 5 to 10 entries, with at least one for each scenario.
Alibaba Cloud Model Studio evaluates the prompt based on its performance with the evaluation data and the selected inference model. For best results, we recommend providing at least 20 data entries. The more data you provide, the better the optimization.
Case study
Background
To improve content management efficiency on an automotive forum, you plan to use a large language model to classify articles. The classification criteria are as follows:
Which of the following categories does the car-related article below belong to:
"Product analysis",
"Car dealer sales",
"Classic nostalgia",
"Quality complaints",
"Sales performance",
"Other".
Please provide the final result in the JSON format: {"type":"<category_result>"}.After using the preceding instructions as your prompt, you find the app fails to classify articles correctly. For example, an article that should be classified as "Sales performance" is miscategorized.
After some research, you realize that prompt engineering is the right solution. To improve the app's classification accuracy, you manually classify a set of typical articles. The following is your sample data:
Manually classified data
You decide to use the prompt feedback optimization feature in Alibaba Cloud Model Studio to generate an improved prompt from your manually classified data.
Procedure
On the page in Alibaba Cloud Model Studio, click Create Optimization Task.
Step 1: Select an Inference Model. Alibaba Cloud Model Studio uses this model to perform multiple rounds of prompt evaluation. |
Step 2: Enter the Original Prompt you want to optimize. Just describe the goal of your task. |
Step 3 (Optional): Select sample data. You can either upload a file directly or select data from a sample library. The system adds the sample data to the optimized prompt. We recommend providing 5 to 10 data entries, with at least one entry for each scenario. For this case study, we use the sample data in sample.xlsx. |
Step 4: Upload Evaluation Data. This data is used as the benchmark to evaluate the optimal prompt. We recommend providing at least 20 data entries for your evaluation dataset. The more data you provide, the better the optimization results. For this case study, we use the evaluation data in evaluation.xlsx. |
Step 5: Start optimization. |
Using the optimized prompt
You can save the optimized prompt as a prompt template or directly create an agent app based on it.