Image-to-text template-Platform For AI(PAI)-阿里云帮助中心

In AIGC content generation, traditional manual annotation is inefficient, costly, and makes cross-modal alignment difficult. The image-to-text template uses the Segment Anything Model (SAM) and proprietary segmentation algorithms to quickly segment images. It combines these segments with the semantic understanding and text generation capabilities of large language models. This helps you build an intelligent annotation workflow and significantly improves multimodal content production efficiency.

Data format examples

Note

In CSV and XLSX formats, each column represents a dataset field. In the Manifest format, each subfield of the data field corresponds to a dataset field. The field names are customizable. You can select the corresponding field names when you configure the dataset fields. Common image file formats such as JPG and PNG are supported.

CSV and XLSX formats

image_url	caption
oss://****.oss-cn-hangzhou.aliyuncs.com/iTAG/pic/1.jpg	caption1xxxxx
oss://****.oss-cn-hangzhou.aliyuncs.com/iTAG/pic/2.jpg	caption2xxxxx

Sample file: image-to-text_data_demo.csv

Manifest format (JSONL format)

{"data":{"url":"oss://****.oss-cn-hangzhou.aliyuncs.com/iTAG/pic/1.jpg","caption":"caption1xxxxxx"}}
{"data":{"url":"oss://****.oss-cn-hangzhou.aliyuncs.com/iTAG/pic/10.jpg","caption":"caption2xxxxxx"}}
{"data":{"url":"oss://****.oss-cn-hangzhou.aliyuncs.com/iTAG/pic/11.jpg","caption":"caption3xxxxxx"}}

Sample file: image-to-text_data_demo.manifest

Configuration description

Annotation area (Image) (Required)

Use the Annotation area (Image) to configure the URLs of the images to annotate.

Click Annotation area (Image) and select a dataset. Then, in the Configuration area on the right, configure the parameters. The parameters are described as follows:

Parameter	Description
Title	The default title does not need to be modified.
Dataset field	From the dataset fields, select the field for the image URLs to annotate. Field names are customizable.
Prompt	The default prompt does not need to be modified.
Data pre-processing	No configuration is required. This parameter has no effect.
Watermark configuration	No configuration is required. This parameter has no effect.
Watermark configuration	No configuration is required. This parameter has no effect.
Card style configuration	No configuration is required. This parameter has no effect.

Scenario description (Text) (Optional)

Use the Scenario description (Text) to configure the scenario descriptions in the dataset.

Click Scenario description (Text). Then, in the Configuration area on the right, configure the parameters. The parameters are described as follows:

Parameter	Description
Title	The default title does not need to be modified.
Dataset field	From the dataset fields, select the field that corresponds to the scenario description.
Prompt	The default prompt does not need to be modified.
Click link to navigate	No configuration is required. This parameter has no effect.
Text highlight	No configuration is required. This parameter has no effect.
Card style configuration	No configuration is required. This parameter has no effect.

Global configuration

Use Global configuration to configure the User-Defined Functions (UDFs) for the template.

Click . In the Configuration area on the right, configure the parameters. The parameters are described as follows:

Parameter	Description
Title	The default title does not need to be modified.
Detail description	The title name cannot be modified. Configure the official UDF and its parameters as shown in the image.
Generate text	The title name cannot be modified. Configure the official UDF and its parameters as shown in the image.
General configurations	No configuration is required. This parameter has no effect.

Important

You must configure the UDF exactly as shown in the image. The input_data and MarkResult parameters ensure that the UDF receives the correct input and must not be changed.

Operation demo

Scenario description
You can generate a description from the entire image. The description can also be retrieved from the dataset. You can edit or regenerate it.
Detail description
1. In the image tool area, you can use the cutout tool to segment the image and create a new selection.
  - Click Confirm to generate the detail description. You can modify the detail description at any time. A unique color is used for each annotation.
  - Click Create only to generate a new detail area without a description. You must enter the detail description manually. You can modify the detail description at any time. A unique color is used for each annotation.
2. Click Correct mask to modify and save the mask area for the corresponding ID.
3. Click Hide or Show to control the visibility of the mask layer for the corresponding ID.
4. Click Delete to remove the form content and the corresponding mask area for that ID.
Generate text
Click Generate to combine the scenario description and detail descriptions to create the final text.
Interactive command line
Enter the @ sign to select a command. The @BLIP command controls the generation of scenario and detail descriptions. The @LLM command controls text generation.