Add Multimodal Evaluation Data

Updated at:

Multimodal data structure and how to add data to multimodal evaluation datasets

Multimodal evaluation assesses how well an Agent understands and generates images, audio, video, and mixed content. After preparing a multimodal evaluation dataset, you can select multimodal graders and create evaluation tasks against multimodal data, covering scenarios such as visual question answering, invoice recognition, voice customer service, and video content understanding.

What is multimodal data

Multimodal data can be an image, audio, or video, or a combination of text, images, audio, and video.

Multimodal fields are stored on the server as Array<Object>. Each Object in the array declares its element type with type and carries content in the corresponding payload field. Image, audio, and video payloads are URLs. Array order is content order, and the array supports repeated elements of the same type as well as mixed text and media.

[
  {
    "type": "text",
    "text": "Describe this image and summarize the main content of the audio and video."
  },
  {
    "type": "image_url",
    "image_url": {
      "url": "https://example.com/image.png"
    }
  },
  {
    "type": "audio_url",
    "audio_url": {
      "url": "https://example.com/audio.mp3"
    }
  },
  {
    "type": "video_url",
    "video_url": {
      "url": "https://example.com/video.mp4"
    }
  }
]

A multimodal record does not need to contain all four element types. It can contain a single modality or any mix. For example, image-only, image-and-text, audio-and-text, and video-and-text are all valid multimodal data.

Creating a multimodal evaluation dataset

When creating an evaluation dataset, edit the table schema and set the column type to Multimodal. For example:

ColumnData typeRequiredDescription
inputMultimodal or StringYesUser input
reference_outputMultimodal or StringNoIdeal output

Defining an input field as Multimodal is common when evaluating multimodal content understanding. Defining an output field as Multimodal is common when evaluating multimodal content generation.

For detailed steps to create and publish an evaluation dataset, see Create and Manage Datasets.

Adding multimodal evaluation data

Go to the dataset details page and click Add Data > Add Single Record:

  • For a multimodal field, click Add Content and add text, image, audio, or video elements as needed. This release supports local file upload only.
  • Elements form an array in the order they are added. You can reorder elements, delete individual elements, or append more.
  • Save after filling in the data. The system validates required fields and adds the record after validation passes.

NoteMultimodal data validation:

  • Text: non-empty, up to 10,000 characters.
  • Image: JPG, JPEG, or PNG, up to 20 MB per file.
  • Audio: MP3, up to 100 MB per file.
  • Video: MP4 or MOV, up to 100 MB per file.

Backfilling evaluation data from Traces

Traces in observability can be backfilled into a multimodal evaluation dataset. In the Trace list, select traces in batch, map Trace input and output fields (including multimodal image content) to dataset fields, and import them into the dataset. For details, see Observability and its "Add to Dataset" section.