Generate content descriptions in a specified language

Use cases

• Global business operations: Generate content descriptions for images and videos in target languages such as English or Japanese. You can use these descriptions directly for display and search on multilingual websites.

• Multilingual asset libraries: Generate Insights in different languages as needed within the same asset library to standardize search criteria.

How it works

1. Enable content awareness and specify a language: In the DatasetConfig of the target Dataset, enable Insights (Caption) for images or videos, and set Insights.Language to the target language.

2. Generate descriptions upon ingestion: After a file is uploaded and routed to the Dataset, OSS asynchronously generates a content description in the specified language.

3. Write to metadata: OSS writes the generated content description to the file's metadata (Insights) in the Dataset.

4. Query and read: Use SimpleQuery with WithFields to retrieve and read the Insights field.

Limitations

• The caller must have the oss:MetaQuery and oss:PutObject permissions.

• To receive notifications when file indexing is complete, you must first create an MNS queue and ensure that the RAM role used by OpenMetaQuery has permission to send messages to this queue.

• To route different objects to different Datasets, configure RouteRule when you enable MetaQuery.

Step 1: Enable MetaQuery

Enable the metadata management feature for a bucket in semantic mode. The following examples are based on OSS Java SDK V2 (com.aliyun:alibabacloud-oss-v2) and OSS Go SDK V2 (github.com/aliyun/alibabacloud-oss-go-sdk-v2), and read access credentials from the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables.

import com.aliyun.sdk.service.oss2.OSSClient;
import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.models.OpenMetaQueryRequest;
import com.aliyun.sdk.service.oss2.models.OpenMetaQueryResult;

public class OpenMetaQuerySample {
    public static void main(String[] args) throws Exception {
        try (OSSClient client = OSSClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .build()) {
            OpenMetaQueryResult result = client.openMetaQuery(OpenMetaQueryRequest.newBuilder()
                    .bucket("examplebucket")
                    .mode("semantic")
                    .build());
            System.out.println("open meta query status: " + result.statusCode());
        }
    }
}

package main

import (
	"context"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

func main() {
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion("cn-hangzhou")

	client := oss.NewClient(cfg)

	_, err := client.OpenMetaQuery(context.TODO(), &oss.OpenMetaQueryRequest{
		Bucket: oss.Ptr("examplebucket"),
		Mode:   oss.Ptr("semantic"),
	})
	if err != nil {
		log.Fatalf("open meta query failed: %v", err)
	}
	log.Println("open meta query success")
}

Step 2: Create a dataset and specify language

Call CreateDataset to create a Dataset and configure Insights by using DatasetConfig.

The following example creates a Dataset that outputs Insights in English and enables captions for both images and videos.

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.dataprocess.OSSDataProcessClient;
import com.aliyun.sdk.service.oss2.dataprocess.models.CreateDatasetRequest;

public class CreateDatasetSample {
    public static void main(String[] args) throws Exception {
        String datasetConfig = "{\n" +
                "  \"Insights\": {\n" +
                "    \"Language\": \"en\",\n" +
                "    \"Image\": {\"Caption\": {\"Enable\": true}},\n" +
                "    \"Video\": {\"Caption\": {\"Enable\": true}}\n" +
                "  }\n" +
                "}";
        try (OSSDataProcessClient client = OSSDataProcessClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .build()) {
            client.createDataset(CreateDatasetRequest.newBuilder()
                    .bucket("examplebucket")
                    .datasetName("photos-en")
                    .datasetConfig(datasetConfig)
                    .build());
            System.out.println("create dataset success");
        }
    }
}

package main

import (
	"context"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/dataprocess"
)

func main() {
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion("cn-hangzhou")

	client := dataprocess.NewClient(cfg)

	datasetConfig := `{
  "Insights": {
    "Language": "en",
    "Image": {"Caption": {"Enable": true}},
    "Video": {"Caption": {"Enable": true}}
  }
}`

	_, err := client.CreateDataset(context.TODO(), &dataprocess.CreateDatasetRequest{
		Bucket:        oss.Ptr("examplebucket"),
		DatasetName:   oss.Ptr("photos-en"),
		DatasetConfig: oss.Ptr(datasetConfig),
	})
	if err != nil {
		log.Fatalf("create dataset failed: %v", err)
	}
	log.Println("create dataset success")
}

The supported language codes are as follows:

Step 3: Upload and route a file

When you upload a file, set the OSS object tag routing-dataset=photos-en to route the file to the target Dataset. After the file is indexed, Insights are generated in the language specified by Insights.Language in the Dataset. To override the Dataset's default language for an individual file, add the x-oss-metaquery-insights-language tag during the same upload operation. Language settings are prioritized in the following order: file-level tag > DatasetConfig > system default language.

import com.aliyun.sdk.service.oss2.OSSClient;
import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.models.PutObjectRequest;
import com.aliyun.sdk.service.oss2.transport.BinaryData;
import java.nio.file.Files;
import java.nio.file.Paths;

public class PutObjectSample {
    public static void main(String[] args) throws Exception {
        try (OSSClient client = OSSClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .build()) {
            client.putObject(PutObjectRequest.newBuilder()
                    .bucket("examplebucket")
                    .key("photos/snow.jpg")
                    .tagging("routing-dataset=photos-en&x-oss-metaquery-insights-language=zh-Hans")
                    .body(BinaryData.fromStream(Files.newInputStream(Paths.get("/local/path/snow.jpg"))))
                    .build());
            System.out.println("put object success");
        }
    }
}

package main

import (
	"context"
	"log"
	"os"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

func main() {
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion("cn-hangzhou")

	client := oss.NewClient(cfg)

	file, err := os.Open("/local/path/snow.jpg")
	if err != nil {
		log.Fatalf("open file failed: %v", err)
	}
	defer file.Close()

	_, err = client.PutObject(context.TODO(), &oss.PutObjectRequest{
		Bucket:  oss.Ptr("examplebucket"),
		Key:     oss.Ptr("photos/snow.jpg"),
		Tagging: oss.Ptr("routing-dataset=photos-en&x-oss-metaquery-insights-language=zh-Hans"),
		Body:    file,
	})
	if err != nil {
		log.Fatalf("put object failed: %v", err)
	}
	log.Println("put object success")
}

4. Query files and read Insights

After the files are indexed, call SimpleQuery to query files in a specified Dataset and use WithFields to return the Insights field. Alternatively, if you configured Message Service (MNS) notifications when you enabled MetaQuery, you can consume MNS messages to check if file indexing is complete and read Insights from the notification message.

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.dataprocess.OSSDataProcessClient;
import com.aliyun.sdk.service.oss2.dataprocess.models.File;
import com.aliyun.sdk.service.oss2.dataprocess.models.SimpleQuery;
import com.aliyun.sdk.service.oss2.dataprocess.models.SimpleQueryRequest;
import com.aliyun.sdk.service.oss2.dataprocess.models.SimpleQueryResult;
import java.util.Arrays;

public class SimpleQueryInsightsSample {
    public static void main(String[] args) throws Exception {
        try (OSSDataProcessClient client = OSSDataProcessClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .build()) {
            SimpleQueryResult result = client.simpleQuery(SimpleQueryRequest.newBuilder()
                    .bucket("examplebucket")
                    .datasetName("photos-en")
                    .query(SimpleQuery.newBuilder()
                            .field("Filename")
                            .value("snow")
                            .operation("match")
                            .build())
                    .withFields(Arrays.asList("Filename", "URI", "Insights"))
                    .maxResults(10)
                    .build());
            for (File file : result.files()) {
                System.out.println(file.filename() + " => " + file.insights());
            }
        }
    }
}

package main

import (
	"context"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/dataprocess"
)

func main() {
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion("cn-hangzhou")

	client := dataprocess.NewClient(cfg)

	result, err := client.SimpleQuery(context.TODO(), &dataprocess.SimpleQueryRequest{
		Bucket:      oss.Ptr("examplebucket"),
		DatasetName: oss.Ptr("photos-en"),
		Query:       oss.Ptr(`{"Field":"Filename","Value":"snow","Operation":"match"}`),
		WithFields:  oss.Ptr(`["Filename","URI","Insights"]`),
		MaxResults:  oss.Ptr(int32(10)),
	})
	if err != nil {
		log.Fatalf("simple query failed: %v", err)
	}
	for _, file := range result.Files {
		log.Printf("%s => %+v", oss.ToString(file.Filename), file.Insights)
	}
}

The Insights field in the response is the content description generated in the configured language. For example:

<File>
  <Filename>photos/snow.jpg</Filename>
  <Insights>
    <Image>
      <Caption>Snow-covered mountains with sunlight breaking through clouds.</Caption>
    </Image>
  </Insights>
</File>

Related APIs

• Enable metadata management: OpenMetaQuery

• Create a Dataset and configure the Insights language: CreateDataset

• Query files and read Insights: SimpleQuery

• Data structure for Insights configuration: DatasetConfig