QuerySmarttagJob-阿里云帮助中心

Operation description

This operation returns the results of an intelligent analysis of people, scenes, and keywords in videos.
The DataQ - Smart Tag Service is available only in the China North 2 (Beijing), China East 2 (Shanghai), and China East 1 (Hangzhou) regions.

QPS limit

The maximum number of queries per second (QPS) for this operation is 100 per user. If this limit is exceeded, rate limiting is triggered, which may affect your business. For more information, see QPS limits.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

mts:QuerySmarttagJob

get

*All Resource

*

None

Request parameters

Parameter	Type	Required	Description	Example
JobId	string	Yes	The ID of the DataQ - Smart Tag Service job to query. Get this value from the response of SubmitSmarttagJob.	39f8e0bc005e4f309379701645f4****
Params	string	No	Additional request parameters as a JSON string. For example: `{"labelResultType":"auto"}`. The `labelResultType` parameter is a string with these values: auto: Machine-generated tags hmi: Human-machine collaborative tags	{"labelResultType":"auto"}

Response elements

Element	Type	Description	Example
	object	Response parameters.
JobStatus	string	The job status. Valid values: Success: The job succeeded. Fail: The job failed. Processing: The job is running. Submitted: The job was submitted and is waiting to run.	Success
Message	string
RequestId	string	The request ID.	7B117AF5-2A16-412C-B127-FA6175ED1AD0
Results	object
Result	array<object>	An array of analysis result objects.
	object	Detailed information about each analysis result object.
Data	string	The specific analysis result, formatted as a JSON string. For the structure of different Type values, see Result parameter description below.	{"title":"example-title-****"}
Type	string	The type of analysis result. Tag 1.0 result types: TextLabel: Text labels VideoLabel: Video labels ASR: Raw speech recognition results (not returned by default) OCR: Raw OCR results (not returned by default) Process: Link to the raw algorithm result JSON file (not returned by default). For the JSON structure, see Additional notes. Tag 2.0 result types: CPVLabel (human-machine) Meta: Video title and other metadata (not returned by default) Tag 2.0-custom result types: CPVLabel (human-machine) Meta: Video title and other metadata (not returned by default)	Meta
UserData	string	Information passed through when using MNS callbacks. For the message format, see Callback message format definition below.	example UserData ****

Callback message format

When the state of a DataQ - Smart Tag Service job changes, ApsaraVideo Media Processing (MPS) sends a message to the queue that you specify. You can call the UpdatePipeline operation to specify the callback queue. The message body is a JSON string that contains the following fields:

Name	Type	Description
Type	String	The fixed string "smarttag", indicating a DataQ - Smart Tag Service job
JobId	String	The unique job ID
State	String	The current job status. This matches the JobStatus value in QuerySmarttagJob.
UserData	String	The UserData value you passed in SubmitSmarttagJob.

Result parameter description

VideoLabel data structure

Name	Type	Description
persons	JSONArray	Detected people
persons.name	String	The detected person's name
persons.category	String	The person's category. Valid values: `celebrity` (celebrity); `politician` (politician); `sensitive` (sensitive person); `unknown` (unknown person); or a custom person library ID (returned if the person is in your custom library)
persons.ratio	double	The person's appearance ratio. Valid values: 0 to 1
persons.occurrences	JSONArray	Detailed occurrence information
persons.occurrences.score	double	Confidence score
persons.occurrences.from	double	Start time of the person's appearance, in seconds
persons.occurrences.to	double	End time of the person's appearance, in seconds
persons.occurrences.position	JSONObject	Face coordinates
persons.occurrences.position.leftTop	int[]	x and y coordinates of the top-left corner
persons.occurrences.position.rightBottom	int[]	x and y coordinates of the bottom-right corner
persons.occurrences.timestamp	double	Timestamp of the face coordinates, in seconds
persons.occurrences.scene	String	Shot type. Valid values: `closeUp` (close-up); `medium-closeUp` (medium close-up); `medium` (medium shot); `medium-long` (long shot)
tags	JSONArray	Detected object and scene labels. See the table below for examples.
tags.mainTagName	String	Main tag
tags.subTagName	String	Subtag
tags.ratio	double	Appearance ratio. Valid values: 0 to 1
tags.occurrences	JSONArray	Occurrence details
tags.occurrences.score	double	Confidence score
tags.occurrences.from	double	Start time, in seconds
tags.occurrences.to	double	End time, in seconds
classifications	JSONArray	Video classification information
classifications.score	double	Confidence score
classifications.category1	String	Level 1 category, such as Life, Animation, or Automobile
classifications.category2	String	Level 2 category, such as Health or Home under Life

Example video tags

mainTagName	subTagName
Program	For example: Where Are We Going, Dad? or Happy Comedians
Character	For example: doctor, nurse, or teacher
Object	For example: piano, water glass, table, tomato and egg stir-fry, car, or cosmetics
Logo	For example: CCTV1, CCTV2, Youku, or Dragon TV
Action or event	For example: dancing, kissing, hugging, meeting, singing, making a phone call, horseback riding, or fighting
Region	For example: Tiananmen Square in Beijing, Statue of Liberty, Leshan Giant Buddha, China, or United States
Scene	For example: bedroom, subway station, terraced fields, beach, or desert

ImageLabel data structure

Name	Type	Description
persons	JSONArray	Detected people
persons.name	String	The detected person's name
persons.category	String	The person's category. Valid values: celebrity (celebrity); politician (politician); sensitive (sensitive person)
persons.score	double	Confidence score for the person
persons.position	JSONObject	Face coordinates
persons.position.leftTop	int[]	x and y coordinates of the top-left corner
persons.position.rightBottom	int[]	x and y coordinates of the bottom-right corner
persons.scene	String	Shot type. Valid values: closeUp (close-up); medium-closeUp (medium close-up); medium (medium shot); medium-long (long shot)
tags	JSONArray	Detected object and scene labels. See the table below for examples.
tags.mainTagName	String	Main tag
tags.subTagName	String	Subtag
tags.score	double	Confidence score

Example image tags

mainTagName	subTagName
Character	For example: doctor, nurse, or teacher
Region	For example: Tiananmen Square in Beijing, Statue of Liberty, Leshan Giant Buddha, China, or United States
Action or event	For example: speaking
Logo	For example: CCTV1, CCTV2, Youku, or Dragon TV
Action or event	For example: dancing, kissing, hugging, meeting, singing, making a phone call, horseback riding, or fighting
Object	For example: piano, water glass, table, tomato and egg stir-fry, car, or cosmetics
Scene	For example: bedroom, subway station, terraced fields, beach, or desert

TextLabel data structure (from automatic speech recognition (ASR) and optical character recognition (OCR))

Name	Type	Description
tags	JSONArray	Text label information. See the table below for examples.
tags.name	String	Label category
tags.value	String	Label value. Multiple values are separated by commas (,).

Example text tags

name	value
Region	For example: Tiananmen Square in Beijing, Statue of Liberty, Leshan Giant Buddha, China, or United States
Organization	For example: China Wildlife Conservation Association or China Media Group
Logo	For example: Nike or Li-Ning
Keyword	For example: backbone force

CPVLabel data structure

cates: Categories (level 1, level 2, and level 3)
entities: Category properties (with knowledge graph information)
hotwords: Hot words (your watchlist)
freeTags: Free tags (keywords)

Name	Type	Example value	Description
type	String	hmi	Result type. Valid values: hmi (human-machine collaborative result) or autp (machine-generated result)
cates	JSONArray	-	Category results
cates.labelLevel1	String	Travel	Level 1 label
cates.labelLevel2	String	Scenic spots	Level 2 label
cates.label	String	""	Label name. The algorithm may return an empty string.
cates.appearanceProbability	double	0.96	Appearance ratio
cates.detailInfo	JSONArray	-	-
cates.detailInfo.score	double	0.9	Confidence score
cates.detailInfo.startTime	double	0.021	Start time
cates.detailInfo.endTime	double	29.021	End time
entities	JSONArray	-	-
entities.labelLevel1	String	Region	Level 1 label
entities.labelLevel2	String	Landmark	Level 2 label
entities.label	String	Huangguoshu Waterfall	Label name
entities.appearanceProbability	double	0.067	Appearance ratio
entities.knowledgeInfo	String	{"name": "Huangguoshu Waterfall", "nameEn": "Huangguoshu Waterfall", "description": "One of Asia's four great waterfalls"}	Knowledge graph information. All fields are listed in the tables at the end: IP Graph, Music Graph, Person Graph, Landmark Graph, and Object Graph.
entities.detailInfo	JSONArray	-	-
entities.detailInfo.score	double	0.33292606472969055	Confidence score
entities.detailInfo.startTime	double	6.021	Start time
entities.detailInfo.endTime	double	8.021	End time
entities.detailInfo.trackData	JSONArray	-	Structured entity label information
entities.detailInfo.trackData.score	double	0.32	Confidence score
entities.detailInfo.trackData.bbox	integer[]	23, 43, 45, 67	Bounding box
entities.detailInfo.trackData.timestamp	double	7.9	Timestamp
hotwords	JSONArray	-	-
hotwords.labelLevel1	String	Keyword	Level 1 label
hotwords.labelLevel1	String	Keyword	Level 2 label
hotwords.labelLevel2	String	""	Level 2 label
hotwords.label	String	China Meteorological Administration	Hot word content
hotwords.appearanceProbability	double	0.96	Appearance ratio
hotwords.detailInfo	JSONArray
hotwords.detailInfo.score	double	1.0	Confidence score
hotwords.detailInfo.startTime	double	0.021	Start time
hotwords.detailInfo.endTime	double	29.021	End time
freeTags	JSONArray
freeTags.labelLevel1	String	Keyword	Level 1 label
freeTags.labelLevel2	String	""	Level 2 label
freeTags.label	String	Central Meteorological Observatory	Hot word content
freeTags.appearanceProbability	double	0.96	Appearance ratio
freeTags.detailInfo	JSONArray
freeTags.detailInfo.score	double	0.9	Confidence score
freeTags.detailInfo.startTime	double	0.021	Start time
freeTags.detailInfo.endTime	double	29.021	End time

Automatic speech recognition (ASR) results

Name	Type	Description
details	JSONArray	Detailed task results
details.from	double	Start timestamp, in seconds
details.to	double	End timestamp, in seconds
details.content	String	Recognized text content

Optical character recognition (OCR) results

Name	Type	Description
details	JSONArray	Detailed task results
details.timestamp	double	Timestamp, in seconds
details.info	JSONArray	Specific information recognized at that timestamp
details.info.score	double	Confidence score
details.info.position	JSONObject	Text coordinates
details.info.position.leftTop	int[]	x and y coordinates of the top-left corner
details.info.position.rightBottom	int[]	x and y coordinates of the bottom-right corner
details.info.content	String	Recognized text content
details.info.textType	String	Text type. Valid values: 0 (normal caption), 1 (scrolling caption), or 2 (fixed-length identifying caption, duration longer than 3 seconds)

Meta tagging results

Note

If you do not use human-machine tagging and set needMetaData to true in a SubmitSmarttagJob request, QuerySmarttagJob returns the original title.

Name	Type	Description
title	String	Title

Caption extraction results

Name	Type	Description
details	JSONArray	Detailed task results
details.allResultUrl	String	URL for all captions. This URL remains valid for six months after the job completes.
details.chResultUrl	String	URL for Chinese captions. This URL remains valid for six months after the job completes.
details.engResultUrl	String	URL for English captions. This URL remains valid for six months after the job completes.

Note

The content of the file at the caption result URL is formatted as follows: Sequence number + Time range + Caption content (each caption on one line).

Examples

Success response

JSON format

{
  "JobStatus": "Success",
  "Message": "",
  "RequestId": "7B117AF5-2A16-412C-B127-FA6175ED1AD0",
  "Results": {
    "Result": [
      {
        "Data": "{\"title\":\"example-title-****\"}",
        "Type": "Meta"
      }
    ]
  },
  "UserData": "example UserData ****"
}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.