Picture description

Updated at:

This document provides technical information about the English picture description speech evaluation for developers who want to integrate this feature.

Scenarios for this question type:

  • A picture is provided, and the user describes its content in English.

Parameter description

Field

Type

Optional

Description

coreType

String

Required

en.pict.score indicates the English picture description question type.

rank

int

Optional

The scoring system for the total score.

precision

double

Optional

Sets the scoring precision. Supported values are 0.1, 0.5, and 1. The default value is 1. If you set the value to anything other than 0.1 or 0.5, the system uses 1.

grade

int

Optional

Specifies the education level. Supported values are 1 and 2. The default value is 1.1: Indicates the junior high school level. A full score requires covering about 90% of the key points.2: Indicates the high school level. A high score requires covering about 80% of the key points because the topics are more difficult.

lm

array

Required

The sample compositions for the picture. Each text subfield represents one sample composition. Note: The lm field is the lowercase form of LM.

phones

object

Optional

Specifies the pronunciation of a word. If a word has multiple pronunciations, separate them with a pipe symbol (|). Use a space between each phoneme, and between phonemes and the syllable separator (-). Example: "phones":{"directory":"d ih - 'r eh k - tr ih|d ax - 'r eh k - t ax - r ih"}.

points

array

Optional

Sets the key points for the answer. Each key point is a separate text object.

attachAudioUrl

int

Optional

Specifies whether to return the audio URL in the evaluation response.
0: Does not return the URL.
1: Returns the URL.

rateScale

float

Optional

The scoring leniency. The value ranges from 0.8 to 1.5. The default value is 1.0. If the value is less than 0.8, the system uses 0.8. If the value is greater than 1.5, the system uses 1.5. This parameter acts as a multiplier. A higher value results in a higher score.

Parameter example

{
    "coreType": "en.pict.score",
    "rank": 100,
    "precision": 0.5,
    "attachAudioUrl":1,
    "lm": [
        {
            "text": "My aunt Judy was a single mother of fifty, working as a secretary. When doctors discovered she had a very serious heart disease, she didn't give in. She began studying medical reports and found an article in a magazine about a well known heart doctor, Dr. Michael Baker, of Houston, Texas. The article said Dr. Baker's fees were very high; Aunt Judy couldn't afford them. But could he tell her of someone whose fee she could pay? So Aunt Judy wrote to him. She simply listed her reasons for wanting to live: her three children, who would be independent in three or four more years, her little girl dream of travelling and seeing the world. There wasn't a word of self pity only warmth and humor and the joy of living. She mailed the letter, not really expecting an answer. A few days later, Aunt Judy rushed into my house, and read aloud a letter: your beautiful letter moved me very deeply. If you can come to Houston, there will be no charge for either the hospital or the operation. Michael Baker. That was seven years ago. Since then, Aunt Judy has been around the world. Her three children are happily married."
        }
    ],
    "points": [
        {
            "text": "My aunt Judy was a single mother of fifty, working as a secretary."
        },
        {
            "text": "When doctors discovered she had a very serious heart disease, she didn't give in."
        },
        {
            "text": "She began studying medical reports and found an article in a magazine about a well known heart doctor, Dr. Michael Baker, of Houston, Texas."
        },
        {
            "text": "The article said Dr. Baker's fees were very high; Aunt Judy couldn't afford them."
        },
        {
            "text": "But could he tell her of someone whose fee she could pay? So Aunt Judy wrote to him."
        },
        {
            "text": "She simply listed her reasons for wanting to live: her three children, who would be independent in three or four more years, her little girl dream of travelling and seeing the world."
        },
        {
            "text": "There wasn't a word of self pity only warmth and humor and the joy of living."
        },
        {
            "text": "She mailed the letter, not really expecting an answer."
        },
        {
            "text": "A few days later, Aunt Judy rushed into my house, and read aloud a letter: your beautiful letter moved me very deeply. If you can come to Houston, there will be no charge for either the hospital or the operation. Michael Baker."
        },
        {
            "text": "That was seven years ago. Since then, Aunt Judy has been around the world. Her three children are happily married."
        }
    ]
}

Response example

{
    "tokenId": "612f3d6c88bf5b0000040001",
    "applicationId": "a148",
    "audioUrl": "http:\/\/files.cloud.ssapi.cn:8080\/a148\/11ec0b00cf7d709f94a4a14872616788",
    "connect": {
        "param": {
            "app": {
                "timestamp": "1630485868",
                "applicationId": "a148",
                "sig": "342befbf5b7ccb03d4562fad7b21d7261908d5da"
            },
            "sdk": {
                "os": "6.2(9200)1",
                "product": "",
                "source": 7,
                "protocol": 1,
                "os_version": "6.2(9200)",
                "arch": "586",
                "version": 16778752
            }
        },
        "cmd": "connect"
    },
    "params": {
        "app": {
            "timestamp": "1630485868",
            "userId": "ssound_text",
            "sig": "342befbf5b7ccb03d4562fad7b21d7261908d5da",
            "connect_id": "612f3d6c88bf5b0000060002",
            "clientId": "",
            "applicationId": "a148"
        },
        "audio": {
            "sampleBytes": 2,
            "audioType": "ogg",
            "sampleRate": 16000,
            "channel": 1
        },
        "request": {
            "lm": [
                {
                    "text": "My aunt Judy was a single mother of fifty, working as a secretary. When doctors discovered she had a very serious heart disease, she didn't give in. She began studying medical reports and found an article in a magazine about a well known heart doctor, Dr. Michael Baker, of Houston, Texas. The article said Dr. Baker's fees were very high; Aunt Judy couldn't afford them. But could he tell her of someone whose fee she could pay? So Aunt Judy wrote to him. She simply listed her reasons for wanting to live: her three children, who would be independent in three or four more years, her little girl dream of travelling and seeing the world. There wasn't a word of self pity only warmth and humor and the joy of living. She mailed the letter, not really expecting an answer. A few days later, Aunt Judy rushed into my house, and read aloud a letter: your beautiful letter moved me very deeply. If you can come to Houston, there will be no charge for either the hospital or the operation. Michael Baker. That was seven years ago. Since then, Aunt Judy has been around the world. Her three children are happily married."
                }
            ],
            "tokenId": "612f3d6c88bf5b0000040001",
            "request_id": "612f3d6c88bf5b0000050001",
            "precision": 0.5,
            "attachAudioUrl": 1,
            "coreType": "en.pict.score",
            "rank": 100,
            "points": [
                {
                    "text": "My aunt Judy was a single mother of fifty, working as a secretary."
                },
                {
                    "text": "When doctors discovered she had a very serious heart disease, she didn't give in."
                },
                {
                    "text": "She began studying medical reports and found an article in a magazine about a well known heart doctor, Dr. Michael Baker, of Houston, Texas."
                },
                {
                    "text": "The article said Dr. Baker's fees were very high; Aunt Judy couldn't afford them."
                },
                {
                    "text": "But could he tell her of someone whose fee she could pay? So Aunt Judy wrote to him."
                },
                {
                    "text": "She simply listed her reasons for wanting to live: her three children, who would be independent in three or four more years, her little girl dream of travelling and seeing the world."
                },
                {
                    "text": "There wasn't a word of self pity only warmth and humor and the joy of living."
                },
                {
                    "text": "She mailed the letter, not really expecting an answer."
                },
                {
                    "text": "A few days later, Aunt Judy rushed into my house, and read aloud a letter: your beautiful letter moved me very deeply. If you can come to Houston, there will be no charge for either the hospital or the operation. Michael Baker."
                },
                {
                    "text": "That was seven years ago. Since then, Aunt Judy has been around the world. Her three children are happily married."
                }
            ]
        }
    },
    "recordId": "11ec0b00cf7d709f94a4a14872616788",
    "eof": 1,
    "result": {
        "keypoints_hit": [
            0.25,
            0,
            0,
            0.166667,
            0.142857,
            0.1875,
            0.222222,
            0,
            0.090909,
            0.5
        ],
        "overall": 11.5,
        "wavetime": 7990,
        "precision": 0.5,
        "systime": 8098,
        "res": "eng.retell.mfcc.0.1",
        "version": "0.0.80.2021.8.27.04:29:39",
        "pron": 85.5,
        "info": {
            "tipId": 0,
            "clip": 0,
            "snr": 14.624933,
            "volume": 84
        },
        "fluency": 84,
        "rank": 100,
        "rhythm": 0,
        "integrity": 8,
        "pretime": 586,
        "delaytime": 375
    },
    "dtLastResponse": "2021-09-01 16:44:34:784",
    "cloud_platform": {
        "origin_audio_length": 30457
    }
}

Response description

Key

Type

Description

Suggestion

recordId

string

The unique ID of the audio file.

For reference

tokenId

string

The unique ID of the request.

For reference

applicationId

string

The provided authorization account, which is the AppKey.

Internal parameter

audioUrl

string

The URL of the user's audio. By default, the URL is stored in the cloud for one month. Save the audio to your product server.

For reference

dtLastResponse

string

The timestamp of the server response.

Internal parameter

eof

Int

0: The response is not complete. More results will follow.1: All results for this evaluation have been returned.

Internal parameter

params

object

Echoes the evaluation request parameters from the client.

- app

object

- - applicationId

string

The provided authorization account, which is the AppKey.

- - userId

string

The user ID. This is a custom string that helps with troubleshooting.

- - timestamp

string

The start time.

- - sig

string

The verification code.

- - clientId

string

- request

object

The section for scoring request parameters.

- audio

object

The node for the audio parameters of the scoring request.

result

object

The evaluation result.

- res

string

The name of the resource used for this evaluation type.

Internal parameter

- version

string

The engine version, including the release date and time.

Internal parameter

- rank

int

The scoring system.

Internal parameter

- precision

float

The scoring precision.

Internal parameter

- pretime

Int

The time, in ms, that the server took to call the Start API.

Internal parameter

- systime

int

The total time, in ms, for the entire evaluation process.

Internal parameter

- delaytime

int

The time, in ms, from when the server finished processing the audio feed to when the result was obtained.

Internal parameter

- wavetime

int

The audio duration, in ms.

For reference

- overall

float

The overall score.

Recommended field

- fluency

float

The fluency score. This score reflects the smoothness of speech and is not related to the correct answer.

Recommended field

- pron

float

The pronunciation score. This score reflects pronunciation quality. It is not related to the correct answer and depends only on the accuracy of word pronunciation.

Recommended field

- rhythm

float

The rhythm score. This parameter is not currently used.

Internal parameter

- integrity

float

The integrity score.

Internal parameter

- keypoints_hit

array

The hit status for each key point, in the same order as the points parameter. This field is not returned if the points field is not specified.

Recommended field

- fake_words

object

This field is returned only for out-of-vocabulary (OOV) words. To evaluate OOV words, pass the phonetic transcription in the refText parameter.

- -char

string

- info

object

- - snr

float

The signal-to-noise ratio (SNR). A higher value indicates clearer audio. Range: 0 dB to 40 dB. If this parameter affects the score, a corresponding tipId is set.

For reference

- - clip

float

Indicates audio clipping due to high volume. Range: 0 to 1. If this parameter affects the score, a corresponding tipId is set.

Internal parameter

- - volume

int

The recording volume. Range: 0 dB to 180 dB.

For reference

- - tipId

int

The audio quality ID. For more information, see tipId prompt information.

Recommended field

Suggestions for displaying evaluation results

Dimension

Description

Details

overall

Overall score

The evaluation score for the question type.

pron

Pronunciation score

fluency

Fluency

keypoints_hit

Hit rate for each key point

Response for a failed evaluation

If the result structure in the response is empty, the evaluation failed. Retrieve the errId and error from the response and handle the error based on the corresponding error code.

Error codes

For a list of error codes, see Evaluation error codes.