Automated device testing-阿里云帮助中心

The IoT Platform provides an automated testing feature that helps you identify device connection issues and improve product integration efficiency. This feature thoroughly tests the device to improve quality control. It eliminates tedious manual testing, helps you quickly find problems, and provides clear and visual test results. Currently, this feature is available only for products in Tmall Genie projects.

Prerequisites

You have added a test device and completed device debugging. For more information, see Device debugging overview.

Background information

When you run automated tests, note the following:

Ensure a stable operating environment during automated testing. Avoid power outages or network disconnections.
Stability tests run for a long time. Ensure the device can operate safely for extended periods to prevent security issues, such as overheating.

Important For the speaker network provisioning success rate test, you must prepare a network provisioning firmware. This firmware must cause the device to automatically enter the pending provisioning state after it receives a detach instruction from the platform. To ensure a high success rate, avoid broadcast interference from other devices during the test. If possible, run the test in a shielded box.

Procedure

Log on to the IoT Platform console.
Go to the Device Debugging page of the product.
Click Online Debugging or Debug for the device to open the device debugging page.
Select the Automated Testing tab and click Log on with Taobao Account. Log on with the Taobao account that is attached to the speaker and the test device.
From the Test Speaker drop-down list, select the speaker to test. Then, click Select Attached Device to select the test device.

Complete each automated test item in order.

The system determines whether the test passes based on the results. If a test fails, fix the issue based on the provided feedback. You can also contact the review team when you submit the product for review. If a test fails due to special circumstances, you can add a note to provide an explanation. We will contact you for confirmation. For more information about how to add a note, see Add a note.

Test evaluation rules

The evaluation rules for automated test results are described in the following table:

System evaluation result	Description	Example test status
Incomplete	Not all required test items were executed.
Failed	All required test items were executed, but one or more items failed. Note If your device cannot pass the test due to special circumstances, click Add Note on the product publishing page to add a note. The review team will determine whether the product meets the standards based on the report.
Passed	All required test items were executed and passed.

Test item descriptions

The automated testing interface is shown in the following figure. The test items are described in the following table.

Note

Functional integrity tests (TSL model test and corpus test) are required.
For products jointly customized with Tmall Genie, you must also complete stability tests (speaker network provisioning success rate test and voice control success rate test). If a test fails, you can re-run the failed item.
Check whether features that can block other operations, such as a power switch or child lock, were enabled in previous tests. Manually disable these features before you re-run the failed test item.

Test item	Test objective	Use case generation rule	Passing standard	Example test report
TSL model test (Required)	Verifies that the product's defined Thing Specification Language (TSL) model is consistent with the end-to-end feature implementation.	Generates use cases based on each property and service defined in the product's TSL model. Note: Enumeration and Boolean types: Traverses all enumeration value scenarios in order. Integer, float, and double types: Verifies scenarios for the corresponding property, such as minimum value, maximum value, and minimum value + step size. Note After the test is complete, if you modify the TSL model, you must run the test again.	100%
Corpus test (Required)	Verifies that the TSL model-based corpus can be executed correctly.	Generate test cases by leveraging all corpora supported by the product.	100%
Speaker network provisioning success rate test Note This item is required if the product is jointly customized with Tmall Genie.	Tests the number of successful and failed network provisioning attempts for the speaker device.	The platform runs a stress test with 20 attempts to obtain the device's network provisioning success rate. To improve test efficiency, the test item stops if the number of failures exceeds the threshold (2). Subsequent attempts are marked as failed.	Success rate ≥ 95.0%
Voice control success rate test Note This item is required if the product is jointly customized with Tmall Genie.	Tests the success rate data for voice control of the device.	The platform runs a stress test with 1,000 attempts to obtain the voice control success rate. To improve test efficiency, the test item stops if the number of failures exceeds the threshold (3). Subsequent attempts are marked as failed.	Success rate ≥ 99.7%

(Optional) Add a note.
If an automated test fails after multiple attempts, or if the failure occurs for a special reason, you can add a note to provide an explanation. The review team will determine whether to skip the test or contact you for more details based on your description.
1. On the Online Debugging page, click Go to Add Note to open the Mass Production product publishing page.
2. On the Mass Production page, in the device testing step, click Add Note to add a note.
3. Follow the prompts in the Note text box to enter the required information. After you confirm that the information is correct, click OK.
Note Automated testing helps you quickly and easily identify product issues and ensure product quality. You must complete automated testing before you publish a product.
After a product is published, you cannot add or edit notes. You must revoke the publication to add a note.