Data Service system configuration

Permissions

Only super administrators and system administrators can modify system configurations.

Access system configuration

1. On the Dataphin home page, in the top menu bar, choose Service > Service Management.

2. In the left navigation pane, click System Configuration. The System Configuration page opens.

API call authentication configuration

Limits

• Alibaba Cloud API Gateway (dedicated or shared instance) and Apsara Stack API Gateway support token-based authentication.

• For Alibaba Cloud API Gateway-Dedicated Instance or Alibaba Cloud API Gateway-Shared Instance, the authentication section is hidden after you enable it. Republish existing APIs to apply token-based authentication.

• If you use an Apsara Stack API Gateway, you can modify the API call authentication configuration.

API cached data storage location

1. In the API Cached Data Storage Location section, click Modify.

   • System Redis + Application Memory: Stores cached data in the system public Redis instance, shared with other modules. Suitable for small amounts of cached data.

   • Application Memory Only: Not recommended for large cache volumes because high memory usage affects system response time. Suitable only when few APIs require caching with minimal data.

     Note

     The cache duration is determined by the memory data cache time set during Dataphin deployment. The cache duration defined when creating an API does not take effect.

   • Specified Redis Instance: Stores cached data in a dedicated Redis instance. Recommended when many APIs use caching or the cached data volume is large. Create a Redis data source.

     Important

     Do not delete the Redis instance used for API cached data. Deletion causes cache storage failure and invalidates the API cache.

2. Click Save.

SQL injection check

1. In the SQL Injection Check section, click Modify to enable or disable the SQL injection check.

   • Enable: Input SQL is validated during API calls. Failed checks block the call, and the error code on the page shows the reason.

   • Disable: No SQL validation during API calls.

2. Click Save.

Log and O&M statistics settings

Limits

The log collection framework does not guarantee full high availability. Some logs may be lost or duplicated during application restarts or API call failures.

• If you use an Alibaba Cloud API Gateway (dedicated or shared instance), call logs for previously published APIs do not contain requestid information. You must republish the APIs to include this information in the logs.

• If you use an Alibaba Cloud API Gateway-Shared Instance, the gateway does not support collecting information such as requestQueryString,requestHeaders,requestBody,responseHeaders, or responseBody. When authentication, throttling, or other failures occur on the gateway, call logs do not contain request parameters, request SQL, or other information.

• Without gateway log collection, failed gateway calls are excluded from statistics and their logs cannot be collected. Configure log collection for the gateway to ensure complete log data.

• When the storage table for call detail logs or statistics logs is changed, logs are queried from the new table. Data in the previous table can no longer be searched. Plan table changes accordingly to avoid losing access to historical logs.

Procedure

Changes take effect the same day. Logs exceeding the storage duration are automatically purged. Long retention periods consume significant storage, so configure with caution.

1. In the Log And O&M Statistics Settings section, click Modify to configure storage location and duration for call detail logs and call statistics.

   • Call Detail Logs: Disabled by default. When enabled, configure the storage location and duration for successful and failed API call detail logs.

     • Storage Database: Select the storage location for call detail logs. Options include Built-in Storage and External Data Source.

       • Built-in Storage: Stores up to 500,000 successful and 500,000 failed call detail logs. Successful logs can be stored for 3 days and failed logs for 7 days. These values cannot be modified.

       • External Data Source: Stores up to 100 million successful and 100 million failed call detail logs. You must select the data source type, data source, and log table name for log storage. Available storage durations are 7 Days, 14 Days, 30 Days, Custom (1 to 9,999 days), Do Not Store, and Unlimited.

         • Data Source Type: Only PostgreSQL data sources are supported.

         • Data Source: Select a data source instance of the specified type. Click Test Connectivity to test the connection for the TP application. To create a data source, click Create to go to Management Center > Data Source Management.

         • Log Table Name - Primary Table: Select the table for storing API call log information. You can also click Create Table With One Click. In the Create Log Table dialog box, click Check Table Name Existence. The system then creates the log table and its related index tables.

           The log table contains the following fields. The length of each field cannot exceed 10,240 characters.

           Field name	Data type	Field description
           biz_code_describe	VARCHAR(1024)	Description of the business code.
           tenant_id	INT4	Tenant ID.
           status_code	VARCHAR(16)	HTTP status code.
           biz_code	VARCHAR(64)	Business code.
           create_time	TIMESTAMP(3)	Time when the log was created.
           api_no	INT4	API number.
           ip	VARCHAR(32)	Client IP address.
           end_time	TIMESTAMP(3)	End time of the query.
           response_size	INT4	Response size in bytes.
           sql	TEXT	SQL statement for the query.
           start_time	TIMESTAMP(3)	Start time of the query.
           app_key	INT4	Application key.
           response_parameter	TEXT	Response parameters.
           status_describe	VARCHAR(1024)	Description of the HTTP status code.
           cost_time	INT4	Query duration in milliseconds.
           id	INT8	Auto-incrementing ordinal number.
           request_size	INT4	Request size in bytes.
           result_count	INT4	Number of API query results
           request_id	VARCHAR(64)	Unique ID of the query.
           request_parameter	TEXT	Request parameters.
           successful	Boolean	Indicates whether the query was successful.
           job_id	VARCHAR(64)	Task ID of the asynchronous call.
           execute_mode	INT2	Execution mode of the call.
           execute_cost_time	INT4	SQL execution duration of the asynchronous call in milliseconds.

         • Supplementary Information Table: Stores log information for calls to public APIs in asynchronous invocation mode. You can also click Create Table With One Click. In the Create Log Table dialog box, click Check Table Name Existence. The system then creates a supplementary information table (public API call detail log table) for you.

           The supplementary information table contains the following fields.

           Field name	Data type	Field description
           id	INT8	Auto-incrementing ordinal number.
           api_no	INT4	API number.
           job_id	VARCHAR(64)	Task ID of the asynchronous call.
           ext_content	TEXT	Result of the public API call.
           create_time	TIMESTAMP(3)	Time when the log record was created.
           update_time	TIMESTAMP(3)	Time when the log record was updated.
           tenant_id	INT4	Tenant ID.

       Note

       • Successful and failed logs are stored in the same table. The table name is unique for each tenant. You can use the `successful` field to distinguish between them.

       • Expired logs are deleted by the system daily, starting at 02:00. The expiration time is calculated as follows: current date - logs older than storage duration - 1 < creation time.

       • Built-in storage suits short-term scenarios such as successful logs for troubleshooting slow API calls. External data sources suit long-term scenarios such as failed logs for troubleshooting API errors.

       • After an on-premises upgrade (public or private cloud), the call detail log table name changes from dataservice_api_log to dataservice_api_log_${tenantid}. Update the log table name in DataService Studio > Management > System Configuration > Call detail logs > Log table name.

   • Call Statistics: Configure the storage location and duration for 1-minute and 5-minute API call statistics logs.

     • Storage Database: Select the storage location for call statistics logs. Options include Built-in Storage and External Data Source.

       • Built-in Storage: Stores up to 500,000 1-minute and 500,000 5-minute call statistics logs. 1-minute call statistics logs can be stored for 7 days, and 5-minute call statistics logs can be stored for 183 days. These values cannot be modified.

       • External Data Source: Stores up to 100 million 1-minute and 100 million 5-minute call statistics logs. You must select the data source type, data source, and table name for call statistics log storage. Available storage durations for 1-minute call statistics logs are 7 Days, 14 Days, 30 Days, and Custom (1 to 9,999 days). Available storage durations for 5-minute call statistics logs are 183 Days, 366 Days, 2 Years, 3 Years, and Custom (1 to 9,999 days).

         • Data Source Type: Only PostgreSQL data sources are supported.

         • Data Source: Select a data source instance of the specified type. Click Test Connectivity to test the connection in the TP application. To create a data source, click Create to go to Management Center > Data Source Management.

         • Table Name: Select the table for storing call statistics logs. You can also click Create Table With One Click. In the Create Log Table dialog box, click Check Table Name Existence. The system then creates the log table and its related index tables.

           The log table contains the following fields.

           Field name	Data type	Field description
           tenant_id	INT4	Tenant ID.
           client_ips	VARCHAR(256)	Summary of client IP addresses (up to 10).
           biz_error_count	INT8	Number of business errors in API queries.
           create_time	TIMESTAMP(3)	Time when the log record was created.
           total_count	INT8	Total number of API queries.
           api_no	INT4	API number.
           total_time_cost	INT8	Total time consumed by API queries in milliseconds.
           offline_count	INT8	Number of server-side failures in API queries.
           total_succ_time_cost	INT8	Total time consumed by successful API queries in milliseconds.
           minute	VARCHAR(16)	Time window for statistics.
           update_time	TIMESTAMP(3)	Time when the log record was updated.
           app_key	INT4	Application key.
           client_fail_count	INT8	Number of client-side failures in API queries.
           execute_mode	INT2	Execution mode of the API call.

     Note

     • 1-minute and 5-minute call statistics logs are stored in different tables. The table name is unique for each tenant.

     • Expired logs are deleted by the system daily, starting at 02:00. The expiration time is calculated as follows: current date - logs older than storage duration - 1 < creation time.

     • Built-in storage suits short-term scenarios such as 1-minute statistics for troubleshooting slow API calls. External data sources suit long-term scenarios such as 5-minute statistics for error analysis.

     • The dws_dataphin_service_api_mi table (Data Service call count statistics table in the Dataphin metadata warehouse shared model) records only built-in storage logs. For external data source logs, use an integration task.

2. Click Save.