Import Google Cloud Storage data to SLS

Alibaba Cloud

1. Confirm your account permissions. If you are using a RAM user, ensure that the AliyunLogFullAccess policy is attached to the user. For more fine-grained control, see Create a custom policy.

2. Create a Project and a LogStore to store the imported data.

GCS

1. Obtain the following key configuration information from GCS. You will need it to create the SLS data import task.

   Parameter	Description
   GCS Endpoint	By default, the global endpoint is https://storage.googleapis.com. To access a bucket in a specific region, you can use a regional endpoint, such as https://storage.googleapis.com/storage/v1.
   Bucket name	The name of the GCS bucket that stores the log files.
   AccessKeyId / AccessKeySecret	The HMAC key for GCS, used for authentication via the S3-compatible protocol. For instructions on obtaining this key, see Step 3 below.

2. Get the bucket name. In the Google Cloud Console, navigate to Cloud Storage > Buckets. Find your target bucket name (for example, demoh01) in the list and record it for later configuration.

3. Get the AccessKeyId and AccessKeySecret. In the Google Cloud Storage console, click Settings in the left navigation pane. At the bottom of the page, in the "Access keys for your user account" section, click Create key to generate an HMAC key. Then, copy the corresponding access key (AccessKeyId) and secret (AccessKeySecret).

Step 1: Create an SLS data import task

1. Log on to the Simple Log Service console.

2. Click the name of the target Project to open its details page.

3. In the left navigation pane, click Task Management and switch to the Data Import tab.

4. Click Create Data Import Task. On the GCS-Data Import card, click Import Now.

5. Select the target LogStore for the data import and click Next.

6. On the Import Configurations page, configure the data source information. The following table describes the parameters.

   Parameter	Description	Example
   Display Name	A name to identify and manage the import task.	gcs-log-import
   GCS Endpoint	The GCS access endpoint. The default, https://storage.googleapis.com, typically does not require changes.	https://storage.googleapis.com
   Bucket	The name of the GCS bucket that stores the log files.	my-log-bucket
   AccessKeyId	The access key ID of the GCS HMAC key. For instructions, see Prerequisites.	GOOG1E***
   AccessKeySecret	The HMAC secret key that is paired with the AccessKeyId.	bGoa***
   File Prefix (Optional)	Specifies a prefix to filter the files to be imported. If left empty, all files in the bucket are imported.	logs/2026/
   Data Format	The data format of the log files. Supported formats include JSON, CSV, and delimiter-separated values. Select the format that matches the actual logs stored in GCS.	JSON string

7. Click Preview. SLS attempts to connect to GCS and read the files.

   Note

   If the connection fails, check your network connectivity, ensure that the bucket name, AccessKeyId, and AccessKeySecret are correct, and verify that the service account associated with the HMAC key has read permissions.

8. After confirming that the preview data is correct, click Next.

Step 2: View and manage the import task

After the task is created, it starts automatically and runs continuously.

1. In the left navigation pane, click Task Management.

2. Select the Data Import tab and click the name of the target task to open its overview page. Here, you can view monitoring metrics such as task status, processing rate, and error information.

Failed to connect to GCS

If a connection fails after you click Preview during task creation, perform the following checks:

• Incorrect endpoint configuration: Ensure that the GCS Endpoint is set to https://storage.googleapis.com. Include the https:// prefix.

• Incorrect bucket name: Confirm that the bucket name is identical to the name displayed in the GCS console. Bucket names are case-sensitive.

• Network connectivity issues: SLS accesses GCS over the public internet. Ensure that your GCS bucket does not have policies that restrict access from external IP addresses.

Authentication failure (AccessKey-related)

If an authentication failure or insufficient permissions error occurs during preview or task execution, perform the following checks:

• Invalid HMAC key: Ensure that the AccessKeyId and AccessKeySecret are the HMAC keys obtained from the GCS interoperability page. Do not use a Google Cloud API Key or OAuth credentials.

• Insufficient permissions: Verify that the service account associated with the HMAC key has at least the storage.objects.get and storage.objects.list permissions for the target bucket. Granting the Storage Object Viewer role is usually sufficient.

• Key disabled or deleted: Log on to the GCS console and check that the HMAC key status is Active.

Data parsing errors

If the import task runs correctly but log data is not parsed into fields as expected, perform the following checks:

• Incorrect data format selected: Ensure that the data format specified in the import configuration matches the actual format of the log files in GCS. For example, if the log files are in JSON format, the data format should be set to JSON string.

• File encoding mismatch: Ensure that the log files are encoded in UTF-8. If another encoding is used, you must select the corresponding encoding format in the import configuration.

• Improper file prefix filtering: If a file prefix is configured, ensure that the prefix path is correct and does not exclude the target log files.