Import Google Cloud Storage data

Prerequisites

• Log files have been uploaded to GCS.

• A Project and a Logstore have been created.

Create a data import configuration

1. Log on to the Log Service console.

2. In the right pane, click Import Data. On the Data Import tab, select GCS - data import.

3. Select the destination Project and Logstore, and then click Next.

4. In the data import configuration, configure the following parameters.

   Parameter	Description
   task name	The unique name of the Log Service task.
   display name	The display name of the task.
   task description	A description of the data import task.
   GCS endpoint	The service endpoint for GCS. The global endpoint, https://storage.googleapis.com, is used by default. Specify a different endpoint if needed.
   bucket	The name of the GCS bucket.
   AccessKeyId	The AccessKeyId for your GCS account.
   AccessKeySecret	The AccessKeySecret for your GCS account. For more information, see the GCS documentation.
   file path prefix filter	Filters GCS files to import based on a file path prefix. For example, if your files are in the test/ directory, specify the prefix as test/. If this parameter is not set, the entire bucket is scanned. Note Recommended. Scanning an entire bucket with many files reduces import efficiency.
   file path regex filter	Imports only files with paths matching the specified regular expression. If empty, no regex filtering is applied. For example, for the file path test1/test.txt, set the regular expression to tes.*.
   file modification time filter	Imports only GCS files based on their last modified time. All: Imports all files that meet the criteria. From a specific time: Imports only files modified after a specified time. Specific time range: Imports only files modified within a specified time range.
   data format	The format used to parse the log files. Supported formats include: CSV: Delimiter-separated text files. You can use the first row as field names or specify field names manually. Each subsequent row is parsed as field values. single-line JSON: Each line is treated as a separate JSON object. The object's keys and values are mapped to log fields and their corresponding values. single-line text log: Parses each line as a single log entry. multiline text log: Combines multiple physical lines into a single log entry based on a regular expression that identifies the start or end of an entry. JSON array: Parses the entire file as a JSON array, where each object in the array becomes a single log entry. CloudTrail: Parses the entire file based on the standard CloudTrail data structure.
   compression format	The compression format of the source files. Files are automatically decompressed before parsing.
   encoding format	The encoding format of the GCS files. UTF-8 and GBK are supported.
   new file check interval	The interval for checking new files. When set, the task runs continuously to import new files as they appear, processing each file only once. For a one-time import, select Never Check. The task stops after processing all existing files that match the filter criteria.
   write processor	Preprocesses imported data, including data filtering, field extraction, field enrichment, and data masking. For more information, see Manage write processors.

   Log time configuration

   Parameter	Description
   time field	The field in the source data that contains the log timestamp, used as the log time in Log Service. Available for CSV, single-line JSON, JSON array, and CloudTrail formats.
   time extraction regex	A regular expression used to extract the time value from a log entry. For example, if a log entry is 127.0.0.1 - 2026-04-16 17:25:42 audit-log, you can set the regular expression to [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}. Note Also useful for extracting a specific time format from a time field that contains extraneous text.
   time field format	The time format used to parse the value in the time field. Supports time formats based on Java's SimpleDateFormat syntax, such as yyyy-MM-dd HH:mm:ss. Supports epoch formats, including epoch, epochMillis, epochMicro, and epochNano.
   time zone	The time zone of the timestamp in the source logs. This is not required for epoch formats. To account for Daylight Saving Time (DST), select a UTC-based time zone; otherwise, a GMT-based time zone is sufficient. Note The default time zone is UTC+8.

   When you select CSV or multiline text log as the data format, you must configure the following additional parameters.

   CSV-specific parameters

   Parameter	Description
   delimiter	The character that separates fields in the CSV file. Default: comma (,).
   quote character	The quote character for strings in the CSV file.
   escape character	The escape character for the log. The default is a backslash (\).
   maximum lines per log	The maximum number of lines that a single log entry can span. The default is 1. Note The maximum value is 10,000 lines.
   use first row as field names	When enabled, the first row of the CSV file is used as field names.
   custom field list	If the first-row-as-field-names option is turned off, specify custom field names separated by commas (,).
   lines to skip	The number of lines to skip from the beginning of the file. For example, if you set this to 1, data ingestion starts from the second line.

   Multiline text log-specific parameters

   Parameter	Description
   regex matching position	The position where the regular expression is matched: Start-of-line Regex: Uses a regular expression to match the first line of a log entry. Unmatched lines are considered part of the current log entry until the maximum line count is reached. End-of-line Regex: Uses a regular expression to match the last line of a log entry. Unmatched lines are considered part of the next log entry until the maximum line count is reached.
   regular expression	The regular expression for matching log entries.
   maximum lines	The maximum number of lines for a single log entry. Note The maximum value is 10,000 lines.

5. Click Preview to view the import results.

6. After you confirm the settings, click Next.

7. Click Query Logs to open the Query and Analysis page. It may take up to a minute for the imported data to appear. The import is successful when you see the GCS data in the query results.

View an import configuration

After creating a data import configuration, view its details and statistical reports.

1. In the Project list, click the target Project.

2. Under Log Storage, locate the target Logstore and select Data Collection > Data Import. Then, click the name of your import configuration.

3. View the basic information and statistical reports for the import configuration.

4. You can modify, start, stop, or delete the configuration.

   Warning

   A deleted configuration cannot be recovered. Proceed with caution.

Billing

The data import feature is free in Log Service. However, Google Cloud charges for internet egress and API requests from GCS. Refer to your Google Cloud bill for details. The pricing model and relevant parameters are as follows:

FAQ

Error handling

GCS import APIs

• Create a GCS import task

• Update a GCS import task

• Get a GCS import task

• List GCS import tasks

• Start a GCS import task

• Stop a GCS import task