DocsICP FilingConsole
Documentation
Home Page Simple Log Service Query and Analysis Query and analysis best practices Run a context query with a Scan search

Run a context query with a Scan search

Updated at:
Copy as MD
Product
My Favorites

You can search for logs by keyword without accessing a server. Simple Log Service lets you use a Scan search to view a log's context within its original file.

Scan syntax for context queries

{Index Search Query} | {Scan Query} | with_pack_meta

Prerequisites

The server uses a PackId to associate logs with their context. Your logs must include a PackId. For more information, see Associate a log with its context by using the PackId mechanism.

Procedure

Java SDK

Code example

  1. Add the following dependency to the pom.xml file.

    <!-- https://mvnrepository.com/artifact/com.aliyun.openservices/aliyun-log -->
<dependency>
    <groupId>com.aliyun.openservices</groupId>
    <artifactId>aliyun-log</artifactId>
    <version>0.6.120</version>
</dependency>

  2. Use the following code. Replace the values of theproject,logStore,endpoint,accessKeyId, andaccessKeySecret parameters to match your setup. To find the public endpoint for Simple Log Service, see Endpoints. To obtain an AccessKey, see Create an AccessKey.

    import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.response.GetLogsResponse;
public class DoScanTest {
    // This example shows how to obtain an AccessKey ID and an AccessKey secret from environment variables.
    static String accessKeyId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
    static String accessKeySecret = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
    // The endpoint for Simple Log Service. This example uses the China (Hangzhou) region. For other regions, use the correct endpoint.
    static String host = "cn-hangzhou.log.aliyuncs.com";
    // Initialize the Simple Log Service client.
    static Client client = new Client(host, accessKeyId, accessKeySecret);
    public static void main(String[] args) throws LogException {
        String project = "test-project";
        String logStore = "test-logstore";
        int fromTime = 1740985125;      // event-time, [from, to)
        int toTime = 1740985125 + 7200; // event-time, [from, to)
        // String query = "Status:404 | where http_host = 'www.yt.mock.com'";      // Context information is not returned.
        String query = "Status:404 | where http_host = 'www.yt.mock.com' | with_pack_meta";    // Context information is returned.
        int totalCount = 0;
        // boolean reverse = false;  // Scan from oldest to newest.
        boolean reverse = true; // Scan from newest to oldest.
        boolean forward = true;
        int offset = 0;
        while (true) {
            GetLogsResponse resp = client.GetLogs(project, logStore, fromTime, toTime, "", query, 100, offset, reverse, forward, "mode=scan;");
            for (QueriedLog log : resp.getLogs()) {
                System.out.println(log.GetLogItem().ToJsonString());
            }
            System.out.println("[response of this scan]\tbegin offset: " + resp.GetBeginOffset() + "\tend offset: " + resp.GetEndOffset() + "\tresult logs: " + resp.getLogs().size() + "\tis finished: " + resp.IsScanAll());
            totalCount += resp.getLogs().size();
            if (resp.IsScanAll()) {
                break;
            }
            offset = forward ? (int) resp.GetEndOffset() : (int) resp.GetBeginOffset();
        }
        System.out.println("totally scanned logs\t: " + totalCount);
    }
}

    SDK parameters

    Reverse

    Forward

    Offset

    Behavior

    false

    true

    0 or the endOffset from the previous response.

    Paginates forward from older logs (smaller timestamp) to newer logs (larger timestamp).

    true

    true

    0 or the endOffset from the previous response.

    Paginates backward from newer logs (larger timestamp) to older logs (smaller timestamp).

    false

    false

    The total count of indexed logs, or the beginOffset from the previous response.

    Paginates backward from newer logs (larger timestamp) to older logs (smaller timestamp).

    true

    false

    The total count of indexed logs, or the beginOffset from the previous response.

    Paginates forward from older logs (smaller timestamp) to newer logs (larger timestamp).

  3. The output is as follows.

    {
    "referer": "www.xxx.xxx.xxx",
    "slbid": "slb-01",
    "scheme": "https",
    "vpc_id": "8c093000-9f68-2c0f-a904-5c612483505a",
    "upstream_addr": "125.36.xx.xx",
    "owner_id": "owner-01",
    "body_bytes_sent": "1733",
    "request_method": "GET",
    "http_host": "www.yt.mock.com",
    "http_user_agent": "Mozilla/4.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/11.0.1245.0 Safari/537.36",
    "remote_user": "0iu9d",
    "upstream_status": "200",
    "vip_addr": "139.207.xxx.xxx",
    "request_time": "24",
    "__pack_meta__": "0|MTczNjkzNzIxNzg2NDA5NT****==|287|277",
    "__tag__:__pack_id__":"5253859C5169****-3",
    "host": "www.xxx.xxx.xxx",
    "client_ip": "58.19.XXX.XX",
    "user_agent": "Mozilla/5.0 (Windows NT 7.1) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30",
    "remote_addr": "183.70.XX.XXX",
    "__topic__": "nginx_access_log",
    "instance_name": "instance-01",
    "time_local": "03/Mar/2025:08:02:34",
    "request_uri": "/request/path-2/file-4",
    "instance_id": "i-01",
    "request_length": "4458",
    "http_x_forwarded_for": "103.96.xxx.xxx",
    "upstream_response_time": "30",
    "network_type": "vlan",
    "region": "cn-shanghai",
    "logtime": 1740988954,
    "server_protocol": "HTTP/2.0",
    "status": "404"
}
[response of this scan]	begin offset: 0	end offset: 1	result logs: 1	is finished: true
totally scanned logs	: 1
Process finished with exit code 0

  4. Context information

    If the Scan syntax is{Index Search Query} | {Scan Query}, the response does not include context fields. To include context fields in the response, use the syntax {Index Search Query} | {Scan Query} | with_pack_meta.

    In the preceding code example, the returned log context fields are as follows:

    __pack_meta__: 0|MTczNjkzNzIxNzg2NDA5NT****==|287|277
__tag__:__pack_id__: 5253859C5169****-3

    ​With this context information, you can call the GetContextLogs operation to retrieve the surrounding logs from the original file.

Console

  1. In the target Logstore, enter the following Scan query in the search box. When you use the console to perform a context query with a Scan search, the console automatically addswith_pack_meta to the backend request.

    Status:404 | where http_host = 'www.yt.mock.com'

    After you run the query, the results show the log records that meet the specified criteria. You can expand a log to view the details of fields such as http_host, remote_addr, request_uri, and referer.

  2. On the Raw Log > Raw Data tab, find the target log and click the Context query icon icon.

    The magnifying glass icon appears to the right of the log entry timestamp.

  3. Scroll up or down on the page to view the context of the specified log.

    • Click Old to load earlier logs.

    • Click Update to load later logs.

    • Click All Fields to manage displayed fields. All fields are visible by default. Click a field name to toggle visibility.

    • In the Filter text box, enter a keyword to filter log entries.

    • In the Highlight text box, enter a string to highlight all occurrences.

Previous：Correlate log context with PackIdNext：Synchronize index configuration