Integrate the Loggie Agent with custom runtimes

How it works

The Loggie Agent runs as an independent background process inside the function instance, alongside your application. When the function starts, the bootstrap file launches Loggie in the background, then starts your application. Loggie continuously watches the log files your application writes and ships them to SLS.

Because Loggie runs independently, a malfunctioning agent does not affect your function's execution.

Prerequisites

Before you begin, ensure that you have:

• A Simple Log Service project and Logstore in the same region as your function. See Manage a project and Create a Logstore

Step 1: Create a function

1. Log on to the Function Compute console. In the left navigation bar, choose Function Management > Function.

2. In the top navigation bar, select a region. On the Function page, click Create Function.

3. In the dialog box, select Web Function, then click Create Web Function.

4. On the Create Function page, configure the parameters below. Keep the defaults for all other parameters, then click Create. For the full parameter reference, see Create a function. For Code Upload Method, upload a folder named code that contains app.py. The following sample app.py shows a Flask app that writes logs to /tmp/log/fc-flask.log:

   The filename value in logging.basicConfig must match the sources.paths setting in the pipelines.yml file you create in Step 2.

   Parameter	Value
   Function Name (under Basic Configurations)	Enter a name
   Runtime (under Code)	Custom Runtime
   Code Upload Method (under Code)	Upload Folder
   Startup Command (under Code)	/code/bootstrap
   Listening Port (under Code)	9000

   from flask import Flask
   from flask import request
   import logging
   import os
   
   REQUEST_ID_HEADER = 'x-fc-request-id'
   
   app = Flask(__name__)
   
   format_str = '[%(asctime)s] %(levelname)s in %(module)s: %(message)s'
   logging.basicConfig(filename='/tmp/log/fc-flask.log', filemode='w',
       format=format_str, encoding='utf-8', level=logging.DEBUG)
   
   @app.route("/invoke", methods=["POST"])
   def hello_world():
       rid = request.headers.get(REQUEST_ID_HEADER)
       logger = logging.getLogger()
   
       print("FC Invoke Start RequestId: " + rid)
       logger.info("FC Invoke Start RequestId: " + rid)
   
       data = request.stream.read()
       print(str(data))
       logger.info("receive event: {}".format(str(data)))
   
       print("FC Invoke End RequestId: " + rid)
       logger.info("FC Invoke Start RequestId: " + rid)
       return "Hello, World!"
   
   if __name__ == '__main__':
       app.run(host='0.0.0.0', port=9000)

   The bootstrap file referenced as the startup command will be created in Step 2.

Step 2: Create the bootstrap file

1. After the function is created, go to the Code tab and use the WebIDE to create a bootstrap file in the code directory. The following bootstrap script starts Loggie as a background process, then starts the application:

   • Creates pipelines.yml — the Loggie pipeline configuration file. The sources block collects all .log files from /tmp/log/. Setting addonMeta: true adds default metadata about the log collection state to each log entry. The sink block sends logs to SLS using environment variables set in Step 4. For the full sources configuration reference, see Common source configurations and addonMeta.

   • Creates loggie.yml — the Loggie system configuration file. An empty file uses the default configuration. The loggie.yml file must exist, even if it is empty. For custom parameters, see Loggie system configuration.

   • Starts Loggie as a background process. Loggie's own runtime logs are written to /tmp/loggie.log.

   • Starts the application. This example uses Python. Adjust the command for your actual runtime.

   #!/bin/bash
   
   # 1. Create the pipelines.yml file.
   mkdir -p /tmp/log /code/etc
   cat << EOF > /code/etc/pipelines.yml
   pipelines:
     - name: demo
       sources:
         - type: file
           name: fc-demo
           addonMeta: true
           fields:
             topic: "loggie"
           fieldsUnderRoot: true
           paths:
             - "/tmp/log/*.log"
       sink:
         type: sls
         endpoint: ${LOGGIE_SINK_SLS_ENDPOINT}
         accessKeyId: ${LOGGIE_SINK_SLS_ACCESS_ID}
         accessKeySecret: ${LOGGIE_SINK_SLS_ACCESS_SECRET}
         project: ${LOGGIE_SINK_SLS_PROJECT}
         logstore: ${LOGGIE_SINK_SLS_LOGSTORE}
         topic: ${LOGGIE_SINK_SLS_TOPIC}
   EOF
   
   # 2. Create the loggie.yml file.
   cat << EOF > /code/etc/loggie.yml
   EOF
   
   # 3. Start the Loggie Agent as a background process.
   /opt/bin/loggie -config.system=/code/etc/loggie.yml -config.pipeline=/code/etc/pipelines.yml > /tmp/loggie.log 2>&1 &
   
   # 4. Start the application.
   exec python app.py

2. Set executable permissions on the bootstrap file. In the WebIDE terminal, run:

   chmod 777 bootstrap

3. Click Deploy.

Step 3: Add the Loggie Agent public layer

1. Go to the Configuration tab. Under Advanced Settings, click Modify.

2. In the Layer section, click Add Layer > Add Public Layer, then select the Loggie Agent layer:

   Layer name	Compatible runtime	Version	ARN
   Loggie Agent	custom runtime	1	acs:fc:{region}:official:layers/Loggie13x/versions/1

3. Click Deploy.

Step 4: Set environment variables

1. Go to the Configuration tab. Under Advanced Settings, click Modify.

2. In the Environment Variables section, add the following variables. For configuration details, see Configure environment variables. Set FC_EXTENSION_SLS_LOGGIE=true. This prevents the function instance from freezing immediately after an invocation — it waits 10 seconds to ensure Loggie finishes uploading logs. Set the SLS connection variables referenced in pipelines.yml:

   Important

   The 10-second wait is billable at the same rate as the invocation phase. See Billing.

   Variable	Description	Where to get the value
   LOGGIE_SINK_SLS_ENDPOINT	SLS endpoint	Endpoints
   LOGGIE_SINK_SLS_ACCESS_ID	AccessKey ID	Create an AccessKey pair
   LOGGIE_SINK_SLS_ACCESS_SECRET	AccessKey secret	Create an AccessKey pair
   LOGGIE_SINK_SLS_PROJECT	SLS project name	Your SLS console
   LOGGIE_SINK_SLS_LOGSTORE	Logstore for storing logs	Your SLS console
   LOGGIE_SINK_SLS_TOPIC	Log topic (custom value)	Define your own

3. Click Deploy. After the function configuration is updated, execution logs can be uploaded to Simple Log Service via Loggie.

Step 5: Verify the results

1. On the Code tab, click Test Function. After you complete the configuration, logs may be delayed for the first tests. Invoke the function a few more times to generate log data.

2. Log on to the Simple Log Service console and search for logs using the region, project, and Logstore specified in pipelines.yml. A successful result includes the following fields:

   • body: The log message content.

   • state.*: Metadata about the log collection state. The hostname field is the ID of the function instance.

If no logs appear after a few minutes, see Troubleshooting.

Troubleshooting

The Function Compute platform cannot detect whether the Loggie Agent is running correctly.

What's next

In this example, Loggie uploads collected logs as-is, without any processing.

• To process logs before uploading — such as parsing JSON-formatted entries or filtering out DEBUG logs — add an interceptor to pipelines.yml. See Loggie interceptors.

• For the full Loggie documentation, see the Loggie docs.