Collect data from multiple channels

Background

"I Want Takeaway" (IWT) is an e-commerce platform that connects users, restaurants, and delivery couriers. Users can place orders through various channels such as web pages, mobile apps, WeChat, and Alipay. After a restaurant prepares the order, the system automatically notifies a nearby courier, who then delivers the food to the user.

Operational requirements

During operations, IWT faces the following challenges:

• User acquisition: Despite significant ad spending on marketing channels like web and WeChat push notifications, it is difficult to measure the effectiveness of each channel.

• Slow deliveries: Users frequently complain about delivery times, but the specific bottlenecks—whether in order acceptance, food preparation, or delivery—are unclear, making optimization difficult.

• User engagement: Promotional campaigns often fail to produce the expected results.

• Scheduling: IWT needs a way to help restaurants prepare for peak hours and dispatch more couriers to high-demand areas.

• Customer support: When users report failed orders, support staff need to understand the user's actions and identify potential system errors.

Data collection challenges

To implement data-driven operations, the first step is to centralize scattered logs. This process presents several challenges:

• Multiple channels: Data comes from various sources, including advertisers and offline promotions like flyers.

• Multiple endpoints: Users interact through web browsers, official accounts on social platforms, mobile apps, and mobile websites.

• Heterogeneous networks: The infrastructure spans across VPCs, self-managed data centers, and Alibaba Cloud ECS instances.

• Multiple development languages: The technology stack includes Java for the core system, Nginx for frontend servers, and C++ for the backend payment system.

• Diverse devices: Merchant hardware includes both x86 and ARM-based platforms.

While this process was traditionally complex and labor-intensive, you can now use the collection features of LogHub to streamline the process.

Centralize and configure logs

1. Create a project to manage your logs, such as myorder.

2. Create a Logstore for logs from each data source. For example:

   • wechat-server: Stores access logs from WeChat servers.

   • wechat-app: Stores application logs from WeChat servers.

   • wechat-error: Stores error logs.

   • alipay-server

   • alipay-app

   • deliver-app: Stores status logs from the courier app.

   • deliver-error: Stores error logs.

   • web-click: Stores click events from H5 pages.

   • server-access: Stores server-side access logs.

   • server-app: Stores application logs.

   • coupon: Stores logs related to coupon usage.

   • pay: Stores payment logs.

   • order: Stores order logs.

3. If you need to clean raw data or perform ETL (Extract, Transform, Load) operations, you can create additional Logstores to store intermediate results.

Collect user promotion logs

Collection strategy

You can use two methods to acquire new users:

• Offer coupons directly during website registration.

• Offer coupons through QR codes that users can scan on other channels.

  • QR codes on flyers

  • QR codes on web pages for login

Implementation

Define a registration server URL and generate QR codes for different channels, such as flyers and web pages. When a user scans a QR code to register, you can identify the source channel and record it in a log.

http://example.com/login?source=10012&ref=kd4b

When the server receives a request, it generates a log entry similar to the following:

2016-06-20 19:00:00 e41234ab342ef034,102345,5k4d,467890

The log contains the following fields:

• time: The registration time.

• session: The current browser session, used to track user behavior.

• source: The source channel. For example, Campaign A is 10001, flyers are 10002, and elevator ads are 10003.

• ref: The referral ID. This field is empty if there is no referrer.

• params: Other parameters.

Collection methods:

• The application writes logs to a local disk, and Logtail collects them. For more information, see Logtail (legacy).

• The application writes logs directly by using an SDK. For more information, see SDK reference overview.

Collect server-side data

Collection strategy

Applications for Alipay and WeChat official accounts follow a typical web model and generate four main types of logs:

• Nginx and Apache access logs

  These logs are used for monitoring and real-time statistics.

  10.1.168.193 - - [01/Mar/2012:16:12:07 +0800] "GET /Send?AccessKeyId=8225105404 HTTP/1.1" 200 5 "-" "Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"                    

• Nginx and Apache error logs

  2016/04/18 18:59:01 [error] 26671#0: *20949999 connect() to unix:/tmp/fastcgi.socket failed (111: Connection refused) while connecting to upstream, client: 10.101.1.1, server: , request: "POST /logstores/test_log HTTP/1.1", upstream: "fastcgi://unix:/tmp/fastcgi.socket:", host: "example.com" 

• Application-level logs

  These logs should record details such as the event time, location, result, latency, method, and parameters. Extended fields are typically placed at the end.

  {
      "time":"2016-08-31 14:00:04",
      "localAddress":"192.0.2.8:0",
      "methodName":"load",
      "param":["31851502"],
      "result":....,
      "serviceName":"com.example",
      "startTime":1472623203994,
      "success":true,
      "traceInfo":"88_1472621445126_1092"
  }

• Application-level error logs: These logs record error details, such as the time, line of code, error code, and cause.

  2016/04/18 18:59:01 :/var/www/html/SCMC/routes/example.php:329 [thread:1] errorcode:20045 message:extractFuncDetail failed: account_hsf_service_log                    

Collect end-user logs

Collection strategy

• Mobile devices: Use the SDK for iOS or Android, or integrate with Mobile Analytics (MAN).

• ARM devices: Use native C cross-compilation.

• Merchant platform devices: Use an SDK for x86 platforms or native C cross-compilation for ARM platforms.

Collect web and mobile user behavior

Collection strategy

User behavior on web and mobile pages can be categorized into two types:

• Page interactions that involve the backend server, such as placing an order, logging in, or logging out.

• Page interactions that do not involve the backend server and are handled on the client side, such as scrolling or closing a page.

Implementation

Use WebTracking to collect user behavior data from pages. For more information, see Collect client-side logs with WebTracking.

Collect server O&M logs

Collection strategy

Examples:

• Syslog

  Aug 31 11:07:24 zhouqi-mac WeChat[9676]: setupHotkeyListenning event NSEvent: type=KeyDown loc=(0,703) time=115959.8 flags=0 win=0x0 winNum=7041 ctxt=0x0 chars="u" unmodchars="u" repeat=0 keyCode=32                  

• Application debug logs

  __FILE__:build/release64/sls/shennong_worker/ShardDataIndexManager.cpp
  __LEVEL__:WARNING
  __LINE__:238
  __THREAD__:31502
  offset:816103453552
  saved_cursor:1469780553885742676
  seek count:62900
  seek data redo
  log:pangu://localcluster/redo_data/41/example/2016_08_30/250_1472555483
  user_cursor:1469780553885689973

• Trace logs

  [2013-07-13 10:28:12.772518]    [DEBUG] [26064]  __TRACE_ID__:661353951201    __item__:[Class:Function]_end__  request_id:1734117   user_id:124 context:.....

Collect data in different network environments

LogHub provides an endpoint in each region and supports three access methods:

• Internal network (classic network): Recommended for accessing services within the same region over a high-quality, reliable link.

• Internet (classic network): Allows access from anywhere. Connection speed depends on link quality. We recommend using HTTPS to secure data in transit.

• Private network (VPC): Allows access from within a VPC in the same region.