Export Java observability data to Cloud Database ClickHouse Enterprise Edition

Step 1: Deploy the Otel Collector

1. Configure an ECS instance.

   1. Purchase an ECS instance. We recommend assigning a public IP when you purchase the instance for public network access.

   2. Add the IP address of the ECS instance to the allowlist of your ApsaraDB for ClickHouse instance.

      • Private IP: If an ECS instance and an ApsaraDB for ClickHouse Enterprise Edition instance are in the same VPC, you can add the private IP address of the ECS instance to the allowlist. In the following operations, use the VPC address as the endpoint for the ApsaraDB for ClickHouse instance.

      • Public IP: If an ECS instance cannot connect to an ApsaraDB for ClickHouse Enterprise Edition instance over a private network, you can add the public IP address of the ECS instance to the allowlist. In the following operations, use the public address for the endpoint of the ApsaraDB for ClickHouse instance.

      Connect to the ECS instance and run the following command to test the connection to your ApsaraDB for ClickHouse instance. A response of Ok. indicates a successful connection.

      curl http://cc-xxxxxxx-clickhouse.clickhouseserver.pre.rds.aliyuncs.com:8123

2. Connect to the ECS instance and run the following commands to install otelcol-contrib.

   mkdir llm-demo
   cd llm-demo
   wget -O otelcol-contrib.rpm https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.136.0/otelcol-contrib_0.136.0_linux_amd64.rpm
   yum install otelcol-contrib.rpm

3. Configure the Otel Collector to receive observability data over gRPC and HTTP, and to export it to your ApsaraDB for ClickHouse instance over HTTP.

   Copy the following configuration and modify it as needed:

   • In the receivers section: We recommend replacing the endpoint of the OTLP receiver with the endpoint of your Otel Collector service.

   • In the exporters section:

     • Configure the connection details for your ApsaraDB for ClickHouse instance, including endpoint, username, and password.

     • Configure the database and table names for storing observability data in your ApsaraDB for ClickHouse instance, including database, traces_table_name, logs_table_name, and metrics_tables.

   cat > /etc/otelcol-contrib/config.yaml << EOF
   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318
   processors:
     batch:
       timeout: 5s
       send_batch_size: 5000
   exporters:
     clickhouse:
       endpoint: http://cc-xxxxxx-clickhouse.clickhouseserver.pre.rds.aliyuncs.com:8123?dial_timeout=10s&compress=lz4&async_insert=1
       username: test
       password: password
       # ttl: 72h
       traces_table_name: otel_traces
       logs_table_name: otel_logs
       metrics_tables:
         gauge: 
           name: otel_metrics_gauge
         sum: 
           name: otel_metrics_sum
         summary: 
           name: otel_metrics_summary
         histogram: 
           name: otel_metrics_histogram
         exponential_histogram: 
           name: otel_metrics_exp_histogram
       create_schema: false
       timeout: 5s
       database: clickobserve_service
       sending_queue:
         queue_size: 1000
       retry_on_failure:
         enabled: true
         initial_interval: 5s
         max_interval: 30s
         max_elapsed_time: 300s
   
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [clickhouse]
       metrics:
         receivers: [otlp]
         processors: [batch]
         exporters: [clickhouse]
       logs:
         receivers: [otlp]
         processors: [batch]
         exporters: [clickhouse]
   EOF

4. Start the Otel Collector.

   service otelcol-contrib restart

Step 2: Create a Java application

This section demonstrates how to create the application, using a simple Spring Boot dice-rolling service as an example.

1. Configure the Java environment.

   1. Deploy a Java environment. We recommend installing Java 17 or later.

   2. Download and install Gradle. We recommend installing Gradle 8 or later.

2. Initialize a Gradle project.

   In this example, the directory is named java-simple. After initialization, the directory structure is as follows:

   java-simple
   ├── app
   │   ├── build.gradle.kts
   │   └── src
   │       ├── main
   │       │   ├── java
   │       │   │   └── org
   │       │   │       └── example
   │       │   │           └── App.java
   │       │   └── resources
   │       └── test
   │           ├── java
   │           │   └── org
   │           │       └── example
   │           │           └── AppTest.java
   │           └── resources
   ├── gradle
   │   ├── libs.versions.toml
   │   └── wrapper
   │       ├── gradle-wrapper.jar
   │       └── gradle-wrapper.properties
   ├── gradle.properties
   ├── gradlew
   ├── gradlew.bat
   └── settings.gradle.kts

3. In the java-simple directory, modify or create the following files:

   build.gradle.kts

   plugins {
     id("java")
     id("org.springframework.boot") version "3.0.6"
     id("io.spring.dependency-management") version "1.1.0"
   }
   sourceSets {
     main {
       java.setSrcDirs(setOf("."))
     }
   }
   repositories {
     mavenCentral()
   }
   dependencies {
     implementation("org.springframework.boot:spring-boot-starter-web")
   }
   

   App.java

   package org.example;
   import org.springframework.boot.Banner;
   import org.springframework.boot.SpringApplication;
   import org.springframework.boot.autoconfigure.SpringBootApplication;
   @SpringBootApplication
   public class App {
     public static void main(String[] args) {
       SpringApplication app = new SpringApplication(App.class);
       app.setBannerMode(Banner.Mode.OFF);
       app.run(args);
     }
   }

   RollController.java

   package org.example;
   import java.util.Optional;
   import java.util.concurrent.ThreadLocalRandom;
   import org.slf4j.Logger;
   import org.slf4j.LoggerFactory;
   import org.springframework.web.bind.annotation.GetMapping;
   import org.springframework.web.bind.annotation.RequestParam;
   import org.springframework.web.bind.annotation.RestController;
   @RestController
   public class RollController {
     private static final Logger logger = LoggerFactory.getLogger(RollController.class);
     @GetMapping("/rolldice")
     public String index(@RequestParam("player") Optional<String> player) {
       int result = this.getRandomNumber(1, 6);
       if (player.isPresent()) {
         logger.info("{} is rolling the dice: {}", player.get(), result);
       } else {
         logger.info("Anonymous player is rolling the dice: {}", result);
       }
       return Integer.toString(result);
     }
     public int getRandomNumber(int min, int max) {
       return ThreadLocalRandom.current().nextInt(min, max + 1);
     }
   }

Step 3: Configure the OpenTelemetry Java Agent

1. Download the OpenTelemetry Java Agent.

   curl -L -O https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

2. Configure the environment variables.

   export JAVA_TOOL_OPTIONS="-javaagent:/../opentelemetry-javaagent.jar" \
     OTEL_TRACES_EXPORTER=otlp \
     OTEL_METRICS_EXPORTER=otlp \
     OTEL_LOGS_EXPORTER=otlp \
     OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
     OTEL_EXPORTER_OTLP_ENDPOINT=http://<IP_ADDRESS>:4318 \
     OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<IP_ADDRESS>:4318/v1/traces \
     OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<IP_ADDRESS>:4318/v1/metrics \
     OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://<IP_ADDRESS>:4318/v1/logs \
     OTEL_METRIC_EXPORT_INTERVAL=15000

   • JAVA_TOOL_OPTIONS: Set this to the actual path of the opentelemetry-javaagent.jar file.

   • Endpoint configurations: In http://<IP_ADDRESS>:4318, replace <IP_ADDRESS> with the IP address of the Otel Collector service endpoint accessible from your Java application.

   • For information about other parameters, see properties-exporters.

Step 4: Start and access the Java application

1. In the project's application directory, run the following commands to start the application.

   In this example, the project's application directory is ../java-simple/app.

   gradle assemble
   java -jar ./build/libs/app.jar

2. In your browser, access http://<IP_ADDRESS>:8080/rolldice. Replace <IP_ADDRESS> with the IP address of the server running the Java application.

   After a successful request, the page returns a random number, as shown in the following example:

   5

Step 5: Query and analyze the observability data

1. In the left navigation pane of the Cloud Database ClickHouse instance details page, click All-in-one Observe Suite and follow the on-screen instructions to open the service portal.

2. On the main page, query and analyze the observability data from your Java application.

   On the Search page in HyperDX, set the data source to otel_traces. Enter the SQL query SELECT Timestamp, ServiceName as service, StatusCode as level, round(Duration / 1e6) as duration, SpanName, and set the sort order to Timestamp DESC. In the left Filters panel, filter by SpanName (for example, GET /rolldice or PUBLISH) and ServiceName (for example, app or langgenius/dify). The results table, which includes columns such as Timestamp, service, level, duration, and SpanName, allows you to review latency details for spans across services.