Stream Engine FAQ
This topic provides answers to common questions about resource estimation, network access, dependency conflicts, and JDBC write performance for Lindorm Stream Engine.
How do I estimate job parallelism and cluster resources?
Job parallelism and cluster resources must be evaluated based on peak data volume and computational complexity. Consider the following factors:
Upstream peak write rate, message size, and number of Kafka partitions.
SQL operator complexity, such as JOIN, window, aggregation, UDF, and external service calls.
Job state size and Checkpoint duration.
Write capacity of downstream storage such as Kafka and Lindorm.
Resource headroom for peak traffic, failover, and future growth.
If you cannot estimate the requirements in advance, start with the minimum specification and run jobs at a low parallelism. Monitor the following metrics:
Whether upstream Pending or Kafka Lag continues to increase.
Input and output rates, busy time, and backpressure of each operator.
Cluster CPU, memory, GC, and slot usage.
Checkpoint duration and failure rate.
Downstream write latency and failure rate.
Adjust resources based on monitoring results:
Symptom |
Recommendation |
Backlog keeps increasing while CPU, memory, or slot usage is high |
Scale up or expand the Stream Engine cluster first. |
Backlog keeps increasing but cluster load is low with no obvious backpressure |
Increase job parallelism if the upstream partition count and downstream capacity allow. |
Backlog keeps increasing and Sink shows backpressure or high write latency |
The bottleneck is usually downstream. Scale up or optimize storage first. Increasing parallelism may worsen downstream pressure. |
Only a few concurrent instances have high load |
Check for uneven Kafka partition distribution, data skew, or hot keys. |
After stress testing, estimate the required parallelism:
Job parallelism ≈ Peak target throughput ÷ Measured throughput per parallelismStream Engine cluster resources must cover all job parallelism instances and their resource consumption, with headroom for peak traffic and failover.
The effective parallelism of a Source operator is usually limited by the number of upstream partitions. For example, if a Kafka topic has only 4 partitions, setting the Kafka Source parallelism to a value greater than 4 usually does not increase read throughput.
What do I do if Kafka, Lindorm, or other storage is unreachable?
Use the network detection feature of the Stream Task O&M Platform to probe the target domain, IP address, and port from the actual network environment where the job runs.
Troubleshoot in the following order:
Confirm that Stream Engine and the target storage are in the same VPC, or connected through a supported private link.
Add allowlist rules on Kafka, Lindorm, or other storage for the IP addresses or VSwitch CIDR block used by Stream Engine.
Verify the target domain, IP address, port, and connection string.
Verify the authentication settings such as username, password, AccessKey, and SASL/SSL configuration.
After modifying allowlists, connection addresses, or authentication parameters, resubmit or restart the job to avoid reusing stale connections.
Stream Engine does not support direct access to public network addresses. Use VPC private addresses, Express Connect, or other supported private access methods. Network connectivity tests from a client machine do not represent job-level access. Use the network detection results from the Stream Task O&M Platform as the source of truth.
How do I handle ClassNotFound, NoSuchMethod, or JAR conflicts?
These errors are usually caused by missing dependencies in the runtime environment or version mismatches between job dependencies and Stream Engine server-side dependencies.
Common errors and causes:
Error |
Common Cause |
|
The JAR is missing from the runtime environment, or the dependency was incorrectly set to |
|
Different versions of the dependency were loaded at compile time and runtime. |
|
Multiple versions of the same class exist, or the class was loaded by different ClassLoaders. |
Job submitted successfully but fails at runtime |
The server-side built-in Connector is incompatible with the Connector packaged in the job. |
Follow these steps to resolve the issue:
Save the full exception stack trace and identify the class, method, and dependency that caused the error.
Record the versions of Stream Engine, Flink, Connector, client, and job dependencies.
Run the following command to check the Maven dependency tree and locate duplicate dependencies:
mvn dependency:treeCheck whether the job Fat JAR contains duplicate copies of Flink, Kafka Connector, Log4j, or Lindorm client dependencies.
Align the Connector versions between the server side and the job side. Remove duplicate or incompatible JARs.
Repackage and resubmit the job. If necessary, stop the old job and restart the relevant TaskManagers to clear stale dependencies loaded by the ClassLoader.
When you use the Kafka Connector, note the following:
If the server side uses
flink-sql-connector-kafka, the job side must use the same version that is compatible with the server-side Flink version.If the platform already includes
flink-sql-connector-kafka, the job should not include another version.Do not introduce multiple versions of
flink-sql-connector-kafka,flink-connector-kafka, or their transitive dependencies.
How do I improve JDBC Connector write performance for Lindorm?
When you use the JDBC Connector to write data to Lindorm, add the following parameters to the JDBC connection string:
useServerPrepStmts=true&cachePrepStmts=trueExample connection string:
jdbc:mysql://<host>:<port>/<database>?useServerPrepStmts=true&cachePrepStmts=trueParameter descriptions:
Parameter |
Description |
|
Enables server-side prepared statements to reduce parsing and compilation overhead for repeated SQL statements. |
|
Caches prepared statements to reduce the overhead of creating |
After you modify the connection string, resubmit or restart the job and compare the write throughput, Sink latency, and backpressure before and after the change.
If write performance is still below expectations, check the following:
Batch write size and flush interval of the JDBC Connector.
Whether the Sink parallelism is appropriate.
Lindorm node load, hot primary keys, and write latency.
Network bandwidth and connection stability.
Whether row-by-row commits or overly frequent flushes are used.
These connection string parameters reduce JDBC prepared statement overhead. If the bottleneck is on the network, hot data, or Lindorm storage side, you must address the actual bottleneck.