When you run a Custom Connector in EventBridge, the connector logic runs inside workers -- Java Virtual Machine (JVM) processes that run connector logic.
Each worker creates a set of tasks -- parallel threads that copy data between a source and a sink. Tasks are stateless and can be started, stopped, or restarted at any time.
Auto-scaling
Serverless App Engine (SAE) provides an elastic and scalable data pipeline for workers. SAE automatically scales the number of worker instances within a configured range based on CPU utilization or memory usage thresholds.
Kafka Connect compatibility
Worker configuration parameters are compatible with Kafka Connect. EventBridge partially hosts Kafka Connect, so some parameters are preset, some are customizable, and some are locked by the platform.
The following sections describe each category.
Default configuration
EventBridge applies these defaults to all workers:
key.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=true
value.converter=org.apache.kafka.connect.storage.StringConverter
value.converter.schemas.enable=true
offset.flush.interval.ms=60000
request.timeout.ms=40000
task.shutdown.graceful.timeout.ms=10000
plugin.path=/opt/kafka/connect/plugins
rest.advertised.port=8083
topic.creation.enable=false
listeners=http://:8083Customizable parameters
Required parameters
These parameters are preset in the EventBridge console.
| Parameter | Description | Example |
|---|---|---|
bootstrap.servers | Endpoint for the ApsaraMQ for Kafka instance | alikafka-post-cn-7mz301t5****.alikafka.aliyuncs.com:9092 |
offset.storage.topic | ApsaraMQ for Kafka topic that stores offset data | topic_offset |
config.storage.topic | ApsaraMQ for Kafka topic that stores configuration data | topic_config |
status.storage.topic | ApsaraMQ for Kafka topic that stores status data | topic_status |
group.id | Connect cluster group identifier for the worker | test |
Preset values example:
group.id=connect-eb-cluster-35345
offset.storage.topic=connect-eb-offset-35345
config.storage.topic=connect-eb-config-35345
status.storage.topic=connect-eb-status-35345
consumer.group.id=connector-eb-cluster-mongo-sink
bootstrap.servers=alikafka-pre-cn-zpr3156gn006-1-vpc.alikafka.aliyuncs.com:9092,alikafka-pre-cn-zpr3156gn006-2-vpc.alikafka.aliyuncs.com:9092,alikafka-pre-cn-zpr3156gn006-3-vpc.alikafka.aliyuncs.com:9092Optional parameters
Custom values for these parameters override EventBridge defaults. For full parameter descriptions, see Kafka Connect Configs.
Serialization and schema
| Parameter | Description |
|---|---|
key.converter | Serialization class for message keys |
key.converter.schemas.enable | Whether to include schema in key data |
value.converter | Serialization class for message values |
value.converter.schemas.enable | Whether to include schema in value data |
header.converter | Serialization class for message headers |
Delivery and consistency
| Parameter | Description |
|---|---|
exactly.once.source.support | Enable exactly-once semantics for source connectors |
Timing and coordination
| Parameter | Description |
|---|---|
heartbeat.interval.ms | Heartbeat interval for consumer group coordination |
rebalance.timeout.ms | Timeout for task rebalancing across workers |
session.timeout.ms | Session timeout for consumer group membership |
scheduled.rebalance.max.delay.ms | Maximum delay before rebalancing after a worker departure |
worker.sync.timeout.ms | Timeout for worker synchronization |
worker.unsync.backoff.ms | Backoff interval for unsynced workers |
metadata.max.age.ms | Maximum age of cached metadata before refresh |
Network
| Parameter | Description |
|---|---|
client.dns.lookup | DNS lookup strategy |
connections.max.idle.ms | Maximum idle time before a connection is closed |
receive.buffer.bytes | TCP receive buffer size |
request.timeout.ms | Request timeout |
send.buffer.bytes | TCP send buffer size |
reconnect.backoff.ms | Initial backoff interval for reconnection attempts |
reconnect.backoff.max.ms | Maximum backoff interval for reconnection attempts |
retry.backoff.ms | Backoff interval between retries |
Offset management
| Parameter | Description |
|---|---|
offset.flush.interval.ms | Interval for flushing offsets to storage |
offset.flush.timeout.ms | Timeout for offset flush operations |
Access control and REST API
| Parameter | Description |
|---|---|
access.control.allow.methods | Allowed HTTP methods for CORS |
access.control.allow.origin | Allowed origins for CORS |
admin.listeners | Admin REST API listener endpoints |
Other
| Parameter | Description |
|---|---|
client.id | Client identifier for broker connections |
config.providers | External configuration providers |
connect.protocol | Connect protocol version |
connector.client.config.override.policy | Policy for connector-level client config overrides |
task.shutdown.graceful.timeout.ms | Graceful shutdown timeout for tasks |
topic.tracking.allow.reset | Allow topic tracking offset reset |
topic.tracking.enable | Enable active topic tracking |
Non-customizable parameters
Platform-managed defaults
These parameters are locked to EventBridge defaults. Custom values are ignored.
plugin.path
rest.advertised.port
topic.creation.enable
listenersSecurity and infrastructure parameters
These parameters are not passed to Kafka Connect.
sasl.*
ssl.*
security.*
rest.advertised.host.name
rest.advertised.listener
rest.extension.classes
client.*
inter.worker.*
metrics.*
metrics.context.*
response.http.headers.config
socket.*