DocsICP FilingConsole
Documentation
Home Page Microservices Engine User Guide Task Scheduling Task scheduling with XXL-JOB Executor management

Executor management

Updated at:
Copy as MD
Product
My Favorites

An executor runs the tasks of your XXL-JOB application. After you create an application in the MSE console, you must connect an executor before tasks can run. Connect a standard application through an SDK or a deployed agent. Connect an HTTP application through a domain name or a Kubernetes service.

Connect an executor to a standard application

Connect by using the SDK (high-code)

The high-code method integrates the XXL-JOB software development kit (SDK). You implement the task handling logic in code and register the corresponding Bean name. The console requires only that Bean name to trigger execution. This method suits complex business logic and supports Java, Go, and Python.

Both automatic registration and manual entry are supported.

Automatic registration

Automatic registration requires your service to use the XXL-JOB SDK. After you configure the connection, the application automatically registers when it starts.

View the connection configuration

  1. Log on to the MSE XXL-JOB console.

  2. In the upper-left corner of the page, select the region where your instance resides Region.

  3. Click the ID of your instance. In the left-side navigation pane, choose Application Management.

  4. In the Number of Executors column of your application, click Integrate.

  5. Set Connection Type to Automatic Registration and modify the configuration.

Import the SDK to connect the executor

Java SDK

  1. In the pom.xml file, add the "xxl-job-core" Maven dependency.

  2. Initialize the executor.

    @Configuration
public class XxlJobConfig {
    private Logger logger = LoggerFactory.getLogger(XxlJobConfig.class);

    @Value("${xxl.job.admin.addresses}")
    private String adminAddresses;

    @Value("${xxl.job.accessToken}")
    private String accessToken;

    @Value("${xxl.job.executor.appname}")
    private String appname;

    @Value("${xxl.job.executor.address}")
    private String address;

    @Value("${xxl.job.executor.ip}")
    private String ip;

    @Value("${xxl.job.executor.port}")
    private int port;

    @Value("${xxl.job.executor.logpath}")
    private String logPath;

    @Value("${xxl.job.executor.logretentiondays}")
    private int logRetentionDays;

    @Bean
    public XxlJobSpringExecutor xxlJobExecutor() {
        logger.info(">>>>>>>>>>> xxl-job config init.");
        XxlJobSpringExecutor xxlJobSpringExecutor = new XxlJobSpringExecutor();
        xxlJobSpringExecutor.setAdminAddresses(adminAddresses);
        xxlJobSpringExecutor.setAppname(appname);
        xxlJobSpringExecutor.setAddress(address);
        xxlJobSpringExecutor.setIp(ip);
        xxlJobSpringExecutor.setPort(port);
        xxlJobSpringExecutor.setAccessToken(accessToken);
        xxlJobSpringExecutor.setLogPath(logPath);
        xxlJobSpringExecutor.setLogRetentionDays(logRetentionDays);

        return xxlJobSpringExecutor;
    }

}

Go SDK

  1. Run the following command to install the XXL-JOB SDK for Go. Replace {latest_tag} with the tag of the version that you want to use.

    go get github.com/xxl-job/xxl-job-executor-go@{latest_tag}

  2. Write the business code.

    package main

import (
    "context"
    "fmt"
    xxl "github.com/xxl-job/xxl-job-executor-go"
    "github.com/xxl-job/xxl-job-executor-go/example/task"
    "log"
)

func main() {
    exec := xxl.NewExecutor(
        xxl.ServerAddr("xxxxxx"),       // Request address. Obtain it from the connection configuration in the Application Management section of the console.
        xxl.AccessToken("xxxxxxx"),     // Request token. Obtain it from the connection configuration in the Application Management section of the console.
        xxl.ExecutorPort("9999"),       // Default is 9999. This parameter is optional.
        xxl.RegistryKey("golang-jobs"), // Executor name.
        xxl.SetLogger(&logger{}),       // Custom logger.
    )
    exec.Init()
    exec.Use(customMiddleware)
    // Set the handler for viewing logs.
    exec.LogHandler(customLogHandle)
    // Register the task handler.
    exec.RegTask("task.test", task.Test)
    exec.RegTask("task.test2", task.Test2)
    exec.RegTask("task.panic", task.Panic)
    log.Fatal(exec.Run())
}

// Custom log handler.
func customLogHandle(req *xxl.LogReq) *xxl.LogRes {
    return &xxl.LogRes{Code: xxl.SuccessCode, Msg: "", Content: xxl.LogResContent{
        FromLineNum: req.FromLineNum,
        ToLineNum:   2,
        LogContent:  "This is a custom log handler",
        IsEnd:       true,
    }}
}

// Implementation of the xxl.Logger interface.
type logger struct{}

func (l *logger) Info(format string, a ...interface{}) {
    fmt.Println(fmt.Sprintf("Custom log - "+format, a...))
}

func (l *logger) Error(format string, a ...interface{}) {
    log.Println(fmt.Sprintf("Custom log - "+format, a...))
}

// Custom middleware.
func customMiddleware(tf xxl.TaskFunc) xxl.TaskFunc {
    return func(cxt context.Context, param *xxl.RunReq) string {
        log.Println("I am a middleware start")
        res := tf(cxt, param)
        log.Println("I am a middleware end")
        return res
    }
}

Python SDK

  1. Pull the dependencies.

    pip install pyxxl

 # To write logs to Redis
pip install "pyxxl[redis]"

 # To load configurations from a .env file
pip install "pyxxl[dotenv]"

 # Install all features
pip install "pyxxl[all]"

  2. Write the business code.

    import asyncio
import time

from pyxxl import ExecutorConfig, PyxxlRunner
from pyxxl.ctx import g

config = ExecutorConfig(
    xxl_admin_baseurl="http://xxljob-1b3fd8196eb.schedulerx.mse.aliyuncs.com/api/",
    executor_app_name="xueren-test",
    access_token="default_token",
 #    executor_listen_host="0.0.0.0",  # If xxl-admin can directly connect to the executor's IP address, you do not need to specify executor_listen_host.
)

app = PyxxlRunner(config)

@app.register(name="demoJobHandler")
async def test_task():
    # you can get task params with "g"
    g.logger.info("get executor params: %s" % g.xxl_run_data.executorParams)
    for i in range(10):
        g.logger.warning("test logger %s" % i)
    await asyncio.sleep(5)
    return "Success..."

@app.register(name="sync_func")
def test_task4():
    # To view execution logs in xxl-admin, you must use g.logger for log printing. By default, only logs of the INFO level and higher are printed.
    n = 1
    g.logger.info("Job %s get executor params: %s" % (g.xxl_run_data.jobId, g.xxl_run_data.executorParams))
    # If a sync task contains a loop, you must check g.cancel_event in each iteration to support the cancel operation.
    while n <= 10 and not g.cancel_event.is_set():
        # If you do not need to view logs from xxl-admin, you can use your own logger.
        g.logger.info(
            "log to {} logger test_task4.{},params:{}".format(
                g.xxl_run_data.jobId,
                n,
                g.xxl_run_data.executorParams,
            )
        )
        time.sleep(2)
        n += 1
    return "Success3"

if __name__ == "__main__":
    app.run_executor()

Manual entry

Manual entry lets you maintain the address information of executors. The address format is http://192.168.0.1:9999/.

  1. Log on to the MSE XXL-JOB console.

  2. In the upper-left corner of the page, select the region where your instance resides Region.

  3. Click the ID of your instance. In the left-side navigation pane, choose Application Management.

  4. In the Number of Executors column of your application, click Integrate.

  5. Set Connection Type to Manual Entry and enter the Actuator Address.

Connect by deploying an agent (low-code)

The low-code method connects an executor by deploying an agent. You can write the business logic directly in the console. This method suits script tasks, big data tasks, SQL tasks, and AI tasks.

Three deployment options are supported: installation package, Docker, and Kubernetes.

Installation package

Prerequisites

JDK 17 or later is installed.

  1. Download the installation package.

    wget https://schedulerx3.oss-cn-hangzhou.aliyuncs.com/xxljob/schedulerx3-agent-1.0.0-bin.tar.gz

  2. Decompress the package.

     # Decompress
tar -zxvf schedulerx3-agent-1.0.0-bin.tar.gz
cd schedulerx3-agent-1.0.0-bin

    The directory structure after decompression:

    schedulerx3-agent-1.0.0-bin/
├── bin/              # Startup script directory
├── conf/             # Configuration file directory
│   ├── application.yml      # Application configuration
│   └── logback-spring.xml   # Log configuration
├── lib/              # Dependency JAR directory
└── logs/             # Log directory (created automatically at runtime)
    ├── stdout.log    # Standard output log
    ├── stderr.log    # Standard error log
    ├── worker.log    # Application log
    ├── error.log     # Error log
    ├── gc.log        # GC log
    └── archive/      # Archived log directory

  3. Edit the configuration file conf/application.yml and set the following parameters for your XXL-JOB instance:

    xxl:
  job:
    admin-addresses: {service endpoint}
    access-token: {application AccessToken}
    executor:
      appname: {application AppName}

  4. Start the service.

    • Linux/Mac

      # Start in the background
./bin/start.sh

# Start in the foreground (debugging)
./bin/start.sh -f

# Stop
./bin/stop.sh

# Restart
./bin/restart.sh

# Check the status
./bin/status.sh

# View the logs
tail -f logs/worker.log

    • Windows

      REM Start in the background
.\bin\start.cmd

REM Start in the foreground (debugging)
.\bin\start.cmd -f

REM Stop
.\bin\stop.cmd

REM Restart
.\bin\restart.cmd

REM Check the status
.\bin\status.cmd

REM View the logs
type logs\worker.log

  5. (Optional) Configure logging.

    • The main log files are located in the logs/ directory. Task execution logs are located in ${user.home}/applogs/xxl-job/jobhandler by default.

    Log fileDescriptionRotation policy
    stdout.logStandard output log (startup log)Redirected by the script
    stderr.logStandard error log (exception stack traces)Redirected by the script
    worker.logApplication log (INFO and above)100 MB per file, retained for 30 days
    error.logError log (ERROR level)50 MB per file, retained for 60 days
    gc.logGC logConfigured by JVM parameters
    heap_dump.hprofHeap dump file, generated on out-of-memory-
    archive/Archived log directory, compressed to .gz automatically-

    • Edit conf/logback-spring.xml to adjust log printing.

      <!-- Modify the root log level -->
<root level="INFO">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="FILE" />
</root>
<!-- Modify the log level of a specific package -->
<logger name="com.aliyun.schedulerx" level="DEBUG" />
<logger name="com.xxl.job" level="DEBUG" />

  6. (Optional) Configure JVM parameters. Adjust the JVM memory size to the actual load.

 # Linux/Mac - Specify temporarily
JAVA_OPTS="-Xms2g -Xmx4g" ./bin/start.sh

 # Linux/Mac - Change permanently
vim bin/start.sh  # Edit the JAVA_OPTS variable

 # Windows - Specify temporarily
set JAVA_OPTS=-Xms2g -Xmx4g
.\bin\start.cmd

 # Windows - Change permanently
notepad bin\start.cmd  # Edit the JAVA_OPTS variable

Docker

Method 1: Deploy from the public image

The public image provides runtime support for common scripts and ships with Python 3, Node.js, and Go preinstalled. Pull and run it directly from the image repository. No build is required.

 # Pull the image
docker pull schedulerx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx3/schedulerx3-agent:1.0.0

 # Run with a custom configuration
 # Configure JVM parameters as needed
docker run -d \
  --name schedulerx3-agent \
  -p 9999:9999 \
  -e JAVA_OPTS="-Xms1g -Xmx2g" \
  -e SCHEDULERX3_ADMIN_ADDRESSES="{service endpoint}" \
  -e SCHEDULERX3_EXECUTOR_APPNAME="{application AppName}" \
  -e SCHEDULERX3_ACCESS_TOKEN="{application AccessToken}" \
  -v $(pwd)/logs:/opt/schedulerx3-agent/logs \
  --restart unless-stopped \
  schedulerx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx3/schedulerx3-agent:1.0.0

Method 2: Build your own image from the tarball

If your service has extra external component dependencies or a custom base image, build an image from the downloaded tar package and publish it to your own image repository.

 # Download the installation package
wget https://schedulerx3.oss-cn-hangzhou.aliyuncs.com/xxljob/schedulerx3-agent-1.0.0-bin.tar.gz
 # Build the Docker image
docker build -t schedulerx3-agent:1.0.0 -f Dockerfile .

The corresponding Dockerfile is as follows:

 ############################################

### Install the components your business needs in this Dockerfile

 ############################################

 # Configure the base image. Replace the following placeholder address with the address of your image registry.
FROM hub.docker.xxx.com/library/openjdk:17.0.1-jdk-bullseye

LABEL maintainer="SchedulerX Team"
LABEL description="SchedulerX3 Agent - XXL-Job Executor"
LABEL version="2.4.2"

 # Configure the Alibaba Cloud mirror source
RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list && \
    sed -i 's|security.debian.org/debian-security|mirrors.aliyun.com/debian-security|g' /etc/apt/sources.list

 # Install basic tools, Python 3, Node.js, and Go
RUN apt-get update && \
    apt-get install -y python3 python3-distutils curl wget ca-certificates nodejs npm golang-go && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

 # Install pip by using the official script
RUN curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py && \
    python3 /tmp/get-pip.py && \
    rm -f /tmp/get-pip.py && \
    ln -sf /usr/bin/python3 /usr/bin/python

 # Set the Go environment variables
ENV GOPATH=/root/go
ENV PATH=$GOPATH/bin:$PATH
ENV GO111MODULE=on

 # Copy the tar package into the image
COPY schedulerx3-agent-*-bin.tar.gz /tmp/schedulerx3-agent.tar.gz

 # Decompress the tar package to the specified directory (strip the top-level directory)
RUN mkdir -p /opt/schedulerx3-agent && \
    tar -xzf /tmp/schedulerx3-agent.tar.gz --strip-components=1 -C /opt/schedulerx3-agent && \
    chmod +x /opt/schedulerx3-agent/bin/*.sh && \
    mkdir -p /opt/schedulerx3-agent/logs && \
    rm -f /tmp/schedulerx3-agent.tar.gz

 # Set the working directory
WORKDIR /opt/schedulerx3-agent

 # Expose the port
EXPOSE 9999

 # Startup command (foreground mode of start.sh)
CMD ["bin/start.sh", "-f"]

Kubernetes

  1. Create a schedulerx3-agent.yaml file and deploy it through a Deployment.

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: schedulerx3-agent
  labels:
    app: schedulerx3-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: schedulerx3-agent
  template:
    metadata:
      labels:
        app: schedulerx3-agent
    spec:
      containers:
        - name: schedulerx3-agent
          image: schedulerx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx3/schedulerx3-agent:1.0.0
          imagePullPolicy: Always
          ports:
            - containerPort: 9999
          env:
            - name: "SCHEDULERX3_ADMIN_ADDRESSES"
              value: "{service endpoint}"
            - name: "SCHEDULERX3_EXECUTOR_APPNAME"
              value: "{application AppName}"
            - name: "SCHEDULERX3_ACCESS_TOKEN"
              value: "{application AccessToken}"
          livenessProbe:
            tcpSocket:
              port: 9999
            timeoutSeconds: 30
            initialDelaySeconds: 30

  2. Deploy to Kubernetes.

     # Deploy
kubectl apply -f schedulerx3-agent.yaml

Connect an executor to an HTTP application

No SDK is required for HTTP applications. Backend nodes are automatically discovered when you configure a domain name or a Kubernetes service. The HTTP protocol is used for scheduling.

Connect to a Kubernetes service

If your HTTP application is deployed in Alibaba Cloud Container Service for Kubernetes (ACK), use the "Connect to a Kubernetes service" method.

  1. (Optional) Deploy the application in ACK. The following cluster types are supported:

    • ACK managed cluster that uses the Terway network plug-in

    • ACK Serverless cluster

    • ACS cluster

  2. (Optional) Create a service for the application in ACK. The following service types are supported:

    • ClusterIP

    • LoadBalancer

  3. Log on to the MSE XXL-JOB console.

  4. In the upper-left corner of the page, select the region where your instance resides Region.

  5. Click the ID of your instance. In the left-side navigation pane, choose Application Management.

  6. In the Number of Executors column of your application, click Integrate.

  7. On the Integrate Actuator page, set Connection Type to Integrate K8s Service, complete the parameter settings for the Kubernetes service, and then click OK to complete the connection.

After the connection is established, the number of executors changes. You can click the number to view the list of backend pods.

Manually enter a domain name

If your application is not deployed in ACK, such as on an ECS instance, you can schedule HTTP tasks using an internal domain name.

  1. (Optional) Create a gateway for the HTTP application, such as a Network Load Balancer (NLB). An internal domain name is automatically generated.

  2. In the upper-left corner of the page, select the region where your instance resides Region.

  3. Click the ID of your instance. In the left-side navigation pane, choose Application Management.

  4. In the Number of Executors column of your application, click Integrate.

  5. On the Integrate Actuator page, set Connection Type to Enter a domain name and configure the internal domain name.

Previous：Application managementNext：Tasks