Job FAQ

Spring application cannot find a bean

1. On the application management page, connect to the machine to check the startup method. Ensure that the application is started in Spring or Spring Boot mode.

2. Inject JobProcessor as a bean, for example, by adding the @Component annotation.

3. Check your Pom dependencies. If the spring-boot-devtools dependency is included, exclude it.

4. If the JobProcessor class and the process method have AOP annotations, upgrade the SchedulerX client to the latest version. Earlier versions do not support AOP.

5. A bean type mismatch can occur due to an extra proxy layer. You can set a breakpoint in the DefaultListableBeanFactory class. The beanDefinitionNames member variable lists the beans registered with Spring, where you can check if a bean is proxied by an aspect. This issue can occur if a faulty internal library is indirectly imported. To resolve this issue, exclude the library.

If these solutions do not resolve the issue, debug the ThreadContainer.start method. If a class.forName error is reported but the class exists, this can happen in certain frameworks where the class loaders for your business logic and SchedulerX are different. To resolve this issue, call the SchedulerxWorker.setClassLoader method.

"Unable to make field private" error

MapReduce relies on serialization and deserialization frameworks. Java 9 and later restrict reflective access to private fields by default. Add the following JVM parameter:

--add-opens java.base/java.lang=ALL-UNNAMED

"submit jobInstanceId to worker timeout" error

You can ignore this error if it occurs during an application release or only occasionally.

If the error persists and the workerAddr value is always the same machine, the persistent connection between the server and the client is broken. Restart the worker node or upgrade the SchedulerX client to the latest version. After you upgrade, the client can automatically recover disconnected persistent connections.

"used space beyond 90.0%!" error

The disk is full. Clear disk space on the ECS or container.

Job fails with a "ClassNotFoundException" error

The class does not exist on the worker that runs the job. Ensure that the Fully Qualified Class Name configured for the Java job is the fully qualified class name, not an abbreviated name.

If the configured Fully Qualified Class Name is correct, the class is not present on the worker, typically because an incorrect package was deployed or the application is connected to another user's machine. Log on to the worker and use a decompiler to verify.

"jobInstance=xxx don't update progress more than 60s" error

When a worker running a job stops or is released and does not report progress for more than 60 seconds, the server forcibly terminates the job. Ignore this error if you intentionally stopped the worker or it is no longer available.

Job fails without an error message

• Symptom:

  A job fails to run, and no error message is displayed.

• Possible causes:

  Machine issues or business logic failures.

• Solution:

  • Go to the Instances page, click the Task instance List tab, find the job, and click Details in the Actions column. On the Task instance details page, find the machine where the job failed.

    If no task details are available, the job is a standalone job. Check the workAddr value in the basic information. The address indicates the business machine that runs the job.

  • Log on to the business machine and open the ~/logs/schedulerx/worker.log file.

    Run the grep <instance ID> worker.log command to view logs for the instance. If an ERROR-level exception is found, check the thread stack for the specific cause.

  • If the error description is empty, the business logic has likely failed without returning an error message. Troubleshoot your business logic first.

  • If the error description contains a framework exception, join the DingTalk group (ID: 23103656) to contact SchedulerX technical support.

Troubleshooting job failures

• If a standalone job throws an exception, go to the Instances page, click the Task instance List tab, find the target task instance, and then click Details in the Actions column to view the error message.

• If the job does not throw an exception or is a distributed job, you can use Log Service in the Professional Edition to troubleshoot the issue.

• If you use the Basic Edition, log on to the worker node and check the SchedulerX and business logs to troubleshoot the issue.

Job gets stuck during execution

• Symptom:

  A scheduled job remains in the Running state and does not complete.

• Possible causes:

  • An issue with your business logic.

  • An issue with SchedulerX.

• Solution: Troubleshoot service-related issues with the solutions below. For other issues, contact SchedulerX technical support by joining the DingTalk group (Group ID: 23103656).

  • For Professional Edition applications, use the View Stack feature in the console to check the thread stack of the stuck job. This feature is available in client versions 1.4.2 and later.

  • For Basic Edition applications, log on to the stuck worker node and use the jstack command to view the thread stack. Run the following command:

    jstack <pid> | grep <task instance id> -A 20

    

Troubleshooting slow jobs

Enable Tracing Analysis in the Professional Edition. For more information, see Enable Tracing Analysis.

Task instance limit reached

• Symptom:

  On the job management page, clicking Run once displays the message The number of running job instances has reached the upper limit. Please try again later..

• Possible causes:

  • A task instance of the job is already running.

  • The number of running task instances has reached the maximum instance concurrency configured for the job.

• Solution:

  • If the instance concurrency is set to a reasonable value, no action is required. On the job management page, choose More > Historical records to view the running task instances.

  • If the value is not appropriate, click Edit in the Actions column for the target job and set the Instance concurrency in the advanced configuration section.

Subsequent run behavior for an unfinished job

• The default instance concurrency is 1, which means jobs run serially. If a job takes a long time to complete and has not finished when the next run is scheduled, the next run is dropped, not queued.

• If you set the instance concurrency to 2 and the previous job has not finished when the next run is scheduled, a new instance can still run. A maximum of two task instances can run concurrently.

Setting up a one-time job

SchedulerX 2.0 supports one-time jobs. Set Time Type to one_time. One-time jobs do not retain execution records.

Viewing the history of one-time jobs

A one-time job is automatically destroyed after it runs to prevent data accumulation, and no historical records are retained. To save historical records, enable Log Service, which retains execution logs of all jobs from the last two weeks for troubleshooting. For more information, see Application management.

Configuring second-delay scheduling

SchedulerX supports second-delay scheduling. cron and fix_rate do not support second-level scheduling. Set the time type to second_delay to run the job again a specified number of seconds after the previous run completes.

Job not scheduled at the expected time

If a standalone job is not scheduled to run at a specific time, check if there are machines in the machine list and if all machines are busy. If no machines are available or all workers are busy, troubleshoot the issue. For more information, see No worker available and What to do if all workers are busy?.

Configure an alarm for the "no worker available" event for your job. For more information, see Job management.

Setting a timeout in SchedulerX

SchedulerX supports only job-level timeouts, not task-level timeouts. You can modify the timeout period dynamically in the console. For more information, see Job management.

Task continues to run after being stopped

• Symptom: A task instance continues to run after being stopped.

• Possible cause: When a task instance is stopped, SchedulerX sends a kill message to the client. Upon receiving this message, the client stops dispatching new tasks and destroys the instance context, including its thread pools. However, tasks already in progress are not immediately terminated. Their threads are only interrupted, allowing the tasks to run to completion.

• Solution:

  • In most cases, no action is required. Wait for the tasks to complete.

  • To ensure tasks terminate immediately when a job is stopped, modify your task processing logic to handle the Interrupt state of the current thread.

Using advanced job configurations

For more information, see Advanced job configuration parameters.

All workers are busy

On the application management page, view instances to locate the busy worker. Then, click Busy to view the metrics that have exceeded the configured threshold.

The busy threshold is configured in the Edit Application Group panel on the Application Management page.

If the workers are busy due to high load, check whether your application is deployed in a container (Kubernetes). If it is, you must configure the following two parameters. Otherwise, the collected CPU utilization may be inaccurate. For more information, see Connect a Spring Boot application to SchedulerX.

Enabling tracing analysis

Job scheduling supports end-to-end tracing. For more information, see Enable Tracing Analysis.

Job becomes stuck or slow during release

• Symptom: A job gets stuck or slows down during an application release.

• Possible cause: For a distributed job, if a machine that processes tasks goes offline, the tasks are redistributed. During this period, the system polls to check if machines are online, which can slow down the entire process.

• Solution: Upgrade the client to the latest version. This issue is addressed in versions 1.7.9 and later.

Handling the instance parameters prompt

On the job management page, click Run once in the Actions column to run the job immediately. In the dialog box that appears, you can select a specific machine instance and specify Instance Parameters. This parameter is optional and primarily used for testing.

@Component
public class JavaDemoProcessor extends JavaProcessor {


    private static final Logger LOGGER = LoggerFactory.getLogger("schedulerxLog");


    @Override
    public ProcessResult process(JobContext jobContext) throws InterruptedException {

        LOGGER.info(JSON.toJSONString(jobContext));
        // Get job parameters
        String jobParameters = jobContext.getJobParameters();
        // Get instance parameters
        String instanceParameters = jobContext.getInstanceParameters();
        LOGGER.info("Job parameters: " + jobParameters);
        LOGGER.info("Instance parameters: " + instanceParameters);
        return new ProcessResult(InstanceStatus.SUCCESS);
    }

}

@Component
public class JavaDemoProcessor extends JavaProcessor {

    private static final Logger LOGGER = LoggerFactory.getLogger("schedulerxLog");

    @Override
    public ProcessResult process(JobContext jobContext) throws InterruptedException {
        String params = null;
        if (StringUtils.isNotBlank(jobContext.getInstanceParameters())) {
            params = jobContext.getInstanceParameters();
        } else {
            params = jobContext.getJobParameters();
        }
        LOGGER.info("JavaDemoProcessor params:{}", params);
        return new ProcessResult(InstanceStatus.SUCCESS);
    }

}