Monitor GPUs with OpenClaw

Scenario

Monitoring a long-running training task, such as a fine-tuning task that takes several hours, typically requires periodic status checks. To eliminate manual monitoring, configure OpenClaw to automatically send a DingTalk message when the training task finishes.

Method

Add the OpenClaw message sending command to the end of your training script to automatically send a notification when the training task completes or fails.

Basic training script example

# Run the training command
python train.py --model qwen-7b --lr 2e-5 --epochs 3

# Send a notification after training ends ($? is the exit code of the training script)
openclaw message send --channel dingtalk --target "user:<staffid>" -m "Training task finished with exit code: $?"

Enhanced training script example (with result information)

python train.py --model qwen-7b --lr 2e-5 --epochs 3
EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ]; then
  FINAL_LOSS=$(tail -n 5 train.log | grep -oP 'loss=\K[\d.]+' | tail -1)
  openclaw message send --channel dingtalk --target "user:<staffid>" -m "Training complete! Final loss: ${FINAL_LOSS}. Please check the results."
else
  openclaw message send --channel dingtalk --target "user:<staffid>" -m "Training task terminated unexpectedly with exit code: ${EXIT_CODE}. Please check the log file."
fi

Scenario

During execution, a training task may encounter anomalies such as a sudden drop in GPU utilization, video memory usage nearing its limit, or abnormal temperatures. These issues often signal that the training task is failing or running inefficiently. By setting up a scheduled task for monitoring, OpenClaw can send timely alerts when anomalies occur, preventing wasted computing power.

Method

Create a GPU monitoring script at /mnt/workspace/gpu_monitor.py:

import subprocess
import sys

def get_gpu_stats():
    """Get key GPU metrics."""
    result = subprocess.run(
        ['nvidia-smi', '--query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu',
         '--format=csv,noheader,nounits'],
        capture_output=True, text=True
    )
    stats = []
    for i, line in enumerate(result.stdout.strip().split('\n')):
        util, mem_used, mem_total, temp = [x.strip() for x in line.split(',')]
        stats.append({
            'gpu_id': i,
            'utilization': int(util),
            'memory_used': int(mem_used),
            'memory_total': int(mem_total),
            'memory_pct': round(int(mem_used) / int(mem_total) * 100, 1),
            'temperature': int(temp)
        })
    return stats

def check_anomaly(stats):
    """Check for anomalies and return a list of alert messages."""
    alerts = []
    for s in stats:
        gid = s['gpu_id']
        # Low GPU utilization (training tasks should maintain high utilization)
        if s['utilization'] < 20:
            alerts.append(f"GPU {gid} utilization is only {s['utilization']}%, which may indicate a blocked task.")
        # Video memory is nearing the limit
        if s['memory_pct'] > 90:
            alerts.append(f"GPU {gid} video memory usage is at {s['memory_pct']}%, risk of OOM.")
        # Temperature is too high
        if s['temperature'] > 85:
            alerts.append(f"GPU {gid} temperature is {s['temperature']}°C. Check cooling.")
    return alerts

if __name__ == '__main__':
    stats = get_gpu_stats()
    alerts = check_anomaly(stats)

    if alerts:
        # If anomalies are detected, print the alerts and return a non-zero exit code.
        for alert in alerts:
            print(alert)
        sys.exit(1)
    else:
        print("HEARTBEAT_OK")
        sys.exit(0)

Configure a scheduled monitoring task in OpenClaw:

openclaw cron add \
  --name "GPU Health Check" \
  --cron "*/15 * * * *" \
  --tz "Asia/Shanghai" \
  --session isolated \
  --message "Perform a GPU health check:

1. Run the command: python /mnt/workspace/gpu_monitor.py
2. If the output is HEARTBEAT_OK, all is well. Do not notify me.
3. If an anomaly is detected, do the following:
   - Send me the complete alert message via DingTalk.
   - Assess the current training process status and determine if my intervention is needed.
   - Provide your recommended actions." \
  --announce

Verify the scheduled task

# View all scheduled tasks
openclaw cron list

# Manually trigger a test run (replace <job_id> with the actual job ID)
openclaw cron run <job_id>

Scenario

CUDA out of memory (OOM) is a common problem in model training. By configuring an OOM auto-recovery mechanism, OpenClaw can automatically adjust training parameters, such as reducing the batch_size, and restart the training task when an OOM error is detected. This process requires no manual intervention.

Method

Create a training configuration file at /mnt/workspace/train_config.json:

{
  "model": "qwen-7b",
  "batch_size": 16,
  "lr": 2e-5,
  "epochs": 3,
  "max_length": 512,
  "oom_retry_count": 0
}

Create an OOM auto-recovery script at /mnt/workspace/auto_recover.py:

import json
import subprocess
import sys
from pathlib import Path

CONFIG_PATH = Path("/mnt/workspace/train_config.json")
LOG_PATH = Path("/mnt/workspace/train.log")

def load_config():
    return json.loads(CONFIG_PATH.read_text())

def save_config(config):
    CONFIG_PATH.write_text(json.dumps(config, indent=2))

def check_oom():
    """Check for OOM errors in recent logs."""
    if not LOG_PATH.exists():
        return False
    recent_lines = LOG_PATH.read_text().split('\n')[-50:]
    oom_keywords = ['CUDA out of memory', 'OutOfMemoryError', 'OOM']
    return any(kw in line for line in recent_lines for kw in oom_keywords)

def recover():
    config = load_config()

    if config['oom_retry_count'] >= 3:
        print(f"Auto-retried {config['oom_retry_count']} times and reduced batch_size to {config['batch_size']}, but OOM still occurs. Manual intervention required.")
        sys.exit(2)  # Exit code 2 indicates that manual intervention is required.

    # Halve the batch_size
    old_batch = config['batch_size']
    config['batch_size'] = max(1, old_batch // 2)
    config['oom_retry_count'] += 1
    save_config(config)

    print(f"OOM detected. Adjusted batch_size from {old_batch} to {config['batch_size']} (auto-recovery attempt #{config['oom_retry_count']}). Restarting training task...")

    # Restart the training task (run in the background)
    subprocess.Popen([
        'python', 'train.py',
        '--batch_size', str(config['batch_size']),
        '--lr', str(config['lr']),
        '--epochs', str(config['epochs'])
    ], stdout=open(str(LOG_PATH), 'a'), stderr=subprocess.STDOUT)

    sys.exit(0)

if __name__ == '__main__':
    if check_oom():
        recover()
    else:
        print("HEARTBEAT_OK")

Configure an OpenClaw OOM monitoring task:

openclaw cron add \
  --name "OOM Auto-Recovery Guard" \
  --cron "*/5 * * * *" \
  --tz "Asia/Shanghai" \
  --session isolated \
  --message "Execute an OOM self-healing check:

1. Run: python /mnt/workspace/auto_recover.py
2. If the output is HEARTBEAT_OK, no action is needed.
3. If the output includes 'Adjusted batch_size', notify me via DingTalk with the adjustment details and confirm that training has been automatically restarted.
4. If the exit code is 2 (manual intervention required), notify me immediately via DingTalk and include the last 20 lines of the training log." \
  --announce

Automatic recovery workflow

1. An OOM error occurs during the training task.

2. Within 5 minutes, OpenClaw detects the OOM error.

3. It automatically halves the batch_size and restarts the training task.

4. A notification with the result is sent via DingTalk.

5. If auto-recovery fails three consecutive times, OpenClaw notifies you that manual intervention is required.

Scenario

By configuring a daily training report, you can automatically receive a global status summary of your DSW instance at a fixed time each day, such as 9:00 AM. This report includes currently running training tasks, GPU resource usage, and recent experiment results, helping you track the overall training progress.

Method

Create a daily report generation script at /mnt/workspace/daily_report.py:

import subprocess
import json
from datetime import datetime, timedelta
from pathlib import Path

def get_running_jobs():
    """Get the currently running training processes."""
    result = subprocess.run(['ps', 'aux'], capture_output=True, text=True)
    jobs = [line for line in result.stdout.split('\n')
            if 'train.py' in line or 'finetune.py' in line]
    return jobs

def get_gpu_summary():
    result = subprocess.run(
        ['nvidia-smi', '--query-gpu=index,name,utilization.gpu,memory.used,memory.total',
         '--format=csv,noheader'],
        capture_output=True, text=True
    )
    return result.stdout.strip()

def get_experiment_log_summary():
    """Reads the experiment log."""
    log_path = Path("/mnt/workspace/experiment_log.md")
    if not log_path.exists():
        return "No experiment log found."
    # Return the last 30 lines
    lines = log_path.read_text().split('\n')
    return '\n'.join(lines[-30:])

if __name__ == '__main__':
    now = datetime.now().strftime('%Y-%m-%d %H:%M')
    running = get_running_jobs()
    gpu = get_gpu_summary()
    exp_log = get_experiment_log_summary()

    report = f"""=== DSW Daily Report {now} ===

[Currently Running Tasks]
{chr(10).join(running) if running else 'No training tasks are running.'}

[GPU Status]
{gpu}

[Recent Experiment Log]
{exp_log}
"""
    print(report)

Configure the OpenClaw daily report task:

openclaw cron add \
  --name "Daily Training Report" \
  --cron "0 9 * * *" \
  --tz "Asia/Shanghai" \
  --session isolated \
  --message "Generate today's DSW work report:

1. Run: python /mnt/workspace/daily_report.py
2. Based on the output, generate a concise daily report that includes:
   - A brief description of currently running tasks.
   - GPU resource usage.
   - A summary of recent experiment results (if any).
   - A warning about any anomalies found.
3. Send this report via DingTalk in a clean, readable format." \
  --announce

Q: How to troubleshoot DingTalk notifications?

# Check the OpenClaw service status
openclaw status

# Send a test message
openclaw message send -m "Test message. If you receive this, your DingTalk configuration is correct."

Q: Script not running on trigger

# View all scheduled tasks
openclaw cron list

# View the execution logs for a specific task (replace <job_id> with the actual job ID)
openclaw cron logs <job_id> --tail 20

Q: How to bulk disable monitoring tasks?

You can use the following commands to disable or enable all scheduled tasks in bulk. This requires the jq tool to be installed.

# Disable all tasks in bulk
openclaw cron list --json | jq -r '.[].id' | xargs -I {} openclaw cron disable {}

# Enable all tasks in bulk
openclaw cron list --json | jq -r '.[].id' | xargs -I {} openclaw cron enable {}

Alternatively, you can instruct OpenClaw directly to perform the bulk operation:

Disable all my cron scheduled tasks, then re-enable all of them on Monday at 9 AM.