Deploy ossimport in standalone mode

更新时间:
复制 MD 格式

ossimport migrates data to Object Storage Service (OSS). In standalone mode, the Master, Worker, TaskTracker, and Console modules run within a single JAR package on one machine. This mode supports Linux and Windows.

Prerequisites

Ensure the following requirements are met:

  • Java 1.7 or 1.8 is installed.

Download and configure ossimport

  1. Download and decompress ossimport-2.3.7.zip. After decompression, the directory structure is as follows:

       ossimport
       ├── bin
       │   └── ossimport2.jar  # JAR package containing the Master, Worker, TaskTracker, and Console modules
       ├── conf
       │   ├── local_job.cfg   # Migration job configuration file
       │   └── sys.properties  # System parameters configuration file
       ├── console.bat         # Windows command-line tool for running tasks
       ├── console.sh          # Linux command-line tool for running tasks
       ├── import.bat          # Windows one-click import script (start, migration, verification, and retry)
       ├── import.sh           # Linux one-click import script (start, migration, verification, and retry)
       ├── logs                # Log directory
       └── README.md           # ossimport description file
  2. Edit conf/sys.properties and conf/local_job.cfg for your migration. Do not modify the following items. Configuration file examples. > Important: Verify all parameters in sys.properties and local_job.cfg before submitting a job. Parameters cannot be changed after submission.

    • The workingDir, workerUserName, workerPassword, and privateKeyFile parameters in conf/sys.properties

    • The name and path of the conf/local_job.cfg file

    • The jobName parameter in conf/local_job.cfg

Run a migration job

ossimport provides two ways to run migration jobs:

  • One-click import -- Runs all migration steps through a single script. Best for first-time users.

  • Step-by-step import -- Runs each migration step individually through Console commands.

One-click import

  1. Run the import script. > Note: If the last job is incomplete, ossimport prompts you to resume from the checkpoint or restart. Choose restart if you want to run a different migration job or if you changed the migration source or destination.

    • Windows: Double-click import.bat.

    • Linux: Run bash import.sh.

  2. Monitor the job progress.

    • Windows: A new Command Prompt window opens to run the job and display logs. ossimport displays job status every 10 seconds. Do not close either window during execution.

    • Linux: The job runs in the background.

  3. Handle failed tasks. If a task fails, ossimport prompts you to retry. Enter y to retry or n to skip and exit.

  4. Investigate failures. Check master/jobs/local_test/failed_tasks/<taskid>/audit.log to identify the cause.

Step-by-step import

Console commands for step-by-step import:

Step Command Windows Linux
1. Clear existing jobs clean console.bat clean bash console.sh clean
2. Submit a migration job submit console.bat submit bash console.sh submit
3. Start the service start console.bat start bash console.sh start
4. View job status stat console.bat stat bash console.sh stat
5. Retry failed tasks retry console.bat retry bash console.sh retry
6. Stop the service stop Close %JAVA_HOME%/bin/java.exe window bash console.sh stop

Step 1: Clear existing jobs

Run clean only when reusing a job name from a previous run. Do not run clean for new jobs or when retrying failed tasks.

Step 2: Submit a migration job

The job configuration file is conf/local_job.cfg. The default job name is local_test. Ensure the job name is unique before submitting.

Step 3: Start the service

Start ossimport to process submitted jobs.

Step 4: View job status

Check the current status of the migration job.

Step 5: Retry failed tasks

Tasks may fail due to network issues or other reasons. The retry command retries only failed tasks.

Step 6: Stop the service

  • Windows: Close the %JAVA_HOME%/bin/java.exe window.

  • Linux: Run bash console.sh stop.

View job status and logs

After a job is submitted, the Master splits it into tasks, the Worker executes the tasks, and the TaskTracker collects task status. The following directory structure shows key files generated after a job completes:

ossimport
├── bin
│   └── ossimport2.jar          # JAR package for standalone mode
├── conf
│   ├── local_job.cfg           # Migration job configuration file
│   └── sys.properties          # System parameters configuration file
├── console.sh                  # Command-line tool
├── import.sh                   # One-click import script
├── logs
│   ├── import.log              # Migration logs
│   ├── job_stat.log            # Job status logs
│   ├── ossimport2.log          # ossimport run logs for standalone mode
│   └── submit.log              # Job submission logs
├── master
│   ├── jobqueue                # Jobs not yet split into tasks
│   └── jobs
│       └── local_test          # Job name
│           ├── checkpoints     # Checkpoints from Master splitting the job into tasks
│           │   └── 0
│           │       └── 034DC9DD2860B0CFE884242BC6FF92E7.cpt
│           ├── dispatched      # Tasks dispatched to the Worker but not yet complete
│           │   └── localhost
│           ├── failed_tasks    # Failed tasks
│           ├── pending_tasks   # Tasks not yet dispatched
│           └── succeed_tasks   # Successfully completed tasks
│               └── A41506C07BF1DF2A3EDB4CE31756B93F_1499744514501@localhost
│                   ├── audit.log   # Task run logs (use for error diagnosis)
│                   ├── DONE        # Mark file indicating task completion
│                   ├── error.list  # Task errors
│                   ├── STATUS      # Task status: Failed or Completed
│                   └── TASK        # Task description
└── worker                      # Tasks currently running (managed by Master after completion)
    └── jobs
        └── local_test
            └── tasks
Important
  • To view job information, check logs/ossimport2.log or logs/import.log.

  • To troubleshoot a failed task, check master/jobs/${JobName}/failed_tasks/${TaskName}/audit.log.

  • To view errors for a failed task, check master/jobs/${JobName}/failed_tasks/${TaskName}/error.list.

  • These logs are for reference only. Do not base your service or application deployment on these logs.

Verify migration results

ossimport does not verify data consistency or integrity after migration. After a job completes, verify that the data at the source and destination match.

Warning

If you delete source data without verifying data consistency, you are responsible for any resulting losses.

Troubleshooting

Common causes of migration failures:

Failure scenario Cause Details
SIZE_NOT_MATCH error in log/audit.log Source data modified during migration The original data was uploaded, but subsequent modifications were not uploaded to OSS.
Migration job fails Source data deleted during migration The migration job cannot proceed without the source data.
Migration to OSS fails Source file names violate OSS naming rules Object names in OSS cannot start with a forward slash (/) and cannot be empty.
Source data fails to download Network issues or insufficient permissions Check logs/ossimport2.log or logs/import.log for details.
Job state shows Abort ossimport exits unexpectedly during migration Contact .