Deploy ossimport in standalone mode
ossimport migrates data to Object Storage Service (OSS). In standalone mode, the Master, Worker, TaskTracker, and Console modules run within a single JAR package on one machine. This mode supports Linux and Windows.
Prerequisites
Ensure the following requirements are met:
-
Java 1.7 or 1.8 is installed.
Download and configure ossimport
-
Download and decompress ossimport-2.3.7.zip. After decompression, the directory structure is as follows:
ossimport ├── bin │ └── ossimport2.jar # JAR package containing the Master, Worker, TaskTracker, and Console modules ├── conf │ ├── local_job.cfg # Migration job configuration file │ └── sys.properties # System parameters configuration file ├── console.bat # Windows command-line tool for running tasks ├── console.sh # Linux command-line tool for running tasks ├── import.bat # Windows one-click import script (start, migration, verification, and retry) ├── import.sh # Linux one-click import script (start, migration, verification, and retry) ├── logs # Log directory └── README.md # ossimport description file -
Edit
conf/sys.propertiesandconf/local_job.cfgfor your migration. Do not modify the following items. Configuration file examples. > Important: Verify all parameters insys.propertiesandlocal_job.cfgbefore submitting a job. Parameters cannot be changed after submission.-
The
workingDir,workerUserName,workerPassword, andprivateKeyFileparameters inconf/sys.properties -
The name and path of the
conf/local_job.cfgfile -
The
jobNameparameter inconf/local_job.cfg
-
Run a migration job
ossimport provides two ways to run migration jobs:
-
One-click import -- Runs all migration steps through a single script. Best for first-time users.
-
Step-by-step import -- Runs each migration step individually through Console commands.
One-click import
-
Run the import script. > Note: If the last job is incomplete, ossimport prompts you to resume from the checkpoint or restart. Choose restart if you want to run a different migration job or if you changed the migration source or destination.
-
Windows: Double-click
import.bat. -
Linux: Run
bash import.sh.
-
-
Monitor the job progress.
-
Windows: A new Command Prompt window opens to run the job and display logs. ossimport displays job status every 10 seconds. Do not close either window during execution.
-
Linux: The job runs in the background.
-
-
Handle failed tasks. If a task fails, ossimport prompts you to retry. Enter
yto retry ornto skip and exit. -
Investigate failures. Check
master/jobs/local_test/failed_tasks/<taskid>/audit.logto identify the cause.
Step-by-step import
Console commands for step-by-step import:
| Step | Command | Windows | Linux |
|---|---|---|---|
| 1. Clear existing jobs | clean |
console.bat clean |
bash console.sh clean |
| 2. Submit a migration job | submit |
console.bat submit |
bash console.sh submit |
| 3. Start the service | start |
console.bat start |
bash console.sh start |
| 4. View job status | stat |
console.bat stat |
bash console.sh stat |
| 5. Retry failed tasks | retry |
console.bat retry |
bash console.sh retry |
| 6. Stop the service | stop |
Close %JAVA_HOME%/bin/java.exe window |
bash console.sh stop |
Step 1: Clear existing jobs
Run clean only when reusing a job name from a previous run. Do not run clean for new jobs or when retrying failed tasks.
Step 2: Submit a migration job
The job configuration file is conf/local_job.cfg. The default job name is local_test. Ensure the job name is unique before submitting.
Step 3: Start the service
Start ossimport to process submitted jobs.
Step 4: View job status
Check the current status of the migration job.
Step 5: Retry failed tasks
Tasks may fail due to network issues or other reasons. The retry command retries only failed tasks.
Step 6: Stop the service
-
Windows: Close the
%JAVA_HOME%/bin/java.exewindow. -
Linux: Run
bash console.sh stop.
View job status and logs
After a job is submitted, the Master splits it into tasks, the Worker executes the tasks, and the TaskTracker collects task status. The following directory structure shows key files generated after a job completes:
ossimport
├── bin
│ └── ossimport2.jar # JAR package for standalone mode
├── conf
│ ├── local_job.cfg # Migration job configuration file
│ └── sys.properties # System parameters configuration file
├── console.sh # Command-line tool
├── import.sh # One-click import script
├── logs
│ ├── import.log # Migration logs
│ ├── job_stat.log # Job status logs
│ ├── ossimport2.log # ossimport run logs for standalone mode
│ └── submit.log # Job submission logs
├── master
│ ├── jobqueue # Jobs not yet split into tasks
│ └── jobs
│ └── local_test # Job name
│ ├── checkpoints # Checkpoints from Master splitting the job into tasks
│ │ └── 0
│ │ └── 034DC9DD2860B0CFE884242BC6FF92E7.cpt
│ ├── dispatched # Tasks dispatched to the Worker but not yet complete
│ │ └── localhost
│ ├── failed_tasks # Failed tasks
│ ├── pending_tasks # Tasks not yet dispatched
│ └── succeed_tasks # Successfully completed tasks
│ └── A41506C07BF1DF2A3EDB4CE31756B93F_1499744514501@localhost
│ ├── audit.log # Task run logs (use for error diagnosis)
│ ├── DONE # Mark file indicating task completion
│ ├── error.list # Task errors
│ ├── STATUS # Task status: Failed or Completed
│ └── TASK # Task description
└── worker # Tasks currently running (managed by Master after completion)
└── jobs
└── local_test
└── tasks
-
To view job information, check
logs/ossimport2.logorlogs/import.log. -
To troubleshoot a failed task, check
master/jobs/${JobName}/failed_tasks/${TaskName}/audit.log. -
To view errors for a failed task, check
master/jobs/${JobName}/failed_tasks/${TaskName}/error.list. -
These logs are for reference only. Do not base your service or application deployment on these logs.
Verify migration results
ossimport does not verify data consistency or integrity after migration. After a job completes, verify that the data at the source and destination match.
If you delete source data without verifying data consistency, you are responsible for any resulting losses.
Troubleshooting
Common causes of migration failures:
| Failure scenario | Cause | Details |
|---|---|---|
SIZE_NOT_MATCH error in log/audit.log |
Source data modified during migration | The original data was uploaded, but subsequent modifications were not uploaded to OSS. |
| Migration job fails | Source data deleted during migration | The migration job cannot proceed without the source data. |
| Migration to OSS fails | Source file names violate OSS naming rules | Object names in OSS cannot start with a forward slash (/) and cannot be empty. |
| Source data fails to download | Network issues or insufficient permissions | Check logs/ossimport2.log or logs/import.log for details. |
Job state shows Abort |
ossimport exits unexpectedly during migration | Contact . |