Import data from OSS

Prerequisites

• Create a GDB instance. For more information, see Create a primary instance.
• Create an OSS bucket. For more information, see Create buckets in the console.
• Obtain the ARN of the service-linked role. For more information, see Authorize GDB to access OSS.
• Obtain the connection endpoint and port of the GDB instance. For more information, see View the information of a GDB instance.

  If you import data from a local client, you must connect using a public IP address. For instructions on how to apply for one, see Apply for a public endpoint.

• Create a database account. For more information, see Create an account.
• Add the IP address of the client that you will use to access GDB to an allowlist. For more information, see Configure an allowlist.

Prepare test data

This topic uses the air-routes dataset as an example. GDB data is divided into vertex files and edge files. You can download them from the following links:

• Download vertex files.
• Download edge files.

You can also use your own data for the import test. GDB has specific format requirements for CSV data files. For more information, see CSV data file format.

Upload data to OSS

This section describes how to upload data using the OSS console. To upload data by using ossutil, see Use ossutil to upload data files.

1. Log on to the OSS console.
2. In the left-side navigation pane, click Bucket List and click the name of the destination bucket.
3. Click Upload File.
4. In the Upload File panel, configure the storage directory and permissions, and select the files to upload.
5. Click Upload File.

After the upload is complete, you can view the file path on the File Management page. You will need this path when you import the data into GDB.

In this example, the vertex files are in the gdbOssDemo/node path, and the edge files are in the gdbOssDemo/edge path. If the bucket name is gdbbucket, the OSS paths for the vertex and edge files are as follows:

• Vertex files:

  oss://gdbbucket/gdbOssDemo/node/

• Edge files:

  oss://gdbbucket/gdbOssDemo/edge/

Import data into GDB

The following example shows how to import data into GDB. For more information about the parameters, see Use cURL commands to import data files.

1. On your client, such as an Elastic Compute Service (ECS) instance or a local client, run the curl command to import the vertex files. You can use either ramRoleArn or accessKey and secretKey for authentication.

   curl -u gdb_user:PassWord -X POST \
   -H 'Content-Type: application/json' \
   http://gds-*****.graphdb.rds.aliyuncs.com:8182/loader -d '
   {
      "source" : "oss://gdbbucket/gdbOssDemo/node/",
      "ramRoleArn" : "acs:ram::1***************:role/aliyunserviceroleforgdb"
   }'

   The ramRoleArn method is limited by the 10-hour validity period of a temporary security token (STS). Exceeding this period interrupts the task and results in an incomplete data import. For large data imports, use the accessKey and secretKey method.

   curl -u gdb_user:PassWord -X POST \
   -H 'Content-Type: application/json' \
   http://gds-*****.graphdb.rds.aliyuncs.com:8182/loader -d '
   {
      "source" : "oss://gdbbucket/gdbOssDemo/node/",
      "accessKey": "XXXXXX",
      "secretKey": "XXXXXX"
   }'

   Note Replace the parameters in the examples with your actual database information. If you use a local client, use the public IP address to connect.

   After the GDB loader verifies the parameters, it returns the import task ID, loadId. Sample response:

   {
     "status" : "200 OK",
     "payload" : {
         "loadId" : "552617AF-4F1E-4CD8-9533-************"
     }
   }

2. On your client, run the curl command to import the edge files. Sample command:

   curl -u gdb_user:PassWord -X POST \
   -H 'Content-Type: application/json' \
   http://gds-*****.graphdb.rds.aliyuncs.com:8182/loader -d '
   {
      "source" : "oss://gdbbucket/gdbOssDemo/edge/",
      "ramRoleArn" : "acs:ram::1***************:role/aliyunserviceroleforgdb"
   }'

   Note Replace the parameters in the example with your actual database information. If you use a local client, use the public IP address to connect.

   After the GDB loader verifies the parameters, it returns the import task ID, loadId. Sample response:

   {
     "status" : "200 OK",
     "payload" : {
         "loadId" : "552617AF-4F1E-4CD8-9533-************"
     }
   }

After starting the import, you can use the loadId to check the task's status. The following command is an example:

curl -u gdb_user:PassWord -X GET 'http://gds-*****.graphdb.rds.aliyuncs.com:8182/loader/<loadId>'

FAQ

• Q: When I re-import the same file, the system returns the LOAD_NOT_SUPPORT TO RESUME LOAD error.

  A: Delete the loadId from the previous import. The following command deletes a loadId:

  curl -u gdb_account:******** -X DELETE 'http://gds-123.graphdb.rds.aliyuncs.com:****/loader/552617AF-4F1E-4CD8-9533-A2EC154688DC'

  If you have many loadIds to delete, you can use the following script to delete them in bulk:

  #!/bin/bash
  GDB_HOST=YOU_GDB_HOST
  GDB_PORT=YOU_GDB_PORT
  GDB_USER=YOUR_GDB_USER
  GDB_PASSWORD=YOUR_GDB_PASSWORD
  curl -u "${GDB_USER}:${GDB_PASSWORD}" "http://${GDB_HOST}:${GDB_PORT}/loader"| python3 -c 'import json,sys;print("\n".join(json.load(sys.stdin)["payload"]["loadIds"]))'| while read TASK;
  do
    echo delete task: $TASK
    curl -u "${GDB_USER}:${GDB_PASSWORD}" -XDELETE  "http://${GDB_HOST}:${GDB_PORT}/loader/${TASK}"
  done
  curl -u "${GDB_USER}:${GDB_PASSWORD}" "http://${GDB_HOST}:${GDB_PORT}/loader"

• Q: When I import a file, the system returns the Commit record Error(21-0) error.

  A: You need to set the import parameter failOnError to false. For more information, see Import API Introduction.

For more information about error messages, see List of error messages.