Configure datasets for Python/Shell nodes

Limits

• You can add a dataset only after you purchase the unstructured data feature for the current instance.

• Only Python and Shell nodes in Basic projects support adding datasets. You can add a maximum of five datasets to each node.

Procedure

1. In the top menu bar of the Dataphin homepage, choose Develop > Data Development.

2. On the Develop page, select a Project from the top menu bar. In Dev-Prod mode, select an Environment.

3. In the left navigation pane, select Data Processing > Compute Job.

4. In the list of compute nodes, click the target Python or Shell node to open its tab.

5. In the sidebar on the right, click Property to open the Property panel. In the Dataset area, click Add Dataset and configure the following parameters.

   • Dataset: Select a file dataset or a mixed dataset from the current project.

   • Version: Select any version of the selected dataset.

     Note

     You cannot select the same version of a dataset more than once.

   • Mount Path: The mount path for the selected dataset version is automatically entered. You can modify this path. The path must start with /mnt/data/. When you reference multiple datasets, their mount paths cannot be nested. For paths like /mnt/data/{dir}, the {dir} directories cannot contain one another.

   • Read-only: This option is disabled by default, which allows read and write operations. If you enable this option, only read operations are supported.

   You can view or delete the datasets you added.

   • View: Click the View icon to open the dataset editing page. On this page, you can view the dataset information, modify its basic information, and edit its versions. For more information about editing, see Datasets.

   • Delete: Click the Delete icon to delete the corresponding dataset.

   After you reference a dataset, its path is mounted to the specified mount path in the node's runtime environment. The development path is used in the development environment, and the production path is used in the production environment. In your code, you can use the mount path as a local path.