DocsICP FilingConsole
Documentation
Home Page Dataphin Intelligent Data construction and governance Dataphin-sharing mode (fully managed version) User Guide Data ingestion Data integration Offline Integration Widget library development Input component Configure the Milvus input component

Configure the Milvus input component

Updated at:
Copy as MD
Product
My Favorites

The Milvus input component reads data from a Milvus data source. To synchronize data from a Milvus source to a different destination, configure this input component to read from the source and an output component for the destination.

Prerequisites

  • You have created a Milvus data source. For more information, see .

  • Your account must have the read permission for synchronization for the data source. If not, apply for this permission. For more information, see Apply for, renew, and return data source permissions.

Procedure

  1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

  2. In the top menu bar of the integration page, select Project (or an environment if you are in Dev-Prod mode).

  3. In the navigation pane on the left, click Batch integration. In the Batch integration list, click the batch pipeline to open its configuration page.

  4. In the upper-right corner of the page, click Component library to open the Component library panel.

  5. In the navigation pane on the left of the Component library panel, select Input. In the list of input components on the right, find the Milvus component and drag it onto the canvas.

  6. Click the image icon on the Milvus input component card to open the Milvus input configuration dialog box.

  7. In the Milvus input configuration dialog box, configure the following parameters.

    Parameter

    Description

    Step name

    The name for the Milvus input component. Dataphin automatically generates a name, but you can change it.

    • Contain only Chinese characters, letters, underscores (_), and digits.

    • Be no more than 64 characters long.

    Data source

    Select the Milvus data source. The list shows all available Milvus data sources, regardless of your permissions. You can click the image icon to copy the current data source name.

    • For a data source where you lack the read permission for synchronization, click Apply next to the data source to request the required permission. For more information, see Apply for, renew, and return data source permissions.

    • If you do not have a Milvus data source, click New Data Source to create one. For more information, see .

    Database (Optional)

    The database where the collection resides. If you leave this blank, Dataphin uses the database specified when the data source was registered.

    Collection

    The collection in Milvus that stores the vector data.

    Partition (Optional)

    Specify the partition to read from, for example, ds=1000. If you leave this blank, data is read from all partitions.

    Split key (Optional)

    The column used to shard data for parallel reads. Specify a column from the source collection. For optimal performance, use a primary key or an indexed column.

    Important

    If you select a column of a date or time data type, the system identifies the maximum and minimum values and performs a brute-force split based on the total time range and concurrency level. This process may not split the data evenly.

    Batch size (Optional)

    The number of records to read in a single batch. Example: 1024. Batch reading reduces interactions with the data source, improving I/O efficiency and network latency.

    Input filter (Optional)

    Enter a filter expression based on the Milvus filter syntax. You do not need to include the filter keyword. System variables are supported. Example: status = 'active' AND create_time >= '${bizdate}'.

    Consistency level

    • Strong: Ensures you read the most up-to-date data, at the cost of lower query performance.

    • Eventually: Provides faster query performance, but reads may not return recently inserted data.

    Output field

    The Output field area displays all fields from the selected table that match the filter criteria. You can perform the following operations:

    • Field management: If you do not need to output certain fields to downstream components, you can delete the corresponding fields:

      • Deleting fields individually: If you need to delete a small number of fields, you can click the sgaga icon in the Actions column to delete the unnecessary fields.

      • Bulk deleting fields: To delete a large number of fields, click Field Management. In the Field Management dialog box, select multiple fields, click the image left-arrow icon to move the selected input fields to the unselected input fields, and then click OK to complete the bulk deletion.

        image..png

    • Batch Add: Click Batch Add to configure items in batches by using the JSON, TEXT, or DDL format.

      Note

      Clicking OK to complete the batch addition will overwrite the configured field information.

      • Batch configure in JSON format. For example:

        [
    {
        "index": 1,
        "name": "product_id",
        "type": "Int64",
        "mapType": "Long",
        "comment": ""
    },
    {
        "index": 2,
        "name": "product_name",
        "type": "VarChar",
        "mapType": "String",
        "comment": ""
    },
    {
        "index": 3,
        "name": "price",
        "type": "Float",
        "mapType": "Double",
        "comment": ""
    },
    {
        "index": 4,
        "name": "tags",
        "type": "Array",
        "mapType": "String",
        "comment": ""
    },
    {
        "index": 5,
        "name": "ratings",
        "type": "Array",
        "mapType": "String",
        "comment": ""
    },
    {
        "index": 6,
        "name": "warehouse_stock",
        "type": "Array",
        "mapType": "String",
        "comment": ""
    },
    {
        "index": 7,
        "name": "embedding",
        "type": "FloatVector",
        "mapType": "String",
        "comment": ""
    }
]
        Note

        index specifies the column number of the specified object, name specifies the name of the field after it is imported, and type specifies the type of the field after it is imported.

        For example, "index":3,"name":"user_id","type":"String" indicates that the fourth column of the file is imported, the field name is user_id, and the field type is String.

      • Configure multiple items at once in TEXT format. For example:

        1,product_id,Int64,Long,
2,product_name,VarChar,String,
3,price,Float,Double,
4,tags,Array,String,
5,ratings,Array,String,
6,warehouse_stock,Array,String,
7,embedding,FloatVector,String,

        • A row delimiter is used to separate the information in each field. The default is the newline character (\n). Supported delimiters are the newline character (\n), a semicolon (;), and a period (.).

        • The column delimiter is used to separate field names from field types. The default is a comma (,).The supported delimiter is ','. The field type is optional, and the default is ','.

      • Configure in batches using DDL format. For example:

        CREATE TABLE tablename (
	user_id serial,
	username VARCHAR(50),
	password VARCHAR(50),
	email VARCHAR (255),
	created_on TIMESTAMP,
);

    • Create an output field: Click + Create Output Field, and then enter the Field, Type, and Description and select a Mapping Type. After you complete the configuration for the current row, click the image icon to save.

  8. Click Confirm to complete the property configuration for the Milvus input component.

Previous：Configure the TDSQL for PostgreSQL input componentNext：Output component