Semantic vector distance-Platform For AI(PAI)-阿里云帮助中心

The Semantic Vector Distance component finds the nearest neighbors of word vectors generated by word embedding models such as Word2Vec. It computes pairwise distances between vectors using Euclidean distance, cosine similarity, or Manhattan distance, and returns the top-N closest matches. Common use cases include synonym generation, text similarity computation, and semantic search.

Configure the component

Method 1: Configure the component on the pipeline page

Add a Semantic Vector Distance component on the pipeline page and configure the following parameters:

Category	Parameter	Description
Fields setting	ID column	The name of the ID column. Leave blank to use all vectors in the input table for calculation. The ID column maps to the ID list imported through the second input port, where each ID occupies a cell. Example:<br>`1`<br>`2`<br>`4`<br>`6`<br>`8`
Fields setting	Vector columns	The names of columns that contain vectors. Example: `f1,f2`.
Parameters setting	Number of closest vectors to output	The number of nearest vectors returned per query vector. Default value: `5`.
	Distance calculation mode	The method used to compute the distance between vectors. Valid values: `euclidean`, `cosine`, `manhattan`. Default value: `euclidean`.
	Distance threshold	Filters the output to include only vector pairs whose distance is within this value. Default value: `+∞` (no filtering). For example, if a cosine similarity threshold is set at `0.8`, only vectors with a similarity greater than or equal to `0.8` will be regarded as similar vectors.
Tuning	Computing cores	The number of cores used for calculation. Automatically allocated.
Tuning	Memory size per core (unit: MB)	The memory size per core. Automatically allocated.

Method 2: Use PAI commands

Run the component using a PAI command in the SQL Script component. For more information, see SQL Script.

PAI -name SemanticVectorDistance
    -project algo_public
    -DinputTableName="test_input"
    -DoutputTableName="test_output"
    -DidColName="word"
    -DvectorColNames="f0,f1,f2,f3,f4,f5"
    -Dlifecycle=30

Parameter	Required	Default value	Description
`inputTableName`	Yes	None	The name of the input table.
`inputTablePartitions`	No	All partitions	The partitions selected from the input table for calculation.
`outputTableName`	Yes	None	The name of the output table.
`idTableName`	No	None	The name of the vector ID table. The table must contain a single column where each row holds one vector ID. Leave blank to use all vectors in the input table.
`idTablePartitions`	No	All partitions	The partitions selected from the ID table for calculation.
`idColName`	Yes	3	The name of the ID column.
`vectorColNames`	No	None	The names of columns that contain vectors. Example: `f1,f2`.
`topN`	No	`5`	The number of nearest vectors returned per query vector. Valid values: `[1, +∞)`.
`distanceType`	No	`euclidean`	The method used to compute the distance between vectors. Valid values: `euclidean`, `cosine`, `manhattan`.
`distanceThreshold`	No	`+∞`	The maximum distance value for a vector pair to be included in the output. Valid values: `(0, +∞)`.
`lifecycle`	No	None	The lifecycle of the output table. Must be a positive integer.
`coreNum`	No	System-determined	The number of cores used for calculation. Must be a positive integer.
`memSizePerCore`	No	System-determined	The memory size per core in MB. Must be a positive integer.

Sample output

The output table contains four columns.

Column	Description
`original_id`	The ID of the query vector
`near_id`	The ID of a neighboring vector
`distance`	The computed distance between the two vectors
`rank`	The rank of this neighbor (1 = closest)

Example output:

original_id	near_id	distance	rank
hello	hi	0.2	1
hello	xxx	xx	2
Man	Woman	0.3	1
Man	xx	xx	2