Document similarity

Choose a similarity method

Each method outputs either a distance (higher = more different) or a similarity (higher = more alike). For distance methods, similarity is derived as Similarity = 1 − Distance.

Distance-to-similarity formulas:

• Levenshtein and LCS: Similarity = 1 − Distance

• Simhash_Hamming: Similarity = 1 − Distance / 64.0

How Simhash_Hamming works: SimHash maps each document to a 64-bit binary fingerprint. The Hamming distance counts the number of characters of binary fingerprints on the same position that differ. See Similarity Estimation Techniques from Rounding Algorithms and Hamming distance (Wikipedia) for background.

Limitations

The Document Similarity component requires MaxCompute computing resources.

Configure the component

PAI -name doc_similarity
    -project algo_public
    -DinputTableName="pai_test_doc_similarity"
    -DoutputTableName="pai_test_doc_similarity_output"
    -DinputSelectedColName1="col0"
    -DinputSelectedColName2="col1"

Example

This example calculates Levenshtein similarity between two columns of city-name strings.

Step 1: Create the input table

Use an ODPS SQL node to create the input table. For details, see Develop an ODPS SQL task.

drop table if exists pai_doc_similarity_input;
create table pai_doc_similarity_input as
select * from
(
  select 0 as id, "Beijing Shanghai" as col0, "Beijing Shanghai" as col1 from dual
  union all
  select 1 as id, "Beijing Shanghai" as col0, "Beijing Shanghai Shenzhen" as col1 from dual
) tmp

Step 2: Run the similarity calculation

drop table if exists pai_doc_similarity_output;
PAI -name doc_similarity
    -project algo_public
    -DinputTableName=pai_doc_similarity_input
    -DoutputTableName=pai_doc_similarity_output
    -DinputSelectedColName1=col0
    -DinputSelectedColName2=col1
    -Dmethod=levenshtein_sim
    -DinputAppendColNames=id,col0,col1;

Output table: pai_doc_similarity_output

Row 0 scores 1.0 because both strings are identical. Row 1 scores approximately 0.667 because "Beijing Shanghai" shares 2 of 3 words with "Beijing Shanghai Shenzhen" after Levenshtein normalization.

FAQ

My cosine or ssk similarity output is always 0. What's wrong?

When k is greater than the number of words in either input string, both methods treat the two strings as identical in their internal representation and return a similarity of 0. Set k to a value no greater than the minimum word count across your input pairs.

What is the difference between distance and similarity outputs?

Methods named without _sim (for example, levenshtein, lcs, simhash_hamming) output a distance — a higher value means the two documents are more different. Methods ending in _sim output a similarity — a higher value means the two documents are more alike. The conversion is Distance = 1.0 − Similarity (or Distance = 64.0 × (1 − Similarity) for Simhash_Hamming).

Should I use document similarity or string similarity?

Document similarity treats space-separated tokens as comparison units, making it appropriate for multi-word text. If your input is a single string where character-level comparison matters (for example, product codes or short IDs), use the String Similarity component instead.

What's next

• Explore other algorithm components in Machine Learning Designer: Overview of Designer components

• Get an overview of Machine Learning Designer: Designer overview