The Multiclass Classification Evaluation component evaluates a model's performance on classification tasks with more than two classes. It calculates metrics such as accuracy, recall, F1 score, and a confusion matrix to quantify the model's classification accuracy for each class. The confusion matrix shows the relationship between predicted and actual labels, while other metrics provide details on the classification accuracy for each class. These metrics help you understand the model's performance on individual classes and optimize it further.
Component configuration
Method 1: Console
In Designer, add the Multiclass Classification Evaluation component to your workflow and configure its parameters in the panel on the right.
|
Parameter type |
Parameter |
Description |
|
Fields Setting |
Original Classification Result Column |
Select the column that contains the actual labels. The number of classes cannot exceed 1,000. |
|
Predicted Classification Result Column |
Select the column that contains the predicted labels. By default, this column is named |
|
|
Advanced Options |
Select the Advanced Options checkbox to enable the Prediction Result Probability Column parameter. |
|
|
Prediction Result Probability Column |
Specifies the column for calculating the model's log loss. This parameter is valid only for random forest models and can cause errors if used with other models. By default, this column is named |
|
|
Tuning |
Number of Cores |
Works with Memory per Core. By default, the system automatically allocates resources. |
|
Memory per Core |
The amount of memory for each core, in MB. By default, the system automatically allocates resources. |
Method 2: PAI command
To configure the Multiclass Classification Evaluation component with a PAI command, run the command in a SQL Script component. For more information, see Scenario 4: Run a PAI command in the SQL Script component.
PAI -name MultiClassEvaluation -project algo_public
-DinputTableName="test_input"
-DoutputTableName="test_output"
-DlabelColName="label"
-DpredictionColName="prediction_result"
-Dlifecycle=30;|
Parameter |
Required |
Default |
Description |
|
inputTableName |
Yes |
None |
The name of the input table. |
|
inputTablePartitions |
No |
Full table |
The partitions to use from the input table. |
|
outputTableName |
Yes |
None |
The name of the output table. |
|
labelColName |
Yes |
None |
The name of the column containing the actual labels. |
|
predictionColName |
Yes |
None |
The name of the column that contains the predicted labels. |
|
predictionDetailColName |
No |
Empty |
The column that contains the predicted class probabilities. For example, |
|
lifecycle |
No |
None |
The lifecycle of the output table, in days. |
|
coreNum |
No |
System-determined |
The number of cores for the job. |
|
memSizePerCore |
No |
System-determined |
The amount of memory for each core, in MB. |
Usage example
-
Add a SQL Script component and enter the following SQL statement to generate training data.
drop table if exists multi_esti_test; create table multi_esti_test as select * from ( select '0' as id,'A' as label,'A' as prediction,'{"A": 0.6, "B": 0.4}' as detail union all select '1' as id,'A' as label,'B' as prediction,'{"A": 0.45, "B": 0.55}' as detail union all select '2' as id,'A' as label,'A' as prediction,'{"A": 0.7, "B": 0.3}' as detail union all select '3' as id,'A' as label,'A' as prediction,'{"A": 0.9, "B": 0.1}' as detail union all select '4' as id,'B' as label,'B' as prediction,'{"A": 0.2, "B": 0.8}' as detail union all select '5' as id,'B' as label,'B' as prediction,'{"A": 0.1, "B": 0.9}' as detail union all select '6' as id,'B' as label,'A' as prediction,'{"A": 0.52, "B": 0.48}' as detail union all select '7' as id,'B' as label,'B' as prediction,'{"A": 0.4, "B": 0.6}' as detail union all select '8' as id,'B' as label,'A' as prediction,'{"A": 0.6, "B": 0.4}' as detail union all select '9' as id,'A' as label,'A' as prediction,'{"A": 0.75, "B": 0.25}' as detail )tmp; -
Add another SQL Script component and enter the following PAI command to run the evaluation.
drop table if exists ${o1}; PAI -name MultiClassEvaluation -project algo_public -DinputTableName="multi_esti_test" -DoutputTableName=${o1} -DlabelColName="label" -DpredictionColName="prediction" -Dlifecycle=30; -
Right-click the component from the previous step and select View Data > SQL Script's Output to view the evaluation results.
| result | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | { "ActualLabelFrequencyList": [5, 5], "ActualLabelProportionList": [0.5, 0.5], "ConfusionMatrix": [[4, 1], [2, 3]], "LabelList": ["A", "B"], "LabelMeasureList": [{ "Accuracy": 0.7, "F1": 0.7272727272727273, "FalseDiscoveryRate": 0.3333333333333333, "FalseNegative": 1, "FalseNegativeRate": 0.2, "FalsePositive": 2, "FalsePositiveRate": 0.4, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.75, "Precision": 0.6666666666666666, "Sensitivity": 0.8, "Specificity": 0.6, "TrueNegative": 3, "TruePositive": 4}, { "Accuracy": 0.7, "F1": 0.6666666666666666, "FalseDiscoveryRate": 0.25, "FalseNegative": 2, "FalseNegativeRate": 0.4, "FalsePositive": 1, "FalsePositiveRate": 0.2, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.6666666666666666, "Precision": 0.75, "Sensitivity": 0.6, "Specificity": 0.8, "TrueNegative": 4, "TruePositive": 3}], "LabelNumber": 2, "OverallMeasures": { "Accuracy": 0.7, "Kappa": 0.3999999999999999, "LabelFrequencyBasedMicro": { "Accuracy": 0.7, "F1": 0.696969696969697, "FalseDiscoveryRate": 0.2916666666666666, "FalseNegative": 1.5, "FalseNegativeRate": 0.3, "FalsePositive": 1.5, "FalsePositiveRate": 0.3, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.7083333333333333, "Precision": 0.7083333333333333, "Sensitivity": 0.7, "Specificity": 0.7, "TrueNegative": 3.5, "TruePositive": 3.5}, "MacroAveraged": { "Accuracy": 0.7, "F1": 0.696969696969697, "FalseDiscoveryRate": 0.2916666666666666, "FalseNegative": 1.5, "FalseNegativeRate": 0.3, "FalsePositive": 1.5, "FalsePositiveRate": 0.3, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.7083333333333333, "Precision": 0.7083333333333333, "Sensitivity": 0.7, "Specificity": 0.7, "TrueNegative": 3.5, "TruePositive": 3.5}, "MicroAveraged": { "Accuracy": 0.7, "F1": 0.7, "FalseDiscoveryRate": 0.3, "FalseNegative": 3, "FalseNegativeRate": 0.3, "FalsePositive": 3, "FalsePositiveRate": 0.3, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.7, "Precision": 0.7, "Sensitivity": 0.7, "Specificity": 0.7, "TrueNegative": 7, "TruePositive": 7}}, "PredictedLabelFrequencyList": [6, 4], "PredictedLabelProportionList": [0.6, 0.4], "ProportionMatrix": [[0.8, 0.2], [0.4, 0.6]]} |
Appendix
If you run the component in visual mode, you can right-click it and select Visual Analysis to view detailed results. In the list of visual nodes on the left, select the Multiclass Classification Evaluation-1 node and navigate to the Metrics tab on the right to view the results. Key metrics include: accuracy = 0.7 and Kappa ≈ 0.4. The macro-averaged metrics are: F1 score ≈ 0.697, precision ≈ 0.708, sensitivity = 0.7, specificity = 0.7, false positive rate = 0.3, false negative rate = 0.3, true positive = 3.5, and true negative = 3.5.
The results page contains three tabs: Metrics, Confusion Matrix, and Distribution Statistics. On the Confusion Matrix tab, the prediction results for each class are displayed as a heatmap. The vertical axis represents the actual classes, and the horizontal axis represents the predicted classes. Hover over a cell in the matrix to view its specific value (Value) and proportion (Ratio).
The Distribution Statistics tab on the results page displays the classification evaluation metrics for each class in a table, including true positive (TP), true negative (TN), false positive (FP), false negative (FN), sensitivity, specificity, precision, accuracy, F1 score, and Kappa. For example, for class A, the metrics are: TP=4, TN=3, FP=2, FN=1, sensitivity=0.8, specificity=0.6, precision=0.6667, accuracy=0.7, F1 score=0.7273, and Kappa=0.4. For class B, the metrics are: TP=3, TN=4, FP=1, FN=2, sensitivity=0.6, specificity=0.8, precision=0.75, accuracy=0.7, F1 score=0.6667, and Kappa=0.4.