Pearson correlation coefficient
The Pearson Correlation Coefficient measures the linear correlation between two features. The larger the absolute value, the stronger the correlation.
Applicable conditions
Your data must meet all three conditions for Pearson to be valid:
-
Neither variable has a standard deviation of 0.
-
Both variables are continuous and in a linear relationship.
-
Both variables follow a bivariate normal distribution, or a unimodal distribution that approximates a normal distribution.
Common use case: In machine learning, if two features are highly correlated, they are likely interchangeable. Dropping one reduces redundancy and can improve model performance.
Syntax
/*polar4ai*/CREATE FEATURE feature_name
WITH (
feature_class = 'pearson',
x_cols = '',
parameters = ()
)
AS (SELECT select_expr [, select_expr] ... FROM table_reference)Parameters
| Parameter | Description | Example |
|---|---|---|
feature_name |
Name of the feature to create | pearson_001 |
feature_class |
Feature type. Set to pearson. |
pearson |
x_cols |
Columns to include in the correlation analysis. Each column must contain floating-point or integer values. Separate multiple column names with commas. | dx1,dx2 |
parameters |
Custom configuration. See sub-parameters below. | — |
select_expr |
Column expression in the SELECT clause used to build the feature | dx4 |
table_reference |
Table that contains the columns specified in select_expr |
airlines_test_1000 |
Sub-parameters of parameters
| Sub-parameter | Values | Description |
|---|---|---|
null_strategy |
mean | median |
How to handle NULL values in the input columns. mean replaces NULLs with the average value; median replaces NULLs with the median value. |
categorical_feature |
Column names, comma-separated | Columns in x_cols that contain categorical (non-numeric) values. These columns are excluded from the correlation computation. Example: categorical_feature='dx3' |
Example
The following statement creates a Pearson feature using the airlines_test_1000 table. It includes seven columns in x_cols, replaces NULLs with the column mean, and marks five columns as categorical to exclude them from the numeric correlation computation.
/*polar4ai*/CREATE FEATURE pearson_001
WITH (
feature_class = 'pearson',
x_cols = 'Airline,Flight,AirportFrom,AirportTo,DayOfWeek,Time,Length',
parameters = (
null_strategy = 'mean',
categorical_feature = 'Airline,Flight,AirportFrom,AirportTo,DayOfWeek'
)
)
AS (SELECT * FROM airlines_test_1000);