Quick Start

Procedure

1. Optional:Activate the pay-as-you-go MaxCompute service, DataWorks Basic Edition, and the pay-as-you-go Platform for AI service. PAI includes Designer, Deep Learning Containers (DLC), and Elastic Algorithm Service (EAS). MaxCompute, DataWorks, and PAI must be in the same region.

   1. On the Alibaba Cloud MaxCompute product homepage, click Buy Now.

      For more information about activating MaxCompute, see Activate MaxCompute and DataWorks.

      Note

      • If you have not activated the MaxCompute service, this action also activates the DataWorks Basic Edition service for free and the pay-as-you-go MaxCompute service by default.

      • If you have already activated the pay-as-you-go MaxCompute service, you can skip this step.

   2. Go to the DataWorks purchase page to purchase the Basic Edition service.

      For more information about activating DataWorks, see Activate DataWorks.

      Note

      If you have already activated DataWorks Basic Edition, you can skip this step.

   3. Go to the Platform for AI purchase page, activate PAI, and create a default workspace.

      For more information about activating Platform for AI, see Activate PAI and create a default workspace.

      Note

      If you have already activated PAI and created a workspace, you can skip this step.

2. Download the agaricus-lepiota.data file from the Mushroom Data Set and save it as a TXT, CSV, or LOG file, such as agaricus-lepiota.data.txt.

3. Log in to the DataWorks console to create or configure a DataWorks workspace.

   • If you have a DataWorks workspace, configure a MaxCompute compute engine for the workspace and turn on the Schedule PAI Algorithm Tasks switch. Follow these steps:

     1. In the navigation pane on the left, click Workspace to open the workspace list page.

     2. On the Workspace List page, find the target workspace and click Manage in the Actions column.

     3. In the navigation pane on the left, click General Configurations. In the Basic Properties section, turn on Schedule PAI Algorithm Tasks.

     4. In the navigation pane on the left, click Data Source to open the data source page and create a MaxCompute data source. For more information about creating a data source, see Attach a MaxCompute compute engine.

   • If you do not have a DataWorks workspace, create one. Set the compute engine service to MaxCompute and enable the scheduling of PAI algorithm tasks. For more information about creating a DataWorks workspace, see Create a workspace.

4. In DataWorks, create a table named mushroom_classification and import the data from the prepared dataset.

   1. In the Actions column of the target DataWorks workspace, click Quick Access > DataStudio to create the mushroom_classification table.

      For more information about creating a table, see Create and use MaxCompute tables.

      The following is a sample DDL statement for creating a table:

      CREATE TABLE mushroom_classification (
          label      string               comment 'poisonous=p,edible=e',
          cap_shape  string               comment 'bell=b,conical=c,convex=x,flat=f,knobbed=k,sunken=s',
          cap_surface string              comment 'fibrous=f,grooves=g,scaly=y,smooth=s',
          cap_color string                comment 'brown=n,buff=b,cinnamon=c,gray=g,green=r,pink=p,purple=u,red=e,white=w,yellow=y',
          bruises string                  comment 'bruises=t,no=f',
          odor string                     comment 'almond=a,anise=l,creosote=c,fishy=y,foul=f,musty=m,none=n,pungent=p,spicy=s',
          gill_attachment string          comment 'attached=a,descending=d,free=f,notched=n',
          gill_spacing string             comment 'close=c,crowded=w,distant=d',
          gill_size string                comment 'broad=b,narrow=n',
          gill_color string               comment 'black=k,brown=n,buff=b,chocolate=h,gray=g,green=r,orange=o,pink=p,purple=u,red=e,white=w,yellow=y',
          stalk_shape string              comment 'enlarging=e,tapering=t',
          stalk_root string               comment 'bulbous=b,club=c,cup=u,equal=e,rhizomorphs=z,rooted=r,missing=?',
          stalk_surface_above_ring string comment 'fibrous=f,scaly=y,silky=k,smooth=s',
          stalk_surface_below_ring string comment 'fibrous=f,scaly=y,silky=k,smooth=s',
          stalk_color_above_ring string   comment 'brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y',
          stalk_color_below_ring string   comment 'brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y',
          veil_type string                comment 'partial=p,universal=u',
          veil_color string               comment 'brown=n,orange=o,white=w,yellow=y',
          ring_number string              comment 'none=n,one=o,two=t',
          ring_type string                comment 'cobwebby=c,evanescent=e,flaring=f,large=l,none=n,pendant=p,sheathing=s,zone=z',
          spore_print_color string        comment 'black=k,brown=n,buff=b,chocolate=h,green=r,orange=o,purple=u,white=w,yellow=y',
          population string               comment 'abundant=a,clustered=c,numerous=n,scattered=s,several=v,solitary=y',
          habitat string                  comment 'grasses=g,leaves=l,meadows=m,paths=p,urban=u,waste=w,woods=d'
      );

   2. Import the data from the agaricus-lepiota.data.txt file into the mushroom_classification table. For the field matching method, select Match By Position.

      For more information about uploading data, see Upload local data.

   3. In DataWorks, use the ad hoc query feature to create a MaxCompute ODPS SQL node and run an SQL command to verify that the data is imported.

      For more information about ad hoc queries, see Use an ad hoc query to run an SQL statement (optional).

      The following is a sample command:

      SELECT * FROM mushroom_classification;

      The following result is returned:

5. Process the data in the mushroom_classification table using one-hot encoding.

   Because logistic regression binary classification models require numeric fields, you must use one-hot encoding to convert enumerated values to numeric values. For example, the `cap_shape` field has six possible values: b, c, x, f, k, s. One-hot encoding converts these six enumerated values into six columns. Each column corresponds to one value. If the `cap_shape` value for a row matches the value that corresponds to a column, the cell value is set to 1. Otherwise, the cell value is set to 0.

   1. Optional:Create a business flow, such as mc_test.

      For more information about creating a business flow, see Create a periodic business flow.

      Note

      If you already have a business flow, you can use it directly and skip this step.

   2. Create a MaxCompute ODPS Script node. In the node, write code to process the imported data using one-hot encoding and write the processed data to a new table named mushroom_classification_one_hot.

      For more information about creating an ODPS Script node, see Develop an ODPS Script task.

      The following is a sample command:

      CREATE temporary FUNCTION one_hot AS 'onehot.OneHotEncoding' USING
      -- CODE ('lang'='JAVA')
      package onehot;
      
      import com.aliyun.odps.udf.UDFException;
      import com.aliyun.odps.udf.UDTF;
      import com.aliyun.odps.udf.annotation.Resolve;
      
      import java.io.IOException;
      import java.util.ArrayList;
      import java.util.List;
      
      @Resolve({"string,string,string,string,string,string,string,string,string,string," +
              "string,string,string,string,string,string,string,string,string,string,string,string" +
              "->" +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,"+
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint,bigint," +
              "bigint,bigint,bigint,bigint,bigint,bigint"})
      public class OneHotEncoding extends UDTF {
        private static char[][] features = {
                { 'b','c','x','f','k','s'}, //cap-shape
                { 'f','g','y','s'}, //cap-surface
                { 'n','b','c','g','r','p','u','e','w','y'}, //cap-color
                { 't','f'}, //bruises
                { 'a','l','c','y','f','m','n','p','s'}, //odor
                { 'a','d','f','n'}, //gill-attachment
                { 'c','w','d'}, //gill-spacing
                { 'b','n'}, //gill-size
                { 'k','n','b','h','g','r','o','p','u','e','w','y'}, //gill-color
                { 'e','t'}, //stalk-shape
                { 'b','c','u','e','z','r','?'}, //stalk-root
                { 'f','y','k','s'}, //stalk-surface-above-ring
                { 'f','y','k','s'}, //stalk-surface-below-ring
                { 'n','b','c','g','o','p','e','w','y'}, //stalk-color-above-ring
                { 'n','b','c','g','o','p','e','w','y'}, //stalk-color-below-ring
                { 'p','u'}, //veil-type
                { 'n','o','w','y'}, //veil-color
                { 'n','o','t'}, //ring-number
                { 'c','e','f','l','n','p','s','z'}, //ring-type
                { 'k','n','b','h','r','o','u','w','y'}, //spore-print-color
                { 'a','c','n','s','v','y'}, //population
                { 'g','l','m','p','u','w','d'}, //habitat
        };
        @Override
        public void process(Object[] objects) throws UDFException, IOException {
          List<Long> featuresEncoding = new ArrayList<>(126);
          for (int i = 0; i < objects.length; i++) {
            String value = (String)objects[i];
            char[] feature = features[i];
            for (char c : feature) {
              featuresEncoding.add(value.charAt(0) == c ? 1L : 0L);
            }
          }
          forward(featuresEncoding.toArray());
        }
      }
      
      -- END CODE;
      
      CREATE TABLE mushroom_classification_one_hot AS
      SELECT t.*, label FROM mushroom_classification
      LATERAL VIEW 
      one_hot(cap_shape,cap_surface,cap_color,bruises,odor,gill_attachment, 
              gill_spacing, gill_size, gill_color, stalk_shape,stalk_root ,
              stalk_surface_above_ring,stalk_surface_below_ring,stalk_color_above_ring,
              stalk_color_below_ring,veil_type,veil_color,ring_number,ring_type,spore_print_color,
              population,habitat) t
      AS f1,f2,f3,f4,f5,f6,f7,f8,f9,f10,f11,f12,f13,f14,f15,f16,f17,f18,f19,f20,
      f21,f22,f23,f24,f25,f26,f27,f28,f29,f30,f31,f32,f33,f34,f35,f36,f37,f38,f39,f40,
      f41,f42,f43,f44,f45,f46,f47,f48,f49,f50,f51,f52,f53,f54,f55,f56,f57,f58,f59,f60,
      f61,f62,f63,f64,f65,f66,f67,f68,f69,f70,f71,f72,f73,f74,f75,f76,f77,f78,f79,f80,
      f81,f82,f83,f84,f85,f86,f87,f88,f89,f90,f91,f92,f93,f94,f95,f96,f97,f98,f99,f100,
      f101,f102,f103,f104,f105,f106,f107,f108,f109,f110,f111,f112,f113,f114,f115,f116,
      f117,f118,f119,f120,f121,f122,f123,f124,f125,f126;

   3. In DataWorks, use the ad hoc query feature to create a MaxCompute ODPS SQL node and run an SQL command to verify the result of the one-hot encoding.

      The following is a sample command:

      SELECT * FROM mushroom_classification_one_hot;

      The following result is returned:

   4. In DataWorks, use the ad hoc query feature to create a MaxCompute ODPS SQL node and create a training dataset and a test dataset based on the data in the mushroom_classification_one_hot table.

      The following is a sample command:

      -- Training dataset. One-fourth of the data is used for model training.
      CREATE TABLE mushroom_training AS 
      SELECT * FROM mushroom_classification_one_hot WHERE sample(4,1);
      
      -- Test dataset. The remaining three-fourths of the data is used for prediction and evaluation.
      CREATE TABLE mushroom_predict AS 
      SELECT * FROM mushroom_classification_one_hot EXCEPT ALL SELECT * FROM mushroom_training;

6. Create a machine learning model and make predictions.

   1. In DataWorks, use the ad hoc query feature to create a MaxCompute ODPS SQL node and create the logistic regression binary classification model lr_test_model based on the training dataset.

      The following is a sample command:

      CREATE model lr_test_model
      WITH properties('model_type'='logisticregression_binary', 'goodValue'='p','maxIter'='1000')
      AS SELECT * FROM mushroom_training;

      Note

      • You can specify more parameters in properties. The parameters are the same as those on the Platform for AI platform. For more information, see Linear support vector machine.

      • The SQL engine extracts and runs the query statement after the as clause. The results are stored in a temporary table. You can view the results in the summary information of the job's Logview. The temporary table has a lifecycle of one day and is automatically deleted when it expires.

      • To delete the model later, run the drop offlinemodel lr_test_model command.

   2. In DataWorks, use the ad hoc query feature to create a MaxCompute ODPS SQL node and use the built-in function ml_predict to make predictions on the test dataset based on the lr_test_model model.

      The following is a sample command:

      CREATE TABLE mushroom_predict_result AS 
      SELECT * FROM ml_predict(
          lr_test_model, 
          (SELECT * FROM mushroom_predict)
      );

      Note

      • The SQL engine saves the results of the subquery within the ml_predict function to a temporary table. This temporary table has a lifecycle of one day and is automatically deleted when it expires.

      • The results of the ml_predict function can be used in the from clause of an SQL query. You can also use an insert or create table as statement to save the results to another table. For more information about the ml_predict function, see Supported prediction model functions.

   3. In DataWorks, use the ad hoc query feature to create a MaxCompute ODPS SQL node and run an SQL command to view the prediction results in the mushroom_predict_result table.

      The following is a sample command:

      SELECT * FROM mushroom_predict_result;

      The following result is returned:

7. Use the built-in function ml_evaluate to evaluate the prediction accuracy of the model.

   For more information about the ml_evaluate function, see Supported evaluation model functions.