PyODPS column operations

Procedure

1. Download the test dataset and import it into MaxCompute.

   1. Download and decompress the Iris dataset. Rename the iris.data file to iris.csv.

   2. Create a table named pyodps_iris and upload the iris.csv dataset. For more information, see Create a table and upload data.

      Use the following statement to create the table.

      CREATE TABLE if not exists pyodps_iris
      (
      sepallength  DOUBLE comment 'Sepal length (cm)',
      sepalwidth   DOUBLE comment 'Sepal width (cm)',
      petallength  DOUBLE comment 'Petal length (cm)',
      petalwidth   DOUBLE comment 'Petal width (cm)',
      name         STRING comment 'Species'
      );

2. Log in to the DataWorks console and select a region in the upper-left corner.

3. In the left-side navigation pane, click Workspace.

4. In the Actions column of the target workspace, choose Go to > Data Development.

5. On the Data Development page, right-click the created workflow and choose Create Node > MaxCompute > PyODPS 2.

6. In the Create Node dialog box, enter a node name and click OK.

7. In the PyODPS node, enter the sample code.

   from odps import DataFrame
   import numpy as np
   import pandas as pd
   
   iris = DataFrame(o.get_table('pyodps_iris'))
   
   # Check for null values.
   print iris.sepallength.isnull().head(5)
   
   # Conditional logic.
   print (iris.sepallength > 5).ifelse('gt5','lte5').rename('cmp5').head(5)
   
   # Multiple conditions.
   print iris.sepallength.switch(4.9,'eq4.9',5.0,'eq5.0',default='noeq').rename('equalness').head(5)
   from odps.df import switch
   print switch(iris.sepallength == 4.9,'eq4.9',iris.sepallength == 5.0,'eq5.0',default='noeq').rename('equalness').head(5)
   
   # Update a subset of a column based on a condition.
   iris[iris.sepallength > 5,'cmp5'] = 'gt5'
   iris[iris.sepallength <=5,'cmp5'] = 'lte5'
   print iris.head(5)
   
   
   # Mathematical operations.
   print (iris.sepallength * 10).log().head(5)
   fields = [iris.sepallength,(iris.sepallength /2).rename('sepallength/2'),(iris.sepallength ** 2).rename('sepallength_squared')]
   print iris[fields].head(5)
   print  (iris.sepallength < 5).head(5)
   
   
   
   # Operations on collection types.
   data = {'id': [1,2], 'a': [['a1','b1'],['c1']], 'b': [{'a2': 0, 'b2': 1, 'c2': 2},{'d2': 3, 'e2': 4}]}
   df = pd.DataFrame(data)
   print df
   
   df1 = DataFrame(df, unknown_as_string=True, as_type={'a': 'list<string>','b' : 'dict<string,int64>'})
   print df1.dtypes
   print df1.head()
   
   print df1[df1.id,df1.a[0],df1.b.len()].head()
   print df1.a.explode().head()
   print df1.a.explode(pos=True).head()
   print df1.b.explode().head()
   print df1.b.explode(['key','value']).head()
   
   # Use explode to output multiple rows with other columns.
   print df1[df1.id,df1.a.explode()].head()
   
   # isin, notin, and cut operations.
   
   # The isin operation checks if elements in a sequence are in a specified set. The notin operation is the opposite.
   print iris.sepallength.isin([4.9,5.1]).rename('sepallength').head()
   
   # The cut operation discretizes data into specified bins.
   print iris.sepallength.cut(range(6),labels=['0-1','1-2','2-3','3-4','4-5']).rename('sepallength_cut').head(5)
   
   # The include_under and include_over parameters include values below the lowest boundary and above the highest boundary.
   labels = ['0-1', '1-2', '2-3', '3-4', '4-5', '5-']
   iris.sepallength.cut(range(6), labels=labels, include_over=True).rename('sepallength_cut').head(5)

8. Click Run.

   

9. On the Runtime Log tab, view the results.