PyODPS deduplication

Procedure

1. Download the test dataset and import it into MaxCompute.

   1. Download and decompress the Iris dataset. Rename the iris.data file to iris.csv.

   2. Create a table named pyodps_iris and upload the iris.csv dataset. For more information, see Create a table and upload data.

      Use the following statement to create the table.

      CREATE TABLE if not exists pyodps_iris
      (
      sepallength  DOUBLE comment 'Sepal length (cm)',
      sepalwidth   DOUBLE comment 'Sepal width (cm)',
      petallength  DOUBLE comment 'Petal length (cm)',
      petalwidth   DOUBLE comment 'Petal width (cm)',
      name         STRING comment 'Species'
      );

2. Log in to the DataWorks console and select a region in the upper-left corner.

3. In the left-side navigation pane, click Workspaces.

4. In the Actions column, click Go to > Data Development.

5. On the Data Development page, right-click the created workflow and choose Create Node > MaxCompute > PyODPS 2.

6. In the Create Node dialog box, enter a node name and click OK.

7. In the PyODPS node, enter the code to deduplicate data.

   Sample code:

   from odps.df import DataFrame
   iris = DataFrame(o.get_table('pyodps_iris'))
   # Deduplicate by a single column using a column selector.
   print(iris[['name']].distinct())
   # Deduplicate by a single column using a column name as an argument.
   print(iris.distinct('name'))
   # Deduplicate by multiple columns and preview the first three rows.
   print(iris.distinct('name','sepallength').head(3))
   # You can call unique() to deduplicate a Sequence.
   # Note: The Sequence returned by unique() does not support further column selection.
   print(iris.name.unique())

8. Click Run.

9. In the Run Log, view the result.

   name = Column[sequence(string)] 'name' from collection ref_0
   Sql compiled:
   CREATE TABLE tmp_pyodps_07f3dbed_e4e6_4c97_8d85_15becf617755 LIFECYCLE 1 AS
   SELECT DISTINCT t1.`name`, t1.`sepallength`
   FROM my_project_simple.`pyodps_iris` t1
   Instance ID: 20191018031207168gndzywj9
         name  sepallength
   0  Iris-setosa          4.3
   1  Iris-setosa          4.4
   2  Iris-setosa          4.5
   Collection: ref_0
     odps.Table
       name: my_project_simple.`pyodps_iris`
       schema:
         sepallength           : double      # sepal length (cm)
         sepalwidth            : double      # sepal width (cm)
         petallength           : double      # petal length (cm)
         petalwidth            : double      # petal width (cm)
         name                  : string      # class
   Collection: ref_1
     Distinct[collection]
       collection: ref_0
       distinct:
         name = Column[sequence(string)] 'name' from collection ref_0
   name = Column[sequence(string)] 'name' from collection ref_1
   2019-10-18 11:12:10 INFO ================================================================
   2019-10-18 11:12:10 INFO Exit code of the Shell command 0
   2019-10-18 11:12:10 INFO --- Invocation of Shell command completed ---
   2019-10-18 11:12:10 INFO Shell run successfully!
   2019-10-18 11:12:10 INFO Current task status: FINISH
   2019-10-18 11:12:10 INFO Cost time is: 6.932s

   The complete output is as follows:

   Sql compiled:
   CREATE TABLE tmp_pyodps_ed85ebd5_d678_44dd_9ece_bff1822376f6 LIFECYCLE 1 AS
   SELECT DISTINCT t1.`name`
   FROM WB_BestPractice_dev.`pyodps_iris` t1
   Instance ID: 2019101006391142g2cp5692
                 name
   0      Iris-setosa
   1  Iris-versicolor
   2   Iris-virginica
   Sql compiled:
   CREATE TABLE tmp_pyodps_8ce6128f_9c6f_45af_b9de_c73ce9d5ba51 LIFECYCLE 1 AS
   SELECT DISTINCT t1.`name`
   FROM WB_BestPractice_dev.`pyodps_iris` t1
   Instance ID: 20191010063915987gmuws592
                 name
   0      Iris-setosa
   1  Iris-versicolor
   2   Iris-virginica
   Sql compiled:
   CREATE TABLE tmp_pyodps_a3dc338e_0fea_4d5f_847c_79fb19ec1c72 LIFECYCLE 1 AS
   SELECT DISTINCT t1.`name`, t1.`sepallength`
   FROM WB_BestPractice_dev.`pyodps_iris` t1
   Instance ID: 2019101006392210gj056292
             name  sepallength
   0  Iris-setosa          4.3
   1  Iris-setosa          4.4
   2  Iris-setosa          4.5
   Sql compiled:
   CREATE TABLE tmp_pyodps_bc0917bb_f10c_426b_9b75_47e94478382a LIFECYCLE 1 AS
   SELECT t2.`name`
   FROM (
     SELECT DISTINCT t1.`name`
     FROM WB_BestPractice_dev.`pyodps_iris` t1
   ) t2
   Instance ID: 20191010063927189g9fsz192
                 name
   0      Iris-setosa
   1  Iris-versicolor
   2   Iris-virginica