Feature Updates (Before 2022)

Feature Name

Feature Description

Release Date

Release Region

References

Product Billing: Policy Optimization and Data Processing Unit Specification Expansion

The ratio of data processing units (DPUs) for sync tasks and integration tasks changed from 1:1 to 3:1. The first 200 integration tasks are free. This further reduces DataQ - DatatoCloud costs.

One-time tasks and logical aggregate tables no longer consume data processing units. This lowers your logical development costs.

August 19, 2021

China (Beijing), China (Hangzhou)

Billing

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

The system allocates default schedule resources based on your selected data processing unit specifications. Higher specifications provide more schedule resources, support greater task concurrency, and improve development efficiency.

August 19, 2021

China (Beijing), China (Hangzhou)

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

The Intelligent Development Edition now supports up to 200 data processing units.

August 19, 2021

China (Beijing), China (Hangzhou)

Purchasing guide, Billing

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Data Source: Expanded Supported Data Source Types and Versions, Optimized Data Source Configuration Interaction

The new data source page now categorizes data source types and displays recently used types, improving product usability.

August 19, 2021

China (Beijing), China (Hangzhou)

Data Source

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

MySQL data source version support now extends to 8.0. Add data sources that match your business database version.

August 19, 2021

China (Beijing), China (Hangzhou)

Create a MySQL data source

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Hologres data sources are now supported. Import business data from Hologres to Dataphin or export Dataphin data to Hologres. Create real-time metadata tables based on Hologres data sources.

August 19, 2021

China (Beijing), China (Hangzhou)

Create a Hologres data source, Create a metadata table

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Data source permission control is upgraded. It now includes view permission control for connection addresses, improving data source security.

August 19, 2021

China (Beijing), China (Hangzhou)

Data source permission list

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Dataphin Supports Generic IP Addresses

To improve network connectivity configuration efficiency, Dataphin now provides generic IP addresses for all regions in a VPC network environment. Add these generic IP addresses to your business database's whitelist to establish network connectivity.

August 19, 2021

China (Beijing), China (Hangzhou)

Network Connectivity Solutions

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

DataService Studio: Supports Data Query Acceleration for DataService Studio Module, Expanded Supported Data Source Types and Versions

To meet business scenarios requiring fast data queries, Dataphin lets you add accelerated compute sources to projects. After developing a Dataphin data source, manually enable the acceleration switch to accelerate data queries in the DataService Studio module.

August 19, 2021

China (Beijing), China (Hangzhou)

Create a project, Develop Dataphin data sources and accelerate data queries

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Expanded supported data source types and versions. Added Impala, AnalyticDB for PostgreSQL, and AnalyticDB for MySQL 3.0 data sources. HBase data source version support extends to 2.x. Add data sources that match your business database version.

August 19, 2021

China (Beijing), China (Hangzhou)

Create an Impala data source, Create an AnalyticDB for MySQL 3.0 data source, Dataphin supported data sources

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Code Task: Expanded Python Compute Task Versions, Global Search for Code Tasks

Python compute task version support extends to 3.7. Dataphin also supports global search for code tasks, improving compute task management.

August 19, 2021

China (Beijing), China (Hangzhou)

Create a Python compute task

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Data Integration: Expanded Data Integration Read and Write Components, One-Click Table Creation for AnalyticDB for PostgreSQL Target Data Sources, Optimized Data Source Permission Request Interaction

Added Hologres input and output components. For AnalyticDB for PostgreSQL output components, Dataphin supports one-click table creation to quickly create target tables.

In the data source dropdown list, click "Request Permission" next to data sources without permissions to quickly request read and write permission.

August 19, 2021

China (Beijing), China (Hangzhou)

Configure a Hologres input component, Configure a Hologres output component

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Operation Center: Operation Center Supports One-Click Termination of Data Backfill Operations for All Nodes Under a Data Backfill Instance, and Page Interaction Optimization

To improve O&M task efficiency, terminate data backfill operations for all nodes under a running data backfill instance with one click.

August 19, 2021

China (Beijing) and China (Hangzhou)

Generic data backfill instance

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Search for materialized nodes by field name and field English name. Automatically locate the corresponding materialized node.

August 19, 2021

China (Beijing), China (Hangzhou)

Logical table task instance

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Offline Full Database Migration

To support more scenarios, Dataphin now synchronizes databases from your data center or self-built databases on ECS to Hive, MaxCompute, Oracle, and AnalyticDB for PostgreSQL.

August 19, 2021

China (Beijing), China (Hangzhou)

Supported data sources

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Data Security

To immediately scan data using detection rules and identify all sensitive fields, Dataphin supports manually triggering detection rule scans.

August 19, 2021

China (Beijing), China (Hangzhou)

Create, configure, and manually trigger detection rules

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Stream-Batch Integrated Task: Added Property Configuration and Support for Viewing Metadata Table Reference Information

To prevent frequently missing `stat_date` configuration in task parameters, Dataphin now supports adding `stat_date` key-value pairs in the property configuration of real-time tasks.

August 19, 2021

China (Beijing), China (Hangzhou)

Develop stream-batch integrated tasks

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

View metadata table reference information. Click a referenced compute task to view its details.

August 19, 2021

China (Beijing), China (Hangzhou)

Create a metadata table

August 24, 2021

China (Shenzhen)

August 26, 2021

China (Shanghai)

Feature Name

Feature Description

Release Date

Release Region

References

Data Source

Added Loghub, PolarDB, OSS, OceanBase, SAP HANA, DB2, and TeraData data sources.

November 17, 2020

China (Beijing)

Create an offline data source

November 26, 2020

China (Shenzhen)

December 15, 2020

China (Shanghai)

Data Integration

• Added Custom Components. Users can customize components not supported by the system. RDBMS database components connect via Java Database Connectivity (JDBC). Non-RDBMS database components require manual JAR package uploads.

• Added PolarDB, LogHub, OSS, SAP HANA, DB2, OceanBase, TeraData, and local CSV input components. Added script configuration mode.

• Added PolarDB, OSS, SAP HANA, DB2, OceanBase, and TeraData output components. Added script configuration mode.

November 17, 2020

China (Beijing)

• Create a custom component

• Input components

• Output components

November 26, 2020

China (Shenzhen)

December 15, 2020

China (Shanghai)

Data Development

• Added global code search. Quickly search all code within a Dataphin instance.

  Note

  Historical submitted code does not support search. To search historical code, resubmit the code task.

• Added version rollback for functions.

• Added version rollback for resources.

• Developed stream-batch integrated tasks based on the Flink compute engine, improving data development efficiency.

November 17, 2020

China (Beijing)

• Create a user-defined function

• Create a resource

• Data Development Overview

November 26, 2020

China (Shenzhen)

December 15, 2020

China (Shanghai)

Operation Center

• Added an O&M dashboard. It describes all task runs, instance distribution statistics, and abnormal instance statistics. This helps you promptly understand task running status and instance trends.

• Added O&M list and Batch Operations, improving operations management efficiency.

• Added logical table O&M. Manage logical table nodes from a field perspective.

• Monitoring configuration is upgraded. It supports batch monitoring configuration, logical table field monitoring, and custom send counts and frequencies.

November 17, 2020

China (Beijing)

Operation Center Overview

November 26, 2020

China (Shenzhen)

December 15, 2020

China (Shanghai)

Asset Quality

• Added asset quality monitoring rules for data sources, including data source connectivity monitoring and table schema change monitoring.

• Added asset quality monitoring rules for real-time metadata tables, including statistical trend monitoring, real-time multi-link comparison, and real-time offline data reconciliation.

November 17, 2020

China (Beijing)

• Create a data source rule

• Create a real-time metadata table rule

November 26, 2020

China (Shenzhen)

December 15, 2020

China (Shanghai)

Asset Security

Added a data security value-added feature package. Data security defines data categories and security levels, builds sensitive data detection rules, and sets sensitive data masking rules to ensure data asset security.

December 15, 2020

China (Beijing)

Asset Security

December 15, 2020

China (Shenzhen)

December 15, 2020

China (Shanghai)

DataService Studio

• Added Dev-Prod and Basic modes to meet user requirements for asset security control.

• Added API development based on logical tables.

November 17, 2020

China (Beijing)

DataService Studio Overview

November 26, 2020

China (Shenzhen)

December 15, 2020

China (Shanghai)

Feature Name

Feature Description

Release Date

Release Region

References

Create a Project

• When binding an offline compute source, automatically create the corresponding data source.

• The system controls permission requests for reading and writing data in metadata tables and physical tables within projects at a project granularity. This prevents data breaches and security issues caused by approver errors.

September 17, 2020

China (Shanghai), China (Shenzhen), and China (Beijing)

Create a project

Edit Project

In Dev-Prod development mode, add members to both Dev and Prod projects simultaneously.

Manage project permissions and compute sources

Offline Compute Task

• Added task cloning.

• Added version rollback and version comparison.

• Added quick jump to Operation Center.

• Added quick publish.

• Added quick filtering for data processing objects I created.

Manage compute tasks

Real-time Compute Task

• Added quick jump to Operation Center.

• Added quick publish.

• Added quick filtering for data processing objects I created.

Manage compute tasks

Publish Task

• Added a remove object feature to the pending publish object list.

• The system remembers the most recently accessed lists (pipeline scripts, standardized modeling, and data processing).

Manage publish tasks

Business Category

The system supports setting up to 10 business category administrators.

Create a data category

Feature Name

Feature Description

Release Date

Release Region

References

Basic Development Edition

Pricing and purchasing instructions for the Basic Development Edition.

July 14, 2020

China (Shanghai), China (Shenzhen), and China (Beijing).

Purchasing guide, Billing, and Activate Dataphin.

Real-time Custom Source

The system supports customizing data sources by uploading JAR files.

Create a custom source

Real-time Compute Metadata Table

When creating a data source for a real-time compute metadata table, add a real-time custom source.

Renewal

Added cloning for real-time compute metadata tables.

Real-time Compute Template

Added cloning for real-time compute templates.

Create a code template

Publish Task

Added a publish dry run feature.

Manage publish tasks

Real-time Compute Task

Added cloning for real-time compute tasks.

Manage compute tasks

Added creation of Flink_DataStream type real-time compute tasks.

Real-time Compute Function

Added viewing function reference information, saving function version information, and rolling back function versions.

Create a user-defined function

Offline Compute Function

Added viewing function reference information, saving function version information, and rolling back function versions.

Create a user-defined function

Resource Management

Added viewing resource reference information, saving resource version information, rolling back resource versions, and downloading resource plans.

Create a resource

Feature Name

Feature Description

Release Date

Release Region

References

Data Processing Unit

Expanded data processing unit specifications to include 500, 1000, 1500, 2000, 2500, and 3000.

May 9, 2020

China (Shanghai), China (Shenzhen), and China (Beijing).

Purchasing guide, Activate Dataphin, Billing, and Upgradation.

DataService Studio

Expanded DataService Studio specifications to include api.base (max 500 QPS), 1api.s1.small (max 2500 QPS), api.s1.medium (max 5000 QPS), and api.s2.large (max 10000 QPS).

Activate Dataphin and Upgradation.

Auto-renewal

Added auto-renewal. Quickly renew your Dataphin instances.

Renewal

Resource Statistics

Added system resource usage statistics. View resource usage in the Dataphin system.

View resource usage

Behavioral Dashboard

Behavioral data sampling preview is updated. View behavioral data distribution and sampling results.

View behavioral data

Feature Name

Feature Description

Release Date

Release Region

References

Data Synchronization

If you purchased Dataphin after April 2020, data synchronization is upgraded to data integration. Data integration provides a simple, efficient, secure, and reliable data synchronization platform.

April 9, 2020

China (Beijing) and China (Shenzhen)

Import data

Compute Settings

Configure offline compute engines and real-time computing engines.

Set up compute engines for Dataphin instances

Compute Source

Compute sources provide compute and storage resources for offline data processing, and compute resources for real-time data processing.

Compute Source

Data Source

Create offline stores and real-time data sources. Use data sources to import business data into the Dataphin platform.

Data Source

Business Category

When creating a business category, set category administrators to better manage business categories.

Business Category

Release Management

Added publish management for data integration and real-time compute tasks.

Manage publish tasks

O&M Scheduling

Added real-time tasks and real-time instances to global O&M.

Overview

Monitoring and Alerting

Added real-time alerting records and real-time monitoring configuration. Promptly understand task running status.

Configure real-time monitoring and alerting rules

Data Processing

Added real-time data processing. Process real-time compute tasks.

Real-time Data Processing

Data Integration

Committed to providing powerful data pre-processing capabilities and high-speed, stable data synchronization between disparate data sources.

Data Integration

Alert Center

Displays alert events, push records, and on-call schedules for asset quality, real-time computing, and DataService Studio modules.

Alert Center

Asset Quality

Provides a full-link asset quality solution, including quality rule configuration, quality monitoring, and intelligent alerting.

Asset Quality

DataService Studio

Build APIs and Dataphin data sources through interface configuration. This meets various data usage scenarios, lowers data development barriers, and helps enterprises maximize data application value.

DataService Studio Overview

Asset Quality Sales

Added asset quality sales information. Learn how to purchase asset quality value-added features.

Purchasing guide and Activate Dataphin

Asset Quality Billing

Added billing details for asset quality.

Billing

Feature Name

Feature Description

Release Date

Release Region

References

Dataphin Instance Sales

Added Dataphin instance sales in the China (Beijing) region.

December 25, 2019

China (Shanghai), China (Shenzhen), and China (Beijing).

Purchasing guide and Activate Dataphin.

Basic Development Edition

Global basic users can only purchase the Basic Development Edition. The Basic Development Edition supports only 200 data processing units.

Purchasing guide

Upgradation

Upgrade low-specification data processing units of the Intelligent Development Edition to high-specification units. Also, upgrade the Basic Development Edition to the Intelligent Development Edition.

Upgradation

Intelligent Development Edition

Global standard users can only purchase Intelligent Development Edition data processing units with 1500 specifications.

Purchasing guide

Feature Name

Feature Description

Release Date

Release Region

References

Dataphin Instance Sales

Added Dataphin instance sales in the China (Shenzhen) region.

November 29, 2019

China (Shanghai) and China (Shenzhen).

Purchasing guide and Activate Dataphin.

Data Processing Unit

Intelligent Development Edition data processing unit specifications increased from 200 to 500. Added 1500 data processing unit specifications.

China (Shanghai) and China (Shenzhen).

Billing

Upgradation

Upgrade low-specification data processing units to high-specification data processing units.

China (Shanghai) and China (Shenzhen).

Upgradation

DataService Studio

DataService Studio provides efficient, convenient topic-based query features and effective full-link enterprise API lifecycle management. It enables low-barrier API development, helping you better apply data assets for value realization.

November 14, 2019

China (Shanghai)

Service

Data Distilling

Addressing the demand for in-depth data value mining across industries, Data Distilling provides target-object-centric data integration and in-depth mining. This helps you achieve business integration and drive innovation.

Extraction

Asset Governance

Dataphin Asset Governance supports data usage consumption analysis. From a global perspective, control compute and storage costs and improve resource utilization efficiency.

Administration

Metadata

Manually refresh to get the latest metadata information. Refresh specified data table metadata to the Dataphin system.

Refresh metadata

Security

Provides permission management.

Security Center

Notification Center

Collects messages from all workflow tasks within the system and displays them in the Notification Center.

View system messages