Feature overview
Data Development provides end-to-end compute node development capabilities. It integrates the development, execution, and debugging of data transformation workflows into a unified management experience. This lets you orchestrate data transformation workflows and reuse transformation logic, greatly improving data development efficiency.
Data Development helps you optimize your intelligent system architecture and improve engineering standards. It lets you focus on compute logic instead of infrastructure details, which boosts development speed. In addition to centralizing compute node management at the system level to reduce coupling between compute nodes and other system components, Data Development also provides a set of internal implementation standards for compute nodes. In real-world business systems, data comes from many sources. Different data types have different latency and volume requirements, so you often need multiple platforms, such as batch and stream processing environments. After one platform finishes its computation, it passes data to the next platform. These platforms work together to complete the task. Data Development defines standardized data forwarding rules between platforms and methods for sequencing compute node execution. This lets you focus on developing compute logic without worrying about low-level technical details.
Data Development standardizes compute node interfaces, which improves reusability and simplifies team collaboration. It provides cross-platform workflow configuration for compute nodes across various storage and compute systems and includes rapid publishing capabilities. When you build data applications, you can start from a specific business scenario and combine different types of compute nodes into a workflow. With the one-click execution and publishing features of Data Development for cross-platform, heterogeneous tasks, you can implement business workflows quickly and dramatically improve engineering efficiency.
Node mode
Node mode is ideal for simple data processing scenarios where the entire workflow consists of a single, relatively independent compute node. It supports flexible node development and lets you configure and manage scripts, user-defined functions, node inputs, node outputs, and parameters. It supports multiple compute node types, including batch offline, online, and stream processing, and enables task dependencies and unified scheduling across different workspaces within the same tenant.
Development node
You can create, edit, and delete node task categories. You can register offline, online, and stream nodes, and configure and manage their scripts, user-defined functions, inputs, outputs, and parameters. You can also set node dependencies and run and publish node tasks.
Create, edit, and delete node task categories.
Create, edit, and delete node tasks.
Register offline nodes and configure their scripts, user-defined functions, inputs, outputs, and parameters.
Automatically parse inputs and outputs of offline nodes based on SQL.
Select any structure for inputs and outputs for ease of use.
Configure node dependencies.
Run and publish node tasks.
Production node
You can view and unpublish node tasks in production environments. You can also view detailed information such as algorithm details, compute resources, storage resources, scheduling intervals, input parameters, output parameters, and runtime status.
Scenario mode
Scenario mode is ideal for complex algorithm workflows that are composed of multiple orchestrated nodes within a single scenario. It offers a visual and customizable task flow development experience and lets you configure and manage scripts, user-defined functions, node inputs, node outputs, and parameters. It provides end-to-end capabilities for data transformation, algorithm service workflow development, orchestration, debugging, publishing, and maintenance. You can orchestrate multiple compute node types, including stream processing and batch offline computing, within a single workflow. This mode supports global task dependencies and unified scheduling across different workspaces and scenarios within the same tenant. The visual interface lets you drag and drop components to connect compute nodes, which enables rapid editing of data transformation workflows and significantly boosts development efficiency.
Development scenario
You can manage development scenario categories and reference public nodes in development scenarios. You can orchestrate data processing flows by combining multiple compute nodes into a task flow and customize task flows visually and flexibly. You can also mix batch, stream, and other node types within a single scenario. After you run and publish scenarios in development environments, you can deploy the tested scenarios to the production environment.
Create, edit, and delete development scenario folders.
Create, edit, and delete development scenarios.
Reference public nodes in development scenarios and configure scheduling for offline compute nodes.
Create new compute nodes in development scenarios and configure their parameters and offline scheduling settings.
Create new system nodes in development scenarios and configure their settings.
Visually configure dependencies between upstream and downstream compute nodes and map input and output parameter relationships.
Automatically parse external dependency node information based on configured inputs.
Select any structure for input and output parameters.
Run and publish development scenarios.
Automatically generate Data Quality check nodes when a compute node’s output references a logical table that has quality rules configured.
Production scenario
You can run and unpublish scenarios in production environments. You can also view detailed information for each node, including algorithm details, compute resources, storage resources, scheduling intervals, inputs, and outputs.
Public nodes
You can manage public node categories. This feature supports multiple public node types, including batch and stream, and lets you configure and manage public node scripts, inputs, outputs, and parameters. You can also enable version control for public nodes.
Register public nodes and configure their scripts, user-defined functions, inputs, outputs, and parameters.
Automatically parse inputs and outputs of offline nodes based on SQL.
Select any structure for inputs and outputs for ease of use.
Manage public node versions. You can modify and optimize public nodes in development environments. After optimization, you can publish the node with a specified version number to the production environment.
Deprecate public nodes to prevent others from reusing outdated versions.
View scenarios associated with public or compute nodes for a clear overview of their usage.
Support multiple public node types, including batch and stream.