最佳实践:表管理OpenAPI基础实践

DataWorks提供了丰富的OpenAPI,您可以根据需要使用DataWorksOpenAPI等开放能力实现各种业务场景。本文以元数据表管理为例,为您介绍如何串连元数据的OpenAPI来达成查询列表、查询表详情等操作。

前置概念

在进行本实践之前,建议您先参考以下链接,了解DataWorksOpenAPI的基本能力和概念:

一、查询表列表

本实践介绍如何使用OpenAPI查询MaxCompute项目/模式下的表列表,并支持分页查询。主要流程如下:

后端:使用MetaServiceProxy查询表详情

  1. MetaServiceProxy中编写一个ListTables方法,处理前端参数并发送请求到OpenAPI ListTables中,以获取元数据表的列表信息。

    • 支持查询的参数:父层级元数据实体ID、名称(模糊匹配)、注释(模糊匹配)、类型以及排序参数和分页的参数。

      说明

      参考元数据实体相关概念说明,此处的父层级元数据实体ID可能存在两种格式。

      • 父层级元数据实体IDMaxCompute项目ID,格式为:maxcompute-project:${主账号ID}::{projectName}

      • 父层级元数据实体IDMaxCompute模式ID,格式为:maxcompute-schema:${主账号ID}::{projectName}:{schemaName}

    • 查询结果中包含内容:数据表总数、分页信息、每个数据表的ID、父层级元数据实体ID、表名称、注释、数据库名称、模式名称、类型、分区字段列表、创建时间与修改时间。

    • 获取主账号ID:无论是RAM子账号还是主账号都可在页面右上角的账号处查看主账号ID。

      1. 登录主账号或RAM子账号到DataWorks控制台

      2. 鼠标悬浮右上方的头像按钮上即可查看主账号ID。

        • 主账号:账号ID即为需要获取的主账号ID。

        • RAM账号:可直接查看到主账号ID

    • 示例代码:

      /**
       * @author dataworks demo
       */
      @Service
      public class MetaServiceProxy {
      
          @Autowired
          private DataWorksOpenApiClient dataWorksOpenApiClient;
      
          /**
           * DataWorks OpenAPI : ListTables
           *
           * @param listTablesDto
           */
          public ListTablesResponseBodyPagingInfo listTables(ListTablesDto listTablesDto) {
              try {
                  Client client = dataWorksOpenApiClient.createClient();
                  ListTablesRequest listTablesRequest = new ListTablesRequest();
                  // 父层级实体ID,元数据相关实体概念说明。
                  listTablesRequest.setParentMetaEntityId(listTablesDto.getParentMetaEntityId());
                  // 表名称,支持模糊匹配
                  listTablesRequest.setName(listTablesDto.getName());
                  // 表注释,支持模糊匹配
                  listTablesRequest.setComment(listTablesDto.getComment());
                  // 表类型,默认返回所有类型
                  listTablesRequest.setTableTypes(listTablesDto.getTableTypes());
                  // 排序方式,CreateTime (默认) / ModifyTime / Name / TableType
                  listTablesRequest.setSortBy(listTablesDto.getSortBy());
                  // 排序方向,支持 Asc (默认) / Desc
                  listTablesRequest.setOrder(listTablesDto.getOrder());
                  // 分页页码,默认为1
                  listTablesRequest.setPageNumber(listTablesDto.getPageNumber());
                  // 分页大小,默认为10,最大100
                  listTablesRequest.setPageSize(listTablesDto.getPageSize());
                  ListTablesResponse response = client.listTables(listTablesRequest);
                  // 获取符合要求的表总数
                  System.out.println(response.getBody().getPagingInfo().getTotalCount());
                  for (Table table : response.getBody().getPagingInfo().getTables()) {
                      // 表ID
                      System.out.println(table.getId());
                      // 表的父层级实体ID
                      System.out.println(table.getParentMetaEntityId());
                      // 表名称
                      System.out.println(table.getName());
                      // 表注释
                      System.out.println(table.getComment());
                      // 表所属数据库/maxcompute项目名称
                      System.out.println(table.getId().split(":")[3]);
                      // 表所属schema名称
                      System.out.println(table.getId().split(":")[4]);
                      // 表类型
                      System.out.println(table.getTableType());
                      // 表创建时间
                      System.out.println(table.getCreateTime());
                      // 表修改时间
                      System.out.println(table.getModifyTime());
                      // 表的分区字段列表
                      System.out.println(table.getPartitionKeys());
                  }
                  return response.getBody().getPagingInfo();
              } catch (TeaException error) {
                  // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                  // 错误 message
                  System.out.println(error.getMessage());
                  // 诊断地址
                  System.out.println(error.getData().get("Recommend"));
                  com.aliyun.teautil.Common.assertAsString(error.message);
              } catch (Exception _error) {
                  TeaException error = new TeaException(_error.getMessage(), _error);
                  // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                  // 错误 message
                  System.out.println(error.getMessage());
                  // 诊断地址
                  System.out.println(error.getData().get("Recommend"));
                  com.aliyun.teautil.Common.assertAsString(error.message);
              }
              return null;
          }
      }
  2. MetaRestController中添加一个listTables方法作为入口供前端访问。

    /**
     * 演示如何通过DataWorks OpenAPI构建自定义元数据平台
     *
     * @author dataworks demo
     */
    @RestController
    @RequestMapping("/meta")
    public class MetaRestController {
    
        @Autowired
        private MetaServiceProxy metaService;
    
        /**
         * 分页查询表数据
         *
         * @param listTablesDto
         * @return {@link ListTablesResponseBodyPagingInfo}
         */
        @CrossOrigin(origins = "http://localhost:8080")
        @GetMapping("/listTables")
        public ListTablesResponseBodyPagingInfo listTables(ListTablesDto listTablesDto) {
            System.out.println("listTablesDto = " + listTablesDto);
            return metaService.listTables(listTablesDto);
        }
    }

前端:展示表元数据信息

import React from "react";
import cn from "classnames";
import {
  Table,
  Form,
  Field,
  Input,
  Button,
  Pagination,
  Dialog,
  Card,
  Grid,
  Select,
} from "@alifd/next";
import type { TableListInput, TableEntity } from "../services/meta";
import * as services from "../services";
import DetailContent from "./detailContent";
import LineageContent from "./lineageContent";
import classes from "../styles/app.module.css";
import moment from "moment";

export interface Props {}

const { Column } = Table;
const { Item } = Form;
const { Header, Content } = Card;
const { Row, Col } = Grid;
const { Option } = Select;

const App: React.FunctionComponent<Props> = () => {
  const field = Field.useField();
  const [datasource, setDatasource] = React.useState<TableEntity[]>([]);
  const [loading, setLoading] = React.useState<boolean>(false);
  const [total, setTotal] = React.useState<number>(0);
  const [current, setCurrent] = React.useState<number>(1);
  const onSubmit = React.useCallback(
    (pageNumber: number = 1) => {
      field.validate(async (errors, values) => {
        if (errors) {
          return;
        }
        setLoading(true);
        try {
          const response = await services.meta.getTableList({
            pageNumber,
            ...values,
          } as TableListInput);
          setDatasource(response.tables);
          setCurrent(pageNumber);
          setTotal(response.totalCount);
        } catch (e) {
          throw e;
        } finally {
          setLoading(false);
        }
      });
    },
    [field]
  );
  const onViewDetail = React.useCallback((record: TableEntity) => {
    Dialog.show({
      title: "数据表详情",
      content: <DetailContent item={record} />,
      shouldUpdatePosition: true,
      footerActions: ["ok"],
    });
  }, []);
  const onViewLineage = React.useCallback((record: TableEntity) => {
    Dialog.show({
      title: "表血缘关系",
      content: <LineageContent item={record} />,
      shouldUpdatePosition: true,
      footerActions: ["ok"],
    });
  }, []);
  React.useEffect(() => {
    field.setValue("dataSourceType", "odps");
  }, []);
  return (
    <div className={cn(classes.appWrapper)}>
    <Card free hasBorder={false}>
    <Header title="元数据表管理场景Demo(MaxCompute)" />
    <Content style={{ marginTop: 24 }}>
  <Form field={field} colon fullWidth>
  <Row gutter="1">
  <Col>
  <Item
label="MaxCompute项目/模式ID"
name="parentMetaEntityId"
required
help="ID"
  >
  <Input />
  </Item>
  </Col>
  </Row>

  <Row gutter="4">
  <Col>
  <Item label="名称" name="name">
  <Input placeholder="输入表名称,模糊匹配" />
  </Item>
  </Col>
  <Col>
                <Item label="注释" name="comment">
                  <Input placeholder="输入表注释,模糊匹配" />
                </Item>
              </Col>
              <Col>
                <Item label="类型" name="tableTypes">
                  <Select mode="multiple" style={{ width: "100%" }}>
                    <Option value="TABLE">表</Option>
                    <Option value="VIEW">视图</Option>
                    <Option value="MATERIALIZED_VIEW">物化视图</Option>
                  </Select>
                </Item>
              </Col>
              <Col>
                <Item label="排序方式" name="sortBy">
                  <Select defaultValue="CreateTime" style={{ width: "100%" }}>
                    <Option value="CreateTime">创建时间</Option>
                    <Option value="ModifyTime">修改时间</Option>
                    <Option value="Name">名称</Option>
                    <Option value="TableType">类型</Option>
                  </Select>
                </Item>
              </Col>
              <Col>
                <Item label="排序方向" name="order">
                  <Select defaultValue="Asc" style={{ width: "100%" }}>
                    <Option value="Asc">升序</Option>
                    <Option value="Desc">降序</Option>
                  </Select>
                </Item>
              </Col>
            </Row>
            <div
              className={cn(
                classes.searchPanelButtonWrapper,
                classes.buttonGroup
              )}
            >
              <Button type="primary" onClick={() => onSubmit()}>
                查询
              </Button>
            </div>
          </Form>
          <div>
            <Table
              dataSource={datasource}
              loading={loading}
              className={cn(classes.tableWrapper)}
              emptyContent={
                <div className={cn(classes.noDataWrapper)}>暂无数据</div>
              }
            >
              <Column
                title="项目名称"
                dataIndex="id"
                cell={(value) => {
                  const parts = value.split(":");
                  return parts.length > 1 ? parts[parts.length - 3] : "";
                }}
              />
              <Column
                title="schema"
                dataIndex="id"
                cell={(value) => {
                  const parts = value.split(":");
                  return parts.length > 1 ? parts[parts.length - 2] : "";
                }}
              />
              <Column title="表名称" dataIndex="name" />
              <Column title="表注释" dataIndex="comment" />
              <Column title="表类型" dataIndex="tableType" />
              <Column
                title="是否分区表"
                dataIndex="partitionKeys"
                cell={(value) => {
                  return value != null && value.length > 0 ? "是" : "否";
                }}
              />
              <Column
                title="创建时间"
                dataIndex="createTime"
                cell={(value) => {
                  return moment(value).format("YYYY-MM-DD HH:mm:ss");
                }}
              />
              <Column
                title="修改时间"
                dataIndex="modifyTime"
                cell={(value) => {
                  return moment(value).format("YYYY-MM-DD HH:mm:ss");
                }}
              />
              <Column
                title="操作"
                width={150}
                cell={(value, index, record) => (
                  <div className={cn(classes.buttonGroup)}>
                    <Button
                      type="primary"
                      onClick={() => onViewDetail(record)}
                      text
                    >
                      查看详情
                    </Button>
                    <Button
                      type="primary"
                      onClick={() => onViewLineage(record)}
                      text
                    >
                      查看表血缘
                    </Button>
                  </div>
                )}
              />
            </Table>
            <Pagination
              current={current}
              total={total}
              onChange={onSubmit}
              showJump={false}
              className={cn(classes.tablePaginationWrapper)}
            />
          </div>
        </Content>
      </Card>
    </div>
  );
};

export default App;

完成上述代码开发后,您可在通用操作:本地部署运行工程代码。您可以输入MaxCompute项目空间ID及其他参数进行查询,获得项目空间下的所有表的列表,支持分页查询。

二、查找表详情

以下实践将结合元数据中GetTableListColumnsListPartitions三个OpenAPI来实现查询表详情,通过UpdateColumnBusinessMetadataAPI,更新字段的业务说明,实践操作流程如下。

后端:使用MetaServiceProxy查询表详情

  1. MetaServiceProxy中创建getTablelistColumnslistPartitions方法来调用OpenAPI服务获取表的基础信息、表的字段信息与表的分区信息,创建updateColumnBusinessMetadata方法来更新字段的业务说明

    /**
     * @author dataworks demo
     */
    @Service
    public class MetaServiceProxy {
    
        @Autowired
        private DataWorksOpenApiClient dataWorksOpenApiClient;
    
        /**
         * DataWorks OpenAPI : getTableDto
         *
         * @param getTableDto
         */
        public Table getTable(GetTableDto getTableDto) {
            try {
                Client client = dataWorksOpenApiClient.createClient();
                GetTableRequest getTableRequest = new GetTableRequest();
                // 数据表ID
                getTableRequest.setId(getTableDto.getId());
                // 是否返回业务元数据
                getTableRequest.setIncludeBusinessMetadata(getTableDto.getIncludeBusinessMetadata());
    
                GetTableResponse response = client.getTable(getTableRequest);
                // 对应的数据表
                Table table = response.getBody().getTable();
                // 表ID
                System.out.println(table.getId());
                // 表的父层级实体ID
                System.out.println(table.getParentMetaEntityId());
                // 表名称
                System.out.println(table.getName());
                // 表注释
                System.out.println(table.getComment());
                // 表所属数据库 / maxcompute项目名称
                System.out.println(table.getId().split(":")[3]);
                // 表所属schema名称
                System.out.println(table.getId().split(":")[4]);
                // 表类型
                System.out.println(table.getTableType());
                // 表创建时间
                System.out.println(table.getCreateTime());
                // 表修改时间
                System.out.println(table.getModifyTime());
                // 是否分区表
                System.out.println(table.getPartitionKeys() != null && !table.getPartitionKeys().isEmpty());
                // 表的技术元数据
                TableTechnicalMetadata technicalMetadata = table.getTechnicalMetadata();
                // 表的负责人(名称)
                System.out.println(technicalMetadata.getOwner());
                // 是否压缩表
                System.out.println(technicalMetadata.getCompressed());
                // 输入格式(HMS, DLF等类型支持)
                System.out.println(technicalMetadata.getInputFormat());
                // 输出格式(HMS, DLF等类型支持)
                System.out.println(technicalMetadata.getOutputFormat());
                // 表序列化使用的类(HMS, DLF等类型支持)
                System.out.println(technicalMetadata.getSerializationLibrary());
                // 表存储位置(HMS, DLF等类型支持)
                System.out.println(technicalMetadata.getLocation());
                // 其他参数
                Map<String, String> parameters = technicalMetadata.getParameters();
                // 上次DDL表更时间(毫秒级时间戳),仅MaxCompute类型支持
                System.out.println(parameters.get("lastDDLTime"));
                // 生命周期(单位:天),仅MaxCompute类型支持
                System.out.println(parameters.get("lifecycle"));
                // 存储量(单位:字节),非实时,仅MaxCompute类型支持
                System.out.println(parameters.get("dataSize"));
                // 表的业务元数据
                TableBusinessMetadata businessMetadata = table.getBusinessMetadata();
                if (businessMetadata != null) {
                    // 使用说明
                    System.out.println(businessMetadata.getReadme());
                    // 自定义标签信息
                    List<TableBusinessMetadataTags> tags = businessMetadata.getTags();
                    if (tags != null && !tags.isEmpty()) {
                        for (TableBusinessMetadataTags tag : tags) {
                            System.out.println(tag.getKey() + ":" + tag.getValue());
                        }
                    }
                    // 上游产出任务
                    List<TableBusinessMetadataUpstreamTasks> upstreamTasks = businessMetadata.getUpstreamTasks();
                    if (upstreamTasks != null && !upstreamTasks.isEmpty()) {
                        for (TableBusinessMetadataUpstreamTasks upstreamTask : upstreamTasks) {
                            // 任务ID 与任务名称,可通过 GetTask API 获取任务详情信息
                            System.out.println(upstreamTask.getId() + ":" + upstreamTask.getName());
                        }
                    }
                    // 所属类目信息
                    List<List<TableBusinessMetadataCategories>> categories = businessMetadata.getCategories();
                    if (categories != null && !categories.isEmpty()) {
                        // 多级类目的遍历
                        for (List<TableBusinessMetadataCategories> category : categories) {
                            // 输出单个多级类目
                            System.out.println(
                                    category.stream()
                                            .map(TableBusinessMetadataCategories::getName)
                                            .collect(Collectors.joining("->"))
                            );
                        }
                    }
                    // 扩展信息,仅MaxCompute类型支持
                    TableBusinessMetadataExtension extension = businessMetadata.getExtension();
                    if (extension != null) {
                        // 所属项目空间ID
                        System.out.println(extension.getProjectId());
                        // 所属环境类型,Prod 生产 / Dev 开发 
                        System.out.println(extension.getEnvType());
                        // 收藏次数
                        System.out.println(extension.getFavorCount());
                        // 读取次数
                        System.out.println(extension.getReadCount());
                        // 浏览次数
                        System.out.println(extension.getViewCount());
                    }
                }
                return table;
            } catch (TeaException error) {
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            } catch (Exception _error) {
                TeaException error = new TeaException(_error.getMessage(), _error);
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            }
            return null;
        }
    
        /**
         * DataWorks OpenAPI : ListColumns
         *
         * @param listColumnsDto
         */
        public ListColumnsResponseBodyPagingInfo listColumns(ListColumnsDto listColumnsDto) {
            try {
                Client client = dataWorksOpenApiClient.createClient();
                ListColumnsRequest listColumnsRequest = new ListColumnsRequest();
                // 数据表ID,可通过 ListTables 接口获取
                listColumnsRequest.setTableId(listColumnsDto.getTableId());
                // 字段名称,支持模糊匹配
                listColumnsRequest.setName(listColumnsDto.getName());
                // 字段注释,支持分词匹配
                listColumnsRequest.setComment(listColumnsDto.getComment());
                // 排序方式,支持 Name / Position (默认)
                listColumnsRequest.setSortBy(listColumnsDto.getSortBy());
                // 排序方向,支持 Asc (默认) / Desc
                listColumnsRequest.setOrder(listColumnsDto.getOrder());
                // 分页页码,默认为1
                listColumnsRequest.setPageNumber(listColumnsDto.getPageNumber());
                // 分页大小,默认为10,最大100
                listColumnsRequest.setPageSize(listColumnsDto.getPageSize());
                ListColumnsResponse response = client.listColumns(listColumnsRequest);
                // 获取符合要求的字段总数
                System.out.println(response.getBody().getRequestId());
                System.out.println(response.getBody().getPagingInfo().getTotalCount());
                for (Column column : response.getBody().getPagingInfo().getColumns()) {
                    // 字段ID
                    System.out.println(column.getId());
                    // 字段名称
                    System.out.println(column.getName());
                    // 字段注释
                    System.out.println(column.getComment());
                    // 字段类型
                    System.out.println(column.getType());
                    // 字段位置
                    System.out.println(column.getPosition());
                    // 是否分区字段
                    System.out.println(column.getPartitionKey());
                    // 是否主键,仅MaxCompute类型支持
                    System.out.println(column.getPrimaryKey());
                    // 是否外键,仅MaxCompute类型支持
                    System.out.println(column.getForeignKey());
                    // 字段业务说明,目前MaxCompute, HMS(EMR集群), DLF类型支持
                    if (column.getBusinessMetadata() != null) {
                        System.out.println(column.getBusinessMetadata().getDescription());
                    }
                }
                return response.getBody().getPagingInfo();
            } catch (TeaException error) {
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            } catch (Exception _error) {
                TeaException error = new TeaException(_error.getMessage(), _error);
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            }
            return null;
        }
    
        /**
         * DataWorks OpenAPI : UpdateColumnBusinessMetadata
         *
         * @param updateColumnBusinessMetadataDto
         */
        public boolean updateColumnBusinessMetadata(UpdateColumnBusinessMetadataDto updateColumnBusinessMetadataDto) {
            try {
                // 当前支持MaxCompute, DLF, HMS(EMR集群)类型
                Client client = dataWorksOpenApiClient.createClient();
                UpdateColumnBusinessMetadataRequest updateColumnBusinessMetadataRequest = new UpdateColumnBusinessMetadataRequest();
                // 字段ID
                updateColumnBusinessMetadataRequest.setId(updateColumnBusinessMetadataDto.getId());
                // 字段业务说明
                updateColumnBusinessMetadataRequest.setDescription(updateColumnBusinessMetadataDto.getDescription());
                UpdateColumnBusinessMetadataResponse response = client.updateColumnBusinessMetadata(updateColumnBusinessMetadataRequest);
                System.out.println(response.getBody().getRequestId());
                // 是否更新成功
                return response.getBody().getSuccess();
            } catch (TeaException error) {
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            } catch (Exception _error) {
                TeaException error = new TeaException(_error.getMessage(), _error);
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            }
            return false;
        }
    
        /**
         * DataWorks OpenAPI : ListPartitions
         *
         * @param listPartitionsDto
         */
        public ListPartitionsResponseBodyPagingInfo listPartitions(ListPartitionsDto listPartitionsDto) {
            try {
                // 当前支持 MaxCompute类型和HMS(EMR集群)类型
                Client client = dataWorksOpenApiClient.createClient();
                ListPartitionsRequest listPartitionsRequest = new ListPartitionsRequest();
                // 数据表ID,可通过 ListTables 接口获取
                listPartitionsRequest.setTableId(listPartitionsDto.getTableId());
                // 分区名称,支持模糊匹配
                listPartitionsRequest.setName(listPartitionsDto.getName());
                // 排序方式,支持 Name(HMS方式默认,Maxcompute类型支持) / CreateTime (MaxCompute类型默认) / ModifyTime (MaxCompute类型支持)/ RecordCount(MaxCompute类型支持)/ DataSize(MaxCompute类型支持)
                listPartitionsRequest.setSortBy(listPartitionsDto.getSortBy());
                // 排序方向,支持 Asc (默认) / Desc
                listPartitionsRequest.setOrder(listPartitionsDto.getOrder());
                // 分页页码,默认为1
                listPartitionsRequest.setPageNumber(listPartitionsDto.getPageNumber());
                // 分页大小,默认为10,最大100
                listPartitionsRequest.setPageSize(listPartitionsDto.getPageSize());
                ListPartitionsResponse response = client.listPartitions(listPartitionsRequest);
                // 获取符合要求的分区总数
                System.out.println(response.getBody().getPagingInfo().getTotalCount());
                for (Partition partition : response.getBody().getPagingInfo().getPartitionList()) {
                    // 所属数据表ID
                    System.out.println(partition.getTableId());
                    // 分区名称
                    System.out.println(partition.getName());
                    // 创建时间(毫秒级时间戳)
                    System.out.println(partition.getCreateTime());
                    // 修改时间(毫秒级时间戳)
                    System.out.println(partition.getModifyTime());
                    // 分区记录数,仅MaxCompute类型支持
                    System.out.println(partition.getRecordCount());
                    // 分区存储量(单位:字节),仅MaxCompute类型支持
                    System.out.println(partition.getDataSize());
                }
                return response.getBody().getPagingInfo();
            } catch (TeaException error) {
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            } catch (Exception _error) {
                TeaException error = new TeaException(_error.getMessage(), _error);
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            }
            return null;
        }
    }
  2. MetaRestController中提供四个方法分别为getTablelistColumnslistPartitionsupdateColumnBusinessMetadata供前端进行调用。

    /**
     * 演示如何通过DataWorks OpenAPI构建自定义元数据平台
     *
     * @author dataworks demo
     */
    @RestController
    @RequestMapping("/meta")
    public class MetaRestController {
    
        @Autowired
        private MetaServiceProxy metaService;
    
        /**
         * 获取表详情
         *
         * @param getTableDto
         * @return {@link Table}
         */
        @CrossOrigin(origins = "http://localhost:8080")
        @GetMapping("/getTable")
        public Table getTable(GetTableDto getTableDto) {
            System.out.println("getTableDto = " + getTableDto);
            return metaService.getTable(getTableDto);
        }
    
        /**
         * 分页查询字段数据
         *
         * @param listColumnsDto
         * @return {@link ListColumnsResponseBodyPagingInfo}
         */
        @CrossOrigin(origins = "http://localhost:8080")
        @GetMapping("/listColumns")
        public ListColumnsResponseBodyPagingInfo listColumns(ListColumnsDto listColumnsDto) {
            System.out.println("listColumnsDto = " + listColumnsDto);
            return metaService.listColumns(listColumnsDto);
        }
    
        /**
         * 更新字段业务说明
         *
         * @param updateColumnBusinessMetadataDto
         * @return {@link Boolean}
         */
        @CrossOrigin(origins = "http://localhost:8080")
        @PostMapping("/updateColumnBusinessMetadata")
        public Boolean updateColumnBusinessMetadata(@RequestBody UpdateColumnBusinessMetadataDto updateColumnBusinessMetadataDto) {
            System.out.println("updateColumnBusinessMetadataDto = " + updateColumnBusinessMetadataDto);
            return metaService.updateColumnBusinessMetadata(updateColumnBusinessMetadataDto);
        }
    
        /**
         * 分页查询分区数据
         *
         * @param listPartitionsDto
         * @return {@link ListPartitionsResponseBodyPagingInfo}
         */
        @CrossOrigin(origins = "http://localhost:8080")
        @GetMapping("/listPartitions")
        public ListPartitionsResponseBodyPagingInfo listPartitions(ListPartitionsDto listPartitionsDto) {
            System.out.println("listPartitionsDto = " + listPartitionsDto);
            return metaService.listPartitions(listPartitionsDto);
        }
    }

前端:展示表基本信息、表字段信息以及表分区的信息。

import React from "react";
import moment from "moment";
import {
  Form,
  Grid,
  Table,
  Pagination,
  Tag,
  Field,
  Input,
  Button,
  Select,
  Dialog,
} from "@alifd/next";
import cn from "classnames";
import * as services from "../services";
import {
  type ListColumnsInput,
  type TableColumn,
  type TableEntity,
  type TablePartition,
} from "../services/meta";
import classes from "../styles/detailContent.module.css";

export interface Props {
  item: TableEntity;
}

const formItemLayout = {
  labelCol: {
    fixedSpan: 6,
  },
  wrapperCol: {
    span: 18,
  },
  labelTextAlign: "left" as const,
  colon: true,
  className: cn(classes.formItemWrapper),
};
const { Row, Col } = Grid;
const { Item } = Form;
const { Column } = Table;
const { Option } = Select;
const DetailContent: React.FunctionComponent<Props> = (props) => {
  let columnsPageSize = 10;
  let partitionsPageSize = 5;
  const listColumnsField = Field.useField();
  const listPartitionsField = Field.useField();
  const labelAlign = "top";
  const [detail, setDetail] = React.useState<Partial<TableEntity>>({});
  const [columns, setColumns] = React.useState<Partial<TableColumn[]>>([]);
  const [partitions, setPartitions] = React.useState<Partial<TablePartition[]>>(
    []
  );
  const [columnsTotal, setColumnsTotal] = React.useState<number>(0);
  const [partitionTotal, setPartitionTotal] = React.useState<number>(0);
  const [editingKey, setEditingKey] = React.useState<string>("");
  const [editingDescription, setEditingDescription] =
    React.useState<string>("");

  const [updateColumnDialogVisible, setUpdateColumnDialogVisible] =
    React.useState<boolean>(false);
  const onUpdateColumnDialogOpen = () => {
    setUpdateColumnDialogVisible(true);
  };
  const onUpdateColumnDialogOk = () => {
    handleColumnDescriptionSave();
    onUpdateColumnDialogClose();
  };
  const onUpdateColumnDialogClose = () => {
    setUpdateColumnDialogVisible(false);
  };
  const handleEditColumnDescription = async (record: TableColumn) => {
    setEditingKey(record.id);
    setEditingDescription(record.businessMetadata?.description);
    onUpdateColumnDialogOpen();
  };
  const handleColumnDesrciptionChange = async (e: string) => {
    setEditingDescription(e);
  };
  const handleColumnDescriptionSave = async () => {
    try {
      const res = await services.meta.updateColumnDescription({
        id: editingKey,
        description: editingDescription,
      });
      if (!res) {
        console.log("请求更新失败");
      }
    } catch (e) {
      console.error("更新失败", e);
    } finally {
      listColumns();
      setEditingKey("");
    }
  };
  const getTableDetail = React.useCallback(async () => {
    const response = await services.meta.getTableDetail({
      id: props.item.id,
      includeBusinessMetadata: true,
    });
    setDetail(response);
  }, [props.item.id]);
  const listColumns = React.useCallback(
    async (pageNumber: number = 1) => {
      listColumnsField.setValue("tableId", props.item.id);
      listColumnsField.setValue("pageSize", columnsPageSize);
      listColumnsField.validate(async (errors, values) => {
        if (errors) {
          return;
        }
        try {
          const response = await services.meta.getMetaTableColumns({
            pageNumber,
            ...values,
          } as ListColumnsInput);
          setColumns(response.columns);
          setColumnsTotal(response.totalCount);
        } catch (e) {
          throw e;
        } finally {
        }
      });
    },
    [listColumnsField]
  );
  const listPartitions = React.useCallback(
    async (pageNumber: number = 1) => {
      listPartitionsField.setValue("tableId", props.item.id);
      listPartitionsField.setValue("pageSize", partitionsPageSize);
      listPartitionsField.validate(async (errors, values) => {
        if (errors) {
          return;
        }
        try {
          const response = await services.meta.getTablePartition({
            pageNumber,
            ...values,
          } as ListColumnsInput);
          setPartitions(response.partitionList);
          setPartitionTotal(response.totalCount);
        } catch (e) {
          throw e;
        } finally {
        }
      });
    },
    [listPartitionsField]
  );
  React.useEffect(() => {
    if (props.item.id) {
      getTableDetail();
      listColumns();
      listPartitions();
    }
  }, [props.item.id]);
  return (
    <div className={cn(classes.detailContentWrapper)}>
      <Dialog
        v2
        title="更新字段业务元数据"
        visible={updateColumnDialogVisible}
        onOk={onUpdateColumnDialogOk}
        onClose={onUpdateColumnDialogClose}
      >
        <Row>
          <Col>
            <Item
              {...formItemLayout}
              labelAlign={labelAlign}
              label="字段业务说明"
            >
              <Input
                value={editingDescription}
                onChange={(e) => handleColumnDesrciptionChange(e.toString())}
              ></Input>
            </Item>
          </Col>
        </Row>
      </Dialog>
      <Form labelTextAlign="left">
        <Row>
          <Col>
            <Item {...formItemLayout} label="表ID">
              <span className={cn(classes.formContentWrapper)}>
                {detail.id}
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="MaxCompute项目">
              <span className={cn(classes.formContentWrapper)}>
                {detail?.id?.split(":").slice(-3, -2)[0]}
              </span>
            </Item>
          </Col>
          <Col>
            <Item {...formItemLayout} label="schema名称">
              <span className={cn(classes.formContentWrapper)}>
                {detail?.id?.split(":").slice(-2, -1)[0]}
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="表名称">
              <span className={cn(classes.formContentWrapper)}>
                {detail.name}
              </span>
            </Item>
          </Col>
          <Col>
            <Item {...formItemLayout} label="注释">
              <span className={cn(classes.formContentWrapper)}>
                {detail.comment}
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="表类型">
              <span className={cn(classes.formContentWrapper)}>
                {detail.tableType}
              </span>
            </Item>
          </Col>
          <Col>
            <Item {...formItemLayout} label="是否压缩表">
              <span className={cn(classes.formContentWrapper)}>
                {detail.technicalMetadata?.compressed ? "是" : "否"}
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="所属工作空间ID">
              <span className={cn(classes.formContentWrapper)}>
                {detail.businessMetadata?.extension?.projectId}
              </span>
            </Item>
          </Col>
          <Col>
            <Item {...formItemLayout} label="环境">
              <span className={cn(classes.formContentWrapper)}>
                {detail.businessMetadata?.extension?.envType === "Dev"
                  ? "开发"
                  : "生产"}
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="表生命周期">
              <span className={cn(classes.formContentWrapper)}>
                {detail.technicalMetadata?.parameters["lifecycle"]
                  ? `${detail.technicalMetadata?.parameters["lifecycle"]} 天`
                  : "永久"}
              </span>
            </Item>
          </Col>
          <Col>
            <Item {...formItemLayout} label="存储空间">
              <span
                className={cn(classes.formContentWrapper)}
              >{`${detail.technicalMetadata?.parameters["dataSize"]} Bytes`}</span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="DDL上次变更时间">
              <span className={cn(classes.formContentWrapper)}>
                {moment(
                  Number(detail.technicalMetadata?.parameters["lastDDLTime"])
                ).format("YYYY-MM-DD HH:mm:ss")}
              </span>
            </Item>
          </Col>
          <Col>
            <Item {...formItemLayout} label="是否为分区表">
              <span className={cn(classes.formContentWrapper)}>
                {detail.partitionKeys != null && detail.partitionKeys.length > 0
                  ? "是"
                  : "否"}
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="标签">
              <div className={cn(classes.formContentWrapper)}>
                {detail.businessMetadata?.tags?.map((tag, index) => (
                  <span key={index} className={cn(classes.tag)}>
                    <Tag type="primary" size="small">
                      {tag.key + (tag.value ? ":" + tag.value : "")}
                    </Tag>
                  </span>
                ))}
              </div>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="类目">
              <span className={cn(classes.formContentWrapper)}>
                {detail.businessMetadata?.categories?.map((category, index) => (
                  <span key={index} className={cn(classes.tag)}>
                    <Tag type="normal" color="gray" size="small">
                      {category.map((obj) => obj.name).join("->")}
                    </Tag>
                  </span>
                ))}
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="上游产出任务">
              <span className={cn(classes.formContentWrapper)}>
                {detail.businessMetadata?.upstreamTasks?.map((task, index) => (
                  <span key={index} className={cn(classes.tag)}>
                    <Tag type="primary" size="small">
                      {task.name + " (" + task.id + ")"}
                    </Tag>
                  </span>
                ))}
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="浏览次数">
              <span className={cn(classes.formContentWrapper)}>
                <Tag type="primary" size="small">
                  {detail.businessMetadata?.extension?.viewCount}
                </Tag>
              </span>
            </Item>
          </Col>
          <Col>
            <Item {...formItemLayout} label="读取次数">
              <span className={cn(classes.formContentWrapper)}>
                <Tag type="primary" size="small">
                  {detail.businessMetadata?.extension?.readCount}
                </Tag>
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="收藏次数">
              <span className={cn(classes.formContentWrapper)}>
                <Tag type="primary" size="small">
                  {detail.businessMetadata?.extension?.favorCount}
                </Tag>
              </span>
            </Item>
          </Col>
        </Row>
        <Row>
          <Col>
            <Item {...formItemLayout} label="创建时间">
              <span className={cn(classes.formContentWrapper)}>
                {moment(detail.createTime).format("YYYY-MM-DD HH:mm:ss")}
              </span>
            </Item>
          </Col>
          <Col>
            <Item {...formItemLayout} label="修改时间">
              <span className={cn(classes.formContentWrapper)}>
                {moment(detail.modifyTime).format("YYYY-MM-DD HH:mm:ss")}
              </span>
            </Item>
          </Col>
        </Row>
        <br />
        <Item label="字段详情" colon>
          <Form
            field={listColumnsField}
            colon
            fullWidth
            style={{ marginTop: 10, marginBottom: 10 }}
          >
            <Row gutter="4">
              <Col>
                <Item
                  {...formItemLayout}
                  labelAlign={labelAlign}
                  label="名称"
                  name="name"
                >
                  <Input placeholder="输入字段名称,模糊匹配" />
                </Item>
              </Col>
              <Col>
                <Item
                  {...formItemLayout}
                  labelAlign={labelAlign}
                  label="注释"
                  name="comment"
                >
                  <Input placeholder="输入字段注释,模糊匹配" />
                </Item>
              </Col>
              <Col>
                <Item
                  {...formItemLayout}
                  labelAlign={labelAlign}
                  label="排序方式"
                  name="sortBy"
                >
                  <Select defaultValue="Position" style={{ width: "100%" }}>
                    <Option value="Position">位置</Option>
                    <Option value="Name">名称</Option>
                  </Select>
                </Item>
              </Col>
              <Col>
                <Item
                  {...formItemLayout}
                  labelAlign={labelAlign}
                  label="排序方向"
                  name="order"
                >
                  <Select defaultValue="Asc" style={{ width: "100%" }}>
                    <Option value="Asc">升序</Option>
                    <Option value="Desc">降序</Option>
                  </Select>
                </Item>
              </Col>
              <Col>
                <div
                  className={cn(
                    classes.searchPanelButtonWrapper,
                    classes.buttonGroup
                  )}
                  style={{ marginTop: "20px", textAlign: "right" }}
                >
                  <Button type="primary" onClick={() => listColumns()}>
                    查询
                  </Button>
                </div>
              </Col>
            </Row>
          </Form>
          <Table dataSource={columns}>
            <Column title="名称" dataIndex="name" />
            <Column title="类型" dataIndex="type" />
            <Column title="注释" dataIndex="comment" />
            <Column
              title="业务说明"
              dataIndex="businessMetadata"
              cell={(value, index, record) => (
                <div
                  style={{
                    display: "flex",
                    justifyContent: "space-between",
                    alignItems: "center",
                  }}
                >
                  <span>{value?.description}</span>
                  <Button
                    onClick={() => handleEditColumnDescription(record)}
                    size="small"
                    type="primary"
                    text
                    style={{ marginLeft: 8 }}
                  >
                    编辑
                  </Button>
                </div>
              )}
            />
            <Column
              title="是否为主键"
              dataIndex="primaryKey"
              cell={(value) => (value ? "是" : "")}
            />
            <Column
              title="是否为外键"
              dataIndex="foreignKey"
              cell={(value) => (value ? "是" : "")}
            />
            <Column
              title="是否为分区字段"
              dataIndex="partitionKey"
              cell={(value) => (value ? "是" : "")}
            />
            <Column title="位置" dataIndex="position" />
          </Table>
          <Pagination
            total={columnsTotal}
            pageSize={columnsPageSize}
            onChange={listColumns}
            showJump={false}
            className={cn(classes.tablePaginationWrapper)}
          />
        </Item>
        <br />
        <Item label="分区详情" style={{ marginBottom: 32 }} colon>
          <Form
            field={listPartitionsField}
            colon
            fullWidth
            style={{ marginTop: 10, marginBottom: 10 }}
          >
            <Row gutter="4">
              <Col>
                <Item
                  {...formItemLayout}
                  labelAlign={labelAlign}
                  label="名称"
                  name="name"
                >
                  <Input placeholder="输入分区名称,模糊匹配" />
                </Item>
              </Col>
              <Col>
                <Item
                  {...formItemLayout}
                  labelAlign={labelAlign}
                  label="排序方式"
                  name="sortBy"
                >
                  <Select defaultValue="CreateTime" style={{ width: "100%" }}>
                    <Option value="CreateTime">创建时间</Option>
                    <Option value="ModifyTime">修改时间</Option>
                    <Option value="Name">分区名称</Option>
                    <Option value="RecordCount">分区记录数</Option>
                    <Option value="DataSize">分区大小</Option>
                  </Select>
                </Item>
              </Col>
              <Col>
                <Item
                  {...formItemLayout}
                  labelAlign={labelAlign}
                  label="排序方向"
                  name="order"
                >
                  <Select defaultValue="Asc" style={{ width: "100%" }}>
                    <Option value="Asc">升序</Option>
                    <Option value="Desc">降序</Option>
                  </Select>
                </Item>
              </Col>
              <Col>
                <div
                  className={cn(
                    classes.searchPanelButtonWrapper,
                    classes.buttonGroup
                  )}
                  style={{ marginTop: "20px", textAlign: "right" }}
                >
                  <Button type="primary" onClick={() => listPartitions()}>
                    查询
                  </Button>
                </div>
              </Col>
            </Row>
          </Form>
          <Table dataSource={partitions} emptyContent={<span>非分区表</span>}>
            <Column title="名称" dataIndex="name" />
            <Column
              title="创建时间"
              dataIndex="createTime"
              cell={(value) => moment(value).format("YYYY-MM-DD HH:mm:ss")}
            />
            <Column
              title="修改时间"
              dataIndex="modifyTime"
              cell={(value) => moment(value).format("YYYY-MM-DD HH:mm:ss")}
            />
            <Column
              title="分区大小"
              dataIndex="dataSize"
              cell={(value) => `${value} bytes`}
            />
            <Column title="分区记录数" dataIndex="recordCount" />
          </Table>
          <Pagination
            total={partitionTotal}
            pageSize={partitionsPageSize}
            onChange={listPartitions}
            showJump={false}
            className={cn(classes.tablePaginationWrapper)}
          />
        </Item>
      </Form>
    </div>
  );
};

export default DetailContent;

完成上述代码开发后,您可在本地部署并运行工程代码。部署并运行的操作请参见通用操作:本地部署运行

三、查找表血缘

当一张表口径发生变更时,您可以通过DataWorks OpenAPI进行上下游任务的血缘分析,查找表对应的节点。

说明

使用ListLineages来查询数据表的上下游血缘。

后端:使用MetaServiceProxy查询表血缘

  1. MetaServiceProxy中创建listTableLineages方法来调用OpenAPI服务获取表的上下游血缘实体和关系信息。

    /**
     * @author dataworks demo
     */
    @Service
    public class MetaServiceProxy {
    
        @Autowired
        private DataWorksOpenApiClient dataWorksOpenApiClient;
    
        /**
         * DataWorks OpenAPI : ListLineages
         *
         * @param listTableLineagesDto
         */
        public ListLineagesResponseBodyPagingInfo listTableLineages(ListTableLineagesDto listTableLineagesDto) {
            try {
                Client client = dataWorksOpenApiClient.createClient();
                ListLineagesRequest listLineagesRequest = new ListLineagesRequest();
                // 血缘查询方向,是否为上游
                boolean needUpstream = "up".equalsIgnoreCase(listTableLineagesDto.getDirection());
                // 数据表ID,可通过 ListTables 接口获取
                // 分区名称,支持模糊匹配
                if (needUpstream) {
                    listLineagesRequest.setDstEntityId(listTableLineagesDto.getTableId());
                    listLineagesRequest.setSrcEntityName(listTableLineagesDto.getName());
                } else {
                    listLineagesRequest.setSrcEntityId(listTableLineagesDto.getTableId());
                    listLineagesRequest.setDstEntityName(listTableLineagesDto.getName());
                }
                // 排序方式,支持 Name
                listLineagesRequest.setSortBy(listTableLineagesDto.getSortBy());
                // 排序方向,支持 Asc (默认) / Desc
                listLineagesRequest.setOrder(listTableLineagesDto.getOrder());
                // 分页页码,默认为1
                listLineagesRequest.setPageNumber(listTableLineagesDto.getPageNumber());
                // 分页大小,默认为10,最大100
                listLineagesRequest.setPageSize(listTableLineagesDto.getPageSize());
                // 是否需要返回具体的血缘关系信息
                listLineagesRequest.setNeedAttachRelationship(true);
                ListLineagesResponse response = client.listLineages(listLineagesRequest);
                // 获取符合要求的血缘实体总数
                System.out.println(response.getBody().getPagingInfo().getTotalCount());
                if (response.getBody().getPagingInfo().getLineages() == null) {
                    response.getBody().getPagingInfo().setLineages(Collections.emptyList());
                }
                if (response.getBody().getPagingInfo().getLineages() != null) {
                    for (ListLineagesResponseBodyPagingInfoLineages lineage : response.getBody().getPagingInfo().getLineages()) {
                        LineageEntity entity;
                        // 获取对应血缘实体
                        if (needUpstream) {
                            entity = lineage.getSrcEntity();
                        } else {
                            entity = lineage.getDstEntity();
                        }
                        // 所属实体ID
                        System.out.println(entity.getId());
                        // 所属实体名称
                        System.out.println(entity.getName());
                        // 所属实体的属性信息
                        System.out.println(entity.getAttributes());
                        // 实体间血缘关系列表
                        List<LineageRelationship> relationships = lineage.getRelationships();
                        if (relationships != null) {
                            relationships.forEach(relationship -> {
                                // 血缘关系ID
                                System.out.println(relationship.getId());
                                // 创建时间
                                System.out.println(relationship.getCreateTime());
                                // 血缘关系对应的任务
                                LineageTask task = relationship.getTask();
                                if (task != null) {
                                    // 任务ID,对于DataWorks任务,可以通过GetTask获取任务详情信息
                                    System.out.println(task.getId());
                                    // 任务类型
                                    System.out.println(task.getType());
                                    // 任务属性信息
                                    Map<String, String> attributes = task.getAttributes();
                                    if (attributes != null) {
                                        // 对于DataWorks任务,可以从中获取任务和实例的属性信息
                                        // 项目空间ID
                                        System.out.println(attributes.get("projectId"));
                                        // 任务ID
                                        String taskId = attributes.containsKey("taskId") ? attributes.get("taskId") : attributes.get("nodeId");
                                        System.out.println(taskId);
                                        // 实例ID
                                        String taskInstanceId = attributes.containsKey("taskInstanceId") ? attributes.get("taskInstanceId") : attributes.get("jobId");
                                        System.out.println(taskInstanceId);
                                    }
                                }
                            });
                        }
                    }
                    return response.getBody().getPagingInfo();
                }
            } catch (TeaException error) {
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            } catch (Exception _error) {
                TeaException error = new TeaException(_error.getMessage(), _error);
                // 此处仅做打印展示,请谨慎对待异常处理,在工程项目中切勿直接忽略异常。
                // 错误 message
                System.out.println(error.getMessage());
                // 诊断地址
                System.out.println(error.getData().get("Recommend"));
                com.aliyun.teautil.Common.assertAsString(error.message);
            }
            return null;
        }
    }
  2. MetaRestController中提供listTableLineages方法供前端进行调用。

    /**
     * 演示如何通过DataWorks OpenAPI构建自定义元数据平台
     *
     * @author dataworks demo
     */
    @RestController
    @RequestMapping("/meta")
    public class MetaRestController {
    
        @Autowired
        private MetaServiceProxy metaService;
    
        /**
         * 查询表的血缘关系
         *
         * @param listTableLineagesDto
         * @return
         */
        @CrossOrigin(origins = "http://localhost:8080")
        @GetMapping("/listTableLineages")
        public ListLineagesResponseBodyPagingInfo listTableLineages(ListTableLineagesDto listTableLineagesDto) {
            System.out.println("listTableLineagesDto = " + listTableLineagesDto);
            return metaService.listTableLineages(listTableLineagesDto);
        }
    
    }

前端:展示表血缘关系

import React from 'react';
import Graphin, { Behaviors } from '@antv/graphin';
import type { IUserNode, IUserEdge, GraphinData, GraphEvent } from '@antv/graphin';
import cn from 'classnames';
import * as services from '../services';
import type { TableEntity, ListTableLineageOutput, LineageEntity } from '../services/meta';
import classes from '../styles/lineageContent.module.css';
import '@antv/graphin/dist/index.css';

export interface Props {
  item: TableEntity;
}

function transNode(entity: LineageEntity | TableEntity, direction?: string): IUserNode {
  return {
    id: entity.id,
    label: entity.name,
    data: {
      ...entity,
      direction,
    },
    style: {
      label: {
        value: entity.name,
      }
    },
  }
}
function transEdge(source: LineageEntity | TableEntity, target: LineageEntity | TableEntity): IUserEdge {
  return {
    source: source.id,
    target: target.id,
    style: {
      keyshape: {
        lineDash: [8, 4],
        lineWidth: 2,
      },
      animate: {
        type: 'line-dash',
        repeat: true,
      },
    }
  };
}
function parse(
  source: LineageEntity | TableEntity,
  data: ListTableLineageOutput,
  direction: string,
): [IUserNode[], IUserEdge[]] {
  const nodes: IUserNode[] = [];
  const edges: IUserEdge[] = [];
  data.lineages.forEach((lineageRelationship) => {
    if (direction === 'down') {
      nodes.push(transNode(lineageRelationship.dstEntity, direction));
      edges.push(transEdge(source, lineageRelationship.dstEntity));
    }  else {
      nodes.push(transNode(lineageRelationship.srcEntity, direction));
      edges.push(transEdge(lineageRelationship.srcEntity, source));
    }
  });
  return [nodes, edges];
}
function mergeNodes(prev: IUserNode[], next: IUserNode[]) {
  const result: IUserNode[] = prev.slice();
  next.forEach((item) => {
    const hasValue = prev.findIndex(i => i.id === item.id) >= 0;
    !hasValue && result.push(item);
  });
  return result;
}
function mergeEdges(source: IUserEdge[], target: IUserEdge[]) {
  const result: IUserEdge[] = source.slice();
  target.forEach((item) => {
    const hasValue = source.findIndex(i => i.target === item.target && i.source === item.source) >= 0;
    !hasValue && result.push(item);
  });
  return result;
}

const { ActivateRelations, DragNode, ZoomCanvas } = Behaviors;
const LineageContent: React.FunctionComponent<Props> = (props) => {
  const ref = React.useRef<Graphin>();
  const [data, setData] = React.useState<GraphinData>({ nodes: [], edges: [] });
  const getTableLineage = async (
    collection: GraphinData,
    target: TableEntity | LineageEntity,
    direction: string,
  ) => {
    if (!direction) {
      return collection;
    }
    const response = await services.meta.getTableLineage({
      direction,
      tableId: target.id,
    });
    const [nodes, edges] = parse(target, response, direction);
    collection.nodes = mergeNodes(collection.nodes!, nodes);
    collection.edges = mergeEdges(collection.edges!, edges);
    return collection;
  };
  const init = async () => {
    let nextData = Object.assign({}, data, { nodes: [transNode(props.item)] });
    nextData = await getTableLineage(nextData, props.item, 'up');
    nextData = await getTableLineage(nextData, props.item, 'down');
    setData(nextData);
    ref.current!.graph.fitCenter();
  };
  React.useEffect(() => {
    ref.current?.graph && init();
  }, [
    ref.current?.graph,
  ]);
  React.useEffect(() => {
    const graph = ref.current?.graph;
    const event = async (event: GraphEvent) => {
      const source = event.item?.get('model').data;
      let nextData = Object.assign({}, data);
      nextData = await getTableLineage(nextData, source, source.direction);
      setData(nextData);
    };
    graph?.on('node:click', event);
    return () => {
      graph?.off('node:click', event);
    };
  }, [
    data,
    ref.current?.graph,
  ]);
  return (
    <div className={cn(classes.lineageContentWrapper)}>
      <Graphin
        data={data}
        ref={ref as React.LegacyRef<Graphin>}
        layout={{
          type: 'dagre',
          rankdir: 'LR',
          align: 'DL',
          nodesep: 10,
          ranksep: 40,
        }}
      >
        <ActivateRelations />
        <DragNode/>
        <ZoomCanvas enableOptimize />
      </Graphin>
    </div>
  );
};

export default LineageContent;

完成上述代码开发后,您可在本地部署并运行工程代码。部署并运行的操作请参见通用操作:本地部署运行

四、查找表对应的任务

当一张表口径发生变更时,您可以通过DataWorks OpenAPIOpenData消息订阅的方式进行下游任务的血缘分析,查找表对应的节点。具体操作如下。

说明

消息目前支持表变更、任务变更等。企业版用户可以对接表变更的消息,当接收到表变更的时候,您可以查看表的血缘关系。

血缘任务

  1. 通过ListLineages,查看表的血缘关系,从血缘关系的Task 中获取DataWorks 任务ID (以及任务实例ID)。

  2. 根据获取的任务ID,使用GetTask获取任务详情。

产出任务

  1. 根据实体ID,通过 GetTableAPI获取表的业务元数据,其中的上游产出任务中包含有DataWorks任务ID。

  2. 根据指定任务ID,通过GetTask获取任务详情。

通用操作:本地部署运行

  1. 准备依赖环境。

    您需准备好以下依赖环境:java8及以上、maven构建工具、node环境、pnpm工具。您可以执行以下命令来确定是否具备上述环境:

    npm -v // 如果已安装成功,执行命令将展示版本号,否则会报没有命令错误
    java -version // 如果已安装成功,执行命令将展示版本号,否则会报没有命令错误
    pnpm -v // 如果已安装成功,执行命令将展示版本号,否则会报没有命令错误
  2. 下载工程代码并执行以下命令。

    工程代码下载链接:meta-api-demo.zip

    pnpm i
  3. 在范例工程中的backend/src/main/resources路径下找到application.properties文件,修改文件中的核心参数。

    • api.access-key-idapi.access-key-secret需修改为您阿里云账号的AccessKey ID 和 AccessKey Secret。

      说明

      您可以参考访问密钥管理(AccessKey)获取AccessKey等阿里云账号的相关信息。

    • api.region-id、api.endpoint需修改为待调用OpenAPI的地域信息。

    其他参数可根据实际情况修改,修改后的填写示例如下。本地部署示例

  4. 在工程根目录下执行以下命令,运行示例实践代码。

    npm run dev