Streaming Tunnel是MaxCompute的流式数据通道,您可以通过Streaming Tunnel向MaxCompute中上传数据,本文为您介绍使用Python版Streaming Tunnel SDK上传数据的简单示例。

注意事项

Streaming Tunnel仅支持上传数据,下文为您介绍一个简单的数据上传的Python SDK示例。

上传示例

from odps.tunnel import TableTunnel


from odps import ODPS

o = ODPS('**your-access-id**', '**your-secret-access-key**', '**your-default-project**',  endpoint='**your-end-point**')
table = o.get_table('my_table')

tunnel = TableTunnel(odps)
upload_session = tunnel.create_stream_upload_session(table.name, partition_spec='pt=test')

with upload_session.open_record_writer() as writer:
      record = table.new_record()
      record[0] = 'test1'
      record[1] = 'id1'
      writer.write(record)

      record = table.new_record(['test2', 'id2'])
      writer.write(record)

import pandas as pd
import pyarrow as pa

with upload_session.open_arrow_writer() as writer:
      df = pd.DataFrame({"name": ["test1", "test2"], "id": ["id1", "id2"]})
      batch = pa.RecordBatch.from_pandas(df)
      writer.write(batch)