Manage shards using the Python SDK
Shards control the read and write capacity for a Logstore or Metricstore in Simple Log Service, and all data is stored in shards. You can query, split, and merge shards by using the Python SDK.
Prerequisites
Simple Log Service SDK for Python is initialized.
Simple Log Service SDK for Python is installed. For more information, see Install Simple Log Service SDK for Python.
A project and a Standard logstore are created. For more information, see Manage projects and Create a basic LogStore.
To split or merge a shard, its status must be
readwrite.
Precautions
In this example, the public Simple Log Service endpoint for the China (Hangzhou) region is used. Endpoint: https://cn-hangzhou.log.aliyuncs.com.
If you want to access Simple Log Service from other Alibaba Cloud services that reside in the same region as your project, you can use the internal Simple Log Service endpoint, which is https://cn-hangzhou-intranet.log.aliyuncs.com.
For more information about the supported regions and endpoints of Simple Log Service, see Endpoint.
Billing
For more information about shard billing, see Shard usage fees.
Query shards
The following example queries all shards in a Logstore.
from aliyun.log import LogClient
import os
# This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')
# The endpoint for Simple Log Service. This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
endpoint = "cn-hangzhou.log.aliyuncs.com"
# Create a LogClient instance.
client = LogClient(endpoint, accessKeyId, accessKey)
# The name of the project.
project_name = "ali-test-project"
# The name of the Logstore.
logstore_name = "ali-test-logstore"
if __name__ == '__main__':
# Query all shards in the specified Logstore.
print("ready to list shards")
res = client.list_shards(project_name, logstore_name)
print("Shards info is :%s" % res.get_shards_info())
print("list shards success")Expected output:
ready to list shards
Shards info is :[{'shardID': 0, 'status': 'readwrite', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1670999677}, {'shardID': 1, 'status': 'readwrite', 'inclusiveBeginKey': '7f000000000000000000000000000000', 'exclusiveEndKey': 'ffffffffffffffffffffffffffffffff', 'createTime': 1670999677}]
list shards successSplit a shard
You can also split and merge shards in the Simple Log Service console. For more information, see Split a shard.
The following example splits a shard in a Logstore.
import time
import os
from aliyun.log import LogClient
# This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')
# The endpoint for Simple Log Service. This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
endpoint = "cn-hangzhou.log.aliyuncs.com"
# Create a LogClient instance.
client = LogClient(endpoint, accessKeyId, accessKey)
# The name of the project.
project_name = "ali-test-project"
# The name of the Logstore.
logstore_name = "ali-test-logstore"
if __name__ == '__main__':
# Query the shards before the split operation.
res1 = client.list_shards(project_name, logstore_name)
print("Before split shard, Shards info is :%s" % res1.get_shards_info())
# Split shard 0.
print("ready to split shards")
client.split_shard(project_name, logstore_name, shardId=0, split_hash="3f000000000000000000000000000000")
# Wait for 60 seconds, then query the shards again to see the result.
time.sleep(60)
res2 = client.list_shards(project_name, logstore_name)
print("After split Shards, Shards info is :%s" % res2.get_shards_info())
print("list shards success")Expected output:
Before split shard, Shards info is :[{'shardID': 0, 'status': 'readwrite', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1670999677}, {'shardID': 1, 'status': 'readwrite', 'inclusiveBeginKey': '7f000000000000000000000000000000', 'exclusiveEndKey': 'ffffffffffffffffffffffffffffffff', 'createTime': 1670999677}]
ready to split shards
After split Shards, Shards info is :[{'shardID': 2, 'status': 'readwrite', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '3f000000000000000000000000000000', 'createTime': 1672042813}, {'shardID': 3, 'status': 'readwrite', 'inclusiveBeginKey': '3f000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1672042813}, {'shardID': 1, 'status': 'readwrite', 'inclusiveBeginKey': '7f000000000000000000000000000000', 'exclusiveEndKey': 'ffffffffffffffffffffffffffffffff', 'createTime': 1670999677}, {'shardID': 0, 'status': 'readonly', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1670999677}]
list shards successMerge shards
You can also split and merge shards in the Simple Log Service console. For more information, see Merge shards.
The following example merges a shard with the adjacent shard in a Logstore.
import time
import os
from aliyun.log import LogClient
# This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')
# The endpoint for Simple Log Service. This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
endpoint = "cn-hangzhou.log.aliyuncs.com"
# Create a LogClient instance.
client = LogClient(endpoint, accessKeyId, accessKey)
# The name of the project.
project_name = "ali-test-project"
# The name of the Logstore.
logstore_name = "ali-test-logstore"
if __name__ == '__main__':
# Query the shards before the merge operation.
res1 = client.list_shards(project_name, logstore_name)
print("Before merge shard, Shards info is :%s" % res1.get_shards_info())
# Merge shard 2 with its adjacent shard. The service automatically finds the adjacent shard to merge.
print("ready to merge shards")
client.merge_shard(project_name, logstore_name, shardId=2)
# Wait for 60 seconds, then query the shards again to see the result.
time.sleep(60)
res2 = client.list_shards(project_name, logstore_name)
print("After merge Shards, Shards info is :%s" % res2.get_shards_info())
print("merge shards success")Expected output:
Before merge shard, Shards info is :[{'shardID': 2, 'status': 'readwrite', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '3f000000000000000000000000000000', 'createTime': 1672042813}, {'shardID': 3, 'status': 'readwrite', 'inclusiveBeginKey': '3f000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1672042813}, {'shardID': 1, 'status': 'readwrite', 'inclusiveBeginKey': '7f000000000000000000000000000000', 'exclusiveEndKey': 'ffffffffffffffffffffffffffffffff', 'createTime': 1670999677}, {'shardID': 0, 'status': 'readonly', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1670999677}]
ready to merge shards
After merge Shards, Shards info is :[{'shardID': 4, 'status': 'readwrite', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1672043611}, {'shardID': 1, 'status': 'readwrite', 'inclusiveBeginKey': '7f000000000000000000000000000000', 'exclusiveEndKey': 'ffffffffffffffffffffffffffffffff', 'createTime': 1670999677}, {'shardID': 0, 'status': 'readonly', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1670999677}, {'shardID': 2, 'status': 'readonly', 'inclusiveBeginKey': '00000000000000000000000000000000', 'exclusiveEndKey': '3f000000000000000000000000000000', 'createTime': 1672042813}, {'shardID': 3, 'status': 'readonly', 'inclusiveBeginKey': '3f000000000000000000000000000000', 'exclusiveEndKey': '7f000000000000000000000000000000', 'createTime': 1672042813}]
merge shards successView shards in the console
Log on to the Simple Log Service console.
In the Projects section, click the one you want.
On the tab, hover over your Logstore, and then select
> Modify.
Related documentation
In addition to its native SDK, SLS also supports the common Alibaba Cloud SDKs. For more information, see Log Service_SDKcenter-Alibaba CloudOpenAPIDeveloper Portal.
For more information about shard API operations, see the following topics: