阿里云CDN为您提供刷新预热自动化脚本,帮助您对文件或目录快速进行刷新和预热。本章节为您介绍了该脚本的功能简介、使用说明,并以Windows系统示例为您说明。

功能简介

刷新预热自动化脚本可以帮助您分批进行刷新或预热任务,替代手动分批提交的繁琐操作。

当您指定刷新或预热URL列表文件后,脚本按照指定的并发刷新或预热数量对URL列表文件进行切割,分批进行刷新或预热。任务运行后会自动进行判断,等待上一个任务完成,脚本自动进行下一个刷新或预热任务的操作。

如果您有以下情况,建议您使用刷新预热自动化脚本:

  • 无开发人员,需手动提交刷新预热任务,运维成本高。
  • 刷新或预热URL过多,分批提交导致刷新或预热效率低。
  • 需要人工或程序判断刷新预热任务是否正常进行,费时费力。

使用说明

在使用刷新预热自动化脚本之前,您需要在本机安装如下环境:

  • Python CDN SDK模块包:目前使用版本为v20180510,建议您使用:pip install aliyun-python-sdk-cdn命令安装。
  • Python阿里云核心包:目前使用版本为 2.6.0,建议您使用:pip install aliyun-python-sdk-core命令安装。
  • Windows/Linux系统安装Python 2.75版本及以上。
  • 将如下代码保存为Refresh.py脚本。
    #!/usr/bin/env python
    #coding=utf-8
    # __author__ = 'hanli.zyb'
    
    '''Check Package'''
    
    try:
     import os, sys, getopt, time, json
     from aliyunsdkcore.client import AcsClient
     from aliyunsdkcore.acs_exception.exceptions import ClientException
     from aliyunsdkcore.acs_exception.exceptions import ServerException
     from aliyunsdkcdn.request.v20180510.RefreshObjectCachesRequest import RefreshObjectCachesRequest
     from aliyunsdkcdn.request.v20180510.PushObjectCacheRequest import PushObjectCacheRequest
     from aliyunsdkcdn.request.v20180510.DescribeRefreshTasksRequest import DescribeRefreshTasksRequest
     from aliyunsdkcdn.request.v20180510.DescribeRefreshQuotaRequest import DescribeRefreshQuotaRequest
    except:
     sys.exit("[Error] Please pip install aliyun-python-sdk-cdn and aliyun-java-sdk-core ,please install now......")
    
    class Refresh(object):
    
      '''init func'''
    
      def __init__(self):
    
       self.lists = []
       self.param = {}
    
      '''
      描述:调度的主函数
      resP:检测入参结果,如果类型不是 bool 说明有报错
      '''
    
      def main(self,argv):
        if len(argv) < 1 :
          sys.exit("\nusage: " + sys.argv[0] + " -h ")
        try:
          opts,args = getopt.getopt(argv,"hi:k:n:r:t:a:o:")
        except Exception as e :
          sys.exit("\nusage: " + sys.argv[0] + " -h ")
      
        for opt,arg in opts:
          if opt == '-h':
            self.helps()
            sys.exit()
          elif opt == '-i':
            self.param['-i'] = arg
          elif opt == '-k':
            self.param['-k'] = arg
          elif opt == '-r':
            self.param['-r'] = arg
          elif opt == '-t':
            self.param['-t'] = arg
          elif opt == '-a':
            self.param['-a'] = arg 
          elif opt == '-o':
            self.param['-o'] = arg
          elif opt == '-n':
            self.param['-n'] = arg
          else:
            sys.exit("\nusage: " + sys.argv[0] + " -h ")
      
        resP = self.doCheck(self.param)
        if not isinstance(resP,bool): sys.exit(resP)
        
        try:
          client = AcsClient(self.param['-i'], self.param['-k'], 'cn-hangzhou')
        except NameError:
          sys.exit("[Error]: SDK module not detected")
    
        for g in self.doProd(self.param):
          self.lists = []
          self.doRefresh(''.join(g),self.param['-t'],client)
    
      '''
      描述:检测入参数
      '''
      def doCheck(self,param):
    
        try:
          for key1 in ('-i','-k','-r','-t'):
            if not key1 in param.keys():
              return "[Error]: {0} Must be by parameter".format(key1)
    
          try:
            if not param.has_key('-n'):
              self.param['-n'] = 50
            if not (abs(int(param['-n'])) <= 100 and abs(int(param['-n'])) > 0):
              return "[Error]: 0 < -n <= 100"
            else:
              self.param['-n'] = int(param['-n'])
          except ValueError as e:
            return "[Error]: -n Must be int Type ,{0}".format(str(e))
    
          if not param['-t'] in ("push","clear"): return "[Error]: taskType Error"
          if param.has_key('-a') and param.has_key('-o'): return "[Error]: -a and -o cannot exist at same time"
    
          if param.has_key('-a'):
              if not param['-a'] in ("domestic","overseas"): 
                return "[Error]: Area value Error"
              if param['-t'] == 'clear':
                return "[Error]: -t must be push and 'clear' -o use together"
    
          if param.has_key('-o'):
              if not param['-o'] in ("File","Directory"):
                return "[Error]: ObjectType value Error"
              if param['-t'] == 'push':
                return "[Error]: -t must be clear and 'push' -a use together"
    
        except KeyError as e:
           return "[Error]: Parameter {0} error".format(str(e))
        return True
      
      '''
      描述:生成器切分文件,对每行文件进行处理 '\n'
      gop:每次读取 URL 数量
      '''
      def doProd(self,params):
        gop = params['-n']
        mins = 1
        maxs = gop
    
        with open(params['-r'], "r") as f:
          for line in f.readlines():
            if mins != maxs:
             line = line.strip("\n") + "\n"
            else:
             line = line.strip("\n")
            self.lists.append(line)
            if mins >= maxs:
             yield self.lists
             mins = maxs
             maxs = gop + maxs -1
            else:
             mins += 1
          if len(self.lists) > 0: yield self.lists
           
      '''
      描述:刷新/预热任务
      '''
      def doRefresh(self,lists,types,client):
        try:
          if types == 'clear':
            taskID = 'RefreshTaskId'
            request = RefreshObjectCachesRequest()
            if self.param.has_key('-o'):
              request.set_ObjectType(self.param['-o'])
          elif types == 'push':
            taskID = 'PushTaskId'
            request = PushObjectCacheRequest()
            if self.param.has_key('-a'):
              request.set_Area(self.param['-a'])
    
          taskreq = DescribeRefreshTasksRequest()
          request.set_accept_format('json')
          request.set_ObjectPath(lists)
          response = json.loads(client.do_action_with_exception(request))
          print(response)
        
          while True:
            count = 0
            taskreq.set_accept_format('json')
            taskreq.set_TaskId(int(response[taskID]))
            taskresp = json.loads(client.do_action_with_exception(taskreq))
            print("[" + response[taskID] + "]" + "is doing... ...")
            for t in taskresp['Tasks']['CDNTask']:
              if t['Status'] != 'Complete':
                count += 1
            if count == 0:
              break
            else:
              continue
            time.sleep(5)
        except Exception as e:
          sys.exit("[Error]" + str(e))
    
      '''
      描述:帮助信息
      '''
      def helps(self):
        print("\nscript options explain: \
                \n\t -i <AccessKey>                  访问阿里云凭证,访问控制台上可以获得; \
                \n\t -k <AccessKeySecret>            访问阿里云密钥,访问控制台上可以获得; \
                \n\t -r <filename>                   文件名称,每行一条 URL,有特殊字符先做 URLencode,以 http/https 开头; \
                \n\t -t <taskType>                   任务类型 clear 刷新,push 预热; \
                \n\t -n [int,[..100]]                可选项,每次操作文件数量,做多 100 条; \
                \n\t -a [String,<domestic|overseas>  可选项,预热范围,不传是默认是全球;\
                \n\t    domestic                     仅中国大陆; \
                \n\t    overseas                     全球(不包含中国大陆); \
                \n\t -o [String,<File|Directory>]    可选项,刷新的类型; \
                \n\t    File                         文件刷新(默认值); \
                \n\t    Directory                    目录刷新")
    #TODO 入口
    
    if __name__ == '__main__':
      fun = Refresh()
      fun.main(sys.argv[1:])

安装成功后,在CMD窗口使用脚本帮助命令:python $scripte -h。帮助参数如下所示。

script options explain:
              -i <AccessKey>               //访问阿里云凭证,访问控制台获得;
              -k <AccessKeySecret>    //访问阿里云密匙,访问控制台获得;
              -r <filename>                 //文件名称,每行一条URL,有特殊字符先做URLencode,以http/https开头;
              -t <taskType>                 //任务类型:clear刷新,push预热;
              -n [nums,[..100]]             //可选项,每次操作文件数量,最多100条;
              -a [String,<domestic|overseas>   //可选项,预热范围,不传是默认是全球;            
                   domestic                   //仅中国大陆;             
                   overseas                    //全球(不包含中国大陆);             
              -o [String,<File|Directory>]    可选项,刷新的类型;             
                   File                            //文件刷新(默认值);             
                   Directory                   //目录刷新;

脚本示例

在CMD窗口,使用命令:python Refresh.py -i yourAccessKey -k yourAccessKeySecret -r filename -t clear。运行示例如下所示。

python Refresh.py -i yourAccessKey -k yourAccessKeySecret -r /root/test/test.lst -t clear
{u'RefreshTaskId': u'8211135448', u'RequestId': u'093602B9-6192-4137-8CCA-0D8C85729DF6'}
[8211135448]is doing... ...
[8211135448]is doing... ...
[8211135448]is doing... ...
[8211135448]is doing... ...
[8211135448]is doing... ...
...
{u'RefreshTaskId': u'8211136701', u'RequestId': u'C1896DDD-003B-41F9-B481-93BC509ED5A3'}
[8211136701]is doing... ...
[8211136701]is doing... ...
...