Use Function Compute to package and download multiple files

更新时间:
复制 MD 格式

Deploy a Function Compute function to compress multiple Object Storage Service (OSS) objects into a ZIP file and download them in a single operation—eliminating the overhead of downloading files one by one.

How it works

When you invoke the function, it runs the following steps:

image
  1. You invoke the function, specifying the OSS bucket and the objects to compress.

  2. Function Compute downloads the specified objects from the bucket and compresses them into a ZIP file.

  3. Function Compute uploads the ZIP file back to OSS.

  4. Function Compute returns the ZIP file URL to you.

  5. You download the ZIP file from OSS using the returned URL.

The following figure shows the internals of steps 2 and 3:

image

The function uses two techniques to handle large workloads efficiently:

  • Streaming download and upload: Objects are streamed directly into the compression pipeline, minimizing memory usage and avoiding disk space limits in the function's runtime environment.

  • Multipart upload: The ZIP file is uploaded to OSS in concurrent parts using a queued, multithreaded approach, reducing total upload time.

Prerequisites

Before you begin, make sure you have:

Deploy and invoke the function

Step 1: Initialize the project

Run the following command to initialize the project:

sudo s init start-zip-oss-v3 -d start-zip-oss-v3

Follow the prompts to select a region, set the function name, and enter the Resource Access Management (RAM) role for the function.

Step 2: Deploy the function

Run the following command to deploy the project:

cd start-zip-oss-v3 && s deploy - y

After deployment, the output looks similar to the following. Record the system_url value—you'll need it to invoke the function.

region:         cn-hangzhou
description:
functionArn:    acs:fc:cn-hangzhou:1034354682****:functions/zip-oss-func-2m1d
functionName:   zip-oss-func-jbip
handler:        main.main_handler
internetAccess: true
memorySize:     3072
role:           acs:ram::1034354682****:role/aliyunfcdefaultrole
runtime:        python3.9
timeout:        1800
triggers:
  -
    description:
    qualifier:     LATEST
    triggerConfig:
      methods:
        - GET
        - POST
        - PUT
      authType:           anonymous
      disableURLInternet: false
    triggerName:   http-test
    triggerType:   http
url:
  system_url:          https://zip-ossunc-jbip-gvdzx****.cn-hangzhou.fcapp.run
  system_intranet_url: https://zip-ossunc-jbip-gvdzx****.cn-hangzhou-vpc.fcapp.run
__component:    fc3

Step 3: Invoke the function

  1. In the start-zip-oss-v3 directory, create an event.json file. Specify the bucket name and the objects to download—either by directory or by individual file paths. To download all objects in a directory, use source-dir:

    {
      "bucket": "bucketname",
      "source-dir": "filepath/"
    }

    To download specific objects, use source-files:

    {
      "bucket": "bucketname",
      "source-files": ["files1.txt", "filepath/files2.txt"]
    }
  2. Invoke the function using curl. Replace the endpoint with the system_url from your deployment output.

    • The ZIP file my.zip is saved to /tmp on your local machine.

    • The ZIP file is also stored in the output directory of your OSS bucket. Log in to the OSS console to confirm.

    curl -v -L -o /tmp/my.zip -d @./event.json https://zip-oss-func-zip-oss-****.cn-hangzhou.fcapp.run

    After a successful invocation:

Performance reference

The following table shows execution results from two representative test runs:

Test Objects Size before compression Size after compression Duration
1 7 1.2 MB 1.16 MB 0.4s
2 57 1.06 GB 0.91 GB 63s

Test 1 shows that compression reduces storage size even for small batches. Test 2 shows that the function can process over 1 GB of data across 57 objects in about one minute.

What's next

You can deploy a function in Function Compute to automatically decompress ZIP files uploaded to OSS, or you can configure automatic decompression directly in OSS. For more information, see Upload and automatically decompress a ZIP package.

References