How to store large files generated by a function?

更新时间:
复制 MD 格式

Function Compute limits the /tmp directory to a default size. When your function generates or processes files larger than 500 MB — such as intermediate ETL outputs, machine learning model downloads, or media files — you need additional storage capacity. Choose one of the following approaches based on whether you need persistence and how much disk space you require.

Choose an approach

ApproachBest forProsConsPersistence
Expand disk storageTemporary large files within a single invocation (ETL jobs, ML inference, media processing)Simple to configure; no extra service requiredCleared after each invocationTemporary
Mount File Storage NASFiles that must persist across invocations or be shared between function instancesPersistent; accessible from multiple instancesRequires NAS setupPersistent
Stream dataProcessing large inputs or outputs without writing them to diskLowest disk and memory footprintRequires streaming-capable libraries or APIsNone

Expand disk storage

Set Disk to 10 GB in the Function Compute console. Each function instance then has 10 GB of /tmp space available.

For configuration steps, see Create a function.

Typical use cases:

  • Extract-transform-load (ETL) jobs that require intermediate scratch space

  • Machine learning (ML) inference that downloads large model weights to /tmp

  • Media transcoding or image processing that generates large intermediate files

Mount File Storage NAS

Mount a File Storage NAS (NAS) volume when files must survive across invocations — for example, shared model weights, reference datasets, or output files consumed by downstream services. NAS provides persistent file storage accessible from your function instances.

For setup steps, see Mount a NAS file system.

When to choose NAS over expanded disk storage:

  • Multiple function instances need access to the same files

  • Output files must be available after the function invocation ends

  • File access patterns involve repeated reads of the same large files (for example, reading a model on every invocation)

Stream data

If the file is too large to hold on disk at once, process it as a stream. Streaming reads and writes data incrementally, which reduces memory consumption. No local file is written during the streaming process.

Streaming is effective for:

  • Transcoding or compressing media files

  • Transforming large CSV or JSON datasets

  • Uploading or downloading objects from object storage