Optimizations for the copy operations of TCMU-Alibaba Cloud Linux(Alinux)-阿里云帮助中心

Target Core Module (TCM) is another name for Linux-IO Target (LIO), an in-kernel Internet Small Computer Systems Interface (iSCSI) target. TCM in Userspace (TCMU) is the userspace implementation of LIO. It lets userspace programs work with various backend implementations through a common interface. Using the TCMU framework and the LIO loopback (tcm_loop) module, you can implement userspace iSCSI targets.

By default, every TCMU read/write operation passes data through an intermediate data area, adding two copy steps that limit throughput. Alibaba Cloud Linux 3 provides two optimizations to reduce these copies:

bypass data area: copies data directly between a userspace buffer and a kernel Scatter-Gather List (SGL) page, bypassing the data area
zero copy: maps kernel SGL pages into userspace so tcmu-runner can access them directly without any copy

Supported environments

Operating system: Alibaba Cloud Linux 3
Kernel version: 5.10.134-14 or later

Choose an optimization

Use the following table to select the right optimization for your backstore.

Optimization	Use when	Key constraint
`bypass data area`	The `backstore` manages its own data `buffer`	Works for all I/O sizes
`zero copy`	The `backstore` uses the TCMU `data area` directly (no own buffer)	I/O command size and each `SGL` page offset must be 4 KB aligned

If both optimizations are enabled, zero copy takes precedence. bypass data area applies only when zero copy cannot be used (for example, when the alignment requirements are not met).

How each optimization works

bypass data area

Without this optimization, the data path involves two copies through the data area:

DMA_FROM_DEVICE (read, inquiry): userspace buffer → data area → SGL page
DMA_TO_DEVICE (write, writesame): SGL page → data area → userspace buffer

With bypass data area enabled, the data area step is skipped:

DMA_FROM_DEVICE: userspace buffer → SGL page (direct)
DMA_TO_DEVICE: SGL page → userspace buffer (direct)

zero copy

With zero copy enabled, kernel SGL pages are mapped into userspace. tcmu-runner accesses the pages directly — no copy occurs.

Warning

If an I/O command freezes in tcmu-runner and times out, the corresponding memory mapping is unmapped. Any subsequent access to that command triggers a SIGBUS. Evaluate this risk before enabling zero copy in production.

Note

Zero copy activates only when both the I/O command size and each SGL page offset are 4 KB aligned. 4 KB alignment means the starting offset and the length are multiples of 4 KB. For most workloads, zero copy delivers significant performance gains when the I/O command size is 256 KB or larger. Evaluate your I/O model before setting the threshold.

Configure the optimizations

Configuration attributes are stored under the device's sysfs attrib path. The examples below use the user_0/testblk device path — replace it with your actual device path.

Configuration parameters

Parameter	Direction	Default	How to enable
`read_bypass_data_area`	DMA_FROM_DEVICE (read)	`false`	Set to `1`
`write_bypass_data_area`	DMA_TO_DEVICE (write)	`false`	Set to `1`
`read_zc_size`	DMA_FROM_DEVICE (read)	`0` KB	Set to a non-zero value (KB); zero copy activates when command size exceeds this value and alignment requirements are met
`write_zc_size`	DMA_TO_DEVICE (write)	`0` KB	Set to a non-zero value (KB); zero copy activates when command size exceeds this value and alignment requirements are met

Enable bypass data area

Query the current values:

sudo cat /sys/kernel/config/target/core/user_0/testblk/attrib/read_bypass_data_area
sudo cat /sys/kernel/config/target/core/user_0/testblk/attrib/write_bypass_data_area

Enable the optimization:

sudo bash -c "echo 1 > /sys/kernel/config/target/core/user_0/testblk/attrib/read_bypass_data_area"
sudo bash -c "echo 1 > /sys/kernel/config/target/core/user_0/testblk/attrib/write_bypass_data_area"

Enable zero copy

Query the current values:

sudo cat /sys/kernel/config/target/core/user_0/testblk/attrib/read_zc_size
sudo cat /sys/kernel/config/target/core/user_0/testblk/attrib/write_zc_size

Enable the optimization (example threshold: 256 KB):

sudo bash -c "echo 256 > /sys/kernel/config/target/core/user_0/testblk/attrib/read_zc_size"
sudo bash -c "echo 256 > /sys/kernel/config/target/core/user_0/testblk/attrib/write_zc_size"

User programming interfaces

Both optimizations expose ioctl interfaces defined in /usr/include/linux/target_core_user.h.

All three ioctl commands use one of the following structs:

struct tcmu_data_xfer {
  __u16 cmd_id;
  __u16 __pad1;
  __u32 iov_cnt;
  struct iovec __user *iovec;
};

struct tcmu_cmd_zerocopy {
  struct iovec __user *iov;
  __u32 iov_cnt;
  __u16 cmd_id;
};

The available ioctl commands are:

#define TCMU_IOCTL_CMD_COPY_TO_SGL    _IOW('T', 0xe0, struct tcmu_data_xfer)
#define TCMU_IOCTL_CMD_COPY_FROM_SGL  _IOR('T', 0xe1, struct tcmu_data_xfer)
#define TCMU_IOCTL_CMD_ZEROCOPY       _IOW('T', 0xe2, struct tcmu_cmd_zerocopy)

TCMU_IOCTL_CMD_COPY_TO_SGL: copies data from a userspace buffer to a kernel SGL page (used with bypass data area for DMA_FROM_DEVICE commands)
TCMU_IOCTL_CMD_COPY_FROM_SGL: copies data from a kernel SGL page to a userspace buffer (used with bypass data area for DMA_TO_DEVICE commands)
TCMU_IOCTL_CMD_ZEROCOPY: maps kernel SGL pages into userspace for direct access by tcmu-runner