Video merging-ApsaraVideo VOD(VOD)-阿里云帮助中心

The short video SDK provides the `AliyunIMixComposer` interface for merging multiple video clips into a single video offline. You can create effects such as Picture-in-Picture (PiP), nine-square grid, and split screens, with support for multiple video tracks. This reference covers the video merging process for Android and provides sample code.

Supported editions

Edition	Supported
Professional Edition	Supported
Standard Edition	Supported
Basic Edition	Not supported

Class	Feature
AliyunIMixComposer	The core merging class. Provides functions for setting output parameters, creating tracks, adding video streams, starting the merge, and setting callbacks.
AliyunMixComposerCreator	A factory class for creating an instance of the `AliyunIMixComposer` implementation class.
AliyunMixTrack	A video track. Add video streams to a track and configure parameters such as layout and volume.
AliyunMixStream	A video track stream. Retrieve the display mode, file path, and end time of a video stream.
AliyunMixOutputParam	The output parameters for merging. Configure the width, height, output bitrate, and quality level of the merged video.
AliyunMixCallback	A merging callback. Set callbacks for merge completion, progress, and failure.

Merging structure diagram

Merging process

Phase	Step	Description	Sample code
Basic	1	Create a merging instance.	Create an instance
	2	Create multiple tracks and video streams, and then add the video streams to their respective tracks.	Create tracks
	3	Configure output parameters, such as the output path, width, and height of the merged video.	Configure output parameters
	4	Set callbacks and start merging.	Start merging
	5	Destroy the instance and release resources.	Release resources
Advanced	6	Cancel, pause, or resume merging as needed.	Control merging

Create an instance

Create a merging instance.

For details about the parameters used in the code, see the API reference linked in Related classes.

// Create an instance.
AliyunIMixComposer mixComposer = AliyunMixComposerCreator.createMixComposerInstance();

Create tracks

Create multiple tracks and video streams, and then add the video streams to their respective tracks.For details about the parameters used in the code, see the API reference linked in Related classes

// Create tracks.

// Create Track 1.
// The layout of Track 1.
AliyunMixTrackLayoutParam track1Layout = new AliyunMixTrackLayoutParam.Builder()
        .centerX(0.25f)
        .centerY(0.5f)
        .widthRatio(0.5f)
        .heightRatio(1.f)
        .build();
// Create an instance for Track 1.        
AliyunMixTrack track1 = mixComposer.createTrack(track1Layout);        
// Create the first video stream for Track 1.
AliyunMixStream stream11 = new AliyunMixStream
        .Builder()
        .displayMode(VideoDisplayMode.FILL)
        .filePath("/storage/emulated/0/lesson_01.mp4")
        .streamEndTimeMills(20000)
        .build();
// Add the video stream to Track 1. Note: You can add only one stream. The last added stream is used.
track1.addStream(stream11);


// Create Track 2.
// Parameters for Track 2.
AliyunMixTrackLayoutParam track2Layout = new AliyunMixTrackLayoutParam.Builder()
        .centerX(0.75f)
        .centerY(0.5f)
        .widthRatio(0.5f)
        .heightRatio(1.f)
        .build();
// Create an instance for Track 2.
AliyunMixTrack track2 = mixComposer.createTrack(track2Layout);        
// Create the first video stream for Track 2.
AliyunMixStream stream21 = new AliyunMixStream
        .Builder()
        .displayMode(VideoDisplayMode.FILL)
        .filePath("/storage/emulated/0/lesson_02.mp4")
        .streamStartTimeMills(10000)
        .streamEndTimeMills(30000)
        .build();
// Add the video stream to Track 2. Note: You can add only one stream. The last added stream is used.
track2.addStream(stream21);

Configure output parameters

Configure the output path, video width, video height, and other parameters for the merged video.For details about the parameters used in the code, see the API reference linked in Related classes

// Configure output parameters.
AliyunMixOutputParam outputParam = new AliyunMixOutputParam.Builder()
        .outputPath("/sdcard/output.mp4") // The path of the merged video.
        .outputAudioReferenceTrack(track2)// Use the audio from Track 2 as the final audio. Only one audio stream is supported.
        .outputDurationReferenceTrack(track2)// Use the duration of Track 2 as the duration of the output video. If the duration of Track 1 is shorter, its video stops at the last frame.
        .crf(6)
        .videoQuality(VideoQuality.HD)
        .outputWidth(720) // Video width.
        .outputHeight(1280) // Video height.
        .fps(30) // fps
        .gopSize(30) // gop
        .build();
mixComposer.setOutputParam(outputParam);

Start merging

Set a callback and start the video merge.For details about the parameters used in the code, see the API reference linked in Related classes

// Start merging.
AliyunMixCallback callback = new AliyunMixCallback() {
            @Override
            public void onProgress(long progress) {// Merging progress.
                Log.e("MixRecord", "onProgress " + progress);
            }

            @Override
            public void onComplete() {
                Log.e("MixRecord", "onComplete"); // Merging completed.
                runOnUiThread(new Runnable() {
                    @Override
                    public void run() {
                        // Do not call this interface directly in the callback thread.
                        // Release the instance after merging is complete.
                        mixComposer.release();
                    }
                });
            }

            @Override
            public void onError(int errorCode) { // Merging failed.
                Log.e("MixRecord", "onError " + errorCode);
            }
};

Release resources

After the merge is complete, destroy the instance to release resources. Do not destroy the instance while merging is in progress.For details about the parameters used in the code, see the API reference linked in Related classes

mixComposer.release();

Control merging

Cancel, pause, or resume merging as needed.For details about the parameters used in the code, see the API reference linked in Related classes

// Pause merging.
mixComposer.pause();

// Resume merging.
mixComposer.resume();

// Cancel merging.
mixComposer.cancel();

Video merging sample code

/**
 * Video merging example
 */
class MixActivity : AppCompatActivity() {
    private val REQUEST_TRACK1_STREAM = 1001
    private val REQUEST_TRACK2_STREAM = 1002

    private var mMixComposer : AliyunIMixComposer? = null

    private lateinit var mVideoTrack1 : AliyunMixTrack
    private lateinit var mVideoTrack2 : AliyunMixTrack

    private var mVideoTrack1Duration = 0L
    private var mVideoTrack2Duration = 0L
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_mix)

        findViewById<Button>(R.id.btnReset).setOnClickListener {
            findViewById<Button>(R.id.btnMix).isEnabled = false

            mMixComposer?.release()

            init()
        }

        findViewById<Button>(R.id.btnAddTrack1Stream).setOnClickListener {
            PermissionX.init(this)
                .permissions(
                    Manifest.permission.READ_EXTERNAL_STORAGE,
                    Manifest.permission.WRITE_EXTERNAL_STORAGE
                )
                .request { allGranted, _, _ ->
                    if (allGranted) {
                        PictureSelector.create(this)
                            .openGallery(PictureMimeType.ofVideo())
                            .forResult(REQUEST_TRACK1_STREAM)
                    }
                }
        }

        findViewById<Button>(R.id.btnAddTrack2Stream).setOnClickListener {
            PermissionX.init(this)
                .permissions(
                    Manifest.permission.READ_EXTERNAL_STORAGE,
                    Manifest.permission.WRITE_EXTERNAL_STORAGE
                )
                .request { allGranted, _, _ ->
                    if (allGranted) {
                        PictureSelector.create(this)
                            .openGallery(PictureMimeType.ofVideo())
                            .forResult(REQUEST_TRACK2_STREAM)
                    }
                }
        }

        findViewById<Button>(R.id.btnMix).setOnClickListener {
            // Start composing.
            val callback: AliyunMixCallback = object : AliyunMixCallback {
                override fun onProgress(progress: Long) { // Composing progress.
                    Log.e("MixActivity", "onProgress $progress")
                }

                override fun onComplete() {
                    Log.e("MixActivity", "onComplete")
                    ToastUtil.showToast(it.context, "Video composed successfully")
                }

                override fun onError(errorCode: Int) {
                    Log.e("MixActivity", "onError $errorCode")
                    ToastUtil.showToast(it.context, "Failed to compose video: $errorCode")
                }
            }

            // Configure output parameters.
            val outputParamBuilder = AliyunMixOutputParam.Builder()
            outputParamBuilder
                .outputPath("/storage/emulated/0/DCIM/Camera/svideo_mix_demo.mp4")
                .crf(6)
                .videoQuality(VideoQuality.HD)
                .outputWidth(720)
                .outputHeight(1280)
                .fps(30)
                .gopSize(30)
            if(mVideoTrack1Duration > mVideoTrack2Duration) {
                outputParamBuilder.outputAudioReferenceTrack(mVideoTrack1)
                outputParamBuilder.outputDurationReferenceTrack(mVideoTrack1)
            } else {
                outputParamBuilder.outputAudioReferenceTrack(mVideoTrack2)
                outputParamBuilder.outputDurationReferenceTrack(mVideoTrack2)
            }

            mMixComposer!!.setOutputParam(outputParamBuilder.build())
            mMixComposer!!.start(callback)
        }

        init()
    }

    private fun init() {
        mMixComposer = AliyunMixComposerCreator.createMixComposerInstance()
        // Create Track 1.
        val track1Layout = AliyunMixTrackLayoutParam.Builder()
            .centerX(0.25f)
            .centerY(0.25f)
            .widthRatio(0.5f)
            .heightRatio(0.5f)
            .build()

        mVideoTrack1 = mMixComposer!!.createTrack(track1Layout)

        // Create Track 2.
        val track2Layout = AliyunMixTrackLayoutParam.Builder()
            .centerX(0.75f)
            .centerY(0.75f)
            .widthRatio(0.5f)
            .heightRatio(0.5f)
            .build()

        mVideoTrack2 = mMixComposer!!.createTrack(track2Layout)
    }
    override fun onResume() {
        super.onResume()
        mMixComposer?.resume()
    }

    override fun onPause() {
        super.onPause()
        mMixComposer?.pause()
    }

    override fun onDestroy() {
        super.onDestroy()
        mMixComposer?.release()
    }

    override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
        super.onActivityResult(requestCode, resultCode, data)
        if (resultCode == Activity.RESULT_OK) {
            when (requestCode) {
                REQUEST_TRACK1_STREAM -> {
                    // onResult Callback
                    val result = PictureSelector.obtainMultipleResult(data)

                    mVideoTrack1Duration = 0
                    for (streamBean in result) {

                        // Create the first video stream for Track 1.
                        val stream1 = AliyunMixStream.Builder()
                            .displayMode(VideoDisplayMode.FILL)
                            .filePath(streamBean.realPath)
                            .streamStartTimeMills(mVideoTrack1Duration)
                            .streamEndTimeMills(streamBean.duration)
                            .build()
                        mVideoTrack1Duration += streamBean.duration
                        if(mVideoTrack1.addStream(stream1) == 0) {
                            ToastUtil.showToast(this, "Video stream added to Track 1 successfully")
                        }

                    }
                }
                REQUEST_TRACK2_STREAM -> {
                    // onResult Callback
                    val result = PictureSelector.obtainMultipleResult(data)

                    mVideoTrack2Duration = 0L
                    for (streamBean in result) {

                        // Create the first video stream for Track 2.
                        val stream1 = AliyunMixStream.Builder()
                            .displayMode(VideoDisplayMode.FILL)
                            .filePath(streamBean.realPath)
                            .streamStartTimeMills(mVideoTrack2Duration)
                            .streamEndTimeMills(streamBean.duration)
                            .build()
                        mVideoTrack2Duration += streamBean.duration
                        if(mVideoTrack2.addStream(stream1) == 0) {
                            ToastUtil.showToast(this, "Video stream added to Track 2 successfully")
                        }
                    }
                }
            }
        }
        if(mVideoTrack1Duration > 0 && mVideoTrack2Duration > 0) {
            findViewById<Button>(R.id.btnMix).isEnabled = true
        }
    }
}

XML file configuration example

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">


    <Button
        android:id="@+id/btnReset"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginTop="76dp"
        android:text="Reset"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

    <Button
        android:id="@+id/btnAddTrack1Stream"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginStart="56dp"
        android:layout_marginTop="64dp"
        android:text="Add Video 1"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@id/btnReset"
        app:layout_constraintEnd_toEndOf="@id/btnReset"
         />

    <Button
        android:id="@+id/btnAddTrack2Stream"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginTop="64dp"
        android:layout_marginEnd="56dp"
        android:text="Add Video 2"
        app:layout_constraintStart_toStartOf="@id/btnReset"
        app:layout_constraintTop_toBottomOf="@id/btnReset"
        app:layout_constraintEnd_toEndOf="parent"
        />

    <Button
        android:id="@+id/btnMix"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginBottom="176dp"
        android:text="Start Composing"
        android:enabled="false"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent" />
</androidx.constraintlayout.widget.ConstraintLayout>