Accelerate Stable Diffusion inference using Intel CPU-based g8i instances

Prepare the environment and model

1. Create an ECS instance

   1. Go to the instance creation page.

   2. Follow the on-screen instructions to configure the parameters and create an ECS instance.

      Note the following parameters. For more information about other parameters, see Create an instance using the wizard.

      • Instance: To ensure stable model operation, select an instance type of at least ecs.g8i.4xlarge (16 vCPU).

      • Image: Alibaba Cloud Linux 3.2104 LTS 64-bit.

      • Public IP: Select Assign Public IPv4 Address, set the bandwidth billing method to Pay-by-traffic, and set the peak bandwidth to 100 Mbps. This setting accelerates the model download speed.

        

      • System Disk: Downloading, transforming, and running the model requires a large amount of storage space. To ensure the model runs smoothly, set the system disk size to 100 GiB.

   3. Add a security group rule.

      Add an inbound rule to the security group of the ECS instance to allow traffic on port 22. This port is used for Secure Shell Protocol (SSH) access. For more information, see Add a security group rule.

2. Download and install Anaconda.

   1. Run the following command to download the Anaconda installation script.

      wget https://repo.anaconda.com/archive/Anaconda3-2023.09-0-Linux-x86_64.sh

   2. Run the following command to install Anaconda.

      bash Anaconda3-2023.09-0-Linux-x86_64.sh

      During the installation, you will be prompted to confirm the license agreement and initialize conda for the current shell. Follow these steps.

      1. When Please, press ENTER to continue appears, press Enter.

      2. Press Enter multiple times. When Do you accept the license terms? [yes/no] appears, enter yes.

      3. When the following message appears, press Enter to install conda in the current directory, or enter the directory where you want to install conda.

      4. When You can undo this by running 'conda init --reverse $SHELL'? [yes/no] appears, enter yes.

      5. When Thank you for installing Anaconda appears, the installation is complete.

         

   3. Run the following command to apply the Anaconda environment variables.

      source ~/.bashrc

3. Create a virtual environment that includes the Transformers, Diffusers, Accelerate, PyTorch, and IPEX libraries.

   conda create -n sd_inference python=3.9 -y
   conda activate sd_inference
   pip install pip --upgrade
   pip install transformers diffusers accelerate torch==2.1.1 intel_extension_for_pytorch==2.1.100

4. Use huggingface-cli to download the pre-trained model Lykon/dreamshaper-8-lcm.

   mkdir /home/hf_models
   cd /home/hf_models/
   pip install -U huggingface_hub
   pip install -U hf-transfer
   export HF_ENDPOINT=https://hf-mirror.com
   export HF_HUB_ENABLE_HF_TRANSFER=1
   huggingface-cli download --resume-download --local-dir-use-symlinks False Lykon/dreamshaper-8-lcm --local-dir dreamshaper-8-lcm

Run the model

1. Create the ds8_lcm_pipe.py file.

   1. Run the following command to create and open ds8_lcm_pipe.py.

      vim ds8_lcm_pipe.py

      This script tests the average latency for generating a single image. Enter the following two sections of code into the script:

      • A benchmark function to calculate the average latency for single-image generation.

        import time
        def elapsed_time(pipeline, prompt, height=512, width=512, guidance_scale=2, test_loops=3, num_inference_steps=10):
            # Warm up
            images = pipeline(prompt, num_inference_steps=10, height=height, width=width, guidance_scale=guidance_scale).images
            start = time.time()
            for _ in range(test_loops):
                _ = pipeline(prompt, num_inference_steps=num_inference_steps, height=height, width=width, guidance_scale=guidance_scale)
            end = time.time()
            return (end - start) / test_loops

      • A StableDiffusionPipeline built with the default float32 data type.

        from diffusers import StableDiffusionPipeline, LCMScheduler
        import torch
        model_id = "/home/hf_models/dreamshaper-8-lcm"
        pipe = StableDiffusionPipeline.from_pretrained(model_id)
        pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
        prompt = "portrait photo of muscular bearded guy in a worn mech suit, light bokeh, intricate, steel metal, elegant, sharp focus, soft lighting, vibrant colors "
        generator = torch.manual_seed(0)
        image = pipe(prompt, num_inference_steps=10, height=512, width=512, guidance_scale=2, generator=generator).images[0]  
        image.save("./fp32_image.png")
        latency = elapsed_time(pipe, prompt, height=512, width=512, guidance_scale=2)
        print("Using data type FP32, average latency for a test loop (10 steps) is ", latency, " s.")
        

      The complete content of the ds8_lcm_pipe.py script is as follows:

      import time
      from diffusers import StableDiffusionPipeline, LCMScheduler
      import torch
      
      # Define the benchmark function
      def elapsed_time(pipeline, prompt, height=512, width=512, guidance_scale=2, test_loops=3, num_inference_steps=10):
          # Warm up
          images = pipeline(prompt, num_inference_steps=10, height=height, width=width, guidance_scale=guidance_scale).images
          start = time.time()
          for _ in range(test_loops):
              _ = pipeline(prompt, num_inference_steps=num_inference_steps, height=height, width=width, guidance_scale=guidance_scale)
          end = time.time()
          return (end - start) / test_loops
      
      # Build and test the StableDiffusionPipeline
      model_id = "/home/hf_models/dreamshaper-8-lcm"
      pipe = StableDiffusionPipeline.from_pretrained(model_id)
      pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
      prompt = "portrait photo of muscular bearded guy in a worn mech suit, light bokeh, intricate, steel metal, elegant, sharp focus, soft lighting, vibrant colors"
      generator = torch.manual_seed(0)
      image = pipe(prompt, num_inference_steps=10, height=512, width=512, guidance_scale=2, generator=generator).images[0]  
      image.save("./fp32_image.png")
      latency = elapsed_time(pipe, prompt, height=512, width=512, guidance_scale=2)
      print("Using data type FP32, average latency for a test loop (10 steps) is ", latency, " s.")

   2. Press the Esc key, enter :wq, and then press the Enter key to save the script and exit.

2. Use jemalloc to optimize image generation speed.

   Image generation is a memory-intensive operation. Installing a high-performance memory allocation library can accelerate memory operations and enable parallel processing across CPU cores. jemalloc and tcmalloc are two commonly used memory optimization libraries. This topic uses jemalloc as an example. You can tune jemalloc for specific workloads, such as maximizing CPU utilization. For more information, see the jemalloc tuning guide.

   1. Install jemalloc and set the environment variables.

      Important

      Replace /path_to_your_conda_environment_location in the export CONDA_LOCATION command with the actual installation path of Anaconda.

      conda install jemalloc -y
      export CONDA_LOCATION=/path_to_your_conda_environment_location
      export LD_PRELOAD=$LD_PRELOAD:$CONDA_LOCATION/lib/libjemalloc.so
      export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms: 60000,muzzy_decay_ms:60000"

   2. Install intel-openmp and set the environment variables.

      Important

      Replace the number after OMP_NUM_THREADS with the number of physical CPU cores of your instance.

      pip install intel-openmp
      export LD_PRELOAD=$LD_PRELOAD:$CONDA_LOCATION/lib/libiomp5.so
      export OMP_NUM_THREADS=16

3. Install numactl and run the ds8_lcm_pipe.py script.

   yum install numactl -y
   numactl -C 0-15 python ds8_lcm_pipe.py

   The output shows that the generation time for a single image is about 21 seconds.

Accelerate image generation

To better leverage CPU performance, apply IPEX optimizations to each module of the pipeline and use the bfloat16 data type.

1. Run the following command to open the ds8_lcm_pipe.py script.

   vim ds8_lcm_pipe.py

2. Modify the ds8_lcm_pipe.py script as follows.

   • Use IPEX to optimize each module of the pipeline.

     For a StableDiffusionPipeline, you must apply IPEX optimizations to each of its modules. These optimizations include converting the data format to channels-last, calling the ipex.optimize function, and using TorchScript mode. Intel has submitted this optimized pipeline as a pull request to the diffusers library, which allows it to be directly called as a custom_pipeline. For more information about optimization, see Stable Diffusion on IPEX.

     The required code changes are simple:

     • Configure custom_pipeline="stable_diffusion_ipex" when loading the pipe.

     • Call the prepare_for_ipex function on the custom_pipeline.

       custom_pipe = StableDiffusionPipeline.from_pretrained(model_id, custom_pipeline="stable_diffusion_ipex")
       #value of image height/width should be consistent with the pipeline inference
       custom_pipe.prepare_for_ipex(prompt, dtype=torch.float32, height=512, width=512) 

   • Optimize the AMX accelerator on the CPU

     To use the AMX accelerator on the CPU, you can use the bfloat16 data type using the Automatic Mixed Precision package.

     custom_pipe = StableDiffusionPipeline.from_pretrained(model_id, custom_pipeline="stable_diffusion_ipex")
     #value of image height/width should be consistent with the pipeline inference
     custom_pipe.prepare_for_ipex(prompt, dtype=torch.bfloat16, height=512, width=512) 
     with torch.cpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
         image = custom_pipe (prompt, num_inference_steps=10, height=512, width=512,  guidance_scale=2, generator=generator).images[0]  
         image.save("./bf16_opt_image.png")
         latency = elapsed_time(custom_pipe, prompt, height=512, width=512, guidance_scale=2)
         print("Using data type BF16, average latency for a test loop (10 steps) w/ optimized pipeline is ", latency, " s.")

   The modified ds8_lcm_pipe.py script is as follows:

   import time
   import torch
   from diffusers import StableDiffusionPipeline
   
   def elapsed_time(pipeline, prompt, height=512, width=512, guidance_scale=2, test_loops=3, num_inference_steps=10):
       # Warm up
       images = pipeline(prompt, num_inference_steps=10, height=height, width=width, guidance_scale=guidance_scale).images
       start = time.time()
       for _ in range(test_loops):
           _ = pipeline(prompt, num_inference_steps=num_inference_steps, height=height, width=width, guidance_scale=guidance_scale)
       end = time.time()
       return (end - start) / test_loops
   
   model_id = "/home/hf_models/dreamshaper-8-lcm"
   prompt = "portrait photo of muscular bearded guy in a worn mech suit, light bokeh, intricate, steel metal, elegant, sharp focus, soft lighting, vibrant colors"
   
   custom_pipe = StableDiffusionPipeline.from_pretrained(model_id, custom_pipeline="stable_diffusion_ipex")
   custom_pipe.prepare_for_ipex(prompt, dtype=torch.bfloat16, height=512, width=512)
   generator = torch.manual_seed(0)
   
   custom_pipe.to(torch.bfloat16)
   with torch.cpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
       image = custom_pipe(prompt, num_inference_steps=10, height=512, width=512, guidance_scale=2, generator=generator).images[0]
       image.save("./bf16_opt_image.png")
   
   latency = elapsed_time(custom_pipe, prompt, height=512, width=512, guidance_scale=2)
   print("Using data type BF16, average latency for a test loop (10 steps) w/ optimized pipeline is ", latency, " s.")
   

3. Press the Esc key, enter :wq, and then press the Enter key to save the script and exit.

4. Run the ds8_lcm_pipe.py script.

   numactl -C 0-15 python ds8_lcm_pipe.py

   The output shows that the generation time for a single image is about 7 seconds.