首页 机器学习平台 PAI 实践教程 AIGC AIGC文生图模型微调及WebUI部署

AIGC文生图模型微调及WebUI部署

更新时间: 2023-09-01 11:12:22

本文为您介绍如何在阿里云交互式建模(PAI-DSW)中,基于Diffusers开源库进行AIGC Stable Diffusion模型的微调训练,以及基于Stable-Diffusion-WebUI开源库启动WebUI进行模型推理。

背景信息

AIGC是指通过人工智能技术自动生成内容的生产方式,已经成为继互联网时代的下一个产业时代风口。其中,文生图(Text-to-image Generation)任务是流行的跨模态生成任务,旨在生成与给定文本对应的图像。

本文实现的功能主要是基于AIGC Stable Diffusion文生图Lora模型进行模型微调实现虚拟上装。WebUI文生图推理效果如图所示。image.png

前提条件

  • 已创建工作空间,详情请参见创建工作空间

  • 已创建DSW实例,其中关键参数配置如下。具体操作,请参见创建及管理DSW实例

    • 实例规格选择:ecs.gn7i-c8g1.2xlarge。

    • 选择镜像

      地域

      镜像地址

      华东1(杭州)

      官方镜像页签,选择stable-diffusion-webui-env:pytorch1.13-gpu-py310-cu117-ubuntu22.04。

      华北2(北京)

      华东2(上海)

      华南1(深圳)

步骤一:安装Diffusers

  1. 交互式建模(DSW)页面,单击实例操作列下的打开,进入DSW开发环境。

  2. Notebook页签的Launcher页面,单击快速开始区域Notebook下的Python3

  3. 在Notebook中执行以下代码,从GitHub下载Diffusers开源库,并安装相关依赖。

    后续将基于Diffusers开源库进行模型微调。安装过程大约需要持续5分钟。

    说明

    预置镜像中已安装Diffusers,如果需要安装最新版本,请运行注释代码。

    ! git clone https://github.com/huggingface/diffusers
    ! cd diffusers && git checkout e126a82cc5d9afbeb9b476455de24dd3e7dd358a
    ! cd diffusers && pip install .
  4. 验证是否安装成功。

    import diffusers

    系统返回如下结果,表示安装成功。

    /usr/local/lib/python3.10/dist-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm
  5. 配置accelerate。

    您可以直接运行以下命令下载默认配置文件,如果需要自定义配置则在Terminal中执行accelerate config命令,并根据DSW实例详情,选择对应配置即可。本案例下载默认配置文件。

    ! mkdir -p /root/.cache/huggingface/accelerate/
    ! wget -c http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com/aigc-data/accelerate/default_config.yaml -O /root/.cache/huggingface/accelerate/default_config.yaml

    系统返回如下结果,表示默认配置文件下载成功。image.png

  6. 在Notebook中执行命令,安装文生图算法相关依赖库。

    ! cd diffusers/examples/text_to_image && pip install -r requirements.txt

    系统输出下图结果,出现的WARNING信息可忽略。

    image
  7. 在Notebook中执行以下命令,下载stable-diffusion-webui开源库。

    import os
    
    ! apt update
    ! apt install -y aria2
    
    def aria2(url, filename, d):
        !aria2c --console-log-level=error -c -x 16 -s 16 {url} -o {filename} -d {d}
        
    url_prefix = {
        "cn-shanghai": "http://pai-vision-data-sh.oss-cn-shanghai-internal.aliyuncs.com",
        "cn-hangzhou": "http://pai-vision-data-hz2.oss-cn-hangzhou-internal.aliyuncs.com",
        "cn-shenzhen": "http://pai-vision-data-sz.oss-cn-shenzhen-internal.aliyuncs.com",
        "cn-beijing": "http://pai-vision-data-bj.oss-cn-beijing-internal.aliyuncs.com", 
    }
    
    dsw_region = os.environ.get("dsw_region")
    prefix = url_prefix[dsw_region] if dsw_region in url_prefix else "http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com"
    
    
    ! git clone https://gitcode.net/mirrors/AUTOMATIC1111/stable-diffusion-webui.git
    %cd stable-diffusion-webui
    ! git checkout a9fed7c364061ae6efb37f797b6b522cb3cf7aa2
    repositories_url = f"{prefix}/aigc-data/code/repositories.tar.gz"
    aria2(repositories_url, repositories_url.split("/")[-1], "./")
    ! tar -xf repositories.tar.gz
    %cd extensions
    ! git clone https://gitcode.net/mirrors/DominikDoom/a1111-sd-webui-tagcomplete.git
    ! git clone https://gitcode.net/ranting8323/stable-diffusion-webui-localization-zh_CN
    %cd .. 
    ! wget -c http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com/aigc-data/webui_config/config_tryon.json -O config.json
    %cd ..

    返回如下结果表明下载完成。

    image.png

步骤二:微调Stable Diffusion模型

  1. 在Notebook中执行以下代码,下载示例数据集和训练代码。后续会使用该数据集和训练代码进行模型训练。

    ! wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/cloth_train_example.tar.gz && tar -xvf cloth_train_example.tar.gz
    ! wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/train_text_to_image_lora.py

    系统输出结果如下图所示,表示代码执行成功。

    --2023-07-10 06:15:45--  http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/cloth_train_example.tar.gz
    Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111
    Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 4755690 (4.5M) [application/gzip]
    Saving to: ‘cloth_train_example.tar.gz’
    
    cloth_train_example 100%[===================>]   4.54M  9.33MB/s    in 0.5s    
    
    2023-07-10 06:15:46 (9.33 MB/s) - ‘cloth_train_example.tar.gz’ saved [4755690/4755690]
    
    cloth_train_example/
    cloth_train_example/train/
    cloth_train_example/train/20230407174322.jpg
    cloth_train_example/train/metadata.jsonl
    cloth_train_example/train/20230407174421.jpg
    cloth_train_example/train/20230407174403.jpg
    cloth_train_example/train/20230407174429.jpg
    cloth_train_example/train/20230407174311.jpg
    cloth_train_example/train/20230407174332.jpg
    cloth_train_example/train/20230407174450.jpg
    cloth_train_example/train/.ipynb_checkpoints/
    cloth_train_example/train/.ipynb_checkpoints/20230407174332-checkpoint.jpg
    cloth_train_example/train/20230407174343.jpg
    cloth_train_example/train/20230407174354.jpg
    cloth_train_example/train/20230407174413.jpg
    cloth_train_example/train/20230407174439.jpg
    cloth_train_example/.ipynb_checkpoints/
    --2023-07-10 06:15:47--  http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/train_text_to_image_lora.py
    Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111
    Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 37688 (37K) [application/octet-stream]
    Saving to: ‘train_text_to_image_lora.py’
    
    train_text_to_image 100%[===================>]  36.80K  --.-KB/s    in 0.03s   
    
    2023-07-10 06:15:47 (1.17 MB/s) - ‘train_text_to_image_lora.py’ saved [37688/37688]

    查看示例服装

    from PIL import Image
    display(Image.open("cloth_train_example/train/20230407174450.jpg"))
    image
  2. 下载预训练模型并转化成diffusers格式,您可以直接运行如下命令进行下载。

    safety_checker_url = f"{prefix}/aigc-data/hug_model/models--CompVis--stable-diffusion-safety-checker.tar.gz"
    aria2(safety_checker_url, safety_checker_url.split("/")[-1], "./")
    ! tar -xf models--CompVis--stable-diffusion-safety-checker.tar.gz -C /root/.cache/huggingface/hub/
    
    clip_url = f"{prefix}/aigc-data/hug_model/models--openai--clip-vit-large-patch14.tar.gz"
    aria2(clip_url, clip_url.split("/")[-1], "./")
    ! tar -xf models--openai--clip-vit-large-patch14.tar.gz -C /root/.cache/huggingface/hub/
    
    model_url = f"{prefix}/aigc-data/sd_models/chilloutmix_NiPrunedFp32Fix.safetensors"
    aria2(model_url, model_url.split("/")[-1], "stable-diffusion-webui/models/Stable-diffusion/")
    
    ! python diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py \
    --checkpoint_path=stable-diffusion-webui/models/Stable-diffusion/chilloutmix_NiPrunedFp32Fix.safetensors \
    --dump_path=chilloutmix-ni --from_safetensors

    返回结果如下。

    [#a9f494 660MiB/685MiB(96%) CN:7 DL:166MiB]m0m]
    Download Results:
    gid   |stat|avg speed  |path/URI
    ======+====+===========+=======================================================
    a9f494|OK  |   110MiB/s|.//models--CompVis--stable-diffusion-safety-checker.tar.gz
    
    Status Legend:
    (OK):download completed.
    [#28f571 0.9GiB/0.9GiB(99%) CN:1 DL:145MiB]m0m]
    Download Results:
    gid   |stat|avg speed  |path/URI
    ======+====+===========+=======================================================
    28f571|OK  |   109MiB/s|.//models--openai--clip-vit-large-patch14.tar.gz
    
    Status Legend:
    (OK):download completed.
    [#a43805 3.9GiB/3.9GiB(99%) CN:3 DL:154MiB]m0m]miB(99%)]
    Download Results:
    gid   |stat|avg speed  |path/URI
    ======+====+===========+=======================================================
    a43805|OK  |   122MiB/s|stable-diffusion-webui/models/Stable-diffusion//chilloutmix_NiPrunedFp32Fix.safetensors
    
    Status Legend:
    (OK):download completed.
    global_step key not found in model
    - This IS expected if you are initializing CLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing CLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  3. 模型训练。

    设置num_train_epochs为200,进行lora模型的训练。

    ! export MODEL_NAME="chilloutmix-ni" && \
    export DATASET_NAME="cloth_train_example" && \
    accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
      --pretrained_model_name_or_path=$MODEL_NAME \
      --dataset_name=$DATASET_NAME --caption_column="text" \
      --width=640 --height=768 --random_flip \
      --train_batch_size=1 \
      --num_train_epochs=200 --checkpointing_steps=5000 \
      --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
      --seed=42 \
      --output_dir="cloth-model-lora" \
      --validation_prompt="cloth1" --validation_epochs=100
  4. 准备WebUI所需模型文件。

    1. 将lora模型转化成WebUI支持格式并拷贝到WebUI所在目录。

      ! wget -c http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/convert-to-safetensors.py
      ! python convert-to-safetensors.py --file='cloth-model-lora/pytorch_lora_weights.bin'
      ! mkdir stable-diffusion-webui/models/Lora
      ! cp cloth-model-lora/pytorch_lora_weights_converted.safetensors stable-diffusion-webui/models/Lora/cloth_lora_weights.safetensors

      返回结果如下。

      --2023-07-10 06:44:45--  http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/convert-to-safetensors.py
      Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111
      Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected.
      HTTP request sent, awaiting response... 200 OK
      Length: 2032 (2.0K) [application/octet-stream]
      Saving to: ‘convert-to-safetensors.py’
      
      convert-to-safetens 100%[===================>]   1.98K  --.-KB/s    in 0s      
      
      2023-07-10 06:44:45 (475 MB/s) - ‘convert-to-safetensors.py’ saved [2032/2032]
      
      device is cuda
      Saving cloth-model-lora/pytorch_lora_weights_converted.safetensors
    2. 准备额外模型文件。为了加速下载,您可以运行如下命令直接下载额外模型文件。

      detection_url = f"{prefix}/aigc-data/codeformer/detection_Resnet50_Final.pth"
      aria2(detection_url, detection_url.split("/")[-1], "stable-diffusion-webui/repositories/CodeFormer/weights/facelib/")
      parse_url = f"{prefix}/aigc-data/codeformer/parsing_parsenet.pth"
      aria2(parse_url, parse_url.split("/")[-1], "stable-diffusion-webui/repositories/CodeFormer/weights/facelib/")
      codeformer_url = f"{prefix}/aigc-data/codeformer/codeformer-v0.1.0.pth"
      aria2(codeformer_url, codeformer_url.split("/")[-1], "stable-diffusion-webui/models/Codeformer/")
      
      embedding_url = f"{prefix}/aigc-data/embedding/ng_deepnegative_v1_75t.pt"
      aria2(embedding_url, embedding_url.split("/")[-1], "stable-diffusion-webui/embeddings/")
      
      model_lora_url = f"{prefix}/aigc-data/lora/koreanDollLikeness_v10.safetensors"
      aria2(model_lora_url, model_lora_url.split("/")[-1], "stable-diffusion-webui/models/Lora/")

      返回结果如下。

      [#75b426 86MiB/104MiB(82%) CN:3 DL:51MiB]0m]
      Download Results:
      gid   |stat|avg speed  |path/URI
      ======+====+===========+=======================================================
      75b426|OK  |    40MiB/s|stable-diffusion-webui/repositories/CodeFormer/weights/facelib//detection_Resnet50_Final.pth
      
      Status Legend:
      (OK):download completed.
      [#445975 78MiB/81MiB(96%) CN:2 DL:49MiB]
      Download Results:
      gid   |stat|avg speed  |path/URI
      ======+====+===========+=======================================================
      445975|OK  |    40MiB/s|stable-diffusion-webui/repositories/CodeFormer/weights/facelib//parsing_parsenet.pth
      
      Status Legend:
      (OK):download completed.
      [#703950 358MiB/359MiB(99%) CN:1 DL:172MiB]m]
      Download Results:
      gid   |stat|avg speed  |path/URI
      ======+====+===========+=======================================================
      703950|OK  |   131MiB/s|stable-diffusion-webui/models/Codeformer//codeformer-v0.1.0.pth
      
      Status Legend:
      (OK):download completed.
      
      Download Results:
      gid   |stat|avg speed  |path/URI
      ======+====+===========+=======================================================
      90e55e|OK  |   4.0MiB/s|stable-diffusion-webui/embeddings//ng_deepnegative_v1_75t.pt
      
      Status Legend:
      (OK):download completed.
      [#6bf021 140MiB/144MiB(97%) CN:1 DL:41MiB]m]
      Download Results:
      gid   |stat|avg speed  |path/URI
      ======+====+===========+=======================================================
      6bf021|OK  |    34MiB/s|stable-diffusion-webui/models/Lora//koreanDollLikeness_v10.safetensors
      
      Status Legend:
      (OK):download completed.

步骤三:在DSW中启动WebUI

  1. 在Notebook中执行命令,启动WebUI。

    说明

    由于Github访问存在不稳定性,如果运行后未出现正常返回结果且提示网络相关原因,例如:Network is unreachableunable to access 'https://github.com/......',您可以重新运行命令。

    ! cd stable-diffusion-webui && python -m venv --system-site-packages --symlinks venv
    ! cd stable-diffusion-webui && \
      sed -i 's/can_run_as_root=0/can_run_as_root=1/g' webui.sh && \
      ./webui.sh --no-download-sd-model --xformers --gradio-queue
  2. 在返回结果中,单击URL链接(http://127.0.0.1:7860),进入WebUI页面。后续您可以在该页面,进行模型推理。

    说明

    由于http://127.0.0.1:7860为内网访问地址,仅支持在当前的DSW实例内部通过单击链接来访问WebUI页面,不支持通过外部浏览器直接访问。

步骤四:推理结果验证

完成以上操作后,您已经成功完成了AIGC文生图模型微调训练及WebUI部署。您可以在WebUI页面,进行模型推理验证。

文生图页签配置以下参数:

  • Prompt:cloth1,<lora:koreanDollLikeness_v10:0.4>, (extremely detailed CG unity 8k wallpaper),(RAW photo, best quality), (realistic, photo-realistic:1.2), a close up portrait photo, 1girl, shopping mall rooftop cafe, outdoor, smile, (high detailed skin:1.4), puffy eyes, gorgeous hair, air bangs, brown black hair, soft lighting, high quality,<lora:cloth_lora_weights:1>

  • Negative prompt:ng_deepnegative_v1_75t,paintings, sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), (grayscale:1.2), skin spots, acnes, skin blemishes, age spot, glans,extra fingers,fewer fingers,(watermark:1.2),(letters:1.2),(nsfw:1.2),teeth

  • 采样方法(Sampler): Euler a

  • 采样迭代步数(Steps): 50

  • 宽度高度: 640,768

  • 随机种子(seed): 1400244389

  • 提示词相关性(CFG Scale):7

  • 选中面部修复复选框

单击生成,输出如图推理结果。image.png

阿里云首页 机器学习平台 PAI 相关技术圈