首页 机器学习平台 PAI 实践教程 AIGC AIGC文生图模型微调及WebUI部署

AIGC文生图模型微调及WebUI部署

更新时间: 2023-09-01 11:12:22

本文为您介绍如何在阿里云交互式建模（PAI-DSW）中，基于Diffusers开源库进行AIGC Stable Diffusion模型的微调训练，以及基于Stable-Diffusion-WebUI开源库启动WebUI进行模型推理。

背景信息

AIGC是指通过人工智能技术自动生成内容的生产方式，已经成为继互联网时代的下一个产业时代风口。其中，文生图（Text-to-image Generation）任务是流行的跨模态生成任务，旨在生成与给定文本对应的图像。

本文实现的功能主要是基于AIGC Stable Diffusion文生图Lora模型进行模型微调实现虚拟上装。WebUI文生图推理效果如图所示。image.png

前提条件

  • 已创建工作空间，详情请参见创建工作空间。

  • 已创建DSW实例，其中关键参数配置如下。具体操作，请参见创建及管理DSW实例。

    • 实例规格选择：ecs.gn7i-c8g1.2xlarge。

    • 选择镜像：

      地域

      镜像地址

      华东1（杭州）

      在官方镜像页签，选择stable-diffusion-webui-env:pytorch1.13-gpu-py310-cu117-ubuntu22.04。

      华北2（北京）

      华东2（上海）

      华南1（深圳）

步骤一：安装Diffusers

  1. 在交互式建模（DSW）页面，单击实例操作列下的打开，进入DSW开发环境。

  2. 在Notebook页签的Launcher页面，单击快速开始区域Notebook下的Python3。

  3. 在Notebook中执行以下代码，从GitHub下载Diffusers开源库，并安装相关依赖。

    后续将基于Diffusers开源库进行模型微调。安装过程大约需要持续5分钟。

    说明

    预置镜像中已安装Diffusers，如果需要安装最新版本，请运行注释代码。

    ! git clone https://github.com/huggingface/diffusers
! cd diffusers && git checkout e126a82cc5d9afbeb9b476455de24dd3e7dd358a
! cd diffusers && pip install .

  4. 验证是否安装成功。

    import diffusers

    系统返回如下结果，表示安装成功。

    /usr/local/lib/python3.10/dist-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

  5. 配置accelerate。

    您可以直接运行以下命令下载默认配置文件，如果需要自定义配置则在Terminal中执行accelerate config命令，并根据DSW实例详情，选择对应配置即可。本案例下载默认配置文件。

    ! mkdir -p /root/.cache/huggingface/accelerate/
! wget -c http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com/aigc-data/accelerate/default_config.yaml -O /root/.cache/huggingface/accelerate/default_config.yaml

    系统返回如下结果，表示默认配置文件下载成功。image.png

  6. 在Notebook中执行命令，安装文生图算法相关依赖库。

    ! cd diffusers/examples/text_to_image && pip install -r requirements.txt

    系统输出下图结果，出现的WARNING信息可忽略。

    image

  7. 在Notebook中执行以下命令，下载stable-diffusion-webui开源库。

    import os

! apt update
! apt install -y aria2

def aria2(url, filename, d):
    !aria2c --console-log-level=error -c -x 16 -s 16 {url} -o {filename} -d {d}
    
url_prefix = {
    "cn-shanghai": "http://pai-vision-data-sh.oss-cn-shanghai-internal.aliyuncs.com",
    "cn-hangzhou": "http://pai-vision-data-hz2.oss-cn-hangzhou-internal.aliyuncs.com",
    "cn-shenzhen": "http://pai-vision-data-sz.oss-cn-shenzhen-internal.aliyuncs.com",
    "cn-beijing": "http://pai-vision-data-bj.oss-cn-beijing-internal.aliyuncs.com", 
}

dsw_region = os.environ.get("dsw_region")
prefix = url_prefix[dsw_region] if dsw_region in url_prefix else "http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com"


! git clone https://gitcode.net/mirrors/AUTOMATIC1111/stable-diffusion-webui.git
%cd stable-diffusion-webui
! git checkout a9fed7c364061ae6efb37f797b6b522cb3cf7aa2
repositories_url = f"{prefix}/aigc-data/code/repositories.tar.gz"
aria2(repositories_url, repositories_url.split("/")[-1], "./")
! tar -xf repositories.tar.gz
%cd extensions
! git clone https://gitcode.net/mirrors/DominikDoom/a1111-sd-webui-tagcomplete.git
! git clone https://gitcode.net/ranting8323/stable-diffusion-webui-localization-zh_CN
%cd .. 
! wget -c http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com/aigc-data/webui_config/config_tryon.json -O config.json
%cd ..

    返回如下结果表明下载完成。

    image.png

步骤二：微调Stable Diffusion模型

  1. 在Notebook中执行以下代码，下载示例数据集和训练代码。后续会使用该数据集和训练代码进行模型训练。

    ! wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/cloth_train_example.tar.gz && tar -xvf cloth_train_example.tar.gz
! wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/train_text_to_image_lora.py

    系统输出结果如下图所示，表示代码执行成功。

    --2023-07-10 06:15:45--  http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/cloth_train_example.tar.gz
Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111
Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4755690 (4.5M) [application/gzip]
Saving to: ‘cloth_train_example.tar.gz’

cloth_train_example 100%[===================>]   4.54M  9.33MB/s    in 0.5s    

2023-07-10 06:15:46 (9.33 MB/s) - ‘cloth_train_example.tar.gz’ saved [4755690/4755690]

cloth_train_example/
cloth_train_example/train/
cloth_train_example/train/20230407174322.jpg
cloth_train_example/train/metadata.jsonl
cloth_train_example/train/20230407174421.jpg
cloth_train_example/train/20230407174403.jpg
cloth_train_example/train/20230407174429.jpg
cloth_train_example/train/20230407174311.jpg
cloth_train_example/train/20230407174332.jpg
cloth_train_example/train/20230407174450.jpg
cloth_train_example/train/.ipynb_checkpoints/
cloth_train_example/train/.ipynb_checkpoints/20230407174332-checkpoint.jpg
cloth_train_example/train/20230407174343.jpg
cloth_train_example/train/20230407174354.jpg
cloth_train_example/train/20230407174413.jpg
cloth_train_example/train/20230407174439.jpg
cloth_train_example/.ipynb_checkpoints/
--2023-07-10 06:15:47--  http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/train_text_to_image_lora.py
Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111
Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37688 (37K) [application/octet-stream]
Saving to: ‘train_text_to_image_lora.py’

train_text_to_image 100%[===================>]  36.80K  --.-KB/s    in 0.03s   

2023-07-10 06:15:47 (1.17 MB/s) - ‘train_text_to_image_lora.py’ saved [37688/37688]

    查看示例服装

    from PIL import Image
display(Image.open("cloth_train_example/train/20230407174450.jpg"))
    image

  2. 下载预训练模型并转化成diffusers格式，您可以直接运行如下命令进行下载。

    safety_checker_url = f"{prefix}/aigc-data/hug_model/models--CompVis--stable-diffusion-safety-checker.tar.gz"
aria2(safety_checker_url, safety_checker_url.split("/")[-1], "./")
! tar -xf models--CompVis--stable-diffusion-safety-checker.tar.gz -C /root/.cache/huggingface/hub/

clip_url = f"{prefix}/aigc-data/hug_model/models--openai--clip-vit-large-patch14.tar.gz"
aria2(clip_url, clip_url.split("/")[-1], "./")
! tar -xf models--openai--clip-vit-large-patch14.tar.gz -C /root/.cache/huggingface/hub/

model_url = f"{prefix}/aigc-data/sd_models/chilloutmix_NiPrunedFp32Fix.safetensors"
aria2(model_url, model_url.split("/")[-1], "stable-diffusion-webui/models/Stable-diffusion/")

! python diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py \
--checkpoint_path=stable-diffusion-webui/models/Stable-diffusion/chilloutmix_NiPrunedFp32Fix.safetensors \
--dump_path=chilloutmix-ni --from_safetensors

    返回结果如下。

    [#a9f494 660MiB/685MiB(96%) CN:7 DL:166MiB]m0m]
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
a9f494|OK  |   110MiB/s|.//models--CompVis--stable-diffusion-safety-checker.tar.gz

Status Legend:
(OK):download completed.
[#28f571 0.9GiB/0.9GiB(99%) CN:1 DL:145MiB]m0m]
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
28f571|OK  |   109MiB/s|.//models--openai--clip-vit-large-patch14.tar.gz

Status Legend:
(OK):download completed.
[#a43805 3.9GiB/3.9GiB(99%) CN:3 DL:154MiB]m0m]miB(99%)]
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
a43805|OK  |   122MiB/s|stable-diffusion-webui/models/Stable-diffusion//chilloutmix_NiPrunedFp32Fix.safetensors

Status Legend:
(OK):download completed.
global_step key not found in model
- This IS expected if you are initializing CLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing CLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

  3. 模型训练。

    设置num_train_epochs为200，进行lora模型的训练。

    ! export MODEL_NAME="chilloutmix-ni" && \
export DATASET_NAME="cloth_train_example" && \
accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$DATASET_NAME --caption_column="text" \
  --width=640 --height=768 --random_flip \
  --train_batch_size=1 \
  --num_train_epochs=200 --checkpointing_steps=5000 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --output_dir="cloth-model-lora" \
  --validation_prompt="cloth1" --validation_epochs=100

  4. 准备WebUI所需模型文件。

    1. 将lora模型转化成WebUI支持格式并拷贝到WebUI所在目录。

      ! wget -c http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/convert-to-safetensors.py
! python convert-to-safetensors.py --file='cloth-model-lora/pytorch_lora_weights.bin'
! mkdir stable-diffusion-webui/models/Lora
! cp cloth-model-lora/pytorch_lora_weights_converted.safetensors stable-diffusion-webui/models/Lora/cloth_lora_weights.safetensors

      返回结果如下。

      --2023-07-10 06:44:45--  http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/convert-to-safetensors.py
Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111
Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2032 (2.0K) [application/octet-stream]
Saving to: ‘convert-to-safetensors.py’

convert-to-safetens 100%[===================>]   1.98K  --.-KB/s    in 0s      

2023-07-10 06:44:45 (475 MB/s) - ‘convert-to-safetensors.py’ saved [2032/2032]

device is cuda
Saving cloth-model-lora/pytorch_lora_weights_converted.safetensors

    2. 准备额外模型文件。为了加速下载，您可以运行如下命令直接下载额外模型文件。

      detection_url = f"{prefix}/aigc-data/codeformer/detection_Resnet50_Final.pth"
aria2(detection_url, detection_url.split("/")[-1], "stable-diffusion-webui/repositories/CodeFormer/weights/facelib/")
parse_url = f"{prefix}/aigc-data/codeformer/parsing_parsenet.pth"
aria2(parse_url, parse_url.split("/")[-1], "stable-diffusion-webui/repositories/CodeFormer/weights/facelib/")
codeformer_url = f"{prefix}/aigc-data/codeformer/codeformer-v0.1.0.pth"
aria2(codeformer_url, codeformer_url.split("/")[-1], "stable-diffusion-webui/models/Codeformer/")

embedding_url = f"{prefix}/aigc-data/embedding/ng_deepnegative_v1_75t.pt"
aria2(embedding_url, embedding_url.split("/")[-1], "stable-diffusion-webui/embeddings/")

model_lora_url = f"{prefix}/aigc-data/lora/koreanDollLikeness_v10.safetensors"
aria2(model_lora_url, model_lora_url.split("/")[-1], "stable-diffusion-webui/models/Lora/")

      返回结果如下。

      [#75b426 86MiB/104MiB(82%) CN:3 DL:51MiB]0m]
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
75b426|OK  |    40MiB/s|stable-diffusion-webui/repositories/CodeFormer/weights/facelib//detection_Resnet50_Final.pth

Status Legend:
(OK):download completed.
[#445975 78MiB/81MiB(96%) CN:2 DL:49MiB]
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
445975|OK  |    40MiB/s|stable-diffusion-webui/repositories/CodeFormer/weights/facelib//parsing_parsenet.pth

Status Legend:
(OK):download completed.
[#703950 358MiB/359MiB(99%) CN:1 DL:172MiB]m]
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
703950|OK  |   131MiB/s|stable-diffusion-webui/models/Codeformer//codeformer-v0.1.0.pth

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
90e55e|OK  |   4.0MiB/s|stable-diffusion-webui/embeddings//ng_deepnegative_v1_75t.pt

Status Legend:
(OK):download completed.
[#6bf021 140MiB/144MiB(97%) CN:1 DL:41MiB]m]
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
6bf021|OK  |    34MiB/s|stable-diffusion-webui/models/Lora//koreanDollLikeness_v10.safetensors

Status Legend:
(OK):download completed.

步骤三：在DSW中启动WebUI

  1. 在Notebook中执行命令，启动WebUI。

    说明

    由于Github访问存在不稳定性，如果运行后未出现正常返回结果且提示网络相关原因，例如：Network is unreachable、unable to access 'https://github.com/......'，您可以重新运行命令。

    ! cd stable-diffusion-webui && python -m venv --system-site-packages --symlinks venv
! cd stable-diffusion-webui && \
  sed -i 's/can_run_as_root=0/can_run_as_root=1/g' webui.sh && \
  ./webui.sh --no-download-sd-model --xformers --gradio-queue

  2. 在返回结果中，单击URL链接（http://127.0.0.1:7860），进入WebUI页面。后续您可以在该页面，进行模型推理。

    说明

    由于http://127.0.0.1:7860为内网访问地址，仅支持在当前的DSW实例内部通过单击链接来访问WebUI页面，不支持通过外部浏览器直接访问。

步骤四：推理结果验证

完成以上操作后，您已经成功完成了AIGC文生图模型微调训练及WebUI部署。您可以在WebUI页面，进行模型推理验证。

在文生图页签配置以下参数：

  • Prompt：cloth1,<lora:koreanDollLikeness_v10:0.4>, (extremely detailed CG unity 8k wallpaper),(RAW photo, best quality), (realistic, photo-realistic:1.2), a close up portrait photo, 1girl, shopping mall rooftop cafe, outdoor, smile, (high detailed skin:1.4), puffy eyes, gorgeous hair, air bangs, brown black hair, soft lighting, high quality,<lora:cloth_lora_weights:1>

  • Negative prompt：ng_deepnegative_v1_75t,paintings, sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), (grayscale:1.2), skin spots, acnes, skin blemishes, age spot, glans,extra fingers,fewer fingers,(watermark:1.2),(letters:1.2),(nsfw:1.2),teeth

  • 采样方法（Sampler）： Euler a

  • 采样迭代步数（Steps）： 50

  • 宽度和高度： 640，768

  • 随机种子（seed）： 1400244389

  • 提示词相关性（CFG Scale）：7

  • 选中面部修复复选框

单击生成，输出如图推理结果。image.png

阿里云首页 机器学习平台 PAI 相关技术圈