AIGC文生图模型微调及WebUI部署
本文为您介绍如何在阿里云交互式建模(PAI-DSW)中,基于Diffusers开源库进行AIGC Stable Diffusion模型的微调训练,以及基于Stable-Diffusion-WebUI开源库启动WebUI进行模型推理。
背景信息
AIGC是指通过人工智能技术自动生成内容的生产方式,已经成为继互联网时代的下一个产业时代风口。其中,文生图(Text-to-image Generation)任务是流行的跨模态生成任务,旨在生成与给定文本对应的图像。
本文实现的功能主要是基于AIGC Stable Diffusion文生图Lora模型进行模型微调实现虚拟上装。WebUI文生图推理效果如图所示。
前提条件
已创建工作空间,详情请参见创建工作空间。
已创建DSW实例,其中关键参数配置如下。具体操作,请参见创建及管理DSW实例。
实例规格选择:ecs.gn7i-c8g1.2xlarge。
选择镜像:
地域
镜像地址
华东1(杭州)
在官方镜像页签,选择stable-diffusion-webui-env:pytorch1.13-gpu-py310-cu117-ubuntu22.04。
华北2(北京)
华东2(上海)
华南1(深圳)
步骤一:安装Diffusers
在交互式建模(DSW)页面,单击实例操作列下的打开,进入DSW开发环境。
在Notebook页签的Launcher页面,单击快速开始区域Notebook下的Python3。
在Notebook中执行以下代码,从GitHub下载Diffusers开源库,并安装相关依赖。
后续将基于Diffusers开源库进行模型微调。安装过程大约需要持续5分钟。
说明预置镜像中已安装Diffusers,如果需要安装最新版本,请运行注释代码。
! git clone https://github.com/huggingface/diffusers ! cd diffusers && git checkout e126a82cc5d9afbeb9b476455de24dd3e7dd358a ! cd diffusers && pip install .
验证是否安装成功。
import diffusers
系统返回如下结果,表示安装成功。
/usr/local/lib/python3.10/dist-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
配置accelerate。
您可以直接运行以下命令下载默认配置文件,如果需要自定义配置则在Terminal中执行
accelerate config
命令,并根据DSW实例详情,选择对应配置即可。本案例下载默认配置文件。! mkdir -p /root/.cache/huggingface/accelerate/ ! wget -c http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com/aigc-data/accelerate/default_config.yaml -O /root/.cache/huggingface/accelerate/default_config.yaml
系统返回如下结果,表示默认配置文件下载成功。
在Notebook中执行命令,安装文生图算法相关依赖库。
! cd diffusers/examples/text_to_image && pip install -r requirements.txt
系统输出下图结果,出现的WARNING信息可忽略。
在Notebook中执行以下命令,下载stable-diffusion-webui开源库。
import os ! apt update ! apt install -y aria2 def aria2(url, filename, d): !aria2c --console-log-level=error -c -x 16 -s 16 {url} -o {filename} -d {d} url_prefix = { "cn-shanghai": "http://pai-vision-data-sh.oss-cn-shanghai-internal.aliyuncs.com", "cn-hangzhou": "http://pai-vision-data-hz2.oss-cn-hangzhou-internal.aliyuncs.com", "cn-shenzhen": "http://pai-vision-data-sz.oss-cn-shenzhen-internal.aliyuncs.com", "cn-beijing": "http://pai-vision-data-bj.oss-cn-beijing-internal.aliyuncs.com", } dsw_region = os.environ.get("dsw_region") prefix = url_prefix[dsw_region] if dsw_region in url_prefix else "http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com" ! git clone https://gitcode.net/mirrors/AUTOMATIC1111/stable-diffusion-webui.git %cd stable-diffusion-webui ! git checkout a9fed7c364061ae6efb37f797b6b522cb3cf7aa2 repositories_url = f"{prefix}/aigc-data/code/repositories.tar.gz" aria2(repositories_url, repositories_url.split("/")[-1], "./") ! tar -xf repositories.tar.gz %cd extensions ! git clone https://gitcode.net/mirrors/DominikDoom/a1111-sd-webui-tagcomplete.git ! git clone https://gitcode.net/ranting8323/stable-diffusion-webui-localization-zh_CN %cd .. ! wget -c http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com/aigc-data/webui_config/config_tryon.json -O config.json %cd ..
返回如下结果表明下载完成。
步骤二:微调Stable Diffusion模型
在Notebook中执行以下代码,下载示例数据集和训练代码。后续会使用该数据集和训练代码进行模型训练。
! wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/cloth_train_example.tar.gz && tar -xvf cloth_train_example.tar.gz ! wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/train_text_to_image_lora.py
系统输出结果如下图所示,表示代码执行成功。
--2023-07-10 06:15:45-- http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/cloth_train_example.tar.gz Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111 Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4755690 (4.5M) [application/gzip] Saving to: ‘cloth_train_example.tar.gz’ cloth_train_example 100%[===================>] 4.54M 9.33MB/s in 0.5s 2023-07-10 06:15:46 (9.33 MB/s) - ‘cloth_train_example.tar.gz’ saved [4755690/4755690] cloth_train_example/ cloth_train_example/train/ cloth_train_example/train/20230407174322.jpg cloth_train_example/train/metadata.jsonl cloth_train_example/train/20230407174421.jpg cloth_train_example/train/20230407174403.jpg cloth_train_example/train/20230407174429.jpg cloth_train_example/train/20230407174311.jpg cloth_train_example/train/20230407174332.jpg cloth_train_example/train/20230407174450.jpg cloth_train_example/train/.ipynb_checkpoints/ cloth_train_example/train/.ipynb_checkpoints/20230407174332-checkpoint.jpg cloth_train_example/train/20230407174343.jpg cloth_train_example/train/20230407174354.jpg cloth_train_example/train/20230407174413.jpg cloth_train_example/train/20230407174439.jpg cloth_train_example/.ipynb_checkpoints/ --2023-07-10 06:15:47-- http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/train_text_to_image_lora.py Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111 Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 37688 (37K) [application/octet-stream] Saving to: ‘train_text_to_image_lora.py’ train_text_to_image 100%[===================>] 36.80K --.-KB/s in 0.03s 2023-07-10 06:15:47 (1.17 MB/s) - ‘train_text_to_image_lora.py’ saved [37688/37688]
查看示例服装
from PIL import Image display(Image.open("cloth_train_example/train/20230407174450.jpg"))
下载预训练模型并转化成diffusers格式,您可以直接运行如下命令进行下载。
safety_checker_url = f"{prefix}/aigc-data/hug_model/models--CompVis--stable-diffusion-safety-checker.tar.gz" aria2(safety_checker_url, safety_checker_url.split("/")[-1], "./") ! tar -xf models--CompVis--stable-diffusion-safety-checker.tar.gz -C /root/.cache/huggingface/hub/ clip_url = f"{prefix}/aigc-data/hug_model/models--openai--clip-vit-large-patch14.tar.gz" aria2(clip_url, clip_url.split("/")[-1], "./") ! tar -xf models--openai--clip-vit-large-patch14.tar.gz -C /root/.cache/huggingface/hub/ model_url = f"{prefix}/aigc-data/sd_models/chilloutmix_NiPrunedFp32Fix.safetensors" aria2(model_url, model_url.split("/")[-1], "stable-diffusion-webui/models/Stable-diffusion/") ! python diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py \ --checkpoint_path=stable-diffusion-webui/models/Stable-diffusion/chilloutmix_NiPrunedFp32Fix.safetensors \ --dump_path=chilloutmix-ni --from_safetensors
返回结果如下。
[#a9f494 660MiB/685MiB(96%) CN:7 DL:166MiB]m0m] Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= a9f494|OK | 110MiB/s|.//models--CompVis--stable-diffusion-safety-checker.tar.gz Status Legend: (OK):download completed. [#28f571 0.9GiB/0.9GiB(99%) CN:1 DL:145MiB]m0m] Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 28f571|OK | 109MiB/s|.//models--openai--clip-vit-large-patch14.tar.gz Status Legend: (OK):download completed. [#a43805 3.9GiB/3.9GiB(99%) CN:3 DL:154MiB]m0m]miB(99%)] Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= a43805|OK | 122MiB/s|stable-diffusion-webui/models/Stable-diffusion//chilloutmix_NiPrunedFp32Fix.safetensors Status Legend: (OK):download completed. global_step key not found in model - This IS expected if you are initializing CLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing CLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
模型训练。
设置num_train_epochs为200,进行lora模型的训练。
! export MODEL_NAME="chilloutmix-ni" && \ export DATASET_NAME="cloth_train_example" && \ accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \ --pretrained_model_name_or_path=$MODEL_NAME \ --dataset_name=$DATASET_NAME --caption_column="text" \ --width=640 --height=768 --random_flip \ --train_batch_size=1 \ --num_train_epochs=200 --checkpointing_steps=5000 \ --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \ --seed=42 \ --output_dir="cloth-model-lora" \ --validation_prompt="cloth1" --validation_epochs=100
准备WebUI所需模型文件。
将lora模型转化成WebUI支持格式并拷贝到WebUI所在目录。
! wget -c http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/convert-to-safetensors.py ! python convert-to-safetensors.py --file='cloth-model-lora/pytorch_lora_weights.bin' ! mkdir stable-diffusion-webui/models/Lora ! cp cloth-model-lora/pytorch_lora_weights_converted.safetensors stable-diffusion-webui/models/Lora/cloth_lora_weights.safetensors
返回结果如下。
--2023-07-10 06:44:45-- http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/convert-to-safetensors.py Resolving pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)... 39.98.1.111 Connecting to pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com (pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com)|39.98.1.111|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 2032 (2.0K) [application/octet-stream] Saving to: ‘convert-to-safetensors.py’ convert-to-safetens 100%[===================>] 1.98K --.-KB/s in 0s 2023-07-10 06:44:45 (475 MB/s) - ‘convert-to-safetensors.py’ saved [2032/2032] device is cuda Saving cloth-model-lora/pytorch_lora_weights_converted.safetensors
准备额外模型文件。为了加速下载,您可以运行如下命令直接下载额外模型文件。
detection_url = f"{prefix}/aigc-data/codeformer/detection_Resnet50_Final.pth" aria2(detection_url, detection_url.split("/")[-1], "stable-diffusion-webui/repositories/CodeFormer/weights/facelib/") parse_url = f"{prefix}/aigc-data/codeformer/parsing_parsenet.pth" aria2(parse_url, parse_url.split("/")[-1], "stable-diffusion-webui/repositories/CodeFormer/weights/facelib/") codeformer_url = f"{prefix}/aigc-data/codeformer/codeformer-v0.1.0.pth" aria2(codeformer_url, codeformer_url.split("/")[-1], "stable-diffusion-webui/models/Codeformer/") embedding_url = f"{prefix}/aigc-data/embedding/ng_deepnegative_v1_75t.pt" aria2(embedding_url, embedding_url.split("/")[-1], "stable-diffusion-webui/embeddings/") model_lora_url = f"{prefix}/aigc-data/lora/koreanDollLikeness_v10.safetensors" aria2(model_lora_url, model_lora_url.split("/")[-1], "stable-diffusion-webui/models/Lora/")
返回结果如下。
[#75b426 86MiB/104MiB(82%) CN:3 DL:51MiB]0m] Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 75b426|OK | 40MiB/s|stable-diffusion-webui/repositories/CodeFormer/weights/facelib//detection_Resnet50_Final.pth Status Legend: (OK):download completed. [#445975 78MiB/81MiB(96%) CN:2 DL:49MiB] Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 445975|OK | 40MiB/s|stable-diffusion-webui/repositories/CodeFormer/weights/facelib//parsing_parsenet.pth Status Legend: (OK):download completed. [#703950 358MiB/359MiB(99%) CN:1 DL:172MiB]m] Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 703950|OK | 131MiB/s|stable-diffusion-webui/models/Codeformer//codeformer-v0.1.0.pth Status Legend: (OK):download completed. Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 90e55e|OK | 4.0MiB/s|stable-diffusion-webui/embeddings//ng_deepnegative_v1_75t.pt Status Legend: (OK):download completed. [#6bf021 140MiB/144MiB(97%) CN:1 DL:41MiB]m] Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 6bf021|OK | 34MiB/s|stable-diffusion-webui/models/Lora//koreanDollLikeness_v10.safetensors Status Legend: (OK):download completed.
步骤三:在DSW中启动WebUI
在Notebook中执行命令,启动WebUI。
说明由于Github访问存在不稳定性,如果运行后未出现正常返回结果且提示网络相关原因,例如:
Network is unreachable
、unable to access 'https://github.com/......'
,您可以重新运行命令。! cd stable-diffusion-webui && python -m venv --system-site-packages --symlinks venv ! cd stable-diffusion-webui && \ sed -i 's/can_run_as_root=0/can_run_as_root=1/g' webui.sh && \ ./webui.sh --no-download-sd-model --xformers --gradio-queue
在返回结果中,单击URL链接(http://127.0.0.1:7860),进入WebUI页面。后续您可以在该页面,进行模型推理。
说明由于
http://127.0.0.1:7860
为内网访问地址,仅支持在当前的DSW实例内部通过单击链接来访问WebUI页面,不支持通过外部浏览器直接访问。
步骤四:推理结果验证
完成以上操作后,您已经成功完成了AIGC文生图模型微调训练及WebUI部署。您可以在WebUI页面,进行模型推理验证。
在文生图页签配置以下参数:
Prompt:cloth1,
<lora:koreanDollLikeness_v10:0.4>
, (extremely detailed CG unity 8k wallpaper),(RAW photo, best quality), (realistic, photo-realistic:1.2), a close up portrait photo, 1girl, shopping mall rooftop cafe, outdoor, smile, (high detailed skin:1.4), puffy eyes, gorgeous hair, air bangs, brown black hair, soft lighting, high quality,<lora:cloth_lora_weights:1>
Negative prompt:ng_deepnegative_v1_75t,paintings, sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), (grayscale:1.2), skin spots, acnes, skin blemishes, age spot, glans,extra fingers,fewer fingers,(watermark:1.2),(letters:1.2),(nsfw:1.2),teeth
采样方法(Sampler): Euler a
采样迭代步数(Steps): 50
宽度和高度: 640,768
随机种子(seed): 1400244389
提示词相关性(CFG Scale):7
选中面部修复复选框
单击生成,输出如图推理结果。