Guide: Workflow for Creating Animations using animatediff-cli-prompt-travel Step-by-Step

Guide: Workflow for Creating Animations using animatediff-cli-prompt-travel Step-by-Step

Stable Diffusion#animatediff#workflow

September 27, 20230

Introduction

Here are some animations I've created using animatediff-cli-prompt-travel.

These video transitions are very smooth and there is almost no flicker, which really surprises me.

I will share a simple workflow so that anyone can join in and have fun. Perhaps we can use animatediff to create more interesting art! Let’s get started.

Before getting started

Before getting started, ensure:

  • You have sufficient knowledge of Stable Diffusion
  • Hardware: GPU should be at least 16GB, recommended 4060TI or above.
  • Be familiarity with Python installation, Git operations, and resolving network issues on Windows.

Installation

Resolving installation issues. If you encounter various weird issues during installation, don't be surprised, I've been using Python for 6 years and still get stuck at various places. 🤪

https://github.com/s9roll7/animatediff-cli-prompt-travel

Run Official Demo

Run the official examples for the first time (highlighted in red). The purpose of this step is to ensure that your hardware and software are sufficient to produce a video. At this time, numerous Python dependencies/models, etc., will be downloaded.

If you can successfully run the official demo, then we can proceed smoothly with the rest of the tutorial.

Workflow

Well, let's get into the main topic

In summary, generating keyframes (or so to speak) first, feeding keyframes to cli using controlnet(canny/softedge) and prompt travel.

Prompt Essential Images

You need to prompt essential keyframe images and make each keyframe consistent

launch a1111, prompt from 1girl, cover as many details as possible (quality words, actions, 1girl, background),

parameters as shown.

Here I'm using the AWPainting from @dynamicwangs

Why use AWPainting? Because the image quality produced by AWPainting is relatively high and the picture is relatively clean.

generate till you select some of the more stable images from these pictures.

Yes, this step is like drawing cards multiple times, there are no shortcuts.

Setup Prompts && Directories

First, create a new folder called project-awpainting-01

Now we select these highlighted pictures as cushion pictures.

Copy them into

  • animatediff-cli-prompt-travel\data\controlnet_image\proj-awpaint-01
  • \controlnet_canny
  • \controlnet_softedge

NOTE: You need to rename images as follows

  • 0000.png
  • 0032.png
  • 0064.png
  • 0096.png
  • 0128.png

0000.png means the first frame, 0032.png means the 33rd frame, and so on.

Create a new file animatediff-cli-prompt-travel\projects\proj-awpaint-01\prompts.json

NOTE: The prompt_map does not need to correspond to 0000.png or 0032.png that came before.

Contents are as follows, If you just want to play, you can consider using my template, modify the prompts and controlnet templates according to your requirements.

https://gist.github.com/hylarucoder/8bf47a351698a2834a5f0e5bf1eed102

For an explanation of the parameters, you can refer to the link.

https://github.com/s9roll7/animatediff-cli-prompt-travel#how-to-use

Draft

Run the script for low resolution video

.\venv\Scripts\python -m animatediff generate -c .\projects\proj-awpaint-01\prompts.json -W 384 -H 576 -L 256 -C 16 -o .\projects\proj-awpaint-01\output

Preview the effect of the image video.

It should take 10min to generate a video

Some parameters are listed below.

-W 384 video width
-H 576 video height
-L 256 video frames, in this case 256 means 12.8s (256 frames / 20 fps) 
-C 16 context, The higher context provided, the higher the GPU cost will be.
-o .\projects\proj-awpaint-01\output, you may see some outputfiles like `2023-09-25T20-51-50-sample-awpainting_v12`

You can change fps in prompts.json

{
  "output": {
    "format": "mp4",
    "fps": 20,
    "encode_param": {
      "crf": 10
    }
  }
}

for detail explanation:

https://github.com/s9roll7/animatediff-cli-prompt-travel#how-to-use

Production

Adjust to the highest quality for observation (note!! This is 384px, not the final product)

If the preview effect is good, start outputting refined video. Configuration is 4060TI 16GB VRAM, -C adjusted to 6, width 768

# change `.\projects\proj-awpaint-01\output\2023-09-25T20-51-50-sample-awpainting_v12\00-341774366206100` to your target folder
.\venv\Scripts\python -m animatediff refine -W 768 -C 6 .\projects\proj-awpaint-01\output\2023-09-25T20-51-50-sample-awpainting_v12\00-341774366206100

It should take 1 hour

If any of you are available, could you please give me duration test on A100 lol

Post-production

  • Upscale Video To 2K with Any Upscale Tools : Topaz AI Video / https://github.com/k4yt3x/video2x
  • Edit Video in capcut (color correction / add music / adjust speed / noise reduction)

Here is the final result.

FAQ

Q01: The More Keyframes, The Better?

  • Q: The More Keyframes, The Better?
  • A: No, put too many keyframes make video more frame freeze, so when i generate a 11s video, i usually use 5 images or 2 images, just prompts

Q02: Only Generate(preview) a few frames?

  • Q: When generating a video, can we skip rendering the entire video and only generate a few frames for the sake of speeding up the process?
  • A: No, need to render entire video to make, can not preview some frames, so animatediff-cli-prompt-travel may not be suitable for some people who want to preview the effect in advance.

Q03: Why use only controlnet_canny / controlnet_softedge

In my experiment, using Canny and Soft Edge proved to be a highly stable approach. By “stable,” I mean that you can replicate the smooth video results just like I did.

For example, if you only use OpenPose, you may sometimes feel the transitions are strange.

You can try other ControlNet models, but they may not be as stable to produce results, which means that you may need more time to fine-tune the models.

The workflow mentioned above is basic, but it can still be easily followed and used to produce results.

Q04: Do I need to preprocessing keyframes in a1111

No, cli automatically preprocess images. you can turn off in prompts.json

005: why put prompts in .\projects\proj-awpaint-01\

I thought it's a good habit to put all you assets into a project file and copy it to cli's working directory.

You can also use those link bellow to automate part of your workflow

bat file link: https://gist.github.com/hylarucoder/8bf47a351698a2834a5f0e5bf1eed102

Conclusion

There might be various adaptation issues during the actual operation, such as hardware/prompts/the model is not suitable for animatediff/ad may not be as controllable as you think. Following this workflow can help everyone avoid some weird pitfalls and create an animation with a certain aesthetic sense.

If you found this post helpful, you can follow me on @simpleaiart or share the article with your friends.