zsxkib/animatediff-prompt-travel π’βπ β πΌοΈ
About
π¨AnimateDiff Prompt Travelπ§ Seamlessly Navigate and Animate Between Text-to-Image Prompts for Dynamic Visual Narratives
Example Output
Output
Performance Metrics
114.15s
Prediction Time
221.22s
Total Time
All Input Parameters
{
"seed": -1,
"steps": 25,
"width": 256,
"frames": 128,
"height": 384,
"context": 16,
"clip_skip": 2,
"scheduler": "k_dpmpp_sde",
"base_model": "majicmixRealistic_v5Preview",
"prompt_map": "0: ship steadily moving,((waves crashing against the ship:1.0)) | 32: (((lightning strikes))), distant thunder, ship rocked by waves | 64: ship silhouette,(((heavy rain))),wind howling, waves rising higher | 96: ship navigating through the storm, rain easing off",
"head_prompt": "masterpiece, best quality, a haunting and detailed depiction of a ship at sea, battered by waves, ominous,((dark clouds:1.3)),distant lightning, rough seas, rain, silhouette of the ship against the stormy sky",
"tail_prompt": "dark horizon, flashes of lightning illuminating the ship, sailors working hard, ship's lanterns flickering, eerie, mysterious, sails flapping loudly, stormy atmosphere",
"output_format": "mp4",
"guidance_scale": 7.5,
"negative_prompt": "(worst quality, low quality:1.4), black and white, b&w, sunny, clear skies, calm seas, beach, daytime, ((bright colors)), cartoonish, modern ships, sketchy, unfinished, modern buildings, trees, island",
"prompt_fixed_ratio": 0.5,
"custom_base_model_url": "",
"playback_frames_per_second": 8
}
Input Parameters
- seed
- Seed for different images and reproducibility. Use -1 to randomise seed
- steps
- Number of inference steps
- width
- Width of generated video in pixels, must be divisable by 8
- frames
- Length of the video in frames (playback is at 8 fps e.g. 16 frames @ 8 fps is 2 seconds)
- height
- Height of generated video in pixels, must be divisable by 8
- context
- Number of frames to condition on (default: max of <length> or 32). max for motion module v1 is 24
- clip_skip
- Skip the last N-1 layers of the CLIP text encoder (lower values follow prompt more closely)
- scheduler
- Diffusion scheduler
- base_model
- Choose the base model for animation generation. If 'CUSTOM' is selected, provide a custom model URL in the next parameter
- prompt_map
- Prompt for changes in animation. Provide 'frame number : prompt at this frame', separate different prompts with '|'. Make sure the frame number does not exceed the length of video (frames)
- head_prompt
- Primary animation prompt. If a prompt map is provided, this will be prefixed at the start of every individual prompt in the map
- tail_prompt
- Additional prompt that will be appended at the end of the main prompt or individual prompts in the map
- output_format
- Output format of the video. Can be 'mp4' or 'gif'
- guidance_scale
- Guidance Scale. How closely do we want to adhere to the prompt and its contents
- negative_prompt
- prompt_fixed_ratio
- Defines the ratio of adherence to the fixed part of the prompt versus the dynamic part (from prompt map). Value should be between 0 (only dynamic) to 1 (only fixed).
- custom_base_model_url
- Only used when base model is set to 'CUSTOM'. URL of the custom model to download if 'CUSTOM' is selected in the base model. Only downloads from 'https://civitai.com/api/download/models/' are allowed
- playback_frames_per_second
Output Schema
Output
Example Execution Logs
--------------------------------------------------------------------------------
{
"name": "sample",
"path": "share/Stable-diffusion/majicmixRealistic_v5Preview.safetensors",
"motion_module": "models/motion-module/mm_sd_v15_v2.ckpt",
"compile": false,
"seed": [
-1
],
"scheduler": "k_dpmpp_sde",
"steps": 25,
"guidance_scale": 7.5,
"clip_skip": 2,
"prompt_fixed_ratio": 0.5,
"head_prompt": "masterpiece, best quality, a haunting and detailed depiction of a ship at sea, battered by waves, ominous,((dark clouds:1.3)),distant lightning, rough seas, rain, silhouette of the ship against the stormy sky",
"prompt_map": {
"0": "ship steadily moving,((waves crashing against the ship:1.0))", "32": "(((lightning strikes))), distant thunder, ship rocked by waves", "64": "ship silhouette,(((heavy rain))),wind howling, waves rising higher", "96": "ship navigating through the storm, rain easing off"
},
"tail_prompt": "dark horizon, flashes of lightning illuminating the ship, sailors working hard, ship's lanterns flickering, eerie, mysterious, sails flapping loudly, stormy atmosphere",
"n_prompt": [
"(worst quality, low quality:1.4), black and white, b&w, sunny, clear skies, calm seas, beach, daytime, ((bright colors)), cartoonish, modern ships, sketchy, unfinished, modern buildings, trees, island"
],
"output":{
"format" : "mp4",
"fps" : 8,
"encode_param":{
"crf": 10
}
}
}
--------------------------------------------------------------------------------
Running command: animatediff generate -c config/prompts/custom_prompt_travel.json -W 256 -H 384 -L 128 -C 16
14:46:30 INFO Using generation config: cli.py:288
config/prompts/custom_prompt_travel.json
14:46:31 INFO is_v2=True util.py:360
INFO Using base model: runwayml/stable-diffusion-v1-5 cli.py:306
INFO Will save outputs to cli.py:314
./output/2023-10-19T14-46-31-sample-majicmixrealist
ic_v5preview
INFO Checking motion module... generate.py:321
INFO Loading tokenizer... generate.py:335
INFO Loading text encoder... generate.py:337
14:46:32 INFO Loading VAE... generate.py:339
INFO Loading UNet... generate.py:341
14:46:41 INFO Loaded 453.20928M-parameter motion module unet.py:578
INFO Using scheduler "k_dpmpp_sde" generate.py:353
(DPMSolverSinglestepScheduler)
INFO Loading weights from generate.py:358
/src/data/share/Stable-diffusion/majicmixReali
stic_v5Preview.safetensors
14:46:45 INFO Merging weights into UNet... generate.py:375
INFO Creating AnimationPipeline... generate.py:421
INFO No TI embeddings found ti.py:102
INFO Sending pipeline to device "cuda" pipeline.py:22
14:46:46 INFO Selected data types: unet_dtype=torch.float16, device.py:90
tenc_dtype=torch.float16,
vae_dtype=torch.bfloat16
INFO Using channels_last memory format for UNet and device.py:111
VAE
14:46:48 INFO Saving prompt config to output directory cli.py:367
INFO Initialization complete! cli.py:375
INFO Generating 1 animations cli.py:376
INFO Running generation 1 of 1 cli.py:386
INFO Generation seed: 2091610190647688252 cli.py:392
100% ββββββββββββββββββββββββββββββββββ 275/275 [ 0:01:17 < 0:00:00 , 4 it/s ]
14:48:10 INFO Generation complete, saving... generate.py:846
Saving frames to 00-2091610190647688252 98% βββββΈ 126/β¦ [ 0:0β¦ < 0:00β¦ , 26 ]
it/s
14:48:15 INFO Creating ffmpeg encoder... generate.py:773
INFO Encoding interpolated frames with ffmpeg... generate.py:783
14:48:17 INFO Saved sample to generate.py:857
output/2023-10-19T14-46-31-sample-majicmixreal
istic_v5preview/00_2091610190647688252_masterp
iece_best-quality_a-haunting-and-detailed-d
INFO Generation complete! cli.py:451
INFO Done, exiting... cli.py:457
Error: Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Downloading (β¦)lve/main/config.json: 0%| | 0.00/4.52k [00:00<?, ?B/s]
Downloading (β¦)lve/main/config.json: 100%|ββββββββββ| 4.52k/4.52k [00:00<00:00, 12.7MB/s]
Downloading (β¦)okenizer_config.json: 0%| | 0.00/905 [00:00<?, ?B/s]
Downloading (β¦)okenizer_config.json: 100%|ββββββββββ| 905/905 [00:00<00:00, 5.39MB/s]
Downloading (β¦)olve/main/vocab.json: 0%| | 0.00/961k [00:00<?, ?B/s]
Downloading (β¦)olve/main/vocab.json: 100%|ββββββββββ| 961k/961k [00:00<00:00, 4.15MB/s]
Downloading (β¦)olve/main/vocab.json: 100%|ββββββββββ| 961k/961k [00:00<00:00, 4.13MB/s]
Downloading (β¦)olve/main/merges.txt: 0%| | 0.00/525k [00:00<?, ?B/s]
Downloading (β¦)olve/main/merges.txt: 100%|ββββββββββ| 525k/525k [00:00<00:00, 10.1MB/s]
Downloading (β¦)cial_tokens_map.json: 0%| | 0.00/389 [00:00<?, ?B/s]
Downloading (β¦)cial_tokens_map.json: 100%|ββββββββββ| 389/389 [00:00<00:00, 2.47MB/s]
Downloading (β¦)/main/tokenizer.json: 0%| | 0.00/2.22M [00:00<?, ?B/s]
Downloading (β¦)/main/tokenizer.json: 100%|ββββββββββ| 2.22M/2.22M [00:00<00:00, 24.5MB/s]
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Input #0, image2, from '/src/output/2023-10-19T14-46-31-sample-majicmixrealistic_v5preview/00-2091610190647688252/%08d.png':
Duration: 00:00:16.00, start: 0.000000, bitrate: N/A
Stream #0:0: Video: png, rgb24(pc), 256x384, 8 fps, 8 tbr, 8 tbn, 8 tbc
Stream mapping:
Stream #0:0 (png) -> fps
fps -> Stream #0:0 (libx265)
Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
x265 [info]: build info [Linux][GCC 11.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-2 (Main tier)
x265 [info]: Thread pool 0 using 64 threads on numa nodes 0,1,2,3,4,5,6,7
x265 [info]: Thread pool 1 using 32 threads on numa nodes 0,1,2,3,4,5,6,7
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 5 / wpp(6 rows)
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
x265 [warning]: Source height < 720p; disabling lookahead-slices
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias : 8 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 20 / 6 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 0.4 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-10.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=0.40 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing deblock(tC=1:B=1) sao
Output #0, mp4, to '/src/output/2023-10-19T14-46-31-sample-majicmixrealistic_v5preview/00_2091610190647688252_masterpiece_best-quality_a-haunting-and-detailed-d.mp4':
Metadata:
encoder : Lavf58.76.100
Stream #0:0: Video: hevc (hev1 / 0x31766568), yuv420p(tv, progressive), 256x384, q=2-31, 8 fps, 16384 tbn
Metadata:
encoder : Lavc58.134.100 libx265
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame= 1 fps=0.0 q=0.0 size= 0kB time=00:00:00.00 bitrate=N/A speed= 0x
frame= 65 fps=0.0 q=12.7 size= 256kB time=00:00:03.62 bitrate= 578.6kbits/s speed=6.88x
frame= 108 fps=103 q=12.4 size= 512kB time=00:00:09.00 bitrate= 466.1kbits/s speed=8.62x
frame= 128 fps= 79 q=12.4 Lsize= 1206kB time=00:00:15.62 bitrate= 632.0kbits/s speed= 9.7x
video:1201kB audio:0kB subtitle:0kB other streams:0kB global headers:2kB muxing overhead: 0.384719%
x265 [info]: frame I: 2, Avg QP:9.70 kb/s: 1038.56
x265 [info]: frame P: 26, Avg QP:9.67 kb/s: 821.73
x265 [info]: frame B: 100, Avg QP:12.11 kb/s: 552.28
x265 [info]: Weighted P-Frames: Y:7.7% UV:7.7%
x265 [info]: consecutive B-frames: 7.1% 0.0% 7.1% 35.7% 17.9% 28.6% 3.6%
encoded 128 frames in 1.56s (81.93 fps), 614.61 kb/s, Avg QP:11.58
Identifying the GIF path from the generated outputs...
Identified directory: output/2023-10-19T14-46-31-sample-majicmixrealistic_v5preview
Identified Media Path: output/2023-10-19T14-46-31-sample-majicmixrealistic_v5preview/00_2091610190647688252_masterpiece_best-quality_a-haunting-and-detailed-d.mp4
Version Details
- Version ID
1b8a8f2725c03b1ff4a0b960079899131c384149d1feba45e4ef43653deb3b5f- Version Created
- October 19, 2023