zsxkib/animate-diff 🔢📝❓ → 🖼️

▶️ 58.8K runs 📅 Sep 2023 ⚙️ Cog 0.8.6 🔗 GitHub 📄 Paper ⚖️ License

About

🎨 AnimateDiff (w/ MotionLoRAs for Panning, Zooming, etc): Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

Example Output

Prompt:

"a medium shot of a vibrant coral reef with a variety of marine life, rainbow, visual effects, prores, cineon, royal, monumental, hyperrealistic, exceptional, visually stunning"

Output

Performance Metrics

107.36s Prediction Time

107.35s Total Time

All Input Parameters

{
  "seed": -1,
  "steps": 25,
  "width": 768,
  "frames": 16,
  "height": 512,
  "prompt": "a medium shot of a vibrant coral reef with a variety of marine life, rainbow, visual effects, prores, cineon, royal, monumental, hyperrealistic, exceptional, visually stunning",
  "base_model": "realisticVisionV20_v20",
  "guidance_scale": 7.5,
  "negative_prompt": "",
  "pan_up_motion_strength": 0,
  "zoom_in_motion_strength": 0,
  "pan_down_motion_strength": 0,
  "pan_left_motion_strength": 0,
  "zoom_out_motion_strength": 0,
  "pan_right_motion_strength": 0.75,
  "rolling_clockwise_motion_strength": 0,
  "rolling_anticlockwise_motion_strength": 0
}

Input Parameters

seed Type: integerDefault: -1: Seed for different images and reproducibility. Use -1 to randomise seed
steps Type: integerDefault: 25Range: 1 - 100: Number of inference steps
width Type: integerDefault: 512: Width in pixels
frames Type: integerDefault: 16Range: 1 - 32: Length of the video in frames (playback is at 8 fps e.g. 16 frames @ 8 fps is 2 seconds)
height Type: integerDefault: 512: Height in pixels
prompt Type: stringDefault: photo of vocano, rocks, storm weather, wind, lava waves, lightning, 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3
base_model Default: realisticVisionV20_v20: Select a base model (DreamBooth checkpoint)
output_format Default: mp4: Output format of the video. Can be 'mp4' or 'gif'
guidance_scale Type: numberDefault: 7.5Range: 0 - 20: Guidance Scale. How closely do we want to adhere to the prompt and its contents
negative_prompt Type: stringDefault: blur, haze, deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, mutated hands and fingers, deformed, distorted, disfigured, poorly drawn, bad anatomy, wrong anatomy, extra limb, missing limb, floating limbs, disconnected limbs, mutation, mutated, ugly, disgusting, amputation
pan_up_motion_strength Type: numberDefault: 0Range: 0 - 1: Strength of Pan Up Motion LoRA. 0 disables the LoRA
zoom_in_motion_strength Type: numberDefault: 0Range: 0 - 1: Strength of Zoom In Motion LoRA. 0 disables the LoRA
pan_down_motion_strength Type: numberDefault: 0Range: 0 - 1: Strength of Pan Down Motion LoRA. 0 disables the LoRA
pan_left_motion_strength Type: numberDefault: 0Range: 0 - 1: Strength of Pan Left Motion LoRA. 0 disables the LoRA
zoom_out_motion_strength Type: numberDefault: 0Range: 0 - 1: Strength of Zoom Out Motion LoRA. 0 disables the LoRA
pan_right_motion_strength Type: numberDefault: 0Range: 0 - 1: Strength of Pan Right Motion LoRA. 0 disables the LoRA
rolling_clockwise_motion_strength Type: numberDefault: 0Range: 0 - 1: Strength of Rolling Clockwise Motion LoRA. 0 disables the LoRA
rolling_anticlockwise_motion_strength Type: numberDefault: 0Range: 0 - 1: Strength of Rolling Anticlockwise Motion LoRA. 0 disables the LoRA

Output Schema

Output

Type: array • Items Type: string • Items Format: uri

Example Execution Logs

--------------------------------------------------------------------------------
Cog:
inference_config: "configs/inference/inference-v2.yaml"
motion_module:
- "models/Motion_Module/mm_sd_v15_v2.ckpt"
motion_module_lora_configs:
- path:  "models/MotionLoRA/v2_lora_PanRight.ckpt"
alpha: 0.75
dreambooth_path: "models/DreamBooth_LoRA/realisticVisionV20_v20.safetensors"
lora_model_path: ""
seed:           -1
steps:          25
guidance_scale: 7.5
prompt:
- "a medium shot of a vibrant coral reef with a variety of marine life, rainbow, visual effects, prores, cineon, royal, monumental, hyperrealistic, exceptional, visually stunning"
n_prompt:
- ""
--------------------------------------------------------------------------------
current seed: 14398808075592566791
sampling a medium shot of a vibrant coral reef with a variety of marine life, rainbow, visual effects, prores, cineon, royal, monumental, hyperrealistic, exceptional, visually stunning ...
  0%|          | 0/25 [00:00<?, ?it/s]
  4%|▍         | 1/25 [00:04<01:38,  4.11s/it]
  8%|▊         | 2/25 [00:08<01:34,  4.12s/it]
 12%|█▏        | 3/25 [00:12<01:30,  4.12s/it]
 16%|█▌        | 4/25 [00:16<01:26,  4.12s/it]
 20%|██        | 5/25 [00:20<01:22,  4.12s/it]
 24%|██▍       | 6/25 [00:24<01:18,  4.12s/it]
 28%|██▊       | 7/25 [00:28<01:14,  4.12s/it]
 32%|███▏      | 8/25 [00:32<01:10,  4.12s/it]
 36%|███▌      | 9/25 [00:37<01:05,  4.12s/it]
 40%|████      | 10/25 [00:41<01:01,  4.12s/it]
 44%|████▍     | 11/25 [00:45<00:57,  4.13s/it]
 48%|████▊     | 12/25 [00:49<00:53,  4.13s/it]
 52%|█████▏    | 13/25 [00:53<00:49,  4.13s/it]
 56%|█████▌    | 14/25 [00:57<00:45,  4.13s/it]
 60%|██████    | 15/25 [01:01<00:41,  4.13s/it]
 64%|██████▍   | 16/25 [01:06<00:37,  4.14s/it]
 68%|██████▊   | 17/25 [01:10<00:33,  4.14s/it]
 72%|███████▏  | 18/25 [01:14<00:28,  4.14s/it]
 76%|███████▌  | 19/25 [01:18<00:24,  4.14s/it]
 80%|████████  | 20/25 [01:22<00:20,  4.14s/it]
 84%|████████▍ | 21/25 [01:26<00:16,  4.14s/it]
 88%|████████▊ | 22/25 [01:30<00:12,  4.14s/it]
 92%|█████████▏| 23/25 [01:35<00:08,  4.14s/it]
 96%|█████████▌| 24/25 [01:39<00:04,  4.14s/it]
100%|██████████| 25/25 [01:43<00:00,  4.14s/it]
100%|██████████| 25/25 [01:43<00:00,  4.13s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|███▏      | 5/16 [00:00<00:00, 26.35it/s]
 50%|█████     | 8/16 [00:00<00:00, 12.33it/s]
 62%|██████▎   | 10/16 [00:00<00:00, 10.37it/s]
 75%|███████▌  | 12/16 [00:01<00:00,  9.33it/s]
 88%|████████▊ | 14/16 [00:01<00:00,  8.72it/s]
 94%|█████████▍| 15/16 [00:01<00:00,  8.47it/s]
100%|██████████| 16/16 [00:01<00:00,  8.21it/s]
100%|██████████| 16/16 [00:01<00:00,  9.73it/s]
save to samples/99a6f7b49d305094d1391b37d153ab22-2023-10-03T15-47-58/sample/amediumshotofa.mp4

Version Details

Version ID: 269a616c8b0c2bbc12fc15fd51bb202b11e94ff0f7786c026aa905305c4ed9fb
Version Created: October 5, 2023

Run on Replicate →