fofr/video-morpher ❓🔢📝🖼️✓ → 🖼️

▶️ 14.9K runs 📅 Apr 2024 ⚙️ Cog 0.9.5 🔗 GitHub ⚖️ License
image-to-video video-morphing video-style-transfer

About

Generate a video that morphs between subjects, with an optional style

Example Output

Prompt:

"bright, vibrant, high contrast"

Output

Performance Metrics

181.96s Prediction Time
181.96s Total Time
All Input Parameters
{
  "mode": "upscaled-and-interpolated",
  "prompt": "bright, vibrant, high contrast",
  "checkpoint": "3D",
  "style_image": "https://replicate.delivery/pbxt/KnxLVfe6BRRu2zHc3gT99mnwaemKfR4JzaZWxMCLsZYSTKzp/2024-03-05--06-47-29-u-q1-fofr_tropical_purple_beksinski_aaad09f0-d194-4e40-b312-51054fc4ebbf.png",
  "aspect_ratio": "3:4",
  "style_strength": 0.5,
  "use_controlnet": true,
  "negative_prompt": "dark, gloomy",
  "subject_image_1": "https://replicate.delivery/pbxt/KnxLX9qnA82YKCkScCJZf5VCX6hy5RuprGwLEAVjp3vu6Oh1/1.webp",
  "subject_image_2": "https://replicate.delivery/pbxt/KnxLWqPg086DnRTUvxDM7gCBYi3W3coIbn3Q8jXnTJsfvZzt/2.webp",
  "subject_image_3": "https://replicate.delivery/pbxt/KnxLWU6aHjc6kVW2QOF0xC4oen8zuPnlsK2o24GtOV2bJEn8/4.webp",
  "subject_image_4": "https://replicate.delivery/pbxt/KnxLVg54ySFfg8s78YyfzK0Qgse8YdTbSmEZKpnkFkmSFlsf/4.webp"
}
Input Parameters
mode Default: medium
Determines if you produce a quick experimental video or an upscaled interpolated one. (small ~20s, medium ~60s, upscaled ~2min, upscaled-and-interpolated ~4min)
seed Type: integer
Set a seed for reproducibility. Random by default.
prompt Type: stringDefault:
The prompt has a small effect, but most of the video is driven by the subject images
checkpoint Default: realistic
The checkpoint to use for the model
style_image Type: string
Apply the style from this image to the whole video
aspect_ratio Default: 2:3
The aspect ratio of the video
style_strength Type: numberDefault: 1Range: 0 - 2
How strong the style is applied
use_controlnet Type: booleanDefault: true
Use geometric circles to guide the generation
negative_prompt Type: stringDefault:
What you do not want to see in the video
subject_image_1 (required) Type: string
The first subject of the video
subject_image_2 (required) Type: string
The second subject of the video
subject_image_3 (required) Type: string
The third subject of the video
subject_image_4 (required) Type: string
The fourth subject of the video
Output Schema

Output

Type: arrayItems Type: stringItems Format: uri

Example Execution Logs
Random seed set to: 4049182242
Checking inputs
✅ /tmp/inputs/2.png
✅ /tmp/inputs/1.png
✅ /tmp/inputs/3.png
✅ /tmp/inputs/4.png
✅ /tmp/inputs/circles.mp4
✅ /tmp/inputs/style.png
====================================
Checking weights
Including weights for IPAdapter preset: PLUS (high strength)
✅ RealESRGAN_x4.pth
✅ film_net_fp32.pt
✅ ip-adapter-plus_sdxl_vit-h.safetensors
✅ CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors
✅ AnimateLCM_sd15_t2v.ckpt
✅ control_v1p_sd15_qrcode_monster.safetensors
✅ vae-ft-mse-840000-ema-pruned.safetensors
✅ ip-adapter-plus_sd15.safetensors
✅ AnimateLCM_sd15_t2v_lora.safetensors
✅ rcnzCartoon3d_v20.safetensors
====================================
Running workflow
got prompt
Executing node 746, title: Load Video (Upload) 🎥🅥🅗🅢, class type: VHS_LoadVideo
Executing node 461, title: 🔧 Simple Math, class type: SimpleMath+
Executing node 454, title: RepeatImageBatch, class type: RepeatImageBatch
Executing node 458, title: Split Image Batch 🎥🅥🅗🅢, class type: VHS_SplitImages
Executing node 125, title: Apply Advanced ControlNet 🛂🅐🅒🅝, class type: ACN_AdvancedControlNetApply
Executing node 80, title: KSampler, class type: KSampler
[AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (96) greater than context_length 16.
[AnimateDiffEvo] - INFO - Using motion module AnimateLCM_sd15_t2v.ckpt:v2.
Requested to load ControlNet
Loading 1 new model
  0%|          | 0/11 [00:00<?, ?it/s]
  9%|▉         | 1/11 [00:01<00:11,  1.11s/it]
 18%|█▊        | 2/11 [00:02<00:09,  1.11s/it]
 27%|██▋       | 3/11 [00:03<00:09,  1.18s/it]
 36%|███▋      | 4/11 [00:04<00:08,  1.21s/it]
 45%|████▌     | 5/11 [00:05<00:06,  1.14s/it]
 55%|█████▍    | 6/11 [00:06<00:05,  1.11s/it]
 64%|██████▎   | 7/11 [00:07<00:04,  1.08s/it]
 73%|███████▎  | 8/11 [00:09<00:03,  1.13s/it]
 82%|████████▏ | 9/11 [00:10<00:02,  1.12s/it]
 91%|█████████ | 10/11 [00:11<00:01,  1.09s/it]
100%|██████████| 11/11 [00:12<00:00,  1.07s/it]
100%|██████████| 11/11 [00:12<00:00,  1.11s/it]
Executing node 84, title: VAE Decode, class type: VAEDecode
Executing node 53, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine
Executing node 203, title: Upscale Image By, class type: ImageScaleBy
Executing node 204, title: VAE Encode, class type: VAEEncode
Executing node 198, title: KSampler, class type: KSampler
[AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (96) greater than context_length 16.
[AnimateDiffEvo] - INFO - Using motion module AnimateLCM_sd15_t2v.ckpt:v2.
  0%|          | 0/11 [00:00<?, ?it/s]
  9%|▉         | 1/11 [00:02<00:21,  2.20s/it]
 18%|█▊        | 2/11 [00:04<00:19,  2.19s/it]
 27%|██▋       | 3/11 [00:06<00:17,  2.19s/it]
 36%|███▋      | 4/11 [00:08<00:15,  2.19s/it]
 45%|████▌     | 5/11 [00:10<00:13,  2.19s/it]
 55%|█████▍    | 6/11 [00:13<00:10,  2.19s/it]
 64%|██████▎   | 7/11 [00:15<00:08,  2.19s/it]
 73%|███████▎  | 8/11 [00:17<00:06,  2.19s/it]
 82%|████████▏ | 9/11 [00:19<00:04,  2.18s/it]
 91%|█████████ | 10/11 [00:21<00:02,  2.18s/it]
100%|██████████| 11/11 [00:24<00:00,  2.18s/it]
100%|██████████| 11/11 [00:24<00:00,  2.19s/it]
Requested to load AutoencoderKL
Loading 1 new model
Executing node 201, title: VAE Decode, class type: VAEDecode
Executing node 205, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine
Executing node 271, title: Upscale Image (using Model), class type: ImageUpscaleWithModel
Executing node 279, title: Upscale Image, class type: ImageScale
Executing node 272, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine
Executing node 770, title: FILM VFI, class type: FILM VFI
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Final clearing cache...
Comfy-VFI: Done cache clearing
Executing node 219, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine
Prompt executed in 176.75 seconds
outputs:  {'53': {'gifs': [{'filename': 'preview_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '205': {'gifs': [{'filename': 'upscaled_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '272': {'gifs': [{'filename': 'upscaled_model_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '219': {'gifs': [{'filename': 'interpolated_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}}
====================================
preview_00001.mp4
upscaled_00001.mp4
upscaled_model_00001.mp4
interpolated_00001.mp4
Version Details
Version ID
e70e975067d2b5dbe9e2d9022833d27230a1bdeb3f4af6fe6bb49a548a3039a7
Version Created
April 24, 2024
Run on Replicate →