camenduru/streaming-t2v 🔢📝✓ → 🖼️

▶️ 4.0K runs 📅 Apr 2024 ⚙️ Cog 0.9.4 🔗 GitHub 📄 Paper ⚖️ License

dynamic-video long-video text-to-video video-enhancement video-generation

About

StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text

Example Output

Prompt:

"Experience the dance of jellyfish: float through mesmerizing swarms of jellyfish, pulsating with otherworldly grace and beauty."

Output

Performance Metrics

561.64s Prediction Time

561.65s Total Time

All Input Parameters

{
  "seed": 33,
  "chunk": 24,
  "prompt": "Experience the dance of jellyfish: float through mesmerizing swarms of jellyfish, pulsating with otherworldly grace and beauty.",
  "enhance": true,
  "overlap": 8,
  "num_steps": 50,
  "num_frames": 120,
  "image_guidance": 9,
  "negative_prompt": ""
}

Input Parameters

seed Type: integerDefault: 33
chunk Type: integerDefault: 24
prompt Type: stringDefault: A cat running on the street
enhance Type: booleanDefault: false
overlap Type: integerDefault: 8
num_steps Type: integerDefault: 50
num_frames Type: integerDefault: 24
image_guidance Type: integerDefault: 9
negative_prompt Type: stringDefault:

Output Schema

Output

Type: array • Items Type: string • Items Format: uri

Example Execution Logs

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/usr/local/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:442: PossibleUserWarning: The dataloader, predict_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 48 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
rank_zero_warn(
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
INFERENCE PARAMS = {'concat_video': True, 'conditioning_from_all_past': False, 'conditioning_type': 'fixed', 'eta': 1.0, 'eval_loss_metrics': False, 'frame_rate': 8, 'guidance_scale': 7.5, 'height': 256, 'mode': 'long_video', 'n_autoregressive_generations': 4, 'negative_prompt': '', 'num_inference_steps': 50, 'result_formats': ['eval_mp4'], 'scheduler_cls': '', 'seed': 33, 'start_from_real_input': False, 'use_dec_scaling': True, 'validation_samples': 80, 'video_length': 16, 'width': 256}
Predicting ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 0:06:16 • 0:00:00 0.00it/s
/usr/local/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py:608: UserWarning: Should have tb<=t1 but got tb=4.164773464202881 and t1=4.164773.
warnings.warn(f"Should have {tb_name}<=t1 but got {tb_name}={tb} and t1={self._end}.")

Version Details

Version ID: 1fe245aad4bb7f209074a231142ac3eceb3b1f2adc9cf77b46e8ffa2662323cf
Version Created: April 10, 2024

Run on Replicate →