andreasjansson/flash-eval 📝🖼️ → ❓

▶️ 1.9K runs 📅 Oct 2024 ⚙️ Cog 0.9.21 🔗 GitHub 📄 Paper

image-evaluation image-quality-assessment prompt-alignment

About

Suite of models to evaluate the image quality of text-to-image models with respect to their input prompts.

Example Output

Output

{"a cat":{"https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png":{"BLIP":0.4326244592666626,"CLIP":0.2218017578125,"Aesthetic":6.099793434143066,"PickScore":21.34879493713379,"ImageReward":0.3194262385368347},"https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp":{"BLIP":0.35405826568603516,"CLIP":0.245849609375,"Aesthetic":5.96907901763916,"PickScore":21.581308364868164,"ImageReward":0.637330174446106}},"a dog":{"https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png":{"BLIP":0.35565632581710815,"CLIP":0.181884765625,"Aesthetic":5.137001991271973,"PickScore":19.575040817260742,"ImageReward":-0.915699303150177},"https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp":{"BLIP":0.399705708026886,"CLIP":0.2403564453125,"Aesthetic":6.546636581420898,"PickScore":21.414165496826172,"ImageReward":1.6094799041748047}}}

Performance Metrics

2.67s Prediction Time

144.08s Total Time

All Input Parameters

{
  "models": "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
  "prompts_and_images": "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png"
}

Input Parameters

models Type: stringDefault: ImageReward,Aesthetic,CLIP,BLIP,PickScore: Comma-separated list of models to use to evaluate. Valid models are ImageReward, Aesthetic, CLIP, BLIP, PickScore
image_separator Type: stringDefault: ,: Separator between image URLs
prompts_and_images (required) Type: string: Newline-separated list of prompt/image URL pairs. Prompt/image URL pairs are formatted as <prompt>:<image1>[,<image2>[,<image3>[,...]]], i.e. a prompt followed by a colon followed by one or more comma-separated image URLs.
prompt_images_separator Type: stringDefault: :: Separator between prompt and list of images

Output Schema

Output

Type: array • Items Type: object

Example Execution Logs

"a cat"
https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp
ImageReward: 0.6373  (took 0.256sec)
Aesthetic : 5.9691  (took 0.193sec)
CLIP      : 0.2458  (took 0.109sec)
BLIP      : 0.3541  (took 0.098sec)
PickScore : 21.5813  (took 0.0812sec)
https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png
ImageReward: 0.3194  (took 0.109sec)
Aesthetic : 6.0998  (took 0.0923sec)
CLIP      : 0.2218  (took 0.0985sec)
BLIP      : 0.4326  (took 0.103sec)
PickScore : 21.3488  (took 0.0753sec)
"a dog"
https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp
ImageReward: 1.6095  (took 0.105sec)
Aesthetic : 6.5466  (took 0.0752sec)
CLIP      : 0.2404  (took 0.0403sec)
BLIP      : 0.3997  (took 0.0446sec)
PickScore : 21.4142  (took 0.0654sec)
https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png
ImageReward: -0.9157  (took 0.0527sec)
Aesthetic : 5.1370  (took 0.0362sec)
CLIP      : 0.1819  (took 0.043sec)
BLIP      : 0.3557  (took 0.0467sec)
PickScore : 19.5750  (took 0.0682sec)

Version Details

Version ID: ef9f9879404379fd05006e888a6bddcab189914c1045b77cbabc565735164ce9
Version Created: October 14, 2024

Run on Replicate →