microsoft/omniparser-v2 🖼️🔢 → ❓

▶️ 182.0K runs 📅 Feb 2025 ⚙️ Cog 0.13.7 🔗 GitHub 📄 Paper ⚖️ License

image-object-detection image-to-text ocr ui-grounding

About

OmniParser is a screen parsing tool to convert general GUI screen to structured elements.

Example Output

Output

{"img":"https://replicate.delivery/czjl/RWs9fb8R8eu2uk9jzo5kY7kdWo9s04hNRxS9SAfOexZv0KDRB/output.png","elements":"icon 0: {'type': 'text', 'bbox': [0.31953126192092896, 0.10987482964992523, 0.41015625, 0.13212795555591583], 'interactivity': False, 'content': 'Type here to search'}
icon 1: {'type': 'text', 'bbox': [0.3101562559604645, 0.19332405924797058, 0.3460937440395355, 0.21279555559158325], 'interactivity': False, 'content': 'Pinned'}
icon 2: {'type': 'text', 'bbox': [0.6585937738418579, 0.2990264296531677, 0.694531261920929, 0.3184979259967804], 'interactivity': False, 'content': 'Settings'}
icon 3: {'type': 'text', 'bbox': [0.38359373807907104, 0.4075104296207428, 0.40703123807907104, 0.42698192596435547], 'interactivity': False, 'content': 'Xbox'}
icon 4: {'type': 'text', 'bbox': [0.34453123807907104, 0.7413073778152466, 0.42109376192092896, 0.7607788443565369], 'interactivity': False, 'content': 'Screenshot (9)-png'}
icon 5: {'type': 'text', 'bbox': [0.34296876192092896, 0.7635604739189148, 0.3804687559604645, 0.7830320000648499], 'interactivity': False, 'content': 'Monday'}
icon 6: {'type': 'icon', 'bbox': [0.4970785975456238, 0.23429809510707855, 0.5733658671379089, 0.34724709391593933], 'interactivity': True, 'content': 'Microsoft '}
icon 7: {'type': 'icon', 'bbox': [0.5727160573005676, 0.4605804979801178, 0.6388594508171082, 0.5722821354866028], 'interactivity': True, 'content': 'Photoshop Express '}
icon 8: {'type': 'icon', 'bbox': [0.2868574261665344, 0.22867657244205475, 0.36240988969802856, 0.3407636880874634], 'interactivity': True, 'content': 'Microsoft Edge '}
icon 9: {'type': 'icon', 'bbox': [0.4359043538570404, 0.23435497283935547, 0.4971795678138733, 0.3499503433704376], 'interactivity': True, 'content': 'Calendar '}
icon 10: {'type': 'icon', 'bbox': [0.43351706862449646, 0.3497304320335388, 0.49976786971092224, 0.44185498356819153], 'interactivity': True, 'content': 'Solitaire '}
icon 11: {'type': 'icon', 'bbox': [0.6395878195762634, 0.4572511911392212, 0.7104716897010803, 0.5708341598510742], 'interactivity': True, 'content': 'Calculator '}
icon 12: {'type': 'icon', 'bbox': [0.572555661201477, 0.34660184383392334, 0.6379275321960449, 0.46141958236694336], 'interactivity': True, 'content': 'Netflix '}
icon 13: {'type': 'icon', 'bbox': [0.49764174222946167, 0.3473391830921173, 0.5722932815551758, 0.4547758400440216], 'interactivity': True, 'content': 'Spotify '}
icon 14: {'type': 'icon', 'bbox': [0.42866915464401245, 0.44782811403274536, 0.5009313821792603, 0.5495033264160156], 'interactivity': True, 'content': 'Twitter '}
icon 15: {'type': 'icon', 'bbox': [0.28697189688682556, 0.34129762649536133, 0.3610140383243561, 0.44780731201171875], 'interactivity': True, 'content': 'Office '}
icon 16: {'type': 'icon', 'bbox': [0.6275193095207214, 0.18070241808891296, 0.6904729604721069, 0.22184278070926666], 'interactivity': True, 'content': 'apps '}
icon 17: {'type': 'icon', 'bbox': [0.5780068635940552, 0.23290468752384186, 0.633935272693634, 0.3376893103122711], 'interactivity': True, 'content': 'Photos '}
icon 18: {'type': 'icon', 'bbox': [0.30243855714797974, 0.6528392434120178, 0.5010311007499695, 0.7274318337440491], 'interactivity': True, 'content': 'Screenshot (12) png ago '}
icon 19: {'type': 'icon', 'bbox': [2.5594234102754854e-05, 0.004066085442900658, 0.05640237405896187, 0.11211475729942322], 'interactivity': True, 'content': 'Regyyle '}
icon 20: {'type': 'icon', 'bbox': [0.515768826007843, 0.7268991470336914, 0.6963880658149719, 0.8197184205055237], 'interactivity': True, 'content': 'Screenshot (8) png Monday '}
icon 21: {'type': 'icon', 'bbox': [0.5144692659378052, 0.649393618106842, 0.6971572041511536, 0.7273026704788208], 'interactivity': True, 'content': 'Screenshot (I1) png ago '}
icon 22: {'type': 'icon', 'bbox': [0.9251968264579773, 0.9405298829078674, 0.9922046661376953, 0.9946867823600769], 'interactivity': True, 'content': '7/9/2021 '}
icon 23: {'type': 'icon', 'bbox': [0.29774585366249084, 0.840213418006897, 0.41091904044151306, 0.9214367270469666], 'interactivity': True, 'content': 'WinObs '}
icon 24: {'type': 'icon', 'bbox': [0.30453747510910034, 0.5904491543769836, 0.5040181875228882, 0.6522935628890991], 'interactivity': True, 'content': 'Recommended '}
icon 25: {'type': 'icon', 'bbox': [0.5019777417182922, 0.4564504027366638, 0.5722671747207642, 0.5763444304466248], 'interactivity': True, 'content': 'Instagram'}
icon 26: {'type': 'icon', 'bbox': [0.3643844723701477, 0.23224841058254242, 0.4287545084953308, 0.3503813147544861], 'interactivity': True, 'content': 'Mail'}
icon 27: {'type': 'icon', 'bbox': [0.6400586366653442, 0.34977295994758606, 0.7092428803443909, 0.449526309967041], 'interactivity': True, 'content': 'To Do'}
icon 28: {'type': 'icon', 'bbox': [0.35472971200942993, 0.9400750994682312, 0.3884946405887604, 0.9976202845573425], 'interactivity': True, 'content': 'the Windows logo.'}
icon 29: {'type': 'icon', 'bbox': [0.5811794996261597, 0.9405949711799622, 0.6154322028160095, 0.9956714510917664], 'interactivity': True, 'content': 'the number 8.'}
icon 30: {'type': 'icon', 'bbox': [0.4504772126674652, 0.939802885055542, 0.48435717821121216, 0.9979066252708435], 'interactivity': True, 'content': 'a window or screen view.'}
icon 31: {'type': 'icon', 'bbox': [0.5166957974433899, 0.9415040612220764, 0.5487085580825806, 0.9976794719696045], 'interactivity': True, 'content': 'a folder.'}
icon 32: {'type': 'icon', 'bbox': [0.3890049457550049, 0.9408557415008545, 0.41887956857681274, 0.9980627298355103], 'interactivity': True, 'content': 'magnifying glass'}
icon 33: {'type': 'icon', 'bbox': [0.548693060874939, 0.9416469931602478, 0.5787683725357056, 0.9969745874404907], 'interactivity': True, 'content': 'a shopping or online shopping app.'}
icon 34: {'type': 'icon', 'bbox': [0.4852524697780609, 0.9409526586532593, 0.5157744288444519, 0.9979084134101868], 'interactivity': True, 'content': 'the Microsoft Edge browser.'}
icon 35: {'type': 'icon', 'bbox': [0.42019978165626526, 0.9420081377029419, 0.4496763348579407, 0.9979120492935181], 'interactivity': True, 'content': 'unanswerable'}
icon 36: {'type': 'icon', 'bbox': [0.6154235005378723, 0.9408984184265137, 0.6439913511276245, 0.9948576092720032], 'interactivity': True, 'content': 'a shortcut.'}
icon 37: {'type': 'icon', 'bbox': [0.28949251770973206, 0.4559950828552246, 0.3608283996582031, 0.5482454299926758], 'interactivity': True, 'content': 'News'}
icon 38: {'type': 'icon', 'bbox': [0.6377259492874146, 0.6000238060951233, 0.6892112493515015, 0.6368022561073303], 'interactivity': True, 'content': 'More'}
icon 39: {'type': 'icon', 'bbox': [0.8495050668716431, 0.9487622380256653, 0.8694871664047241, 0.9926995635032654], 'interactivity': True, 'content': 'an item or symbol.'}
icon 40: {'type': 'icon', 'bbox': [0.9050769805908203, 0.946520209312439, 0.9253349304199219, 0.9947614073753357], 'interactivity': True, 'content': 'the power button.'}
icon 41: {'type': 'icon', 'bbox': [0.8878645896911621, 0.9468819499015808, 0.9053664207458496, 0.9935903549194336], 'interactivity': True, 'content': 'Refresh or reload.'}
icon 42: {'type': 'icon', 'bbox': [0.8692238926887512, 0.9474367499351501, 0.8877871632575989, 0.992974579334259], 'interactivity': True, 'content': 'a loading or buffering indicator.'}
icon 43: {'type': 'icon', 'bbox': [0.8311622738838196, 0.9501104950904846, 0.8494259119033813, 0.9917120337486267], 'interactivity': True, 'content': 'unanswerable'}
icon 44: {'type': 'icon', 'bbox': [0.295014888048172, 0.0986727923154831, 0.3146948516368866, 0.13822725415229797], 'interactivity': True, 'content': 'Receive'}
icon 45: {'type': 'icon', 'bbox': [0.7137465476989746, 0.3793989419937134, 0.7312657237052917, 0.4248334765434265], 'interactivity': True, 'content': 'unanswerable'}
icon 46: {'type': 'icon', 'bbox': [0.2667734622955322, 0.35587266087532043, 0.28785228729248047, 0.384502649307251], 'interactivity': True, 'content': 'unanswerable'}
icon 47: {'type': 'icon', 'bbox': [0.6678226590156555, 0.8546284437179565, 0.6951290965080261, 0.9153355956077576], 'interactivity': True, 'content': 'Refresh or reload.'}
icon 48: {'type': 'icon', 'bbox': [0.3080870807170868, 0.7331666946411133, 0.3407413363456726, 0.7866833806037903], 'interactivity': True, 'content': 'a bookmark.'}
icon 49: {'type': 'icon', 'bbox': [0.6955127716064453, 0.2683064639568329, 0.711844265460968, 0.29865121841430664], 'interactivity': True, 'content': 'unanswerable'}
icon 50: {'type': 'icon', 'bbox': [0.2671732008457184, 0.512174665927887, 0.2871202528476715, 0.5422586798667908], 'interactivity': True, 'content': 'unanswerable'}
icon 51: {'type': 'icon', 'bbox': [0.9931192398071289, 0.9415164589881897, 1.0, 0.9950076937675476], 'interactivity': True, 'content': 'unanswerable'}
icon 52: {'type': 'icon', 'bbox': [0.35741090774536133, 0.43543684482574463, 0.4305238723754883, 0.46342185139656067], 'interactivity': True, 'content': 'unanswerable'}
icon 53: {'type': 'icon', 'bbox': [0.6449296474456787, 0.9434605836868286, 0.6626533269882202, 0.9855495095252991], 'interactivity': True, 'content': 'unanswerable'}"}

Performance Metrics

3.87s Prediction Time

3.89s Total Time

All Input Parameters

{
  "image": "https://replicate.delivery/pbxt/MWb5phPoK0NtfxdKRdd7QkbnvAwJpWAeO7xqOZtrvY5Ned18/win11.jpeg",
  "imgsz": 640,
  "box_threshold": 0.05,
  "iou_threshold": 0.1
}

Input Parameters

image (required) Type: string: Input image to process
imgsz Type: integerDefault: 640Range: 640 - 1920: Icon detection image size
box_threshold Type: numberDefault: 0.05Range: 0.01 - 1: Threshold for removing bounding boxes with low confidence
iou_threshold Type: numberDefault: 0.1Range: 0.01 - 1: Threshold for removing bounding boxes with large overlap

Output Schema

img Type: stringFormat: uri: Img
elements Type: string: Elements

Example Execution Logs

0: 736x1280 53 icons, 43.1ms
Speed: 4.4ms preprocess, 43.1ms inference, 3.3ms postprocess per image at shape (1, 3, 736, 1280)
len(filtered_boxes): 54 25
time to get parsed content: 2.766561269760132

Version Details

Version ID: 49cf3d41b8d3aca1360514e83be4c97131ce8f0d99abfc365526d8384caa88df
Version Created: February 19, 2025

Run on Replicate →