OmniParser v2.0 for Pure Vision Based GUI Agents
Turn any UI screenshot into structured, LLM-readable elements: interactive icons with numbered boxes, their functional captions, and the text boxes found on screen.
Detection is a fine-tuned YOLOv8 (icon_detect), captioning is a fine-tuned
Florence-2-base (icon_caption) run with HF Transformers, and text comes from EasyOCR.
This is the official microsoft/OmniParser-v2.0
pipeline, and the merge logic matches the original util/utils.py exactly.
0.01 1
0.01 1
640 1920
640 3200
Examples
| Screenshot | Box Threshold | IOU Threshold | Icon Detect Image Size | Describe each icon (Florence-2) | Max image side |
|---|
Weights: icon_detect (YOLOv8, AGPL) and icon_caption (Florence-2, MIT) from microsoft/OmniParser-v2.0. Code MIT. Runs on ZeroGPU.