Dataset & Benchmark

In this paper we present WebAgent-90K, web-interaction dataset with around 90K tasks collected via Evol-Instruct and an automated web agent based on GPT-4V, which can be used for training an automated web agent with open-sourced VLM. The Llava-v1.5 we finetuned with WebAgent-90K yielded similar performance as GPT-4V on Webvoyager.

This study introduces HQ-Edit, a high-quality instruction-based image editing dataset with around 200K edits. Unlike prior approaches relying on human feedback, we devise a scalable data collection pipeline leveraging self-instruct with advanced foundation models, namely GPT-4V and DALL-E 3.

This paper presents AQA-Bench, a benchmark for evaluating LLMs’ sequential reasoning abilities via interactive environments requiring model executing algorithms such as Binary search, DFS, and BFS. Our find includes (1) the inverse scaling between model sizes and performance, (2) the nuanced impact of naive in-context examples due to over-fitting in ICL, (3) weak models failing mainly due to incapability of starting well and (4) impressive improvement from a few given predecessor steps following the optimal policy.