This paper presents AQA-Bench, a benchmark for evaluating LLMs’ sequential reasoning abilities via interactive environments requiring model executing algorithms such as Binary search, DFS, and BFS. Our find includes (1) the inverse scaling between model sizes and performance, (2) the nuanced impact of naive in-context examples due to over-fitting in ICL, (3) weak models failing mainly due to incapability of starting well and (4) impressive improvement from a few given predecessor steps following the optimal policy.
