Problem-Solving Behaviour in Reasoning Tests: Impact of Reasoning Ability, Item Difficulty and Item-Position.

Performance on reasoning tests is a valid predictor for future success, yet what creates individual differences in problem-solving behaviour remains unclear. A systematic investigation of the problem matrix with fewer switches to and from response alternatives is linked to higher reasoning ability (e.g., Vigneau et al., 2006). Further evidence (Gonthier & Roulin, 2020) suggests that with increasing item difficulty, individuals switch more often. An analysis based on three eye-tracking metrics (von Gugelberg & Troche, 2025) indicated different direction of effects and a possible impact of item position. The present study examines the effects of reasoning ability, item difficulty, and item position on three eye-tracking metrics commonly used to assess problem-solving behaviour in reasoning tests: toggle rate (TR), proportional time on the problem matrix (PM), and proportional time to first fixation on response alternatives (PR). Complete data of 301 participants solving two different reasoning tests was obtained. For both tests a 30-minute time limit was set; this is standard procedure for the Figural Matrices (Kyllonen et al., 2019), but not the Advanced Raven Progressive Matrices (Raven et al., 1998). Correlations between ability scores and eye-tracking metrics in both tests align with other studies (e.g., Rivollier et al., 2021). Mixed-effects models revealed varying patterns of results across the eye-tracking metrics and the different reasoning tests. It seems likely that the different eye-tracking metrics capture distinct aspects of problem-solving behaviour. While TR and PM summarize behaviour over a whole item, PR only assesses one specific behaviour (i.e., when, during the problem-solving behaviour, one looks at the response alternatives for the first time). Interestingly, for all eye-tracking metrics and both reasoning tests, the effect of item position varied greatly across individuals (random slope) when added as a random effect, whereas item difficulty did not. Different possible explanations for this phenomenon are discussed.