Understanding, Anthropocentrically Defined: What the Competence-Performance Gap in Machine Reasoning Reveals about a Contested Concept

Contemporary epistemology has increasingly argued that understanding, not knowledge, is the more demanding and more central epistemic achievement. Where knowledge concerns individual true beliefs, understanding requires grasping how the elements of a domain depend on and relate to one another (Grimm, 2012; Elgin, 2017; Zagzebski, 2001). This paper uses the discovery of unexplored reasoning routes in artificial systems as a case study for testing what this distinction means once the class of possible "graspers" is no longer confined to human minds.

We introduce the idea of a reasoning space: the set of possible trajectories of hypotheses, abstractions, and actions available when addressing a problem. A human solver explores only a limited portion of this space, shaped by inductive biases, perceptual habits, and recurring patterns of premature commitment to a hypothesis. A machine reasoning system explores its own, generally different, portion of the same space. We characterise human ignorance not as a lack of information but as a structured region of this space that human biases make systematically unlikely to be explored, even when all the facts needed to reach a solution are already available. Using the Abstraction and Reasoning Corpus (ARC) as a controlled testbed for comparing human and machine reasoning, we ask what it would take for a system that reaches into this region to be said to understand something, rather than merely to perform well.

We argue that reaching a single solution to a problem is structurally analogous to acquiring a piece of knowledge, while grasping the overall shape of the reasoning space, including where and why human exploration stops short, is analogous to understanding in the fuller, philosophically demanding sense. On this reading, superior performance is not evidence of understanding: a system can be superhuman on a shared task through brute-force search or specialised heuristics while grasping nothing about why humans fail where they fail. This is a structural cousin of Searle's Chinese Room worry (1980), applied not to symbol manipulation but to the inference from successful exploration of a reasoning space to genuine understanding of that space.

We propose that a stronger, second-order capacity is a better candidate for understanding: not merely reaching a solution humans miss, but predicting, explaining, and reasoning about why humans miss it. This mirrors what psychologists call metacognition (Flavell, 1979). We further argue, drawing on Nagel's (1974) discussion of the limits of understanding a perspective from outside, that even a system satisfying this criterion poses a residual problem for human evaluators, who may be unable to verify from their own cognitive vantage point whether a machine's success reflects grasped structure or merely opaque, successful computation.

We conclude that the philosophically interesting question raised by increasingly capable AI systems is not whether they can solve problems humans cannot, but whether the concept of understanding itself, developed with only human and biological cognisers in view, needs to be reconceived once it is asked to do work outside that domain.