Short abstract

The phenomenal progress of generative AI in recent years, and Large Language Models in particular, reignites discussions of the similarities and differences between human and machine intelligence, with, at their apex, the question of whether such systems are, or have the potential to become, conscious agents. This article approaches these questions from the viewpoint of the overarching explanation for biological and technological information systems provided by Emergent Information Theory. Particular focus is given to the significance of language as a modelling system for internal information storage and processing, and to language-based information transfer between intelligent entities. This approach illuminates the strong ontogenetic and ontological convergence between human intelligence and this new form of AI, but also their remaining and fundamental differences. With respect to the ultimate philosophical question, the conclusion drawn is that while such systems may support consciousness-like phenomena, this is not directly comparable to human phenomenal consciousness.

Extended abstract

Philosophical consideration of the possibility of machine consciousness has a long history, as famously embodied in Turing’s “Imitation Game” [1]. In what was at the time a thought experiment, it was proposed that if the textual responses of a machine were to be indistinguishable from a thinking human we may need to acknowledge that the machine is also thinking.

The development of digital technologies since then have progressively transformed these musings into practical considerations. Computers running programs designed and constructed by humans can produce coherent texts, but the way in which these texts are generated justifies their classification as mere deterministic machines. The instruction PRINT “I am a thinking machine” does not make a thinking machine. Even the first generations of artificial neural networks, which developed functions through machine learning rather than design and construction, do not constitute a serious challenge. While they use deep neural networks that bear functional similarities to biological neural networks, supervised learning towards a predetermined task such as text recognition [2] can be considered little more than an alternative method of constructing a machine with a required informational function.

Where things start to become less clear is with unsupervised learning. Such systems are not given a predetermined input--output relationship, but asked to autonomously find patterns in input data: a process more similar to the thought processes of a human faced with a novel situation requiring analysis. From simple beginnings [3]. such systems have progressed to becoming increasingly similar to biological neural networks [4]. However, such experimental models are generally fed with demarcated input data sets such as the MNIST grayscale image [5], and applications are usually still focused on a pre-specified problem such as analysis of medical data on tumor growth [6]. Such systems therefore still seem far from deserving acknowledgement as ‘thinking machines’.

A whole new chapter in this book has been opened by the recent developments in generative AI, and Large Language Models (LLMs) in particular. While still requiring some form of input (generally a human-derived text prompt) to initiate a response, the output generated demonstrates a significant degree of autonomous creativity. The immense scale and diversity of their training sets allows them to produce convincing responses to almost any question imaginable; their linguistic capabilities allow this to be presented in accessible natural language; and their probabilistic nature prevents the kind of deterministic duplication that was previously a hallmark of digital systems.

Returning to Turing’s Imitation Game, these characteristics of modern LLMs bring them considerably closer to being indistinguishable from a human [7], leading to the following question: to what degree is an LLM that speaks the words “I am a thinking machine” more convincing than a computer program written to output this sequence of characters? This question, and its ethical consequences, have seen renewed interest in popular press [8] and academic circles [9]. However, we may ask ourselves whether this is the relevant question to be asking. The different origins and natures of these systems and the human brain means that this can be considered a distraction from the following deeper question: What the nature is of the higher-level informational entities and processes existing in LLMs?

This article will consider the impact of the use of broad-based symbolic language by LLMs on their higher-level information processes from the viewpoint of Emergent Information Theory [10]. The fact that this relatively new theory provides a generic theoretical framework for both biological and technological information-based systems allows for more direct comparison between the two. Specifically, the question will be placed within the context of the long history of philosophical consideration of the relationship between language, thought, and consciousness in humans [11]. To what extent do generic linguistic abilities over a broad knowledge base, coupled with probabilistic response generation, lead to the type of autonomous creation of conceptual content that can justifiably be characterized as thought? Taking a step further, what means do we have at our disposal of confirming or disproving the existence within these systems of the qualia of phenomenal consciousness?

References

[1] A. Turing, “Computing machinery and intelligence,” Mind, vol. LIX, no. 236, p. 433–460, 1950.

[2] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard and L. D. Jackel, “Backpropagation Applied to Handwritten Zip Code Recognition,” Neural Computation., vol. 1, no. 4, p. 541–551, 1989.

[3] T. Kohonen, “Self-organized formation of topologically correct feature maps,” Biological cybernetics, vol. 43, no. 1, pp. 59-69, 1982 .

[4] N. Ravichandran, A. Lansner and P. Herman, “Unsupervised representation learningwith Hebbian synaptic and structural plasticity inbrain-like feedforward neural networks,” Neurocomputing, vol. 626, p. 129440, 2025.

[5] P. Grother and K. Hanaoka, “NIST special database 19. Handprinted forms and characters database,” National Institute of Standards and Technology, vol. 10, no. 69, 1995.

[6] C. Strack, K. Pomykala, H. Schlemmer, J. Egger and J. Kleesiek, ““A net for everyone”: fully personalized and unsupervised neural networks trained with longitudinal data from a single patient,” BMC Medical Imaging, vol. 23, no. 1, p. 174, 2023.

[7] C. Jones and B. Bergen, “Does GPT-4 pass the Turing test?,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024.

[8] D. Milmo, “AI systems could be ‘caused to suffer’ if consciousness achieved, says research,” The Guardian, 3 2 2025. [Online]. Available: https://www.theguardian.com/technology/2025/feb/03/ai-systems-could-be-caused-to-suffer-if-consciousness-achieved-says-research. [Accessed 114 2 2025].

[9] D. Chalmers, “Could a large language model be conscious?,” arXiv preprint, p. 2303.07103, 2023.

[10] D. Boyd, Existing in the Information Dimension: An Introduction to Emergent Information Theory, London: Routledge, 2024.

[11] P. Carruthers, Language, thought and consciousness: An essay in philosophical psychology, Cambridge: Cambridge University Press, 1998