Us and media

For most of human history we developed media with the goal of storing information and transporting it over space and time (^{Lévy, 1997}). All our somatic communication acts, such as talking, miming, pointing at things, showing intents or emotions through facial expressions or body stances, are transient and ultimately ephemeral. We can talk to someone, and they may later remember the gist of the conversation, perhaps even some of its details, but much of it will at best be fuzzy, blurred by our selective, imprecise, and biased memory.

Media allow us to fixate parts of these messages. A somatic communication act is multimodal, immersed in a complex and ever-changing context that affects the messages conveyed and that ultimately also contributes to them (^{Carvalhais, 2023}). By abstracting parts of those messages to a set of signs that allow to articulate and convey them, and by then fixating these in a physical carrier - stone, wood, parchment, paper, etc. - we manage to achieve a stability - or at least an appearance of stability - that we cannot reach without technological assistance.

Our media have been static vessels for the recording of these signs: drawings, paintings, or other images; marks for tallying and counting; texts that encode speech and language. For most of our history, media were not able to capture processes but only descriptions or instructions for those processes. Before time-based media such as sound recording or cinema were developed, we could at best develop notations for music performance, scripts for theatre, or guides or recipes for other processes, but not quite document the processes themselves. Even media such as cinema or sound recording do not, arguably, capture processes, but simply fixate their appearance for a given span of time.

Media have always been interdependent with humans because humans create the signs that are recorded in them, and humans infuse these signs with meaning during interpretation. Media, therefore, are ideally static and inert conveyors of the outputs of the dynamic processes of creation developed by humans, bringing these to interpreters that will on their side develop dynamic hermeneutical processes (^{Caputo, 2018}). Because of this, and for the most part, media have been unable to create signs or information on their own. If, indeed, the medium is the message (^{McLuhan, 1964}), it is not because the media actively build messages but because they shape the space of possibilities within which messages are created. They do this firstly by forcing us to relinquish multimodality and choose with which modality to convey a message: alphabetic text but no sound, images but no movement, sound but no images, sound and images but no smell or temperature, or any of the many other modalities that even hot media such as television or video games force us to abdicate of. Even in this role as static intermediaries, media alter us, they shape and change how we perceive the world, how we communicate, how our brains work and “we emphasize our senses - seeing versus hearing versus touching” (^{Coupland, 2011}, p. 13).

2. What are media after all?

It’s not always clear what we describe with the term media. On one level it can describe the physical or material resources that allow signs to be recorded: the paint and the cave wall or the canvas; the ink, pen, and paper; or any other set of technologies that are used to construct and stabilise a message. But thinking along these lines quickly brings us to questions about, for example, what are the boundaries, if any, between the tools and the media. Are media permanent and the tools involved only more transiently? Or are these tools, transient as they may be, parts of the media forms?

More important even is how this view also has the potential to muddle things once multiple technologies with similar affordances are used. What happens when a manuscript is typeset and printed in multiple copies? And when a recorded piece of music is broadcast on the radio? Or when a movie, originally shown in cinemas, is broadcast on TV, or streamed on a laptop or on a mobile phone? In such cases, are we interfacing with different media? Are we accessing different contents? Are we accessing the same contents through different technologies? We can, of course, discuss remediation (^{Bolter & Grusin, 1999}), to which we will arrive later in this paper, but we first need to better define what we mean by medium.

Henry Jenkins gives us a useful definition that follows Lisa ^{Gitelman’s model (2014}) with a level of technologies that “enable communication” and another of associated “social and cultural practices” developed around the technologies (^{Jenkins, 2006}, pp. 13-14). This is a model of media as cultural systems. Technologies are central to media, but technologies move in and out of the zeitgeist, while media persist “as layers within an […] information and entertainment stratum.” (^Ibid., p. 14) As such, more than being material devices with which we communicate, media shape the culture and ultimately become a part of it. And media are, of course, also technologies themselves. They are cultural technologies for showing and seeing (^{Noë, 2015}), f or perceiving and thinking.

With media we develop traditions that are only tangentially related to technologies, that are developed from technologies but that take a life of their own and are shaped by a much wider set of forces. Media such as the newspaper or the novel are the result of chains of processes that depend on technical influences - such as paper production, ink, typesetting, binding, printing - but also on the influence of other media being incorporated in their production processes. Besides technologies, media are also the fruit of policies, economics, and other social dynamics. As they are too, let’s not forget, the results of evolutionary pressures from other media (^{Fuller, 2005}) in an ever-changing and complex ecology, that is also the ongoing discourse we call culture.

It is in this media space that surrounds us, and where we are immersed, that we need to develop literacies by understanding and negotiating all these traditions, always in a dialectical tension between static carriers of signs and our dynamic and changing interpretations. Media, thus, not only give us their contents - the signs - as they give us the means to organise our cultures, societies, and personal and collective lives. In this sense, they also play a philosophical role (^{Nöe, 2023}).

It is through media that we expand the ways how we perceive the world. It is through media that we try to shape the world by stabilising something that may not, perhaps, be at any time stabilised: information. Any physical support for information is, ultimately, impermanent, because everything decays, changes, rots, or fades away. However, because the temporal scales at which these processes happen are sometimes too different from our own, and in those cases, we may not be able to recognise these decays, thus perceiving media as static, unchanging, permanent, even eternal. This speaks of a cognitive dissonance between the natural dynamics of objects and how we perceive and conceptualise them. And this may also be connected, as many other things in human life, to the constant effort of resisting death. Most likely, though, this can be the result of a drive to develop homeostasis and negentropy, which is part of what fundamentally defines us as living beings. We too fade, decay, rot, forget, and in many ways our lives are an ongoing effort to counter that. But we know that permanence in nature is nothing but an illusion. And that permanence in media, is a construction.

3. Computational media

Computational media seem to be the dominant forms in our current media ecology. This is due to the ubiquity and low-cost of computers and computer networks, but also to their capacity to shape-shift and to absorb most media forms. Before we look into this potential to act as a universal solvent (^{Finn, 2017}), we should delve into the nature of computation and into its history as media.

When digital computers were initially developed, in the late 1930s and throughout the following decades, they were not conceived of as media but rather as tools. Computers were mathematical tools that could substitute those workers whose name they started by appropriating. When developing the principles of modern digital computation, Alan ^{Turing (1936}) did not call these machines “calculators” but rather “automatic computing engines” (ACE), a choice that hints at their conceptual framing at the time.

To operate, ACEs needed to use media, but they were not media themselves. The first medium ACEs used, we can argue, was an infinite tape, divided in discrete cells, proposed by ^{Turing as the source of their “unlimited memory capacity” (1948}). The medium of the tape is central to the operation of the computational machine, but it just a substrate for data storage. The computer itself, or its processes, were not conceived as media (^{Alt, 2011}).

And for the most part, the media for input, output, or storage of data that computers used, were digital but not computational, as the data stored on punch cards, magnetic tapes, or other technologies, could be read by a computer but these technologies could not themselves compute. There was, therefore, a clear demarcation between the computer and its media, a distinction that is still valid in contemporary media forms, as some technologies are unable to compute themselves and are limited to storing data to be computed by other devices.

With the progressive development of interactive computing during the 1960s, we had the first glimpses of computers becoming media forms. Systems such as Sketchpad (^{Sutherland, 1963}), NLS (^{Engelbart, 1962}), or SAGE (^{Petzold, 2000}), had screens where one could type, program, or otherwise control computers in real time. They allowed computations to be developed and affected in real time, and for information to be read and produced as a result of those interactions. Incidentally, with its capacity to control radar systems and aerial defences, SAGE was also one of the early systems with capacity to act in the world.

It was at this point that computers started to become media, or that media became computational. It is here that we find the confluence of data, computation, interface, information, and human. At this point, the medium ceases to be static and becomes a dynamic system. And it is at this point that, in the background of these transformations, the paradox at the heart of computational media starts to be perceivable. It is here that a big shift starts to happen. A shift from media that are static and ideally neutral carriers, to media that are dynamic. In the first of these, signs are created by dynamic somatic processes, the medium itself is static, and the reception and interpretation of signs is once again dynamic. In the latter, when the medium also becomes a dynamic process, we find a context where hypothetically, the medium can also act as the sender or receiver. If the medium can create signs and information on its own, it may very well replace the somatic acts of creation. If the medium can read signs and information on its own - besides the operational images (^{Parikka, 2023}) destined at it as part of its computational processes - then the medium may very well become the receiver and interpreter. And the contexts where communication is developed are not only computational as they are shaped by computation, in a post-digital culture (^{Cramer, 2013}) where we increasingly live in code/spaces where “software and the spatiality of everyday life [are] mutually constituted” (^{Kitchin & Dodge, 2011}, p. 16). Ultimately, the medium may play all the roles in an act of communication and leave humans out entirely (^{Paglen, 2014}).

As computation becomes media, or media become computational, they are expanded by new affordances. As Janet ^{Murray (2012}) pointed out in her early analysis of this phenomenon, media become procedural, and because of this they also become participatory, spatial, and encyclopaedic. They get increasingly autonomous from humans, and detach from them, while at the same time contributing to accelerate a post-human condition, as ^{Parikka would put it, as they address “a whole other sensorium than that of the human being.” (2011}, p. 256) In a McLuhanian sense, this has always been the case. Media extend humans, expand our innate capabilities and accelerate the ongoing process of emergence of humans (^{Lévy, 1997}).

This capacity to become more than ourselves, something else than us, is a central human feature, as we are what we are through culture and society, never through individual realisation alone (^{Dunbar, 2014}). However, technical media, and chief among them computational media, have a way of abstracting their constraints and mechanics, and of leading us to modes of collaboration where although we may be made to feel to be in charge, we are really not, because we are bound to the black box of the medium in a position of subservience and diminished agency in the relationship with it (^{Flusser, 1983}). If efforts to increase literacy among users and producers of media are certainly useful (^{Heyes, 2018}), often it is the very nature of computational media forms that limits our agency, in many cases simply by replacing it, and acting instead of us. This has been an increasingly strong paradigm in the current generation of systems for creative work based on artificial intelligence, but also in critical software such as semi-autonomous vehicles.

Computation is a process, not an object. Computation is the transformation of information, the way how data, bits, information, are changed according to formal, and finite, processes. This is quite unlike what other machines do when they move or transform matter, because although information always needs to be materialised, it is very unlike matter. To start with, because information has the capacity to act on itself, and to transform the machinery that is producing, reading, changing it, in a very rich and unique way.

Computation starts with data, produces data, but it is not data. Computation is not the physical machine, and it is not its code. Rather, it is something else altogether.

4. The nature of computation

From Turing’s concept of the ACE at a theoretical and abstract level, two fundamental features for its role as media immediately become apparent. The first of these is that an ACE is an imitation machine. An ACE computes by developing a specific kind of information processing that combines the acting of the hardware as the substrate for computation, the software as the set of rules that guide the operation, and the data that is fed to it. If the data changes, so does the computation, but otherwise, the ACE always develops the same process. An adding machine will always and only add, a sorting machine will always and only sort, and so on.

But Turing also discovered that it is possible to create a far more versatile machine that can change its behaviour and become another type of machine. The universal computing engine (UCE), as Turing called it - and to which we nowadays most commonly refer to as the universal Turing machine, after Alonzo Church popularised that term - is a device whose rules do not set it up to perform a single type of computation but rather allow it to read the description of another computing machine and to start operating like that machine. What does this mean? An ACE that, e.g., adds numbers can read some inputs and, interpreting these inputs as numerical quantities, add them to produce a result that is, itself, also a numerical quantity. For such an ACE, any input must be a numerical quantity, because its rules preclude it from ever even understanding other types of data. So, should we want a machine that would, e.g., alphabetise a series of inputs instead of adding them, we would need to build an entirely new machine. “Machine” here is understood as the combination of hardware and operational rules. In Turing’s ACEs the hardware is generic (even fungible) and only the operational rules tell the machines apart.

A UCE, on the other hand, can be fed a description of the machine that adds numbers, and will start acting accordingly and adding numbers. If at some point it is fed a description of the alphabetising machine, it will change its behaviour and start alphabetising. If it is fed with any other description, its behaviour will once again change. These descriptions are part of the software layer of the machine, they describe its rules to be, they are its program.

And programs are fed to the machine through the same input channels that are used for all other data (both in Turing’s model of the machine and in the later Von Neuman’s stored-program architecture, that normalised using a shared memory for program and data). Therefore, one could say, a UCE is able to imitate any other computing machine. This assertion is correct, but also incomplete, as what a UCE does, from a computational point of view, goes beyond imitation. As a UCE reads a program and starts operating as the machine that is described, the UCE becomes that machine. A UCE transforms into something indistinguishable from the machine described to it. A term as imitation doesn’t quite describe this, and neither does simulation. A better term to describe this phenomenon is one that comes to us through its adoption by computer science: emulation.

An imitation is some process or object that tries to replicate the superficial features of another object or process. In ^{Baudrillardian terms (1994}) we could perhaps describe it as a counterfeit in their intention to create illusions that pass as reality. A simulation, on the other hand, is a process that aims to replicate a set of features or behaviours of another system with high precision or fidelity but with no regard to verisimilitude of other aspects such as, e.g., using fundamentally different substrates or operational principles. A good example of this can be found when physical models simulate trajectories or dynamics of moving bodies by using only mathematical functions that abstract them but do not replicate their actual physical processes. An emulation, on the other hand, consists of entirely becoming another process. Not trying to approximate it but replicating it so fully that no possible distinction between two processes can be found.

5. Imitating media

This capacity to transform turns the computer - for simplicity’s sake, and because today’s computers are almost without exception universal, we will refer to any sort of UCE as a computer - into a fluid machine that is able to freely shift its behaviour and actions. A machine that can change itself. Paired with its speed, and with various methods to digitise signs, the computer also becomes able to imitate media, and even to, it seems, imitate anything.

During the 1970s, Alan ^{Kay recognised this as he described the computer as a metamedium (2003}) that is able to transform itself into any other medium, including those media that don’t yet exist and those that are not physically possible. The computer becomes ^{Murray’s digital medium (2012}), a singular source from where all media can spring and from where a continuous deluge of new media forms may arise (^{Manovich, 2001}). The computer becomes the unrivalled medium for remediation. If we can trace a history of media based on remediation, as ^{Bolter and Grusin propose (1999}), it is only in computational media that we find what seems to be a universal solvent for media - something that is able to scan, digitise, absorb all media forms - and a universal remediator - something able to reconstruct and communicate all media forms.

To imitate media, computers need to do something besides imitating their surface signs. They need to also imitate their permanence, their capacity to preserve information and stabilise signs. And let’s keep in mind that a computer may emulate another computer, but when it comes to physical artefacts such as media, the best it can do is to simulate some aspects of them, or often, only imitate them.

And here we arrive at another of Turing’s fundamental realisations: computers have a penchant for unpredictability. As ^{Wolfram puts it, when computational processes are “not obviously simple” (2002}, p. 5), they will most likely be irreducible. As hard as we try, we cannot predict a computation’s future and the “only way to work out how the system will behave is essentially to perform [its] computation” (^{Ibid., p. 750}) and wait until we discover it with the system. And even if the code is simple, it will be enacted atop the complexity of the underlying universal machine, a complexity that should be considered when evaluating the threshold for irreducibility.

In any computation that is “not obviously simple” we may find pockets of reducibility, areas of its phase-space that we can predict, but we can never fully anticipate its behaviours and outputs. Why does this matter? It matters because, as we will see, this characteristic is at the heart of a fundamental ontological paradox in computational media.

6. Imitating stillness and stability

What are the consequences of the tension between imitation and this predisposition for irreducibility? In computational media information is stored in a format that is not directly accessible by readers and that needs to be translated, i.e., transcoded and recreated every time it is mediated. Whenever a digital medium educes a message - a text, an image, a sound, etc. - it must recreate signs from stored data or from a data-generating process. For example, whenever a digital video is reproduced, all its frames need to be recreated, several times per second. And this process needs to happen every time the same video file is played. Every time an image or text are displayed, all pixels, and all letters need to be recreated. And this process may happen dozens of times per second, as a screen refreshes. As Wendy Chun points out, within the computational, every medium becomes a time-based medium, and images and other signs “are frozen for human eyes only” (²⁰¹¹, p. 197). Computational media imitate stillness and stability at the surface level while activity and computation never really stop at the subface level of their computational core.

It is in this surface-subface duality, as Frieder ^{Nake (2018}) calls it, that the essence of computational media resides. Signs in the sensorial surface of the medium are what humans can directly perceive. But these signs are generated by computational activity at the subface, activity that can only be understood indirectly, inferred from the surface and the actions on it. In analogue media that exist on two levels, such as cinema, where images on the film strip are projected onto moving images on a screen, the relationship between these two levels of signs is trivial, as the secondary sign production is “dominated by the material authority of the first level” (^{Aarseth, 1997}, p. 40). In computational media, however, the relationship between the data and processes at the subface and the signs generated at the surface is arbitrary (Ibid.). The same data can generate different signs depending on the processes used to transcode it or the context where it is transcribed. Because of irreducibility, even the same data and processes may lead to varying signs.

Whenever data is transcoded and signs are recreated at the surface, several techniques may allow the medium to check the integrity and completeness of the output signs. A computer can verify if all information is presented, if all signs are in their intended places, and make corrections in case they are not. But in this process many opportunities also arise for divergence. Errors, bugs, glitches, failed transcoding, etc., may not only happen on occasion, as they may happen several times per second in every part of a message: every letter, sample, pixel, etc. Every time a message is recreated within this strange and very peculiar time of the computer, there are opportunities for discorrelation, for time to become “out of joint” (^{Denson, 2020}, p. 164) between the medium and the human. This makes the spectrality of computation concrete in every sign that mutates, in every piece of information that diverges, in every new meaning that these processes may trigger.

Computational media are always dynamic. During their unending processes, computation may find ways to express itself and to reveal its nature. As much as we check-sum messages, and of how many strategies we use to assure data integrity, we cannot stop computational media from potentially diverging, from creating new signs where we did not expect (or want) them, from creating new information. And as we structurally couple with them, be it through interaction (^{Penny, 2017}) or through processes of computational reading and model-making (^{Carvalhais, 2022}), we are not only expanded by new senses (^{Lee, 2018}) but also by new meanings (^{Denson, 2023}). We become privy to images and other signs that are not meant for us, but we are also given a glimpse of the processes behind the curtain (^{Lee, 2023}), the processes at the subface that manage, store, mediate, and ultimately create messages.

7. The paradox of computational media

This is where we find a paradox at the heart of computational media. The goal of any medium is to stabilise information, and computation certainly seems to allow this, however, it ultimately fosters variation and discorrelation (^{Denson, 2020}). Computation brings change and fluidity to where these are not welcome. Does this mean that computational media are inevitably failed as such? That they cannot or should not be used as media? No. But it means that when we use them as media, we should be very aware of their nature and of the influence that it can have on their role as media. And that we can never really trust computational media to fully do their job in stabilising signs. As we digitise all our information, we really do not know whether the computational processes we depend on to retrieve that information can be trusted in the long run.

Classical media also decay and compromise the information they store. But the analogue processes of decay are either much slower, or more predictable and easier to detect and correct. Furthermore, these processes of decay can easily be countered by the mechanical reproduction of multiple copies. This may lead to a loss of Benjaminian aura, but also contributes to gains in stability. Computational media, with their situatedness, their here and now (^{Groys, 2016}), are always in process, creating and recreating, transforming, computing, and infusing things with new meanings.

What can we do about this? How can we, as designers, creators of media, consumers, citizens, deal with this paradox? First, we need to realise that it happens. We need to know about it and to make it known. We must spread media literacy and awareness of how computational media are not inherently bad, faulty, or flawed, but cannot be expected to be exactly like classical media. Consequently, we need to develop a critical stance towards computational media and computational tools. We need to understand them, their nature, and their influence on everything that is made with them.

Ultimately, we need to embrace these media as they are, and for what they are. We need to expose their processes, making them easier to read and understand, and breaking open the black boxes of their subfaces (^{Carvalhais, 2021}). We need to counter the tendency towards abstraction in computation, de-abstract these media and embrace their creative potential (^{Carvalhais & Lee, 2022}). This may make them harder to read, perhaps less friendly to operate, but more transparent in their processes.

We need to engage our computational gaze for a fully realised creation of meaning with computational media and to develop and deploy theories of these systems as hermeneutical tools (^{Lee, 2024}), to develop theories of the system, models of their computational processes (^{Carvalhais, 2022}) that we can use to better understand their operations and behaviour. Ultimately, we cannot regard them as neutral mediators. No medium is neutral, of course, but computational media have a much higher, and ultimately uncontrollable, potential for agency and for the creation of signs and meaning.

8. Conclusion: In tune with the world

Like other media forms, computational media decay, shifting and transforming the information they contain and convey. In analogue media forms this is often such a slow process that it gives us a sense of permanence and stability. In computational media, however, the process is much faster and pervasive at all levels of a medium’s functioning. We developed several technologies to try to assure permanence in computational media forms, that try to counter the constant processes of transformation in computation that clash with a worldview fixated on a notion of perceived, or idealised, permanence. But the world is itself in a constant process of transformation, and as such, computational media are in tune with the world as it is, clashing with the world as we perceive it. Because of this, computational media are frequently understood in biased phenomenological terms that are attached to an understanding of media inherited from analogue technologies, that, although seemingly similar at a superficial level, are ontologically very different from computational counterparts. This creates a dissonance that leads to a certain lack of literacy on computational media, and to limitations of our creative relationships with them. As impermanence is natural and permanence artificial - if it is even achievable -, computational media oddly become both the most artificial of all media we have so far developed but also those that are more natural. It is this paradoxical stand-off that profoundly affects, or even regulates, how we communicate, create, and develop meaning alongside these media. And failure to acknowledge this is to be blind to the other half of the spectrum and refuse the nature of computational media altogether.