Cognitive engineering of mixed reality


iOS 11: user-adaptable control centre, with application and function shortcuts in the lock screen.
iOS 11: user-adaptable control centre, with application and function shortcuts in the lock screen.

In the 1970s and 1980s the concept ‘cognitive engineering’ was used in the industry labs to describe an approach trying to apply cognitive science lessons to the design and engineering fields. There were people like Donald A. Norman, who wanted to devise systems that are not only easy, or powerful, but most importantly pleasant and even fun to use.

One of the classical challenges of making technology suit humans, is that humans change and evolve, and differ greatly in motivations and abilities, while technological systems tend to stay put. Machines are created in a certain manner, and are mostly locked within the strict walls of material and functional specifications they are based on, and (if correctly manufactured) operate reliably within those parameters. Humans, however, are fallible and changeable, but also capable of learning.

In his 1986 article, Norman uses the example of a novice and experienced sailor, who greatly differ in their abilities to take the information from compass, and translate that into a desirable boat movement (through the use of tiller, and rudder). There have been significant advances in multiple industries in making increasingly clear and simple systems, that are easy to use by almost anyone, and this in turn has translated into increasingly ubiquitous or pervasive application of information and communication technologies in all areas of life. The televisions in our living rooms are computing systems (often equipped with apps of various kinds), our cars are filled with online-connected computers and assistive technologies, and in our pockets we carry powerful terminals into information, entertainment, and into the ebb and flows of social networks.

There is, however, also an alternative interpretation of what ‘cognitive engineering’ could be, in this dawning era of pervasive computing and mixed reality. Rather than only limited to engineering products that attempt to adapt to the innate operations, tendencies and limitations of human cognition and psychology, engineering systems that are actively used by large numbers of people also means designing and affecting the spaces, within which our cognitive and learning processes will then evolve, fit in, and adapt into. Cognitive engineering does not only mean designing and manufacturing certain kinds of machines, but it also translates into an impact that is made into the human element of this dialogical relationship.

Graeme Kirkpatrick (2013) has written about the ‘streamlined self’ of the gamer. There are social theorists who argue that living in a society based on computers and information networks produces new difficulties for people. Social, cultural, technological and economic transitions linked with the life in late modern, capitalist societies involve movements from projects to new projects, and associated necessity for constant re-training. There is necessarily no “connecting theme” in life, or even sense of personal progression. Following Boltanski and Chiapello (2005), Kirkpatrick analyses the subjective condition where life in contradiction – between exigency of adaptation and demand for authenticity – means that the rational course in this kind of systemic reality is to “focus on playing the game well today”. As Kirkpatrick writes, “Playing well means maintaining popularity levels on Facebook, or establishing new connections on LinkedIn, while being no less intensely focused on the details of the project I am currently engaged in. It is permissible to enjoy the work but necessary to appear to be enjoying it and to share this feeling with other involved parties. That is the key to success in the game.” (Kirkpatrick 2013, 25.)

One of the key theoretical trajectories of cognitive science has been focused on what has been called “distributed cognition”: our thinking is not only situated within our individual brains, but it is in complex and important ways also embodied and situated within our environments, and our artefacts, in social, cultural and technological means. Gaming is one example of an activity where people can be witnessed to construct a sense of self and its functional parameters out of resources that they are familiar with, and which they can freely exploit and explore in their everyday lives. Such technologically framed play is also increasingly common in working life, and our schools can similarly be approached as complex, designed and evolving systems that are constituted by institutions, (implicit, as well as explicit) social rules and several layers of historically sedimented technologies.

Beyond all hype of new commercial technologies related to virtual reality, augmented reality and mixed reality technologies of various kinds, lies the fact that we have always already lived in complex substrate of mixed realities: a mixture of ideas, values, myths and concepts of various kinds, that are intermixed and communicated within different physical and immaterial expressive forms and media. Cognitive engineering of mixed reality in this, more comprehensive sense, involves involvement in dialogical cycles of design, analysis and interpretation, where practices of adaptation and adoption of technology are also forming the shapes these technologies are realized within. Within the context of game studies, Kirkpatrick (2013, 27) formulates this as follows: “What we see here, then, is an interplay between the social imaginary of the networked society, with its distinctive limitations, and the development of gaming as a practice partly in response to those limitations. […] Ironically, gaming practices are a key driver for the development of the very situation that produces the need for recuperation.” There are multiple other areas of technology-intertwined lives where similar double bind relationships are currently surfacing: in social use of mobile media, in organisational ICT, in so-called smart homes, and smart traffic design and user culture processes. – A summary? We live in interesting times.

– Boltanski, Luc, ja Eve Chiapello (2005) The New Spirit of Capitalism. London & New York: Verso.
– Kirkpatrick, Graeme (2013) Computer Games and the Social Imaginary. Cambridge: Polity.
– Norman, Donald A. (1986) Cognitive engineering. User Centered System Design31(61).

LARP: Art not worthy?

worldcon75Worldcon 75 in Helsinki has generally been an excellent event with multiple cultures, diverse forms of art and innumerable communities of fandom coming together. However, what left bit of a bad taste to the mouth was the organizers’ decision yesterday to cancel a LARP (live action role-play), dealing with old people and dementia. The decision is highly controversial, and apparently based on some (non-Nordic) participants strongly communicating their upset at such a sensitive topic has been even allowed to be submitted in the form of a “game”, into the con program. On the other hand, same people would apparently be completely fine with Altzheimer and similar conditions being handled in form of a novel, for example.

There will be no doubt multiple reactions coming in to this from experts of this field in the future. My short comment: this is an unfortunate case of censorship, based on cultural perception of play and games as inherently trivializing or “fun-based” form of low culture. It seems that for some people, there still are strict cultural hierarchies even within the popular culture, with games at the very bottom – and that handling something sensitive with the form of role-play, for example, can be an insult. Such position completely ignores the work that has been done for decades in Nordic LARP and in digital indie “art games” (and also within the academic traditions of game studies) to expand the range of games and play for cultural expression, and to remove expectation or stigma of automatic trivialism from the interactive forms of art and culture. The organisers have obviously been pressurised by some vocal individuals, but the outcome in this case was a failure to stand up, explain the value and potential of role-playing games, and Nordic LARP in particular to an international audience, and make a difference. A sad day.

Link: Worldcon 75 cancellation statement (currently in updated and revised form) in Facebook regarding “The Old Home” [edit: should be “A Home for the Old”] LARP:

(There has been multiple exchanges regarding this matter in Twitter, for example, but not linking them here.)

(Edit: the documentation for the said LARP is available for download here:

(Edit2: LARP scholars and experts Jaakko Stenros and Markus Montola have published a thorough account of this incident here

(Edit3: Wordcon organisers have now published a more thorough explanation and reasons for their decision here

Three movies

I had some movie tickets that were expiring in Sunday, so I went for it, watching in a row three recent movies in cinema. All of these were transmedia storytelling – two of these were movies based on digital games, one was based on a book. I have no time to write actual reviews but a couple of notes:

Angry Birds Movie: the starting point feels almost like the rumoured Tetris Movie Trilogy – not much narrative material exists in the game to start with, but what little there is, it will be liberally exploited and expanded upon. In this case, we will learn why the birds are angry. In the original games the different birds were colour coded game units that each enabled different slingshot trajectories or other abilities. The movie version does decent work in providing them with personality, and for developing (bit silly and comedy-oriented) backstory for the conflict between the birds and the pigs.

The BFG (Big Friendly Giant): this is probably the strongest of three, when evaluated in terms of its overall cinematic qualities. The combination of Roald Dahl’s innovative children’s book and Steven Spielberg’s skills in high production value adventure movies provides a balanced mixture of humour, sense of wonder and a touch of some darker themes. The most memorable element is the friendly, 24 feet (over 7 meter) giant himself, played by Mark Rylance, and translated into detailed digital version by advanced motion capture technologies and computer generated imagery. The eyes of this friedly figure are particularly lively, deep and expressive.

Warcraft: The Beginning: like the title says, this movie is set to the early stages in the history of Azeroth, the main world of Warcraft game series. Gul’dan, an orc warlock, uses fel magic (evil, vampiric style of magic) to open a portal from Draenor, homeworld of orcs (now destroyed by fel magic) to Azeroth, inhabited by humans, elves and dwarves, and a dramatic conflict ensues. The challenge in Warcraft movie appears to be the exact opposite from the Angry Birds one: here, an abundance of characters, plotlines, wars, races, mythical places etc. has to be reduced into something that resembles more or less coherent, classical movie storyline. The reviews have generally been negative, but I actually rather liked the movie – perhaps due to having spent considerable time in Ironforge, Stormwind etc. myself, as a player of Warcraft RTS and World of Warcraft games in the past. The movie does not get very far in itself: there is perhaps ten or more significant characters, some of them are killed, some plots unravelled and others set into motion, and in the end everything just stops, after this prologue having provided hints at important future developments. But landscapes are impressive, some characters relatable, and there is constant “epic tone” in all of it (that might feel ridiculous or appropriate, depending what one’s tastes in genre fantasy are).

All in all, this day of movies just pointed out how central fantasy as an element, impulse and setting has become for popular culture, and how various storyworld elements cross media boundaries with ever-increasing ease.

Angry Birds Movie © Columbia Pictures and Rovio.
The BFG © Disney.
Warcraft: The Beginning © NBCUniversal
Warcraft: The Beginning © NBCUniversal

Money & Games blog note, #moneygames

A quick note about the Money & Games seminar, based on the first day: I was expecting the relationships between money and games to be diverse and rather complex field, and I was not disappointed by the seminar. The idea that game could be seen as a straightforward product that someone just builds, and then sells to someone else for a fixed sum of money is not how things play out – and, as the historical reviews of the seminar pointed out, is not that typical about how things have been in the past, either. For example, the entire era of game arcades was based on coin-operated games, where the economic incentive was to design for short, micropayment style transactions: every time the player failed, the was room for another coin to be spend (something that Sebastian Deterding’s ambitious “Toward Economic Platform Studies” paper and presentation was particularly emphasising). Value of games and monetary and time-based investments are intricately intertwined, and it is clear that e.g. putting a higher price tag on something can mean that pleyers are more likely to expect it to be of higher quality, or value, than a cheap game. Thus, setting the right price involves theorycrafting practice of game business economics of its own – or “valuecrafting”, like the paper presented by Mia Consalvo suggested about indie developers. Free-to-play business model and the associated monetization strategies were particularly discussed in the seminar, with several interesting case studies focusing on that, plus the more philosophically oriented paper by Olli Heimo et al. used it, plus industry advertising practices as a target of (Aristotelian) virtue ethics based criticism. There were comments expressed in the seminar that the political economy angle of the entire free-to-play sector would be something that would be valuable at this point. On the other hand, while Janne Paavilainen presented the first results from a detailed micro-ethnography in Armoured Warfare game, pointing out the multiple “dark design patterns” or manipulative tricks that tempt the free-riding player to become a paying player, Markus Montola was quick to point out that many of the analysed design choices actually sounded just like good, regular game design that is balanced and appropriately both challenges and rewards the player – and Janne agreed that Armoured Warfare is an example of good game design; free-to-play payments are just used to make an already good game to play even better. Great papers, presentations, and discussions, thanks everyone! Also, our invited commentators, Pauliina Raento and Juho Hamari, did excellent job in providing commentary and guidance, Pauliina also giving a keynote talk of her own about doing gambling studies, about the lessons she personally has learned from her history in this field, and that made the valuable point about importance of bridge building between isolated academic communities. – Link to the seminar program page:


Please spread the word:

CFP DiGRA/FDG 2016 – 1st Joint International Conference of DiGRA and FDG

For the first time, the Digital Games Research Association (DiGRA) and the Foundation of Digital Games (FDG) will partner in an unprecedented gathering of games researchers. We invite researchers and educators within game research, broadly construed, to submit their work.

For more information, please visit the conference’s website:


DiGRA/FDG aims at being a venue for game research from all research disciplines. In line with this, it accepts and encourages submissions in the following six tracks, on a wide range of subjects including, but not limited to:

  • Game design: Design techniques, practices, methods, post mortems, etc.
  • Game criticism and analysis: Close readings, ontologies and frameworks, historical studies, philosophical explorations, and other humanities-informed approaches
  • Play studies + Interaction and player experience: studies of play, observations and interviews of players, and research based on other methods from the social sciences; game interfaces, player metrics, modeling player experience
  • Artificial intelligence: agents, motion/camera planning, navigation, adaptivity, procedural content generation, dialog, authoring tools, general game playing
  • Game technology: engines, frameworks, graphics, networking, animation
  • Game production: studies of game production processes, studio studies, software studies, platform studies and software engineering

Due to the interdisciplinary nature of the DiGRA/FDG conference, authors and reviewers alike will be required to describe their research background and field of study as part of the submission process. The intention for this is to help reviewers be conscious of when they are reviewing work outside their own field as well as making clear the proportions of contributing fields.

Submission categories

DiGRA/FDG 2016 supports two different categories for submitting research:

  • Full Papers (no more than 16 pages)
  • Extended Abstracts with a maximum length of 2-pages

This structure reflects the cross-disciplinary nature and different conference traditions of the conference attendants. A full paper submission is recommended for completed research work, in particular empirical or technical work. The extended abstract format is suitable for discussion topics and ideas. Both full papers and abstracts are subject to a double-blind review process. These two categories are the only ones that will be published in DiGRA’s digital library.

Deadlines full papers and extended abstracts

Submission deadline

  • January 29 2016 (hard deadline)

Acceptance/rejection notification

  • March 21 2016


  • March 25 2016

Notification of final decisions

  • March 31 2016

Camera ready

  • April 29 2016

In addition to this, DiGRA/FDG 2016 accepts submissions for:

  • Events with a maximum length of 2-page abstract
  • Panels with a maximum length of 2-page abstract

These are curated by the local organizers and do not go through an anonymized process.

Deadlines panels and events

Submission deadline

  • January 29 2016 (hard deadline)

Acceptance/rejection notification

  • March 21 2016

Camera ready

  • April 29 2016

Some work does not fit as paper presentations due to its nature or research maturity. For this, DiGRA/FDG 2016 is open to submission to the following categories:

  • Posters with a maximum length of 2-page abstract
  • Demos with a maximum length of 2-page abstract

These categories have late deadlines to allow the most recent research and results to be submitted.

Deadlines posters and demos

Submission deadline

  • April 8 2016 (hard deadline)

Acceptance/rejection notification

  • May 9 2016

Camera ready

  • May 23 2016

DiGRA/FDG 2016 provides a doctoral consortium for PhD students. Those interested in attending this should submit a position paper in the extended abstract format with a maximum length of 2 pages.

Deadlines doctoral consortium

Submission deadline

  • April 22 2016 (hard deadline)

Acceptance/rejection notification

  • May 9 2016

DiGRA/FDG 2016 also welcomes submissions to arrange workshops. These have an earlier deadline than other submission to support workshops that wish to have their own peer reviewing process for submissions. These should be submitted as extended abstracts with a maximum length of 2 pages. Please submit workshop proposals by email to the three program chairs, and place “[FDG/DiGRA 2016 Workshop Submission]” in the subject line.

Deadlines workshops

Submission deadlines

  • November 16 2015 (hard deadline)

Acceptance/rejection notification

  • December 11 2015

Location & Date

  • August 1-6 2016
  • The School of Arts, Media and Computer Games, Abertay University
  • Dundee, Scotland, UK

For more information see:

Program Chairs


DiGRA 2016: joint DiGRA/FDG conference

This actually makes great sense: “For the first time, the Digital Games Research Association (DiGRA) and the Foundation of Digital Games (FDG) will partner in an unprecedented gathering of games researchers. We invite researchers and educators within game research, broadly construed, to submit their work.” More at:

Narrative Theory, Literature, and New Media: new book

Narrative Minds, Virtual Worlds (cover).
Narrative Minds, Virtual Worlds.

New book, edited by Mari Hatavara, Matti Hyvärinen, Maria Mäkelä and myself, is now available for pre-order: Narrative Theory, Literature, and New Media: Narrative Minds and Virtual Worlds (Routledge). This interdisciplinary work discusses and analyses constructions of storyworlds and minds in games as well as in literature and media from multiple perspectives. Here is the table of contents:

Introduction: Minds in Action, Interpretive Traditions in Interaction Mari Hatavara, Matti Hyvärinen, Maria Mäkelä, and Frans Mäyrä

Section I

1. Texts, Worlds, Stories: Narrative Worlds as Cognitive and Ontological Concept Marie-Laure Ryan

2. Storyworlds and Paradoxical Narration: Putting Classifications to a Transmedial Test Liviu Lutas

3The Charge against Classical and Post-Classical Narratologies’ “Epistemic” Approach to Literary FictionGreger Andersson

Section II

4. How You Emerge from This Game Is up to You: Agency, Positioning, and Narrativity in The Mass Effect Trilogy Hanna-Riikka Roine

5. Playing the Worlds of Prom Week Ben Samuel, Dylan Lederle-Ensign, Mike Treanor, Noah Wardrip-Fruin, Josh McCoy, Aaron Reed, and Michael Mateas

6.Scripting Beloved Discomfort: Narratives, Fantasies, and Authenticity in Online Sadomasochism J. Tuomas Harviainen

7.Storyworld in Text-Messages: Sequentiality and Spatialisation Agnieszka Lyons

Section III

8. Defending the Private and the Unnarratable: Doomed Attempts to Read and Write Literary and Cinematic Minds in Marguerite Duras’s India Cycle Tytti Rantanen

9. Of Minds and Monsters: the Eventfulness of Monstrosity and the Poetics of Immersion in Horror LiteratureGero Brümmer

10. Narrative Conventions in Hallucinatory Narratives Tommi Kakko

11.Narrative and Minds in the Traditional Ballads of Early Country Music Alan Palmer

Section IV

12.Mind Reading, Mind Guessing, or Mental-State Attribution? The Puzzle of John Burnside’s A Summer of Drowning Matti Hyvärinen

13 Mind as World in the Reality Game Show Survivor Maria Mäkelä

14 Performing Selves and Audience Design: Interview Narratives on the Internet Jarmila Mildorf

15 Documenting Everyday Life: Mind Representation in the Web Exhibition “A Finnish Winter Day” Mari Hatavara

Afterword: A New Normal? Brian McHale

Publisher book page link: