Cognitive engineering of mixed reality


iOS 11: user-adaptable control centre, with application and function shortcuts in the lock screen.
iOS 11: user-adaptable control centre, with application and function shortcuts in the lock screen.

In the 1970s and 1980s the concept ‘cognitive engineering’ was used in the industry labs to describe an approach trying to apply cognitive science lessons to the design and engineering fields. There were people like Donald A. Norman, who wanted to devise systems that are not only easy, or powerful, but most importantly pleasant and even fun to use.

One of the classical challenges of making technology suit humans, is that humans change and evolve, and differ greatly in motivations and abilities, while technological systems tend to stay put. Machines are created in a certain manner, and are mostly locked within the strict walls of material and functional specifications they are based on, and (if correctly manufactured) operate reliably within those parameters. Humans, however, are fallible and changeable, but also capable of learning.

In his 1986 article, Norman uses the example of a novice and experienced sailor, who greatly differ in their abilities to take the information from compass, and translate that into a desirable boat movement (through the use of tiller, and rudder). There have been significant advances in multiple industries in making increasingly clear and simple systems, that are easy to use by almost anyone, and this in turn has translated into increasingly ubiquitous or pervasive application of information and communication technologies in all areas of life. The televisions in our living rooms are computing systems (often equipped with apps of various kinds), our cars are filled with online-connected computers and assistive technologies, and in our pockets we carry powerful terminals into information, entertainment, and into the ebb and flows of social networks.

There is, however, also an alternative interpretation of what ‘cognitive engineering’ could be, in this dawning era of pervasive computing and mixed reality. Rather than only limited to engineering products that attempt to adapt to the innate operations, tendencies and limitations of human cognition and psychology, engineering systems that are actively used by large numbers of people also means designing and affecting the spaces, within which our cognitive and learning processes will then evolve, fit in, and adapt into. Cognitive engineering does not only mean designing and manufacturing certain kinds of machines, but it also translates into an impact that is made into the human element of this dialogical relationship.

Graeme Kirkpatrick (2013) has written about the ‘streamlined self’ of the gamer. There are social theorists who argue that living in a society based on computers and information networks produces new difficulties for people. Social, cultural, technological and economic transitions linked with the life in late modern, capitalist societies involve movements from projects to new projects, and associated necessity for constant re-training. There is necessarily no “connecting theme” in life, or even sense of personal progression. Following Boltanski and Chiapello (2005), Kirkpatrick analyses the subjective condition where life in contradiction – between exigency of adaptation and demand for authenticity – means that the rational course in this kind of systemic reality is to “focus on playing the game well today”. As Kirkpatrick writes, “Playing well means maintaining popularity levels on Facebook, or establishing new connections on LinkedIn, while being no less intensely focused on the details of the project I am currently engaged in. It is permissible to enjoy the work but necessary to appear to be enjoying it and to share this feeling with other involved parties. That is the key to success in the game.” (Kirkpatrick 2013, 25.)

One of the key theoretical trajectories of cognitive science has been focused on what has been called “distributed cognition”: our thinking is not only situated within our individual brains, but it is in complex and important ways also embodied and situated within our environments, and our artefacts, in social, cultural and technological means. Gaming is one example of an activity where people can be witnessed to construct a sense of self and its functional parameters out of resources that they are familiar with, and which they can freely exploit and explore in their everyday lives. Such technologically framed play is also increasingly common in working life, and our schools can similarly be approached as complex, designed and evolving systems that are constituted by institutions, (implicit, as well as explicit) social rules and several layers of historically sedimented technologies.

Beyond all hype of new commercial technologies related to virtual reality, augmented reality and mixed reality technologies of various kinds, lies the fact that we have always already lived in complex substrate of mixed realities: a mixture of ideas, values, myths and concepts of various kinds, that are intermixed and communicated within different physical and immaterial expressive forms and media. Cognitive engineering of mixed reality in this, more comprehensive sense, involves involvement in dialogical cycles of design, analysis and interpretation, where practices of adaptation and adoption of technology are also forming the shapes these technologies are realized within. Within the context of game studies, Kirkpatrick (2013, 27) formulates this as follows: “What we see here, then, is an interplay between the social imaginary of the networked society, with its distinctive limitations, and the development of gaming as a practice partly in response to those limitations. […] Ironically, gaming practices are a key driver for the development of the very situation that produces the need for recuperation.” There are multiple other areas of technology-intertwined lives where similar double bind relationships are currently surfacing: in social use of mobile media, in organisational ICT, in so-called smart homes, and smart traffic design and user culture processes. – A summary? We live in interesting times.

– Boltanski, Luc, ja Eve Chiapello (2005) The New Spirit of Capitalism. London & New York: Verso.
– Kirkpatrick, Graeme (2013) Computer Games and the Social Imaginary. Cambridge: Polity.
– Norman, Donald A. (1986) Cognitive engineering. User Centered System Design31(61).

Brydge 12.3, Surface Pro 4

Surface Pro 4, with Brydge 12.3 and MS Type Cover
Surface Pro 4, with Brydge 12.3 and MS Type Cover

Getting the input right is one of the most challenging issues in todays world of pervasive, multimodal computing and services. Surface Pro 4 is an excellent multitouch tablet, and with the Surface Pen it is perfect for review and marking (key elements in academic life). The problem with a tablet as a main computer is that much of the productivity oriented tasks really call for a mouse and keyboard style approach.

There are pretty good add-on keyboards for today’s tablet computers, and one can of course also attach to a Surface Pro a full size keyboard and mouse combo. However, a keyboard cover that is always with you is the optimal companion for a tablet user. The official Type Cover by Microsoft is a really good compromise: it is thin, light, has decent keys, excellent touchpad, and backlight, which is really important for business use. There is certain wobbly, flexible quality in the keys though, and writing a whole day with one can create certain strain.

I have now tested a new, much more solid alternative: Brydge 12.3 keyboard cover. It is made of strong aluminium, has 160 degrees rotating hinges that create a firm grip on the corners of the tablet, and its island style keys also are backlighted. According to my experience, the usability issues with Brydge relate to the unreliability of Bluetooth connection on one hand – sometimes I would spend several minutes after tablet wake-up waiting for keyboard to re-establish its connection. Other thing is that the integrated touchpad is rather bad. It is hard to control precisely, pointer movement is wobbly, and not all Windows 10 mouse gestures are supported. It is also very small by today’s standards, and clicks register randomly. The sensible use for the Brydge is to use it alongside a wired or wireless mouse – this, however, diminishes its value as a real laptop replacement option. The trackpad in Type Cover is so much better that in regular use that in the end it trumps Brydge’s better (or at least more solid) keyboard. The plus side of using Brydge is that in tactile terms, it transforms Surface Pro into a (small and heavy) laptop computer.

It is apparently hard to get a 2-in-1 device right. However, multiple manufacturs have recently introduced their own takes on the same theme, so there might be better options out there already.

Surface Pro 4, with Brydge 12.3 and MS Type Cover
Surface Pro 4, with Brydge 12.3 and MS Type Cover

All-in-one: still not there

HP Elite X2 1012 press photo (image © HP).
Some time ago, I blogged about tablets as productivity devices, and then I also have written about some early experiences as a user of Microsoft Surface Pro 4: a Windows 10, 2-in-one tablet PC that relies on combination of touch screen, pen computing, and keyboard and touchpad cover (plus Cortana voice assistant, if you are a US/English user). It just might be that I am restless and curious by nature, but these days I find myself jumping from Microsoft to Apple to Google ecosystems, and not really finding what I am looking for from any of them.

When I am using an iOS or Android tablet, the file management is usually a mess, external keyboard and mouse inputs are not working reliably, and multitasking between several apps and services, copy-pasting or otherwise sharing information between them all is a pain.

When I am on a regular Windows laptop or PC, keyboard and mouse/touchpad usually are just fine, and file management, multitasking and copy-pasting work fine. Touch screen inputs and the ease of use lag behind tablet systems, though. (This is true also to the Apple OS X desktop environment, but I have pretty much given up the use of Macs for my work these days, I just could not configure the system to work and behave in the ways I want – as a Microsoft OS/PC user who has hacked his way around DOS, then Windows 3.0 etc., and thus has certain things pretty much “hard-wired” in the way I work.)

Surface Pro 4 is the most optimal, almost “all-in-one” system I have found so far, but I have started to increasingly dislike its keyboard cover. Surface Pro 4 cover is not that bad, but if you are a touch-typist, it is not perfect. There is still slight flex in the plastic construction and shallow key movement that turns me off, and produces typing errors exactly when you are in a hurry and you’d need to type fast. I am currently trying to find a way to get rid of the type cover, and instead use my favorite, Logitech K810 instead. But: I am not able to attach it to Surface Pro in solid enough way, and there is no touchpad in K810, so workflow with all those mouse right-clicks becomes rather complex.

I really like the simplicity of Chromebooks, and this blog note, for example, is written with my trusty Toshiba Chromebook 2, which has excellent, solid keyboard (though not backlighted), and a good, crisp Full HD IPS screen plus a responsive, large touchpad. However, I keep reaching out and trying to scroll the screen, which is not a touch version. (Asus Chromebook Flip would be one with a touch screen.) And there is nothing comparable to the Surface Pen, which is truly useful when one e.g. reads and makes notes to a pile of student papers in PDF/electronic formats. Also, file management in a Chrome OS is a mess, and web versions of popular apps still respond more slowly and are more limited than real desktop versions.

So, I keep on looking. Recently I tested the HP Elite X2 1012 (pictured), which is pretty identical to the Surface Pro systems that Microsoft produces, but has an excellent, metallic and solid keyboard cover, as well as other productivity oriented enhancements like the optional 4G/LTE sim card slot, USB C port with Thunderbolt technology, and a decent enough screen, pen and kickstand design. However, Elite X2 falls short in using less powerful Intel Core M series processors (Surface Pro 4 goes for regular Core i5 or i7 after the entry-level model), by being rather expensive, and according to the reviews I have read, also the battery life of Elite X2 is not something a real mobile office worker would prefere.

Maybe I can find a way to connect the Elite X2 metallic keyboard cover to the Surface Pro 4? Or maybe not.

(Edit: The battery life of Elite X2 actually appears to be good; the screen on the other hand only so-and-so.)