pervasive computing – frans goes blog

Cognitive engineering of mixed reality

iOS 11: user-adaptable control centre, with application and function shortcuts in the lock screen. — *iOS 11: user-adaptable control centre, with application and function shortcuts in the lock screen*.

In the 1970s and 1980s the concept ‘cognitive engineering’ was used in the industry labs to describe an approach trying to apply cognitive science lessons to the design and engineering fields. There were people like Donald A. Norman, who wanted to devise systems that are not only easy, or powerful, but most importantly pleasant and even fun to use.

One of the classical challenges of making technology suit humans, is that humans change and evolve, and differ greatly in motivations and abilities, while technological systems tend to stay put. Machines are created in a certain manner, and are mostly locked within the strict walls of material and functional specifications they are based on, and (if correctly manufactured) operate reliably within those parameters. Humans, however, are fallible and changeable, but also capable of learning.

In his 1986 article, Norman uses the example of a novice and experienced sailor, who greatly differ in their abilities to take the information from compass, and translate that into a desirable boat movement (through the use of tiller, and rudder). There have been significant advances in multiple industries in making increasingly clear and simple systems, that are easy to use by almost anyone, and this in turn has translated into increasingly ubiquitous or pervasive application of information and communication technologies in all areas of life. The televisions in our living rooms are computing systems (often equipped with apps of various kinds), our cars are filled with online-connected computers and assistive technologies, and in our pockets we carry powerful terminals into information, entertainment, and into the ebb and flows of social networks.

There is, however, also an alternative interpretation of what ‘cognitive engineering’ could be, in this dawning era of pervasive computing and mixed reality. Rather than only limited to engineering products that attempt to adapt to the innate operations, tendencies and limitations of human cognition and psychology, engineering systems that are actively used by large numbers of people also means designing and affecting the spaces, within which our cognitive and learning processes will then evolve, fit in, and adapt into. Cognitive engineering does not only mean designing and manufacturing certain kinds of machines, but it also translates into an impact that is made into the human element of this dialogical relationship.

Graeme Kirkpatrick (2013) has written about the ‘streamlined self’ of the gamer. There are social theorists who argue that living in a society based on computers and information networks produces new difficulties for people. Social, cultural, technological and economic transitions linked with the life in late modern, capitalist societies involve movements from projects to new projects, and associated necessity for constant re-training. There is necessarily no “connecting theme” in life, or even sense of personal progression. Following Boltanski and Chiapello (2005), Kirkpatrick analyses the subjective condition where life in contradiction – between exigency of adaptation and demand for authenticity – means that the rational course in this kind of systemic reality is to “focus on playing the game well today”. As Kirkpatrick writes, “Playing well means maintaining popularity levels on Facebook, or establishing new connections on LinkedIn, while being no less intensely focused on the details of the project I am currently engaged in. It is permissible to enjoy the work but necessary to appear to be enjoying it and to share this feeling with other involved parties. That is the key to success in the game.” (Kirkpatrick 2013, 25.)

One of the key theoretical trajectories of cognitive science has been focused on what has been called “distributed cognition”: our thinking is not only situated within our individual brains, but it is in complex and important ways also embodied and situated within our environments, and our artefacts, in social, cultural and technological means. Gaming is one example of an activity where people can be witnessed to construct a sense of self and its functional parameters out of resources that they are familiar with, and which they can freely exploit and explore in their everyday lives. Such technologically framed play is also increasingly common in working life, and our schools can similarly be approached as complex, designed and evolving systems that are constituted by institutions, (implicit, as well as explicit) social rules and several layers of historically sedimented technologies.

Beyond all hype of new commercial technologies related to virtual reality, augmented reality and mixed reality technologies of various kinds, lies the fact that we have always already lived in complex substrate of mixed realities: a mixture of ideas, values, myths and concepts of various kinds, that are intermixed and communicated within different physical and immaterial expressive forms and media. Cognitive engineering of mixed reality in this, more comprehensive sense, involves involvement in dialogical cycles of design, analysis and interpretation, where practices of adaptation and adoption of technology are also forming the shapes these technologies are realized within. Within the context of game studies, Kirkpatrick (2013, 27) formulates this as follows: “What we see here, then, is an interplay between the social imaginary of the networked society, with its distinctive limitations, and the development of gaming as a practice partly in response to those limitations. […] Ironically, gaming practices are a key driver for the development of the very situation that produces the need for recuperation.” There are multiple other areas of technology-intertwined lives where similar double bind relationships are currently surfacing: in social use of mobile media, in organisational ICT, in so-called smart homes, and smart traffic design and user culture processes. – A summary? We live in interesting times.

References:
– Boltanski, Luc, ja Eve Chiapello (2005) The New Spirit of Capitalism. London & New York: Verso.
– Kirkpatrick, Graeme (2013) Computer Games and the Social Imaginary. Cambridge: Polity.
– Norman, Donald A. (1986) Cognitive engineering. User Centered System Design, 31(61).

Future of interfaces: AirPods

apple-airpods — *Apple AirPods (image © Apple).*

I am a regular user of headphones of various kinds, both wired and wireless, closed and open, with noise cancellation, and without. The latest piece of this technology I invested in are the “AirPods” by Apple.

Externally, these things are almost comically similar to the standard “EarPods” they provide with, or as the upgrade option for their mobile devices. The classic white Apple design is there, just the cord has been cut, leaving the connector stems protruding from the user ears, like small antennas (which they probably also indeed are, as well as directional microphone arms).

There are wireless headphone-microphone sets that have slightly better sound quality (even if AirPods are perfectly decent as wireless earbuds), or even more neutral design. What is here interesting in one part is the “seamless” user experience which Apple has invested in – and the “artificial intelligence” Siri assistant which is another key part of the AirPod concept.

The user experience of AirPods is superior to any other headphones I have tested, which is related to the way the small and light AirPods immediatelly connect with the Apple iPhones, detect when they are placed into the ear, or or not, and work hours on one charge – and quickly recharge after a short session inside their stylishly designed, smart battery case. These things “just work”, in the spirit of original Apple philosophy. In order to achieve this, Apple has managed to create a seamless combination of tiny sensors, battery technology, and a dedicated “W1 chip” which manages the wireless functionalities of AirPods.

The integration with Siri assistant is the other key part of AirPod concept, and the one that probably divides user’s views more than any other feature. A double tap to the side of an AirPod activates Siri, which can indeed understand short commands in multiple languages, and respond to them, carrying out even simple conversations with the user. Talking to an invisible assistant is not, however, part of today’s mobile user cultures – even if Spike Jonze’s film “Her” (2013) shows that the idea is certainly floating around today. Still, mobile devices are often used while on the move, in public places, in buses, trains or in airplanes, and it is just not feasible nor socially acceptable that people carry out constant conversations with their invisible assistants in this kind of environments – not yet today, at least.

Regardless of this, Apple AirPods are actually to a certain degree designed to rely on such constant conversations, which both makes them futuristic and ambitious, but also a rather controversial piece of design and engineering. Most notably, there are no physical buttons or other ways for adjusting volume in these headphones: you just double tap to the side of AirPods, and verbally tell Siri to turn the volume up, or down. This mostly works just fine, Siri does the j0b, but a small touch control gesture would be just so much more user friendly.

There is something engaging in testing Siri with the AirPods, nevertheless. I did find myself walking around the neighborhood, talking to the air, and testing what Siri can do. There are already dozens of commands and actions that can be activated with the help of AirPods and Siri (there is no official listing, but examples are given in lists like this one: https://www.cnet.com/how-to/the-complete-list-of-siri-commands/). The abilities of Siri still fall short in many areas, it did not completely understand Finnish I used in my testing, and the integration of third party apps is often limited, which is a real bottleneck, as these apps are what most of us are using our mobile devices for, most of the time. Actually, Google and the assistant they have in Android is better than Siri in many areas relevant for daily life (maps, traffic information, for example), but the user experience of their assistant is not yet as seamless or integrated whole as that of Apple’s Siri is.

All this considered, using AirPods is certainly another step into the general developmental direction where pervasive computing, AI, conversational interfaces and augmented reality are taking us, in good or bad. Well worth checking out, at least – for more in Apple’s own pages, see: http://www.apple.com/airpods/.

Dumb Smartwatches?

Apple Watch has been much in the headlines lately, and the leading role Apple has long had in fashionable, personal devices explains much of the hype. If Apple enters the product category, then it is ready for mainstream, the conventional wisdom goes. However, in this case Apple’s leadership is not guaranteed.

A smartwatch is possibly even a more personal and intimate device than a smartphone, if such a thing is possible. Rather than pocketed, a watch is tied to your wrist, always just a glance away from the focus of your attention. Granted, many people appear to walk around with their mobile phone constantly in their hand, but that is not the way a phone is originally designed to be used. A wristwatch, on the contrary, is meant to be worn all day long.

A wristwatch is also on your skin. It becomes really intimate part of yourself, and as it is a visible part of one’s attire, it can also be considered as a fashion accessory. Some people invest considerable sums of money to jewelry, and some prefer to make their fashion statements by investing into expensive timepieces. This is one demographic which Apple Watch is aimed at, with its 17 000 dollars top-of-the-line pricing.

One major problem with expensive smartwatches is the speed that information and communication technologies and services evolve. A one year or two years old model might already be obsolete, as it is missing support for some crucial new functionality. A classic wristwatch can be timeless in a manner that a smartwatch never can be.

Another key issue is the service ecosystem integration. Currently there are at least three non-compatible app ecosystems that are competing of the souls (and money) of smartwatch users: Android Wear, Samsung Gear (based on Tizen, Samsung’s own Android-competitor), and the Apple Watch OS. I have personally been testing Samsung Gear S watch for some time, and while it has some arguably superior technical features (for a full list, see here), it is seriously lacking in applications that would support and integrate with key information, communication, entertainment and lifestyle services that most people are already committed into, with their smartphones, tablet devices and personal computers. Support of such services is really essential for a smartwatch to survive in the technologically overpopulated media ecology of today. Having unique content that truly benefits and interacts with e.g. the sensors and contextual information of these things (as in next generation pervasive games) will be another necessary step.

It is still too early to declare any winners in this race to colonize the virgin landscape of mainstream wearable space and associated user cultures. If some guesses can be made, though, I’d bet that Apple Watch will be doing rather well particularly in the US, where the existing user base for Apple devices has traditionally been strong, and the benefits of extending that into a smartwatch are therefore strongest. On the other hand, elsewhere I’d bet Android Wear to have advantage. This is due to the obvious benefits available for Google service users: the direct access to the wrist from Google Calendar and Google Now alerts alone is something that would be rather valuable for any busy professional of today. If all your personal and business contacts are in Google system, and you have saved all your important address points to the Google Maps, then it is natural to extend daily navigation, time keeping and communications filtering tasks to a Google-compatible smartwatch. Samsung appears to be weakest in this service integration area, and it might be a good idea for them to join forces e.g. with Microsoft with its large base of Exchange/Outlook users, as without well-integrated and highly automated access to the backbone personal data services a smartwatch is actually a pretty dumb idea.

From Smartphones to (Playful) Smart Devices?

If you were a techie nerd in the 80s, you might have used a Casio Databank wristwatch: a bulky device that had a small LCD screen, capable of acting as a calculator, address book, as well as a simple gaming device, while also providing advanced clock features. Those watches were (and still do) dividing user opinions, some enjoying their technically advanced, engineer-oriented pleasures, some staying as far as possible from such gadgets. With today’s focus on “smart watches” and “fitness bands”, such bulky appendices may be making a return to thousands of wrists this Christmas.

Another way of looking at these things is to consider them as the coming of the “wristwatch computer”, or manifestations of wearable, pervasive or ubiquitous computing, depending on the more general concept to adopt. As such small, “smart things” start to network and communicate with each other, they are also parts of “Internet of Things”, or “Web of Things” developments. The overall promise is of better services, which are more contextually aware, that provide information and interaction affordances in more convenient ways than the old, “PC-centric” computing paradigm has allowed. Such technologies are on the one hand inherently personal, as they connect with a trusted device (typically your mobile phone), which also may include your calendars, contact information, various social media accounts and other personalization information. On the other, they tap into new types of sensors, location-aware services and proximity beacons to provide novel services and experiences.

The futuristic promises are great, but the reality is still in the making. The current generation of smartwatches are limited in many ways, including unwieldy form factor, limited functionalities, occasional bugs, and typically rather short battery lives. The promise is nevertheless there, and many people appear to be drawn to experiment with such devices on the basis of two key functionalities: accessing smartphone alerts and information from a wristwatch, and for fitness or health information captured by the smartwatch sensors. The more advanced functionalities such as universal NFC payments, or location-based games are still waiting in the future.

My recent experiences on smartwatches are based on setting up the Samsung Galaxy Gear S (apparently a good exercise for your faculties also while lying down with a nasty flu in bed). There are multiple hoops that an early adopter seems to need to hop through: for example, you need to have a particular type of Samsung smartphone in order to use it. Having a custom rom in the phone was also a no-go, so in my case for example, I first had to uninstall the Cyanogenmod 11 I had been using in my Galaxy S4, wipe the phone, install the stock Samsung TouchWiz rom, and after that to proceed to reinstall all my applications, set up all the user accounts and re-authenticate e.g. all two-factor authentication-enabled services – a process that can easily take several hours, and is probably btw enough to turn away few interested testers.

After inserting a nano-SIM card (this device can also double as a stand-alone gsm phone), and charging the Gear S, it is time to install the necessary Galaxy Gear Manager application into Galaxy S4 from Galaxy Store (this is not available from the Google Play store, even if it is an Android app), which makes it possible to install applications and customize the Gear S. The limited selection of Gear apps is one indication of the somewhat problematic, fragmented character of current wearable ecosystems. Rather than supporting Android Wear, the Google ecosystem for wearables, Gear S is based on Tizen, a different Linux based mobile OS, developed by an association of companies, led by Samsung. Next year, the Apple Watch will arrive, opening up yet another key competing ecosystem. Getting support of e.g. your Google Maps favourites and navigation to Gear S soon does not seem likely in this competitive situation, and if you are having your calendar in Microsoft Exchange 365 server, or iCloud, for example, you first need to figure out how to get that information synchronized to the smartphone that acts as the “base station” for the particular smart device you got your eyes on. Everyone is obviously adopting a gold rush tactic, and try to grab as much land in the emerging user base as possible, trying to lock the users to their own, proprietary wearable ecosystem. From the user perspective, the situation is not optimal.

Thus, while it is nice to see e.g. movement information automatically recorded by Gear S in its S Health app, I am already a user of the leading Runkeeper service, and there is no Runkeeper app in Galaxy Gear S, nor is there a way to integrate S Health data with Runkeeper that I know of. Another handy feature would be to have the daily navigational guidance right at your wrist, when you need it. I have already long adopted the habit of including location information to all my important calendar events, so that when I am on the run, one click on the smartphone calendar will automatically open maps, with navigation, helping to choose whether to walk, pick up public transport or a taxi, which is particularly handy in a foreign country or city. Google Maps is particularly good with the public transportation schedule integration, but also Here Maps (ex-Nokia) is pretty decent in this area, at least here in Finland. Gear S does not support Google Maps/navigation, but Here Maps is supported (in “Beta”). It features turn-by-turn navigation, which appears to work and is a very good service. However, while Gear S has a bright and sharp two-inch AMOLED touch screen, which makes it into a very large wristwatch, it is still painful to use for typing in an address, with the tiny QWERTY keys. It is possible to use “Send to Gear” action from the smartphone version of Here Maps, but this seems to work only for beaming the walking instructions, and the entire operation also somewhat negates what is the key idea of wearables – of not needing to dig up the smartphone in a busy situation, with the smartwatch ready in the wrist. Another way around this would be to use “S Voice” input in Gear, but as the Finnish language is not supported, there is currently no way to just speak the local address to the Gear S. While you can get your meetings’ location information displayed in Gear S by including it into the default calendar in your supported Galaxy smartphone, it does no good trying to tap that address line in Gear S, as it is not currently linked to any navigation action.

It finally boils down to practical things such as battery life and form factor of the device, as well as language and application/service support, which of the emerging smartwatches will be a real success among the users. Based on very limited, first experiences, Galaxy Gear S is a good attempt, but finally a borderline case. The plastic-covered wrist computer is so large that at least my skin gets a bit sweaty and irritated after wearing it for several hours; stylistically, the size alone might be a complete turn-off for many potential users. Getting the notifications from text messages, emails (I opted out of those), Facebook or Twitter messages, or occasional Google Now update into the wrist display are sometimes truly useful alarms, but often distracting interferences. If you already have a tendency to lose your concentration easily, the current wearables might not be for you. On the other hand, if your work life relies on following and responding to the flow of various messages and communications and calendar events quickly and efficiently, you might consider one. I have not yet used actively Gear S for a full working day, but I suspect it should make it through a day even with the clock display turned on (the default behaviour is that it is turned off to save battery, and reacts to the movement of your rising hand by lighting up – which it often did, but also failed to do often enough to become really irritating to me at least).

Our research group has been doing studies into the future user cultures of emerging game and media technologies for years, and the ethics and rationale of design is something that we try to pay special attention with. Wearable smart technology holds promise e.g. in health, social and gamification applications of various kinds, potentially communicating the social presence of our important people literally to “our skin” in real time. It can also be used to remind us to balance our lives better, or to help us achieve our important goals by supportive messages or incentives. Gear S did bring up the S Health Pedometer display every now and then while I was writing this thing, reminding me about my physical inactivity and encouraging me to get up and moving. In my case, that was not probably the most efficient rhetoric Samsung could have adopted, but maybe there will be also other, more playful and less efficiency-oriented apps available in the future. And if not in Galaxy Gear platform, then those interesting experiments will be arriving in some other. (There was one, exploration oriented “POI Nearby” style Gear app I could find, but I could not get it to understand Finnish language or place names, either.) In any case, the door for real-world pervasive computing and play applications is now starting to open.

Gear S intro video: