Publications

Abstract: This thesis is about augmented reality (AR). AR is commonly considered a technology that integrates virtual images into a user’s view of the real world. Yet, this thesis is not about such technologies. We believe a technology-based notion of AR is incomplete. In this thesis, we challenge the technology-oriented view, provide new perspectives on AR and propose a different understanding. We argue that AR is characterized by the relationships between the virtual and the real and approach AR from a fundamental, experience-focused view. By doing so, we create an unusually broad and diverse image of what AR is, or arguably could be. We discuss the fundamental characteristics of AR and the many possible manifestations it can take and propose new, imaginative AR environments that have no counterpart in a purely physical world.

Abstract: Metacognitive awareness enables people to make conscious decisions about their own cognitions, and adapt to meet task performance goals. Despite the role of metacognition in task performance, technologies that effectively augment metacognition are scarce. We explore a novel approach to augment metacognition based on making the eye’s pupil dilations, which associate with a variety of cognitions, audible via sonification in real-time. In this exploratory study, we investigated whether pupil dilation sonification can elicit metacognitive awareness. Our findings suggest that correlations between a variety of cognitions, e.g., attentional focus and depth of thinking, and sounds generated by the sonification can emerge spontaneously and by instruction. This justifies further research into the use of pupil dilation sonification as a means to augment metacognitive abilities.

Abstract: Augmented reality (AR) projects typically involve interactive systems that align virtual objects with the real world. This process is called registration and can make it seem as if virtual objects existed in the otherwise real environment. Registration is widely accepted as a defining and necessary characteristic of augmented reality. In this paper, we reconsider the need for registration on two levels. First of all, we argue that the intended presence of virtual objects in real space can be achieved without registration by an interactive AR system. Secondly, we suggest that the perceived spatial presence of virtual content in real space is not necessary for AR in the first place. We illustrate both points with examples and propose a more encompassing view of AR that focuses on relationships between the virtual and the real rather than on registration.

Abstract: People sense the world by exploiting correlations between their physical actions and the changing sensory input that results from those actions. Interfaces that translate non-human sensor data to signals that are compatible with the human senses can therefore augment our abilities to make sense of the world. This insight has recently sparked an increase in projects that explore sensemaking and the creation of novel human experiences across scientific and artistic disci- plines. However, there currently exists no constructive dialogue between artists and scientists that conduct research on this topic. In this position paper, we iden- tify the theory and practice of sensory augmentation as a domain that could benefit from such a dialogue. We argue that artistic and scientific methods can complement each other within research on sensory augmentation and identify six thematic starting points for a dialogue between the arts and sciences. We conducted a case study to explore these conjectures, in which we instigated such a dialogue on a small scale. The case study revealed that the six themes we identified as relevant for a dialogue on sensory augmentation emerge rather spontaneously in such a dialogue and that such an exchange may facilitate progress on questions that are central to the theory and practice of sensory augmentation. Overall, this position paper contributes preliminary evidence for the potential of, and a starting point for, a dialogue between the arts and sciences that advances our understand- ing of sensory augmentation and the development of applications that involve it.

Abstract: Augmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant’s view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case for an experience-focused and modalities-encompassing understanding of AR. We argue that multi-modality in AR is the norm rather than the exception, as AR environments consist of both virtual content and our real, physical, multimodal world. We explore the role multi-modal and non-visual aspects of our physical reality can play when creating AR scenarios and the possibilities and challenges that emerge when approaching AR from a modalities-encompassing perspective.

Abstract: A unique power of virtual objects is that they do not have to look, feel or behave like real objects. With this in mind, we have developed a virtual cube that is part of our real, physical environment but, unlike real objects, is invisible and non-tactile. ‘Touching’ this virtual object triggers binaural sounds that appear to originate from the exact spot where it is touched. Our initial experimentation suggests that this sound-based approach can convey the presence of virtual objects in real space and result in almost-tactile experiences. In this paper, we discuss the concept behind, implementation of and our experience with the sonically tangible cube and place our research in the context of tangible interaction, perception and augmented reality.

Abstract: What is augmented in Augmented Reality (AR)? This fundamental question has received surprisingly little attention in AR research. In this paper, we review exist- ing views and show how little consensus there is on the topic. Subsequently, we approach the question from a theoretical and technology-independent perspective that focuses on the relationships between the virtual and the real. We consider both spatial as well as content-based augmentations and distinguish between augmented environments, augmented objects, augmented humans, augmented content and augmented perception. We dis- cuss our findings and suggest possible future directions, such as research into multimodal and crossmodal AR.

Abstract: In this paper we present a new mode of interaction in ‘Spatial Augmented Reality’ (SAR) setups, using shadows as interaction input as well as display area. We claim that the combination of shadow interaction and SAR offers a novel, enjoyable and interesting way of interacting with information in a physical manner. This is especially relevant for contexts such as museum exhibits, where digital information and physical objects relate to one another. The results of our usability experiment with a zebrafish model show that users enjoy the combination of shadow interaction and SAR, as well as see a use for it in exhibition environments.

Abstract: This paper examines the interaction and influences between the virtual and the real in Augmented Reality (AR). We explore how real objects can affect virtual objects and vice versa. Our work is based on heoretical considerations, a review of existing research and artworks as well as our own initial series of experiments. We argue that virtual and real objects can not only simulate influences that exist between real entities, but also influence each other in new and imaginary ways that have no equivalent in the physical world.

Abstract: In Augmented Reality (AR), virtual and real content coexist in the same physical environment. However, in order to create AR, solely adding virtual content to a real space does not suffice. In this paper we argue that an augmentation adds and relates something virtual to something real. Subsequently, we discuss both existing and promising future relationships between the virtual and the real. We explore what AR is and what it could possibly include from a technology-independent and conceptual point of view. By comparing our take on AR with common manifestations of AR, we identify possible directions for future research and AR (art) works, such as the use of non-visual modalities and the design of novel interactions between the virtual and the real.

Abstract: Interactive art is of great relevance to the arts, sciences and technology alike. A common field of interest among researchers of different disciplines, practising artists and art institutes is the interaction between audience and artwork. This paper reviews existing research concerning interaction in interactive art and discusses its applicability for describing and classifying audience-artwork interaction. In pointing out possible future directions, we identify a need for models describing the relation between the audience’s and artwork’s actions and reactions as well as the necessity for future research looking at interaction as a continuous bi-directional process between work and audience.

Abstract: This work presents One-press control, a tactile input method for pressure-sensitive keyboards based on the detection and classification of pressing movements on the already held-down key. To seamlessly integrate the added control input with existing practices for ordinary computer keyboards, the redefined notion of virtual modifier keys is introduced. A number of application examples are given, especially to point out a potential for simplifying existing interactions by replacing modifier key combinations with single key presses. Also, a new class of interaction scenarios employing the technique is proposed, based on an interaction model named “What You Touch Is What You Get (WYTIWYG)”. Here, the proposed tactile input method is used to navigate interaction options, get full previews of potential outcomes, and then either commit to one or abort altogether – all in the space of one key depress / release cycle. The results of user testing indicate some remaining implementation issues, as well as that the technique can be learned within about a quarter of an hour of hands-on operating practice time.

Abstract: Human decisions are increasingly supported by decision support systems (DSS). Humans are required to remain “on the loop,” by monitoring and approving/rejecting machine recommendations. However, use of DSS can lead to overreliance on machines, reducing human oversight. This paper proposes “reflection machines” (RM) to increase meaningful human control. An RM provides a medical expert not with suggestions for a decision, but with questions that stimulate reflection about decisions. It can refer to data points or suggest counterarguments that are less compatible with the planned decision. RMs think against the proposed decision in order to increase human resistance against automation complacency. Building on preliminary research, this paper will (1) make a case for deriving a set of design requirements for RMs from EU regulations, (2) suggest a way how RMs could support decision-making, (3) describe the possibility of how a prototype of an RM could apply to the medical domain of chronic low back pain, and (4) highlight the importance of exploring an RM’s functionality and the experiences of users working with it.

Abstract: Rapid developments in Artificial Intelligence are leading to an increasing human reliance on machine decision making. Even in collaborative efforts with Decision Support Systems (DSSs), where a human expert is expected to make the final decisions, it can be hard to keep the expert actively involved throughout the decision process. DSSs suggest their own solutions and thus invite passive decision making. To keep humans actively ‘on’ the decision-making loop and counter overreliance on machines, we propose a ‘reflection machine’ (RM). This system asks users questions about their decision strategy and thereby prompts them to evaluate their own decisions critically. We discuss what forms RMs can take and present a proof-of-concept implementation of a RM that can produce feedback on users’ decisions in the medical and law domains. We show that the prototype requires very little domain knowledge to create reasonably intelligent critiquing questions. With this prototype, we demonstrate the technical feasibility to develop RMs and hope to pave the way for future research into their effectiveness and value.

Abstract: Dark patterns are (evil) design nudges that steer people’s behaviour through persuasive interface design. Increasingly found in cookie consent requests, they possibly undermine principles of EU privacy law. In two preregistered online experiments we investigated the effects of three common design nudges (default, aesthetic manipulation, obstruction) on users’ consent decisions and their perception of control over their personal data in these situations. In the first experiment (N = 228) we explored the effects of design nudges towards the privacy-unfriendly option (dark patterns). The experiment re- vealed that most participants agreed to all consent requests regardless of dark design nudges. Unexpectedly, despite generally low levels of perceived control, obstructing the privacy-friendly option led to more rather than less perceived control. In the second experiment (N = 255) we reversed the direc- tion of the design nudges towards the privacy-friendly option, which we title “bright patterns”. This time the obstruction and default nudges swayed peo- ple effectively towards the privacy-friendly option, while the result regarding perceived control stayed the same compared to Experiment 1. Overall, our findings support the notion that the EU’s consent requirement for tracking cookies does not work as intended. Further, we give insights into why this might be the case and recommendations on how to address the issue.

Abstract: In the early months of 2020, the deadly Covid-19 disease spread rapidly around the world. In response, national and regional governments implemented a range of emergency lockdown measures, curtailing citizens’ movements and greatly limiting economic activity. More recently, as restrictions begin to be loosened or lifted entirely, the use of so-called contact tracing apps has figured prominently in many jurisdictions’ plans to reopen society. Critics have questioned the utility of such technologies on a number of fronts, both practical and ethical. However, little has been said about the ways in which the normative design choices of app developers, and the products that result therefrom, might contribute to ethical reflection and wider political debate. Drawing from scholarship in critical design and human–computer interaction, this paper examines the development of a QR code-based tracking app called Zwaai (‘Wave’ in Dutch), where its designers explicitly positioned the app as an alternative to the predominant Bluetooth and GPS-based approaches. Through analyzing these designers’ choices, this paper argues that QR code infrastructures can work to surface a set of ethical–political seams, two of which are discussed here—responsibilization and networked (im)permanence—that more ‘seamless’ protocols like Bluetooth actively aim to bypass, and which may go otherwise unnoticed by existing ethical frameworks.

Abstract: Interactive art is of great relevance to the arts, sciences and technology alike. A common interest among researchers of different disciplines, practising artists and art institutes is the interaction between audience and artwork. This paper introduces the term audience-artwork interaction to facilitate research in this field and raises the question: how can we describe the interaction between audience and artwork? We discuss relevant research such as definitions and taxonomies of interaction, classifications of interactive systems and audience behaviour studies. A working definition of audience-artwork interaction is established. Furthermore, a model which describes interaction as a dialogue between the audience and a dynamic work is presented. We identify possible future directions and point out the need for models which describe the audiences and artworks actions and reactions as well as the relationships among them.

Abstract: What is augmented in Augmented Reality (AR)? In this paper, we review existing opinions and show how little consensus exists on this matter. Subsequently, we approach the question from a theoretical and technology-independent perspective. We identify spatial and content-based relationships between the virtual and the real as being decisive for AR and come to the conclusion that virtual content augments that to which it relates. Subsequently, we categorize different forms of AR based on what is augmented. We distinguish between augmented environments, augmented objects, augmented humans and augmented content and consider the possibility of augmented perception. The categories are illustrated with AR (art) works and conceptual differences between them are pointed out. Moreover, we discuss what the real contributes to AR and how it can shape (future) AR experiences. A summary of our findings and suggestions for future research and practice, such as research into multimodal and crossmodal AR, conclude the paper.