Designing Inclusive Future Augmented Realities

Augmented and mixed reality technology [12] is developing at a rapid pace, thanks to advancements in display technologies (ranging from immersive mixed reality video-passthrough displays such as Quest Pro to advanced virtual production environments, e.g., LED volume walls), sensing technologies (such as cameras and depth sensors), and artificial intelligence for inference (such as increasingly full body tracking including facial expressions). In particular, with generative AI approaches based on large language and text-to-image models, such as GPT and Stable Diffusion as prominent examples, we could potentially not only create new, but also adapt, customize, and personalize existing, augmented and mixed reality experiences. While not without risk, we posit that this could potentially allow us to make both physical and virtual spaces more accessible and inclusive, increasing the range of tasks that provide lasting benefits to users while reducing design effort.

In this workshop, we aim to bring together researchers, designers, and practitioners passionate about a future of inclusive immersive design, who are interested in jointly exploring the benefits and tradeoffs of using augmented and mixed reality for “place making,” creating more inclusive environments that better accommodate users’ individual differences. We will approach this topic from a critical and co-design perspective by inviting diverse participants to collaboratively work on topics of immersive and inclusive design, addressing various physical and social barriers to access and issues of representation and stimulation or user engagement.

To drive the discussion, we will share two examples from workshop organizers, FUTUREGYM [13] and iGYM [4], designed to support individuals with neurodiversity, physical disabilities, as well as the professionals who support them, such as teachers and doctors. Designed for children with special needs, both systems use augmented reality via interactive large-scale floor projection. FUTUREGYM demonstrates how physical space like a gymnasium can be transformed into a place that welcomes and even supports children with neurodevelopmental disorders by virtually painting interactive games onto the physical environment [9]. iGYM demonstrates how children in wheelchairs can playfully engage in inclusive sports, competing against each other, as well as their able-bodied siblings, parents, or friends, by adapting the physics simulation of ball movement, goal size, and behavior of a virtual circle marking the peripersonal space of players and reacting to their movement and interactions in a virtual game of soccer [4].

The workshop will be structured around three main activities: (1) brainstorming to establish the landscape of inclusive immersive design, (2) scenario construction and rapid prototyping to explore and discuss new design ideas and techniques, (3) creating a future research agenda that tackles the issues systematically.

First, we will establish different notions of accessibility and inclusion in different communities within the larger HCI field. This will include both brainstorming and design critique activities to help participants develop a common understanding and identify design requirements when aiming to achieve universal design [10]. The activities will first establish how complex a space inclusive immersive design really is, and what work is needed to better understand what it takes to make a place and create a sense of belonging for a diverse user population in digitally mediated environments.

Second, an important question we ask in this workshop is to what extent generative AI could play a role in inclusive design—to help, assist, and inspire, if not ease at least some of the design effort. To that end, our goal is to better understand the different dimensions and facets of the design space of inclusive future augmented realities. Moving the discussion from user to environment and incorporating hands-on prototyping activities will allow participants to creatively explore how to augment physical spaces to achieve immersion and presence while simultaneously increasing users’ understanding and navigability of physical space. Learning from the organizers’ inclusive design research projects geared towards users with cognitive and physical impairments, FUTUREGYM [9, 13] and iGYM [4], will serve as examples and allow participants to concretize design ideas to make a space communicate directly to the users, actively engaging and simultaneously guiding them through virtual affordances using various cues, not just visual context such as signage and text.

Third, we will leverage the collective expertise of workshop participants to charter a future research agenda, adopting a sociotechnical lens as we explore and articulate possible paths to designing future augmented realities that promote equal opportunities. Throughout we will emphasize the plural notion of augmented realities both for purposes of production of ideas and to remind participants about the need as well as the possibility, thanks to AR/VR and potentially the use of generative AI, to create alternative immersive designs specific to the users. The FUTUREGYM and iGYM examples will also emphasize on the critical role of medically trained professionals and pedagogy and that there are important limits to what can and should be complemented by technology.

With this workshop, we continue an important discussion started in previous venues. In recent years, the intersection of XR advancements and emerging societal challenges has drawn considerable attention. Prior CHI and CSCW workshops have actively explored the social challenges posed by AR/VR technologies to future everyday users, emphasizing their potential impact on society and individuals as they are increasingly used in public [5, 7]. A particular focus of recent work has been the study of dark patterns, which are deceptive design strategies that can influence users’ behavior against their best interests [6]. The proposed workshop underscores the importance of examining the ethical and social implications of XR technologies, especially as they become increasingly integrated into our daily lives, where they can impact a large variety of users. Previously, Ryskeldiev et al. [11] and Biswas et al. [2] focused on how to make AR/VR more accessible and adaptive; here, we build on these workshops but specifically ask how AR and MR can be used to structure physical environments through immersive design with the goal of improving access and increasing inclusion.

At the same time, the accessibility community has recognized the growing role of AR/VR in creating inclusive experiences. Workshops at the ASSETS conference have started to explore the evolving relationship between AR/VR, accessibility, and inclusion, highlighting the need to make these technologies both more usable and also safe. Examples include a workshop by Froehlich et al. [3] that addressed urban accessibility challenges, aiming to enhance city design and equity, seeking out new data methods, standards, and tools for assessing pathway, transit, and building accessibility, facilitating inclusive urban environments at a larger scale. Another important workshop explored the nuanced tradeoffs between privacy and accessibility accommodations [1] by delving into the challenges of achieving a balance between user privacy and providing necessary accessibility features, emphasizing the complex ethical and practical considerations that researchers and designers encounter. Finally, Mankoff et al. [8] aimed to envision the future of accessibility work as accessibility research has explored a more extensive range of topics, recognizing that individuals with disabilities are often over- and under-represented in the same spaces as those without disabilities. Given that technology has similarly pervaded these spaces, their workshop explored novel questions about accessibility and the role advanced technology including AR/VR can play.

These workshops—spanning critical aspects of XR design, social challenges, accessibility, and inclusion—underscore the importance of interdisciplinary collaboration between the XR and accessibility focused communities that our CHI workshop can bring together.

We have assembled an international team of HCI and design researchers with expertise in XR technology, empathic computing, games, accessibility and inclusion, who have worked with diverse user populations including children with autism and children with motor impairments, as well as their parents, teachers, and social workers.

Michael Nebeling is an Associate Professor at the University of Michigan, School of Information, where he leads a research group focused on AR/VR in HCI. His research has contributed new design frameworks and prototyping tools for AR/VR creation, including support for accessibility and privacy to broaden the types of AR/VR interfaces that can be created and benefit their users.

Mika Oki is a Researcher at the University of Tsukuba, Institute of Systems and Information Engineering, and a member of the Artificial Intelligence Laboratory. With extensive experience in FUTUREGYM, she collaborates with specialists from diverse disciplines to develop new models of social interaction and movement, as well as design feedback mechanisms to enrich the environment for AR-aided inclusive education. Her research interests lie in HCI, AR/MR, and assistive technology.

Mirko Gelsomini is a Researcher at the University of Applied Sciences and Arts of Southern Switzerland. He is an expert in digital interactive technologies for physical interfaces, design for kids and toys, technologies for connected products and systems. He is co-author of the book Interactive Technologies and Autism.

Gillian R. Hayes is the Robert A. and Barbara L. Kleist Professor of Informatics in the School of Information and Computer Sciences and in the School of Education and School of Medicine at UC Irvine. Her research interests are in human-computer interaction, ubiquitous computing, assistive and educational technologies, and health informatics. She designs, develops, deploys, and evaluates technologies to empower people to use collected data to address real human needs in sensitive and ethically responsible ways.

Mark Billinghurst is Director of the Empathic Computing Laboratory, and Professor at the University of South Australia in Adelaide, Australia, and also at the University of Auckland in Auckland, New Zealand. He conducts research on how virtual and real worlds can be merged, publishing over 750 papers on Augmented Reality, Virtual Reality, remote collaboration, Empathic Computing, and related topics. In 2013 he was elected as a Fellow of the Royal Society of New Zealand, in 2019 was given the ISMAR Career Impact Award in recognition for lifetime contribution to AR research and commercialization, and in 2023 elected as a Fellow of the IEEE.

Kenji Suzuki is a Professor at the University of Tsukuba, Institute of Systems and Information Engineering, where he leads Artificial Intelligence Laboratory. His primary research interests include AR/MR, assistive robotics, and affective computing. A special emphasis is laid on the design of empowering people, particularly for elderly, adults and children with special needs.

Roland Graf is an Associate Professor at the University of Michigan, Stamps School of Art & Design, where he led the design of iGYM, an augmented reality system for inclusive play and exercise. He studies and designs interactive systems that reframe the body and interactivity in the physical world, enabling new interaction modalities and inspiring playful behavior in public.

The workshop is currently planned as a one-day in-person event because we believe it better facilitates networking as one of our important goals. However, the workshop format allows synchronous hybrid participation, where participants can contribute to one or more workshop activities of their choice, and could be converted to a fully remote workshop should CHI 2024 end up being held only virtually, in which case we would reduce the workshop to a half-day online event (by reducing the times allocated for breaks and activities and making the prototyping activity an optional, asynchronous component). Table 1 shows a tentative schedule for the one-day in-person version.

The workshop is structured into three major activities: (1) brainstorming to understand the inclusive immersive design landscape, (2) scenario construction / rapid prototyping of novel techniques to create inclusive and immersive user experiences around physical spaces, (3) chartering a future research agenda. We plan to start with an icebreaker activity which the workshop organizers will prepare by providing an overview of the workshop submissions, extracting themes from participants’ responses to questions about their experience with diversity, equity, and inclusion in general and accessibility and inclusion in the AR/VR space specifically.

While the workshop is currently planned as a one-day in-person event, the proposed format allows synchronous hybrid participation. As many of the workshop organizers are instructors and have previously organized workshops during the COVID-19 pandemic, we are experienced in orchestrating groups of online participants. In previous workshops, we have used Zoom breakout rooms for different group activities as well as a lounge for informal discussion and breaks. We have also used Miro boards to visually guide participants through activities. These tools can work well in hybrid settings where our workshop organizers will take on the role of facilitator and support both in-person and remote participants. While we have experience running all the types of main activities in in-person, remote, or hybrid settings, ultimately we will be able to finalize the workshop format in line with guidance provided by the CHI workshop chairs.

Participants will be provided with a workshop agenda and schedule allowing them to plan their in-person or hybrid attendance. All workshop activities will be prepared in a format that is accessible asynchronously (minutes, Zoom chat, Miro boards) and will be shared with participants as an archive post-workshop.

An important goal of the workshop is to create a community of inclusive and immersive design specialists, so we will explore opportunities such as creating a Slack team or other interactive workspace post-workshop, if participants would welcome this.

Since our call for participation only asks for position statements answering two questions central to the workshop, we are not planning workshop proceedings. However, we will raise the question if participants could imagine collaborating and contributing to a special issue in a suitable journal. Some of the workshop organizers have previously edited special issues and also negotiated acceptable fees with journals such as Frontiers.

Finally, based on the workshop activities, it is conceivable that some groups may choose to expand their concepts and prototypes towards joint research projects, and as workshop organizers we would especially welcome this and also offer our support and guidance, and if meaningful, also help facilitate collaboration.

One of the key elements of this workshop is creating more accessible and inclusive spaces. We will follow best practices (e.g., using live captions during presentations, taking minutes) but also solicit accessibility needs from participants well in advance of the workshop to determine how to best support them. Using digital tools like Zoom provides access to additional accessibility features including live chat but can also reduce the overall user experience. In addition to implementing the CHI 2024 accessibility guidelines and potentially consulting the accessibility chairs, one important opportunity will be to consult with the experienced staff in our Center for Research on Learning and Teaching (CRLT) at the University of Michigan. They can suggest feasible and effective accommodations depending on the workshop format. Based on past accommodations, these may include notetakers and human live transcription, ASL interpreter, and/or the ability to audio record the workshop for personal use on an accessibility needs basis. They can not only provide the guidelines and additional resources but also offer training to the workshop organizers.

We plan to bring together researchers, designers, and practitioners excited about the opportunities and critical about the potential risks of using AR/VR to create more accessible and inclusive spaces. By fostering dialogue and sharing insights, we can collectively explore the new opportunities afforded by XR technologies and how they can shape the landscape of accessible and inclusive design. To apply to participate in this hybrid workshop, both in-person and remote attendees are invited to submit a position statement in written form (maximum two pages) or a video response (maximum three minutes) addressing the following questions: