Speech recognition is unreliable in noisy places, compromises privacy and security when around strangers, and inaccessible to people with speech disorders. Lip reading can mitigate many of these challenges but the existing silent speech recognizers for lip reading are error prone. Developing new recognizers and acquiring new datasets is impractical for many since it requires enormous amount of time, effort, and other resources. To address these, first, we develop LipType, an optimized version of LipNet for improved speed and accuracy. We then develop an independent repair model that processes video input for poor lighting conditions, when applicable, and corrects potential errors in output for increased accuracy. We then test this model with LipType and other speech and silent speech recognizers to demonstrate its effectiveness.

We propose Auth+Track, a novel authentication model that aims to reduce redundant authentication in everyday smartphone usage. By sparse authentication and continuous tracking of the user’s status, Auth+Track eliminates the “gap” authentication between fragmented sessions and enables “Authentication Free when User is Around”. To instantiate the Auth+Track model, we present PanoTrack, a prototype that integrates body and near field hand information for user tracking. We install a fisheye camera on the top of the phone to achieve a panoramic vision that can capture both user’s body and on-screen hands. Based on the captured video stream, we develop an algorithm to extract 1) features for user tracking, including body keypoints and their temporal and spatial association, near field hand status, and 2) features for user identity assignment. The results of our user studies validate the feasibility of PanoTrack and demonstrate that Auth+Track not only improves the authentication efficiency but also enhances user experiences with better usability.

We present ElectroRing, a wearable ring-based input device that reliably detects both onset and release of a subtle finger pinch, and more generally, contact of the fingertip with the user’s skin. ElectroRing addresses a common problem in ubiquitous touch interfaces, where subtle touch gestures with little movement or force are not detected by a wearable camera or IMU. ElectroRing’s active electrical sensing approach provides a step-function-like change in the raw signal, for both touch and release events, which can be easily detected using only basic signal processing techniques. Notably, ElectroRing requires no second point of instrumentation, but only the ring itself, which sets it apart from existing electrical touch detection methods. We built three demo applications to highlight the effectiveness of our approach when combined with a simple IMU-based 2D tracking system.

We present Project Tasca, a pocket-based textile sensor that detects user input and recognizes everyday objects that a user carries in the pockets of a pair of pants (e.g., keys, coins, electronic devices, or plastic items). By creating a new fabric-based sensor capable of detecting in-pocket touch and pressure, and recognizing metallic, non-metallic, and tagged objects inside the pocket, we enable a rich variety of subtle, eyes-free, and always-available input, as well as context-driven interactions in wearable scenarios. We developed our prototype by integrating four distinct types of sensing methods, namely: inductive sensing, capacitive sensing, resistive sensing, and NFC in a multi-layer fabric structure into the form factor of a jeans pocket. Through a ten-participant study, we evaluated the performance of our prototype across 11 common objects including hands, 8 force gestures, and 30 NFC tag placements. We yielded a 92.3% personal cross-validation accuracy for object recognition, 96.4% accuracy for gesture recognition, and a 100% accuracy for detecting NFC tags at close distance . We conclude by demonstrating the interactions enabled by our pocket-based sensor in several applications.

Gestures are a promising candidate as an input modality for ambient computing where conventional input modalities such as touchscreens are not available. Existing works have focused on gesture recognition using image sensors. However, their cost, high battery consumption, and privacy concerns made cameras challenging as an always-on solution. This paper introduces an efficient gesture recognition technique using a miniaturized 60 GHz radar sensor. The technique recognizes four directional swipes and an omni-swipe using a radar chip (6.5 × 5.0 mm) integrated into a mobile phone. We developed a convolutional neural network model efficient enough for battery powered and computationally constrained processors. Its model size and inference time is less than 1/5000 compared to an existing gesture recognition technique using radar. Our evaluations with large scale datasets consisting of 558,000 gesture samples and 3,920,000 negative samples demonstrated our algorithm’s efficiency, robustness, and readiness to be deployed outside of research laboratories.

We propose a novel modality for active biometric authentication: electrical muscle stimulation (EMS). To explore this, we engineered an interactive system, which we call ElectricAuth, that stimulates the user’s forearm muscles with a sequence of electrical impulses (i.e., EMS challenge) and measures the user’s involuntary finger movements (i.e., response to the challenge). ElectricAuth leverages EMS’s intersubject variability, where the same electrical stimulation results in different movements in different users because everybody’s physiology is unique (e.g., differences in bone and muscular structure, skin resistance and composition, etc.). As such, ElectricAuth allows users to login without memorizing passwords or PINs.

ElectricAuth’s challenge-response structure makes it secure against data breaches and replay attacks, a major vulnerability facing today’s biometrics such as facial recognition and fingerprints. Furthermore, ElectricAuth never reuses the same challenge twice in authentications – in just one second of stimulation it encodes one of 68M possible challenges. In our user studies, we found that ElectricAuth resists: (1) impersonation attacks (false acceptance rate: 0.17% at 5% false rejection rate); (2) replay attacks (false acceptance rate: 0.00% at 5% false rejection rate); and, (3) synthesis attacks (false acceptance rates: 0.2-2.5%). Our longitudinal study also shows that ElectricAuth produces consistent results over time and across different humidity and muscle conditions.

We present BackTrack, a trackpad placed on the back of a smartphone to track fine-grained finger motions. Our system has a small form factor, with all the circuits encapsulated in a thin layer attached to a phone case. It can be used with any off-the-shelf smartphone, requiring no power supply or modification of the operating systems. BackTrack simply extends the finger tracking area of the front screen, without interrupting the use of the front screen. It also provides a switch to prevent unintentional touch on the trackpad. All these features are enabled by a battery-free capacitive circuit, part of which is a transparent, thin-film conductor coated on a thin glass and attached to the front screen. To ensure accurate and robust tracking, the capacitive circuits are carefully designed. Our design is based on a circuit model of capacitive touchscreens, justified through both physics-based finite-element simulation and controlled laboratory experiments. We conduct user studies to evaluate the performance of using BackTrack. We also demonstrate its use in a number of smartphone applications.

Wake-up-free techniques (e.g., Raise-to-Speak) are important for improving the voice input experience. We present ProxiMic, a close-to-mic (within 5 cm) speech sensing technique using only one microphone. With ProxiMic, a user keeps a microphone-embedded device close to the mouth and speaks directly to the device without wake-up phrases or button presses. To detect close-to-mic speech, we use the feature from pop noise observed when a user speaks and blows air onto the microphone. Sound input is first passed through a low-pass adaptive threshold filter, then analyzed by a CNN which detects subtle close-to-mic features (mainly pop noise). Our two-stage algorithm can achieve 94.1% activation recall, 12.3 False Accepts per Week per User (FAWU) with 68 KB memory size, which can run at 352 fps on the smartphone. The user study shows that ProxiMic is efficient, user-friendly, and practical.

We present Pose-on-the-Go, a full-body pose estimation system that uses sensors already found in today’s smartphones. This stands in contrast to prior systems, which require worn or external sensors. We achieve this result via extensive sensor fusion, leveraging a phone’s front and rear cameras, the user-facing depth camera, touchscreen, and IMU. Even still, we are missing data about a user’s body (e.g., angle of the elbow joint), and so we use inverse kinematics to estimate and animate probable body poses. We provide a detailed evaluation of our system, benchmarking it against a professional-grade Vicon tracking system. We conclude with a series of demonstration applications that underscore the unique potential of our approach, which could be enabled on many modern smartphones with a simple software update.

We present FaceSight, a computer vision-based hand-to-face gesture sensing technique for AR glasses. FaceSight fixes an infrared camera onto the bridge of AR glasses to provide extra sensing capability of the lower face and hand behaviors. We obtained 21 hand-to-face gestures and demonstrated the potential interaction benefits through five AR applications. We designed and implemented an algorithm pipeline that segments facial regions, detects hand-face contact (f1 score: 98.36%), and trains convolutional neural network (CNN) models to classify the hand-to-face gestures. The input features include gesture recognition, nose deformation estimation, and continuous fingertip movement. Our algorithm achieves classification accuracy of all gestures at 83.06%, proved by the data of 10 users. Due to the compact form factor and rich gestures, we recognize FaceSight as a practical solution to augment input capability of AR glasses in the future.

Handheld controllers are an essential part of VR systems. Modern sensing techniques enable them to track users’ finger movements to support natural interaction using hands. The sensing techniques, however, often fail to precisely determine whether two fingertips touch each other, which is important for the robust detection of a pinch gesture. To address this problem, we propose AtaTouch, which is a novel, robust sensing technique for detecting the closure of a finger pinch. It utilizes a change in the coupled impedance of an antenna and human fingers when the thumb and finger form a loop. We implemented a prototype controller in which AtaTouch detects the finger pinch of the grabbing hand. A user test with the prototype showed a finger-touch detection accuracy of 96.4%. Another user test with the scenarios of moving virtual blocks demonstrated low object-drop rate (2.75%) and false-pinch rate (4.40%). The results and feedback from the participants support the robustness and sensitivity of AtaTouch.

Capacitive touchscreens are near-ubiquitous in today’s touch-driven devices, such as smartphones and tablets. By using rows and columns of electrodes, specialized touch controllers are able to capture a 2D image of capacitance at the surface of a screen. For over a decade, capacitive “pixels” have been around 4 millimeters in size – a surprisingly low resolution that precludes a wide range of interesting applications. In this paper, we show how super-resolution techniques, long used in fields such as biology and astronomy, can be applied to capacitive touchscreen data. By integrating data from many frames, our software-only process is able to resolve geometric details finer than the original sensor resolution. This opens the door to passive tangibles with higher-density fiducials and also recognition of every-day metal objects, such as keys and coins. We built several applications to illustrate the potential of our approach and report the findings of a multipart evaluation.

In this paper, we propose a reconfigurable electrode, RElectrode, using a microfluidic technique that can change the geometry and material properties of the electrode to satisfy the needs for sensing a variety of different types of user input through touch/touchless gestures, pressure, temperature, and distinguish between different types of objects or liquids. Unlike the existing approaches, which depend on the specific-shaped electrode for particular sensing (e.g., coil for inductive sensing), RElectrode enables capacity, inductance, resistance/pressure, temperature, pH sensings all in a single package. We demonstrate the design and fabrication of the microfluidic structure of our RElectrode, evaluate its sensing performance through several studies, and provide some unique applications. RElectrode demonstrates technical feasibility and application values of integrating physical and biochemical properties of microfluidics into novel sensing interfaces.

We contribute a method to automate parameter configurations for chart layouts by learning from human preferences. Existing charting tools usually determine the layout parameters using predefined heuristics, producing sub-optimal layouts. People can repeatedly adjust multiple parameters (e.g., chart size, gap) to achieve visually appealing layouts. However, this trial-and-error process is unsystematic and time-consuming, without a guarantee of improvement. To address this issue, we develop Layout Quality Quantifier (LQ2), a machine learning model that learns to score chart layouts from paired crowdsourcing data. Combined with optimization techniques, LQ2 recommends layout parameters that improve the charts’ layout quality. We apply LQ2 on bar charts and conduct user studies to evaluate its effectiveness by examining the quality of layouts it produces. Results show that LQ2 can generate more visually appealing layouts than both laypeople and baselines. This work demonstrates the feasibility and usages of quantifying human preferences and aesthetics for chart layouts.

Animation helps viewers follow transitions in data graphics. When authoring animations that incorporate data, designers must carefully coordinate the behaviors of visual objects such as entering, exiting, merging and splitting, and specify the temporal rhythms of transition through staging and staggering. We present Data Animator, a system for authoring animated data graphics without programming. Data Animator leverages the Data Illustrator framework to analyze and match objects between two static visualizations, and generates automated transitions by default. Designers have the flexibility to interpret and adjust the matching results through a visual interface. Data Animator also supports the division of a complex animation into stages through hierarchical keyframes, and uses data attributes to stagger the start time and vary the speed of animating objects through a novel timeline interface. We validate Data Animator’s expressiveness via a gallery of examples, and evaluate its usability in a re-creation study with designers.

Data-driven articles—i.e., articles featuring text and supporting charts—play a key role in communicating information to the public. New storytelling formats like scrollytelling apply compelling dynamics to these articles to help walk readers through complex insights, but are challenging to craft. In this work, we investigate ways to support authors of data-driven articles using such storytelling forms via a text-chart linking strategy. From formative interviews with 6 authors and an assessment of 43 scrollytelling stories, we built VizFlow, a prototype system that uses text-chart links to support a range of dynamic layouts. We validate our text-chart linking approach via an authoring study with 12 participants using VizFlow, and a reading study with 24 participants comparing versions of the same article with different VizFlow intervention levels. Assessments showed our approach enabled a rapid and expressive authoring experience, and informed key design recommendations for future efforts in the space.

Interfaces for creating visualizations typically embrace one of several common forms. Textual specification enables fine-grained control, shelf building facilitates rapid exploration, while chart choosing promotes immediacy and simplicity. Ideally these approaches could be unified to integrate the user- and usage-dependent benefits found in each modality, yet these forms remain distinct.

We propose parameterized declarative templates, a simple abstraction mechanism over JSON-based visualization grammars, as a foundation for multimodal visualization editors. We demonstrate how templates can facilitate organization and reuse by factoring the more than 160 charts that constitute Vega-Lite’s example gallery into approximately 40 templates. We exemplify the pliability of abstracting over charting grammars by implementing—as a template—the functionality of the shelf builder Polestar (a simulacra of Tableau) and a set of templates that emulate the Google Sheets chart chooser. We show how templates support multimodal visualization editing by implementing a prototype and evaluating it through an approachability study.

Interactive visualizations are widely used in exploratory data analysis, but existing systems provide limited support for confirmatory analysis. We introduce PredictMe, a tool for belief-driven visual analysis, enabling users to draw and test their beliefs against data, as an alternative to data-driven exploration. PredictMe combines belief elicitation with traditional visualization interactions to support mixed analysis styles. In a comparative study, we investigated how these affordances impact participants’ cognition. Results show that PredictMe prompts participants to incorporate their working knowledge more frequently in queries. Participants were more likely to attend to discrepancies between their mental models and the data. However, those same participants were also less likely to engage in interactions associated with exploration, and ultimately inspected fewer visualizations and made fewer discoveries. The results suggest that belief elicitation may moderate exploratory behaviors, instead nudging users to be more deliberate in their analysis. We discuss the implications for visualization design.

Current text visualization techniques typically provide overviews of document content and structure using intrinsic properties such as term frequencies, co-occurrences, and sentence structures. Such visualizations lack conceptual overviews incorporating domain-relevant knowledge, needed when examining documents such as research articles or technical reports. To address this shortcoming, we present ConceptScope, a technique that utilizes a domain ontology to represent the conceptual relationships in a document in the form of a Bubble Treemap visualization. Multiple coordinated views of document structure and concept hierarchy with text overviews further aid document analysis. ConceptScope facilitates exploration and comparison of single and multiple documents respectively. We demonstrate ConceptScope by visualizing research articles and transcripts of technical presentations in computer science. In a comparative study with DocuBurst, a popular document visualization tool, ConceptScope was found to be more informative in exploring and comparing domain-specific documents, but less so when it came to documents that spanned multiple disciplines.

Researchers have investigated a number of strategies for capturing and analyzing data analyst event logs in order to design better tools, identify failure points, and guide users. However, this remains challenging because individual- and session-level behavioral differences lead to an explosion of complexity and there are few guarantees that log observations map to user cognition. In this paper we introduce a technique for segmenting sequential analyst event logs which combines data, interaction, and user features in order to create discrete blocks of goal-directed activity. Using measures of inter-dependency and comparisons between analysis states, these blocks identify patterns in interaction logs coupled with the current view that users are examining. Through an analysis of publicly available data and data from a lab study across a variety of analysis tasks, we validate that our segmentation approach aligns with users’ changing goals and tasks. Finally, we identify several downstream applications for our approach.

Visual analytics systems enable highly interactive exploratory data analysis. Across a range of fields, these technologies have been successfully employed to help users learn from complex data. However, these same exploratory visualization techniques make it easy for users to discover spurious findings. This paper proposes new methods to monitor a user’s analytic focus during visual analysis of structured datasets and use it to surface relevant articles that contextualize the visualized findings. Motivated by interactive analyses of electronic health data, this paper introduces a formal model of analytic focus, a computational approach to dynamically update the focus model at the time of user interaction, and a prototype application that leverages this model to surface relevant medical publications to users during visual analysis of a large corpus of medical records. Evaluation results with 24 users show that the modeling approach has high levels of accuracy and is able to surface highly relevant medical abstracts.

Course selection is a crucial activity for students as it directly impacts their workload and performance. It is also time-consuming, prone to subjectivity, and often carried out based on incomplete information. This task can, nevertheless, be assisted with computational tools, for instance, by predicting performance based on historical data. We investigate the effects of showing grade predictions to students through an interactive visualization tool. A qualitative study suggests that in the presence of predictions, students may focus too much on maximizing their performance, to the detriment of other factors such as the workload. A follow-up quantitative study explored whether these effects are mitigated by changing how predictions are conveyed. Our observations suggest the presence of a framing effect that induces students to put more effort into course selection when faced with more specific predictions. We discuss these and other findings and outline considerations for designing better data-driven course selection tools.

Time-series forecasting contributes crucial information to industrial and institutional decision-making with multivariate time-series input. Although various models have been developed to facilitate the forecasting process, they make inconsistent forecasts. Thus, it is critical to select the model appropriately. The existing selection methods based on the error measures fail to reveal deep insights into the model’s performance, such as the identification of salient features and the impact of temporal factors (e.g., periods). This paper introduces mTSeer, an interactive system for the exploration, explanation, and evaluation of multivariate time-series forecasting models. Our system integrates a set of algorithms to steer the process, and rich interactions and visualization designs to help interpret the differences between models in both model and instance level. We demonstrate the effectiveness of mTSeer through three case studies with two domain experts on real-world data, qualitative interviews with the two experts, and quantitative evaluation of the three case studies.

We present CAST, an authoring tool that enables the interactive creation of chart animations. It introduces the visual specification of chart animations consisting of keyframes that can be played sequentially or simultaneously, and animation parameters (e.g., duration, delay). Building on Canis [19], a declarative chart animation grammar that leverages data-enriched SVG charts, CAST supports auto-completion for constructing both keyframes and keyframe sequences. It also enables users to refine the animation specification (e.g., aligning keyframes across tracks to play them together, adjusting delay) with direct manipulation and other parameters for animation effects (e.g., animation type, easing function) using a control panel. In addition to describing how CAST infers recommendations for auto-completion, we present a gallery of examples to demonstrate the expressiveness of CAST and a user study to verify its learnability and usability. Finally, we discuss the limitations and potentials of CAST as well as directions for future research.

Quantifying user performance with metrics such as time and accuracy does not show the whole picture when researchers evaluate complex, interactive visualization tools. In such systems, performance is often influenced by different analysis strategies that statistical analysis methods cannot account for. To remedy this lack of nuance, we propose a novel analysis methodology for evaluating complex interactive visualizations at scale. We implement our analysis methods in reVISit, which enables analysts to explore participant interaction performance metrics and responses in the context of users’ analysis strategies. Replays of participant sessions can aid in identifying usability problems during pilot studies and make individual analysis processes salient. To demonstrate the applicability of reVISit to visualization studies, we analyze participant data from two published crowdsourced studies. Our findings show that reVISit can be used to reveal and describe novel interaction patterns, to analyze performance differences between different analysis strategies, and to validate or challenge design decisions.

Most of the existing scientific visualizations toward interpretive grammar aim to enhance customizability in either the computation stage or the rendering stage or both, while few approaches focus on the data presentation stage. Besides, most of these approaches leverage the existing components from the general-purpose programming languages (GPLs) instead of developing a standalone compiler, which pose a great challenge about learning curves for the domain experts who have limited knowledge about programming. In this paper, we propose IGScript, a novel script-based interaction grammar tool, to help build scientific data presentation animations for communication. We design a dual-space interface and a compiler which converts natural language-like grammar statements or scripts into a data story animation to make an interactive customization on script-driven data presentations, and then develop a code generator (decompiler) to translate the interactive data exploration animations back into script codes to achieve statement parameters. IGScript makes the presentation animations editable, e.g., it allows to cut, copy, paste, append, or even delete some animation clips. We demonstrate the usability, customizability, and flexibility of IGScript by a user study, four case studies conducted by using four types of commonly-used scientific data, and performance evaluations.

Raised-line graphics are tactile documents made for people with visual impairments (VI). Their exploration relies on a complex two-handed behavior. To better understand the cognitive processes underlying this exploration, we proposed a new method based on “tactile fixations”. A tactile fixation occurs when a finger is stationary within a specific spatial and temporal window. It is known that stationary fingers play an active role when exploring tactile graphics, but they have never been defined or studied before. In this study, we first defined the concept of tactile fixation, then we conducted a behavioral study with ten participants with VI in order to assess the role of tactile fixations under different conditions. The results show that tactile fixations vary according to different factors such as the graphic type as well as the involved hand and the aim of the exploration.

Accurate and effective directional feedback is crucial for an electronic traveling aid device that guides visually impaired people in walking through paths. This paper presents Tactile Compass, a hand-held device that provides continuous directional feedback with a rotatable needle pointing toward the planned direction. We conducted two lab studies to evaluate the effectiveness of the feedback solution. Results showed that, using Tactile Compass, participants could reach the target direction in place with a mean deviation of 3.03° and could smoothly navigate along paths of 60cm width, with a mean deviation from the centerline of 12.1cm. Subjective feedback showed that Tactile Compass was easy to learn and use.

Users with visual impairments find it difficult to enjoy real-time 2D interactive applications on the touchscreen. Touchscreen applications such as sports games often require simultaneous recognition of and interaction with multiple moving targets through vision. To mitigate this issue, we propose ThroughHand, a novel tactile interaction that enables users with visual impairments to interact with multiple dynamic objects in real time. We designed the ThroughHand interaction to utilize the potential of the human tactile sense that spatially registers both sides of the hand with respect to each other. ThroughHand allows interaction with multiple objects by enabling users to perceive the objects using the palm while providing a touch input space on the back of the same hand. A user study verified that ThroughHand enables users to locate stimuli on the palm with a margin of error of approximately 13 mm and effectively provides a real-time 2D interaction experience for users with visual impairments.

Teachers of the Visually Impaired (TVIs) teach academic and functional living skills simultaneously to prepare students with vision impairment to be successful and independent. Current educational tools primarily focus on academic instruction rather than this multifaceted approach needed for students. Our work aims to understand how technology can integrate behavioral skills, like independence, and support TVIs in their preferred teaching strategy. We observed elementary classrooms at a school for the blind for six weeks to study how educators design lessons and use technology to supplement their instruction in different subjects. After the observational study, we conducted remote interviews with educators to understand how technology can support students in building academic and behavioral skills in-person and remotely. Educators suggested incorporating audio feedback that motivates students to play and learn consistently, student progress tracking for parents and educators, and designing features that help students build independence and develop collaborative skills.

Visually impaired children (VI) face challenges in collaborative learning in classrooms. Robots have the potential to support inclusive classroom experiences by leveraging their physicality, bespoke social behaviors, sensors, and multimodal feedback. However, the design of social robots for mixed-visual abilities classrooms remains mostly unexplored. This paper presents a four-month-long community-based design process where we engaged with a school community. We provide insights into the barriers experienced by children and how social robots can address them. We also report on a participatory design activity with mixed-visual abilities children, highlighting the expected roles, attitudes, and physical characteristics of robots. Findings contextualize social robots within inclusive classroom settings as a holistic solution that can interact anywhere when needed and suggest a broader view of inclusion beyond disability. These include children’s personality traits, technology access, and mastery of school subjects. We finish by providing reflections on the community-based design process.

Learning to write is challenging for blind and low vision (BLV) people because of the lack of visual feedback. Regardless of the drastic advancement of digital technology, handwriting is still an essential part of daily life. Although tools designed for teaching BLV to write exist, many are expensive and require the help of sighted teachers. We propose LightWrite, a low-cost, easy-to-access smartphone application that uses voice-based descriptive instruction and feedback to teach BLV users to write English lowercase letters and Arabian digits in a specifically designed font. A two-stage study with 15 BLV users with little prior writing knowledge shows that LightWrite can successfully teach users to learn handwriting characters in an average of 1.09 minutes for each letter. After initial training and 20-minute daily practice for 5 days, participants were able to write an average of 19.9 out of 26 letters that are recognizable by sighted raters.

Standing in line is one of the most common social behaviors in public spaces but can be challenging for blind people. We propose an assistive system named LineChaser, which navigates a blind user to the end of a line and continuously reports the distance and direction to the last person in the line so that they can be followed. LineChaser uses the RGB camera in a smartphone to detect nearby pedestrians, and the built-in infrared depth sensor to estimate their position. Via pedestrian position estimations, LineChaser determines whether nearby pedestrians are standing in line, and uses audio and vibration signals to notify the user when they should start/stop moving forward. In this way, users can stay correctly positioned while maintaining social distance. We have conducted a usability study with 12 blind participants. LineChaser allowed blind participants to successfully navigate lines, significantly increasing their confidence in standing in lines.

People with vision impairments access smartphones with the help of screen reader apps such as TalkBack for Android and VoiceOver for iPhone. Prior research has mostly focused on understanding touchscreen phone adoption and typing performance of novice blind users by logging their real-world smartphone usage. Understanding smartphone usage pattern and practices of expert users can help in developing tools and tutorials for transitioning novice and intermediate users to expert users. In this work, we logged smartphone usage data of eight expert Android smartphone users with visual impairments for four weeks, and then interviewed them. This paper presents a detailed analysis that uncovered novel usage patterns, such as extensive usage of directional gestures, reliance on voice and external keyboard for text input, and repurposed explore by touch for single-tap. We conclude with design recommendations to inform the future of mobile accessibility, including hardware guidelines and rethinking accessible software design.

Visual semantics provide spatial information like size, shape, and position, which are necessary to understand and efficiently use interfaces and documents. Yet little is known about whether blind and low-vision (BLV) technology users want to interact with visual affordances, and, if so, for which task scenarios. In this work, through semi-structured and task-based interviews, we explore preferences, interest levels, and use of visual semantics among BLV technology users across two device platforms (smartphones and laptops), and information seeking and interactions common in apps and web browsing. Findings show that participants could benefit from access to visual semantics for collaboration, navigation, and design. To learn this information, our participants used trial and error, sighted assistance, and features in existing screen reading technology like touch exploration. Finally, we found that missing information and inconsistent screen reader representations of user interfaces hinder learning. We discuss potential applications and future work to equip BLV users with necessary information to engage with visual semantics.

Tactile guiding surfaces in the built environment have held a contentious place in the process of navigation by people who are blind or visually impaired. Despite standards for tactile guiding surfaces, problems persist with inconsistent implementation, perception, and geographic orientation. We investigate the role of tactile cues in non-visual navigation and attitudes surrounding guiding surfaces through a survey of 67 people with vision impairments and ten interviews with navigation and public accessibility experts. Our participants revealed several opportunities to augment existing tactile surfaces while envisioning novel multimodal feedback solutions in immediately relevant contexts. We also propose an approach for designing and exploring low cost, multimodal tactile surfaces, which we call navtiles. Finally, we discuss practical aspects of implementation for new design alternatives such as standardization, installation, movability, discoverability, and a need for transparency. Collectively, these insights contribute to the production and implementation of novel multimodal navigation aids.

Keyboard-based emoji entry can be challenging for people with visual impairments: users have to sequentially navigate emoji lists using screen readers to find their desired emojis, which is a slow and tedious process. In this work, we explore the design and benefits of emoji entry with speech input, a popular text entry method among people with visual impairments. After conducting interviews to understand blind or low vision (BLV) users’ current emoji input experiences, we developed Voicemoji, which (1) outputs relevant emojis in response to voice commands, and (2) provides context-sensitive emoji suggestions through speech output. We also conducted a multi-stage evaluation study with six BLV participants from the United States and six BLV participants from China, finding that Voicemoji significantly reduced entry time by 91.2% and was preferred by all participants over the Apple iOS keyboard. Based on our findings, we present Voicemoji as a feasible solution for voice-based emoji entry.

While the growth of financial technologies (FinTech) is making the flow of money faster, easier, and more secure, such technologies are often unable to serve many countries due to the global political environment. Despite its severe impact, this issue has remained understudied in the HCI literature. We address this gap by presenting our findings from a three-month-long ethnography with the Iranian community in Toronto, Canada. We present their struggles in transferring money to and from their home country - a process that entails financial loss, fear, uncertainty, and privacy breaches. We also outline the informal workarounds that allow this community to circumvent these challenges, along with the associated hassles. This paper contributes to broadening the scope of FinTech in the HCI literature by connecting it with the politics surrounding transnational transactions. We discuss the design implications of our findings and their contribution to the broader interests of HCI in mobilities and social justice.

AI models are increasingly applied in high-stakes domains like health and conservation. Data quality carries an elevated significance in high-stakes AI due to its heightened downstream impact, impacting predictions like cancer detection, wildlife poaching, and loan allocations. Paradoxically, data is the most under-valued and de-glamorised aspect of AI. In this paper, we report on data practices in high-stakes AI, from interviews with 53 AI practitioners in India, East and West African countries, and USA. We define, identify, and present empirical evidence on Data Cascades—compounding events causing negative, downstream effects from data issues—triggered by conventional AI/ML practices that undervalue data quality. Data cascades are pervasive (92% prevalence), invisible, delayed, but often avoidable. We discuss HCI opportunities in designing and incentivizing data excellence as a first-class citizen of AI, resulting in safer and more robust systems for all.

Head-worn displays (HWDs) offer their users high mobility, hands-free operation, and “see-what-I-see” features. In the prehospital environment, emergency medical services (EMS) staff could benefit from the unique characteristics of HWDs. We conducted a field study to analyze work practices of EMS staff and the potential of HWDs to support their activities. Based on our observations and the comments of EMS staff, we propose three use cases for HWDs in the prehospital environment. They are (1) enhanced communication between different care providers, (2) hands-free access to clinical monitoring and imaging, (3) and improved realism of training scenarios. We conclude with a set of design considerations and suggest that for the successful implementation of HWDs in EMS environments, researchers, designers, and clinical stakeholders should consider the harsh outdoor environment in which HWDs will be used, the extensive workload of staff, the complex collaboration performed, privacy requirements, and the high variability of work.

With a plethora of off-the-shelf smart home devices available commercially, people are increasingly taking a do-it-yourself approach to configuring their smart homes. While this allows for customization, users responsible for smart home configuration often end up with more control over the devices than other household members. This separates those who introduce new functionality to the smart home (pilot users) from those who do not (passenger users). To investigate the prevalence and impact of pilot-passenger user relationships, we conducted a Mechanical Turk survey and a series of one-hour interviews. Our results suggest that pilot-passenger relationships are common in multi-user households and shape how people form habits around devices. We find from interview data that smart homes reflect the values of their pilot users, making it harder for passenger users to incorporate their devices into daily life. We conclude the paper with design recommendations to improve passenger and pilot user experience.

Voice-based Conversational Agents (VCA) have served as personal assistants that support individuals with special needs. Adolescents with Autism Spectrum Disorder (ASD) may also benefit from VCAs to deal with their everyday needs and challenges, ranging from self-care to social communications. In this study, we explored how VCAs could encourage adolescents with ASD in navigating various aspects of their daily lives through the two-week use of VCAs and a series of participatory design workshops. Our findings demonstrated that VCAs could be an engaging, empowering, emancipating tool that supports adolescents with ASD to address their needs, personalities, and expectations, such as promoting self-care skills, regulating negative emotions, and practicing conversational skills. We propose implications of using off-the-shelf technologies as a personal assistant to ASD users in Assistive Technology design. We suggest design implications for promoting positive opportunities while mitigating the remaining challenges of VCAs for adolescents with ASD.

Most children who are blind live in low-resource settings and attend schools that have poor technical infrastructure, overburdened teachers, and outdated curriculum. Our work explores the role digital games can play to develop digital skills of such children in the early grades. Recognizing the critical role of teachers in introducing children to technology, we conducted a mixed-methods study to examine which attributes of digital games teachers find useful for children and what challenges they perceive in integrating digital games in schools for the blind. Our findings indicate that teachers prefer games that align well with curriculum objectives, promote learning, improve soft skills, and increase engagement with computers. Despite being overburdened and lacking technological support, teachers expressed strong enthusiasm to integrate these games in school curriculum and schedule. We conclude by discussing design implications for designers of accessible games in low-resource settings.

Disability professionals could use digital fabrication tools to provide customised assistive technologies or accessible media beneficial to the education of Blind or visually impaired youth. However, there is little documentation of long-term practices with these tools by professionals in this field, limiting our ability to support their work. We report on such practices in a French organisation, providing disability educational services and using digital fabrication since 2013, for six years. We trace how professionals defined how digital fabrication could and should be used through a range of projects, based on pedagogical uses and the constraints in creation, production and maintenance. We outline new research perspectives going beyond 3D printers and its promises of automation to embrace hybrid approaches currently supported by laser cutters, the learning and documentation process, and the production of accessible tactile media at a regional or national scale.

Technology has become increasingly pervasive in the creative and experimental environment of the avant-garde fashion runway, particularly in relation to its garments. However, several disciplines are often necessary when exploring technologies for the construction of expressive garments (e.g. garments that respond to their environment), creating a barrier for fashion designers that has limited their ability to leverage new technologies. To help overcome this barrier, we designed and deployed Brookdale, a prototyping system for wearable technology consisting of new plug-and-play hardware that can be programmed using drag-and-drop software. Brookdale was created using a 24-week participatory design process with 17 novice fashion-tech designers. At the end of the 24 week process, designers showcased their Brookdale-enhanced garment collections at an avant-garde fashion-tech runway show in New York City. We report on the experiences, outcomes, and lessons learned throughout this process, and describe results from interviews with the fashion-tech designers 16 weeks after the fashion show, demonstrating the lasting positive impact of Brookdale.

The industry for children’s apps is thriving at the cost of children’s privacy: these apps routinely disclose children’s data to multiple data trackers and ad networks. As children spend increasing time online, such exposure accumulates to long-term privacy risks. In this paper, we used a mixed-methods approach to investigate why this is happening and how developers might change their practices. We base our analysis against 5 leading data protection frameworks that set out requirements and recommendations for data collection in children’s apps. To understand developers’ perspectives and constraints, we conducted 134 surveys and 20 semi-structured interviews with popular Android children’s app developers. Our analysis revealed that developers largely respect children’s best interests; however, they have to make compromises due to limited monetisation options, perceived harmlessness of certain third-party libraries, and lack of availability of design guidelines. We identified concrete approaches and directions for future research to help overcome these barriers.

Repeated exposure to poor air quality in indoor environments such as office, home, and classroom can have substantial adverse effects on our health and productivity. The problem is especially recognized in closed indoor spaces shared by several people. We have studied the evolution of carbon dioxide level in office-meeting spaces, during 1007 meeting sessions. The collected data is employed to examine machine learning models aimed to indicate the CO2 evolution pattern and to forecast when fresh air should be supplied. In addition, to gain insight into the relations and interdependencies of social factors in meetings that may influence the users’ perception of an interactive solution, we have conducted a series of online surveys. Building on the results of the two studies, a solution is proposed that predicts the evolution of air quality in naturally-ventilated meeting rooms and engages the users in preventive actions when risk is forecast.

It is well documented that people living with mental health conditions are more likely to experience financial difficulties. When explaining this association, emphasis has often been placed on financial capability, i.e. the capacity of those living with poor mental health to manage their money. This paper challenges such capability-based explanations by reporting on a diary study and interviews with 14 people who self-identify as living with a mental health condition. We focused on their experiences of financial technology use, and explored the role technology played in their strategies to minimise the impact of mental health on their economic circumstances. Rather than lacking capability, participants’ practices revealed shortcomings of existing financial technologies and how they sought to work around these. We conclude by providing a set of design directions for technologies that engage those living with poor mental health not as vulnerable targets for financial inclusion, but as full financial citizens.

Smart Donations is a blockchain-based platform that offers users ‘contracts’ that donate funds to certain causes in response to real-world events e.g., whenever an earthquake is detected or an activist tweets about refugees. We designed Smart donations with Oxfam Australia, trialled it for 8-weeks with 86 people, recorded platform analytics and qualitatively analysed questionnaires and interviews about user experiences. Temporal qualities emerge when automation enforces conditions that contributed to participants’ awareness of events that are usually unconscious, and senses of immediacy in contributing to crisis response and ongoing involvement in situations far-away while awaiting conditions to be met. We suggest data driven automation can reveal diverse temporal registers, in real-world phenomena, sociality, morality and everyday life, which contributes to experiencing a ‘right time’ to donate that is not limited to productivity or efficiency. Thus, we recommend a sensitivity to right time in designing for multiple temporalities in FinTech more generally.

Running is a widely popular physical activity that offers many health benefits. As runners progress with their training, understanding one’s own body becomes a key concern in achieving wellbeing through running. While extensive bodily sensing opportunities exist for runners, understanding complex sensor data is a challenge. In this paper, we investigate how data from shoe-worn sensors can be visualised to empower runners to improve their technique. We designed GraFeet—an augmented running shoe that visualises kinesiological data about the runner’s feet and gait. We compared our prototype with a standard sensor dashboard in a user study where users ran with the sensor and analysed the generated data after the run. GraFeet was perceived as more usable; producing more insights and less confusion in the users. Based on our inquiry, we contribute findings about using data from body-worn sensors to support physically active individuals.

Maximizing system scalability and quality are sometimes at odds. This work provides an example showing scalability and quality can be achieved at the same time in instructional design, contrary to what instructors may believe or expect. We situate our study in the education of HCI methods, and provide suggestions to improve active learning within the HCI education community. While designing learning and assessment activities, many instructors face the choice of using open-ended or close-ended activities. Close-ended activities such as multiple-choice questions (MCQs) enable automated feedback to students. However, a survey with 22 HCI professors revealed a belief that MCQs are less valuable than open-ended questions, and thus, using them entails making a quality sacrifice in order to achieve scalability. A study with 178 students produced no evidence to support the teacher belief. This paper indicates more promise than concern in using MCQs for scalable instruction and assessment in at least some HCI domains.

We designed and tested an attention-aware learning technology (AALT) that detects and responds to mind wandering (MW), a shift in attention from task-related to task-unrelated thoughts, that is negatively associated with learning. We leveraged an existing gaze-based mind wandering detector that uses commercial off the shelf eye tracking to inform real-time interventions during learning with an Intelligent Tutoring System in real-world classrooms. The intervention strategies, co-designed with students and teachers, consisted of using student names, reiterating content, and asking questions, with the aim to reengage wandering minds and improve learning. After several rounds of iterative refinement, we tested our AALT in two classroom studies with 287 high-school students. We found that interventions successfully reoriented attention, and compared to two control conditions, reduced mind wandering, and improved retention (measured via a delayed assessment) for students with low prior-knowledge who occasionally (but not excessively) mind wandered. We discuss implications for developing gaze-based AALTs for real-world contexts.

Learning to recognize and apply programming patterns — reusable abstractions of code — is critical to becoming a proficient computer scientist. However, many introductory Computer Science courses do not teach patterns, in part because teaching these concepts requires significant curriculum changes. As an alternative, we explore how a novel user interface for practicing coding — Faded Parsons Problems — can support introductory Computer Science students in learning to apply programming patterns. We ran a classroom-based study with 237 students which found that Faded Parsons Problems, or rearranging and completing partially blank lines of code into a valid program, are an effective exercise interface for teaching programming patterns, significantly surpassing the performance of the more standard approaches of code writing and code tracing exercises. Faded Parsons Problems also improve overall code writing ability at a comparable level to code writing exercises, but are preferred by students.

Computational thinking (CT) education reaches only a fraction of young children, in part because CT learning tools often require expensive hardware or fluent literacy. Informed by needfinding interviews, we developed a voice-guided smartphone application leveraging storytelling as a creative activity by which to teach CT concepts to 5- to 8-year-old children. The app includes two storytelling games where users create and listen to stories as well as four CT games where users then modify those stories to learn about sequences, loops, events, and variables. We improved upon the app design through wizard-of-oz testing (N = 28) and iterative design testing (N = 22) before conducting an evaluation study (N = 22). Children were successfully able to navigate the app, effectively learn about the target computing concepts, and, after using the app, children demonstrated above-chance performance on a near transfer CT concept recognition task.

Programming error messages play an important role in learning to program. The cycle of program input and error message response completes a loop between the programmer and the compiler/interpreter and is a fundamental interaction between human and computer. However, error messages are notoriously problematic, especially for novices. Despite numerous guidelines citing the importance of message readability, there is little empirical research dedicated to understanding and assessing it. We report three related experiments investigating factors that influence programming error message readability. In the first two experiments we identify possible factors, and in the third we ask novice programmers to rate messages using scales derived from these factors. We find evidence that several key factors significantly affect message readability: message length, jargon use, sentence structure, and vocabulary. This provides novel empirical support for previously untested long-standing guidelines on message design, and informs future efforts to create readability metrics for programming error messages.

Program tracing, or mentally simulating a program on concrete inputs, is an important part of general program comprehension. Programs involve many kinds of virtual state that must be held in memory, such as variable/value pairs and a call stack. In this work, we examine the influence of short-term working memory (WM) on a person’s ability to remember program state during tracing. We first confirm that previous findings in cognitive psychology transfer to the programming domain: people can keep about 7 variable/value pairs in WM, and people will accidentally swap associations between variables due to WM load. We use a restricted focus viewing interface to further analyze the strategies people use to trace through programs, and the relationship of tracing strategy to WM. Given a straight-line program, we find half of our participants traced a program from the top-down line-by-line (linearly), and the other half start at the bottom and trace upward based on data dependencies (on-demand). Participants with an on-demand strategy made more WM errors while tracing straight-line code than with a linear strategy, but the two strategies contained an equal number of WM errors when tracing code with functions. We conclude with the implications of these findings for the design of programming tools: first, programs should be analyzed to identify and refactor human-memory-intensive sections of code. Second, programming environments should interactively visualize variable metadata to reduce WM load in accordance with a person’s tracing strategy. Third, tools for program comprehension should enable externalizing program state while tracing.

HCI has historically provided little support for moving from fieldwork insights or theories to design outcomes. Having witnessed many students struggle and then justify their designs with a form of marketing hype, we developed a supporting approach of “field theories”. A field theory is a working theory about salient interactions in a particular domain and sensitizing concepts in order to frame design investigations. It is presented visually in a field theory diagram to support succinct communication and critique. Studying use of design prototypes that have been informed by a field theory helps to reflect upon and refine the theory. In this paper we present examples from our HCI classes and reflections based on interviews with students. We discuss how field theories offer an orientation in the spirit of a ‘bricoleur’ who harnesses elements of theory and practice to produce deeper understandings and more fitting outcomes for the task at hand.

We present the design and evaluation of a web-based intelligent writing assistant that helps students recognize their revisions of argumentative essays. To understand how our revision assistant can best support students, we have implemented four versions of our system with differences in the unit span (sentence versus sub-sentence) of revision analysis and the level of feedback provided (none, binary, or detailed revision purpose categorization). We first discuss the design decisions behind relevant components of the system, then analyze the efficacy of the different versions through a Wizard of Oz study with university students. Our results show that while a simple interface with no revision feedback is easier to use, an interface that provides a detailed categorization of sentence-level revisions is the most helpful based on user survey data, as well as the most effective based on improvement in writing outcomes.

Tangible interfaces have much potential for engendering shared interaction and reflection, as well as for promoting playful experiences. How can their properties be capitalised on to enable students to reflect on their learning, both individually and together, throughout learning sessions? This Research through Design paper describes our development of EvalMe, a flexible, tangible tool aimed at being playful, enjoyable to use and enabling children to reflect on their learning, both in the moment and after a learning session has ended. We discuss the insights gained through the process of designing EvalMe, co-defining its functionality with two groups of collaborators and deploying it in two workshop settings. Through this process, we map key contextual considerations for the design of technologies for in situ evaluation of learning experiences. Finally, we discuss how tangible evaluation technologies deployed throughout a learning session, can positively contribute to students’ reflection about their learning.

Novice programmers need differentiated assessments (such as adaptive Parsons problems) to maximize their ability to learn how to program. Parsons problems require learners to place mixed-up code blocks in the correct order to solve a problem. We conducted a within-subjects experiment to compare the efficiency and cognitive load of solving adaptive Parsons problems versus writing the equivalent (isomorphic) code. Undergraduates were usually more significantly efficient at solving a Parsons problem than writing the equivalent code, but not when the solution to the Parsons problem was unusual. This has implications for problem creators. This paper also reports on the mean cognitive load ratings of the two problem types and the relationship between efficiency and cognitive load ratings. Lastly, it reports on think-aloud observations of 11 students solving both adaptive Parsons problems and write-code problems and the results from an end-of-course student survey.

Conversational programmers want to learn about code primarily to communicate with technical co-workers, not to develop software. However, existing instructional materials don’t meet the needs of conversational programmers because they prioritize syntax and semantics over concepts and applications. This mismatch results in feelings of failure and low self-efficacy. To motivate conversational programmers, we propose purpose-first programming, a new approach that focuses on learning a handful of domain-specific code patterns and assembling them to create authentic and useful programs. We report on the development of a purpose-first programming prototype that teaches five patterns in the domain of web scraping. We show that learning with purpose-first programming is motivating for conversational programmers because it engenders a feeling of success and aligns with these learners’ goals. Purpose-first programming learning enabled novice conversational programmers to complete scaffolded code writing, debugging, and explaining activities after only 30 minutes of instruction.

Surveillance of communication between incarcerated and non-incarcerated people has steadily increased, enabled partly by technological advancements. Third-party vendors control communication tools for most U.S. prisons and jails and offer surveillance capabilities beyond what individual facilities could realistically implement. Frequent communication with family improves mental health and post-carceral outcomes for incarcerated people, but does discomfort about surveillance affect how their relatives communicate with them? To explore this and the understanding, attitudes, and reactions to surveillance, we conducted 16 semi-structured interviews with participants who have incarcerated relatives. Among other findings, we learn that participants communicate despite privacy concerns that they felt helpless to address. We also observe inaccuracies in participants’ beliefs about surveillance practices. We discuss implications of inaccurate understandings of surveillance, misaligned incentives between end-users and vendors, how our findings enhance ongoing conversations about carceral justice, and recommendations for more privacy-sensitive communication tools.

Increasingly, icons are being proposed to concisely convey privacy-related information and choices to users. However, complex privacy concepts can be difficult to communicate. We investigate which icons effectively signal the presence of privacy choices. In a series of user studies, we designed and evaluated icons and accompanying textual descriptions (link texts) conveying choice, opting-out, and sale of personal information — the latter an opt-out mandated by the California Consumer Privacy Act (CCPA). We identified icon-link text pairings that conveyed the presence of privacy choices without creating misconceptions, with a blue stylized toggle icon paired with “Privacy Options” performing best. The two CCPA-mandated link texts (“Do Not Sell My Personal Information” and “Do Not Sell My Info”) accurately communicated the presence of do-not-sell opt-outs with most icons. Our results provide insights for the design of privacy choice indicators and highlight the necessity of incorporating user testing into policy making.

“Notice and choice” is the predominant approach for data privacy protection today. There is considerable user-centered research on providing effective privacy notices but not enough guidance on designing privacy choices. Recent data privacy regulations worldwide established new requirements for privacy choices, but system practitioners struggle to implement legally compliant privacy choices that also provide users meaningful privacy control. We construct a design space for privacy choices based on a user-centered analysis of how people exercise privacy choices in real-world systems. This work contributes a conceptual framework that considers privacy choice as a user-centered process as well as a taxonomy for practitioners to design meaningful privacy choices in their systems. We also present a use case of how we leverage the design space to finalize the design decisions for a real-world privacy choice platform, the Internet of Things (IoT) Assistant, to provide meaningful privacy control in the IoT.

The deployment of technologies to track and mitigate the spread COVID-19 has surfaced tensions between individual autonomy and the collective good. These tensions reflect a conflict between two central concerns: (i) effectively controlling the spread of the pandemic and (ii) respecting individual rights, values, and freedoms. We explored these tensions in an online experiment (n = 389) designed to identify the influence of social orientation and communicative framing on perceptions and expected use of pandemic-tracking apps. We found that social orientation is a statistically significant predictor of app perception and expected use, with collectivist social orientation associated with higher levels and individualist social orientation with lower levels for both aspects. We found interactions between social orientation and communicative framing, as well as a connection between privacy concerns and expected duration of app use. Our findings hold important implications for the design, deployment, and adoption of technology for the public good. Shaping the post-pandemic social contract requires considering the long-term sociocultural impact of these technological solutions.

Online privacy policies should enable users to make informed decisions. Current text policies, however, lack usability: users often miss crucial information and consent to them without reading. Visual representation formats may increase comprehension, but are rarely used in practice. In an iterative design process we gathered qualitative feedback on typical policy contents and on existing and newly designed representation formats. We developed design guidelines and a Visual Interactive Privacy Policy based on the Privacy Policy Nutrition Label enriched with control options and further interactive elements. In an empirical evaluation, both visual representations received higher ratings of attractiveness, stimulation, novelty and transparency compared to a standard policy long text. Interactivity improved time spent with the policy. There were no effects on conversion rate, perceived control or perceived trust, efficiency and perspicuity. More research is needed, especially with regard to the cost-benefit ratio of visual privacy policies.

This paper addresses the question whether the recently proposed approach of concise privacy notices in apps and on websites is effective in raising user awareness. To assess the effectiveness in a realistic setting, we included concise notices in a fictitious but realistic fitness tracking app and asked participants recruited from an online panel to provide their feedback on the usability of the app as a cover story. Importantly, after giving feedback, users were also asked to recall the data practices described in the notices. The experimental setup included the variation of different levels of saliency and riskiness of the privacy notices. Based on a total sample of 2,274 participants, our findings indicate that concise privacy notices are indeed a promising approach to raise user awareness for privacy information when displayed in a salient way, especially in case the notices describe risky data practices. Our results may be helpful for regulators, user advocates and transparency-oriented companies in creating or enforcing better privacy transparency towards average users that do not read traditional privacy policies.

Homomorphic encryption, secure multi-party computation, and differential privacy are part of an emerging class of Privacy Enhancing Technologies which share a common promise: to preserve privacy whilst also obtaining the benefits of computational analysis. Due to their relative novelty, complexity, and opacity, these technologies provoke a variety of novel questions for design and governance. We interviewed researchers, developers, industry leaders, policymakers, and designers involved in their deployment to explore motivations, expectations, perceived opportunities and barriers to adoption. This provided insight into several pertinent challenges facing the adoption of these technologies, including: how they might make a nebulous concept like privacy computationally tractable; how to make them more usable by developers; and how they could be explained and made accountable to stakeholders and wider society. We conclude with implications for the development, deployment, and responsible governance of these privacy-preserving computation techniques.

We present PriView, a concept that allows privacy-invasive devices in the users’ vicinity to be visualised. PriView is motivated by an ever-increasing number of sensors in our environments tracking potentially sensitive data (e.g., audio and video). At the same time, users are oftentimes unaware of this, which violates their privacy. Knowledge about potential recording would enable users to avoid accessing such areas or not to disclose certain information. We built two prototypes: a) a mobile application capable of detecting smart devices in the environment using a thermal camera, and b) VR mockups of six scenarios where PriView might be useful (e.g., a rental apartment). In both, we included several types of visualisation. Results of our lab study (N=24) indicate that users prefer simple, permanent indicators while wishing for detailed visualisations on demand. Our exploration is meant to support future designs of privacy visualisations for varying smart environments.

The COVID-19 pandemic has fueled the development of smartphone applications to assist disease management. Many “corona apps” require widespread adoption to be effective, which has sparked public debates about the privacy, security, and societal implications of government-backed health applications. We conducted a representative online study in Germany (n = 1003), the US (n = 1003), and China (n = 1019) to investigate user acceptance of corona apps, using a vignette design based on the contextual integrity framework. We explored apps for contact tracing, symptom checks, quarantine enforcement, health certificates, and mere information. Our results provide insights into data processing practices that foster adoption and reveal significant differences between countries, with user acceptance being highest in China and lowest in the US. Chinese participants prefer the collection of personalized data, while German and US participants favor anonymity. Across countries, contact tracing is viewed more positively than quarantine enforcement, and technical malfunctions negatively impact user acceptance.

The shutdown measures necessary to stop the spread of COVID-19 have amplified the role of technology in intimate partner violence (IPV). Survivors may be forced to endure lockdowns with their abusers, intensifying the dangers of technology-enabled abuse (e.g. stalking, harassment, monitoring, surveillance). They may also be forced to rely on potentially compromised devices to reach support networks: a dangerous dilemma for digital safety. This qualitative study examines how technologists with computer security expertise provided remote assistance to IPV survivors during the pandemic. Findings from 24 consults with survivors and five focus groups with technologist consultants show how remote delivery of technology support services raised three fundamental challenges: (1) ensuring safety for survivors and consultants; (2) assessing device security over a remote connection; and (3) navigating new burdens for consultants, including emotional labor. We highlight implications for HCI researchers creating systems that enable access to remote expert services for vulnerable people.

Automated decision support can accelerate tedious tasks as users can focus their attention where it is needed most. However, a key concern is whether users overly trust or cede agency to automation. In this paper, we investigate the effects of introducing automation to annotating clinical texts — a multi-step, error-prone task of identifying clinical concepts (e.g., procedures) in medical notes, and mapping them to labels in a large ontology. We consider two forms of decision aid: recommending which labels to map concepts to, and pre-populating annotation suggestions. Through laboratory studies, we find that 18 clinicians generally build intuition of when to rely on automation and when to exercise their own judgement. However, when presented with fully pre-populated suggestions, these expert users exhibit less agency: accepting improper mentions, and taking less initiative in creating additional annotations. Our findings inform how systems and algorithms should be designed to mitigate the observed issues.

The popularity of racket sports (e.g., tennis and table tennis) leads to high demands for data analysis, such as notational analysis, on player performance. While sports videos offer many benefits for such analysis, retrieving accurate information from sports videos could be challenging. In this paper, we propose EventAnchor, a data analysis framework to facilitate interactive annotation of racket sports video with the support of computer vision algorithms. Our approach uses machine learning models in computer vision to help users acquire essential events from videos (e.g., serve, the ball bouncing on the court) and offers users a set of interactive tools for data annotation. An evaluation study on a table tennis annotation system built on this framework shows significant improvement of user performances in simple annotation tasks on objects of interest and complex annotation tasks requiring domain knowledge.

To ensure accountability and mitigate harm, it is critical that diverse stakeholders can interrogate black-box automated systems and find information that is understandable, relevant, and useful to them. In this paper, we eschew prior expertise- and role-based categorizations of interpretability stakeholders in favor of a more granular framework that decouples stakeholders’ knowledge from their interpretability needs. We characterize stakeholders by their formal, instrumental, and personal knowledge and how it manifests in the contexts of machine learning, the data domain, and the general milieu. We additionally distill a hierarchical typology of stakeholder needs that distinguishes higher-level domain goals from lower-level interpretability tasks. In assessing the descriptive, evaluative, and generative powers of our framework, we find our more nuanced treatment of stakeholders reveals gaps and opportunities in the interpretability literature, adds precision to the design and comparison of user studies, and facilitates a more reflexive approach to conducting this research.

Training datasets fundamentally impact the performance of machine learning (ML) systems. Any biases introduced during training (implicit or explicit) are often reflected in the system's behaviors leading to questions about fairness and loss of trust in the system. Yet, information on training data is rarely communicated to stakeholders. In this work, we explore the concept of data-centric explanations for ML systems that describe the training data to end-users. Through a formative study, we investigate the potential utility of such an approach, including the information about training data that participants find most compelling. In a second study, we investigate reactions to our explanations across four different system scenarios. Our results suggest that data-centric explanations have the potential to impact how users judge the trustworthiness of a system and to assist users in assessing fairness. We discuss the implications of our findings for designing explanations to support users’ perceptions of ML systems.

Generative Adversarial Networks (GANs) can automatically generate quality images from learned model parameters. However, it remains challenging to explore and objectively assess the quality of all possible images generated using a GAN. Currently, model creators evaluate their GANs via tedious visual examination of generated images sampled from narrow prior probability distributions on model parameters. Here, we introduce an interactive method to explore and sample quality images from GANs. Our first two user studies showed that participants can use the tool to explore a GAN and select quality images. Our third user study showed that images sampled from a posterior probability distribution using a Markov Chain Monte Carlo (MCMC) method on parameters of images collected in our first study resulted in on average higher quality and more diverse images than existing baselines. Our work enables principled qualitative GAN exploration and evaluation.

The advent of artificial intelligence (AI) and machine learning (ML) bring human-AI interaction to the forefront of HCI research. This paper argues that games are an ideal domain for studying and experimenting with how humans interact with AI. Through a systematic survey of neural network games (n = 38), we identified the dominant interaction metaphors and AI interaction patterns in these games. In addition, we applied existing human-AI interaction guidelines to further shed light on player-AI interaction in the context of AI-infused systems. Our core finding is that AI as play can expand current notions of human-AI interaction, which are predominantly productivity-based. In particular, our work suggests that game and UX designers should consider flow to structure the learning curve of human-AI interaction, incorporate discovery-based learning to play around with the AI and observe the consequences, and offer users an invitation to play to explore new forms of human-AI interaction.

This paper addresses an under-explored problem of AI-assisted decision-making: when objective performance information of the machine learning model underlying a decision aid is absent or scarce, how do people decide their reliance on the model? Through three randomized experiments, we explore the heuristics people may use to adjust their reliance on machine learning models when performance feedback is limited. We find that the level of agreement between people and a model on decision-making tasks that people have high confidence in significantly affects reliance on the model if people receive no information about the model’s performance, but this impact will change after aggregate-level model performance information becomes available. Furthermore, the influence of high confidence human-model agreement on people’s reliance on a model is moderated by people’s confidence in cases where they disagree with the model. We discuss potential risks of these heuristics, and provide design implications on promoting appropriate reliance on AI.

Data science (DS) projects often follow a lifecycle that consists of laborious tasks for data scientists and domain experts (e.g., data exploration, model training, etc.). Only till recently, machine learning(ML) researchers have developed promising automation techniques to aid data workers in these tasks. This paper introduces AutoDS, an automated machine learning (AutoML) system that aims to leverage the latest ML automation techniques to support data science projects. Data workers only need to upload their dataset, then the system can automatically suggest ML configurations, preprocess data, select algorithm, and train the model. These suggestions are presented to the user via a web-based graphical user interface and a notebook-based programming user interface. Our goal is to offer a systematic investigation of user interaction and perceptions of using an AutoDS system in solving a data science task. We studied AutoDS with 30 professional data scientists, where one group used AutoDS, and the other did not, to complete a data science project. As expected, AutoDS improves productivity; Yet surprisingly, we find that the models produced by the AutoDS group have higher quality and less errors, but lower human confidence scores. We reflect on the findings by presenting design implications for incorporating automation techniques into human work in the data science lifecycle.

For machine learning models to be most useful in numerous sociotechnical systems, many have argued that they must be human-interpretable. However, despite increasing interest in interpretability, there remains no firm consensus on how to measure it. This is especially true in representation learning, where interpretability research has focused on “disentanglement” measures only applicable to synthetic datasets and not grounded in human factors. We introduce a task to quantify the human-interpretability of generative model representations, where users interactively modify representations to reconstruct target instances. On synthetic datasets, we find performance on this task much more reliably differentiates entangled and disentangled models than baseline approaches. On a real dataset, we find it differentiates between representation learning methods widely believed but never shown to produce more or less interpretable models. In both cases, we ran small-scale think-aloud studies and large-scale experiments on Amazon Mechanical Turk to confirm that our qualitative and quantitative results agreed.

Many researchers motivate explainable AI with studies showing that human-AI team performance on decision-making tasks improves when the AI explains its recommendations. However, prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team. Can explanations help lead to complementary performance, where team accuracy is higher than either the human or the AI working solo? We conduct mixed-method user studies on three datasets, where an AI with accuracy comparable to humans helps participants solve a task (explaining itself in some conditions). While we observed complementary improvements from AI augmentation, they were not increased by explanations. Rather, explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness. Our result poses new challenges for human-centered AI: Can we develop explanatory approaches that encourage appropriate trust in AI, and therefore help generate (or improve) complementary performance?

As AI-powered systems increasingly mediate consequential decision-making, their explainability is critical for end-users to take informed and accountable actions. Explanations in human-human interactions are socially-situated. AI systems are often socio-organizationally embedded. However, Explainable AI (XAI) approaches have been predominantly algorithm-centered. We take a developmental step towards socially-situated XAI by introducing and exploring Social Transparency (ST), a sociotechnically informed perspective that incorporates the socio-organizational context into explaining AI-mediated decision-making. To explore ST conceptually, we conducted interviews with 29 AI users and practitioners grounded in a speculative design scenario. We suggested constitutive design elements of ST and developed a conceptual framework to unpack ST’s effect and implications at the technical, decision-making, and organizational level. The framework showcases how ST can potentially calibrate trust in AI, improve decision-making, facilitate organizational collective actions, and cultivate holistic explainability. Our work contributes to the discourse of Human-Centered XAI by expanding the design space of XAI.

Efforts to make machine learning more widely accessible have led to a rapid increase in Auto-ML tools that aim to automate the process of training and deploying machine learning. To understand how Auto-ML tools are used in practice today, we performed a qualitative study with participants ranging from novice hobbyists to industry researchers who use Auto-ML tools. We present insights into the benefits and deficiencies of existing tools, as well as the respective roles of the human and automation in ML workflows. Finally, we discuss design implications for the future of Auto-ML tool development. We argue that instead of full automation being the ultimate goal of Auto-ML, designers of these tools should focus on supporting a partnership between the user and the Auto-ML tool. This means that a range of Auto-ML tools will need to be developed to support varying user goals such as simplicity, reproducibility, and reliability.

While HCI research has often addressed the needs of older adults, they are often framed as being sceptical of digital technologies. We argue that while many older adults are circumspect users of digital technology, they bring rich and critical perspectives on the role of technology in society that are grounded in lived experiences across their life courses. We report on 20 technology life story interviews conducted with retirees over the age of 60. Our analysis shows how experiences of technology across their life courses significantly undermined participants’ sense of competency, independence, resilience, agency and control. Dissonances between what our participants valued and the perceived values of technology have led them to become critical adopters of technology, and resist its intrusion into certain aspects of their lives. We discuss how the critical perspectives of older adults and the value dissonances they experience are valuable for designing future digital technologies.

Critical consumerism is complex as ethical values are difficult to negotiate, appropriate products are hard to find, and product information is overwhelming. Although recommender systems offer solutions to reduce such complexity, current designs are not appropriate for niche practices and use non-personalized intransparent ethics. To support critical consumption, we conducted a design case study on a personalized food recommender system. Therefore, we first conducted an empirical pre-study with 24 consumers to understand value negotiations and current practices, co-designed the recommender system, and finally evaluated it in a real-world trial with ten consumers. Our findings show how recommender systems can support the negotiation of ethical values within the context of consumption practices, reduce the complexity of finding products and stores, and strengthen consumers. In addition to providing implications for the design to support critical consumption practices, we critically reflect on the scope of such recommender systems and its appropriation.

Older adults can struggle to access relevant community expertise when faced with new situations. One such situation is the number of cyberattacks they may face when interacting online. This paper reports on an initiative which recruited, trained, and supported older adults to become community cybersecurity educators (CyberGuardians), tasked with promoting cybersecurity best practice within their communities to prevent older adults falling victim to opportunistic cyberattacks. This initiative utilised an embedded peer-to-peer information dissemination strategy, rather than expert-to-citizen, facilitating the inclusion of individuals who would ordinarily be unlikely to seek cybersecurity information and thus may be vulnerable to cyberattacks. We report on ways the CyberGuardians used informal methods to create more aware communities, served as role models for behaviour change and indirectly improved their personal wellbeing. We discuss considerations for supporting CyberGuardians, including implications for sustainability and for replicating this model in other digital contexts, e.g., recognising misinformation or improving mental health.

Civic tech initiatives dedicated to environmental issues have become a worldwide phenomenon and made invaluable contributions to data, community building, and publics. However, many of them stop after a relatively short time. Therefore, we studied two long-lasting civic tech initiatives of global scale, to understand what makes them sustain over time. To this end, we conducted two mixed-method case studies, combining social network analysis and qualitative content analysis of Twitter data with insights from expert interviews. Drawing on our findings, we identified a set of key factors that help the studied civic tech initiatives to grow and last. Contributing to Digital Civics in HCI, we argue that the civic tech initiatives’ scaling and sustaining are configured through the entanglement of (1) civic data both captured and owned by the citizens for the citizens, (2) the use of open and accessible technology, and (3) the initiatives’ public narrative, giving them a voice on the environmental issue.

On digital media services, uncivil commenting is a persistent issue causing negative emotional reactions. One enabler for such problematic behavior is the user interface, conditioning, and structuring text-based communication online. However, the specific roles and influences of UIs are little understood, which calls for critical analysis of the current UI solutions as well as speculative exploration of alternative designs. This paper reports a research-through-design study on the problematic phenomenon regarding uncivil and inconsiderate commenting on online news, envisioning unconventional solutions with a critical voice. We unpack this problem area and outline critical perspectives to possible solutions by describing and analyzing four designs that propose to support emotion regulation by facilitating self-reflection. The design choices are further discussed in respect to interviews of ten news media experts. The findings are reflected against the question of how can critique meaningfully manifest in this challenging problem area.

HCI research predominantly uses scientific rationality to explain users’ behaviors, decisions, and interactions with statistical models based and data-driven systems. However, such interactions are often more diverse in real life, and may straddle beyond scientific and economic rationality. Building on a ten-month ethnography at seven Bangladeshi villages, we explore the social and cultural factors that influence the online betting practices among the villagers. We describe how bets harmonize with users’ faith, hunch, and cultural practices, along with statistical recommendations. Drawing on a rich body of social science work on gambling, we contribute to the HCI scholarship in rationality, justification, and postcolonial computing. Finally, we present such betting as an under-appreciated site for HCI that contradicts with the ideological hegemony of statistical rationality, and recommend a smooth integration of AI system with the “other” rationalities of the Global South.

From Fanger's seminal work on thermal comfort in the 1970s, standards governing temperatures in the workplace enshrine clothing level calculations based on full business suits, and building regulations developed using only male metabolic data, locking in a default male perspective. Even later work that highlights gender biases with regard to metabolism calculation, inclusive of both genders has focused on younger women, and the voices of older working women are missing from this discourse. We invited women over 45 to explore what they find important in workplace thermal comfort, and how devices and interfaces might meet their needs and also encourage thermal adaptivity. Our study highlights factors such as 'fresh air', and the importance of empathy to fellow inhabitants. We bring new voices to the thermal comfort discourse which supports reducing energy use in the workplace, improving thermal environments and ensuring the needs of a diverse, aging workforce are considered.

This paper joins the growing body of critical HCI work that studies the digitization of the Global South and reports the elements of ‘secularization’ in it. Based on a year-long ethnography on the contemporary transformations in religious practices in Dhaka, Bangladesh, this paper presents how the emerging “digital” cattle marketplaces subdue various forms of traditional manifestations of urban religiosity during Eid-ul-Adha, the second-largest Islamic festival in the city. This paper further depicts how such secularization contributes to diminishing rural-urban linkages, affecting electoral politics, and reducing the tolerance to religious celebrations in a city. Drawing from a rich body of work in critical urban studies, postcolonial computing, and sociology of religions, we explain how such oft-overlooked embedding of secularization in computing affects the religious fabrics in the urban regions of the Global South, and discuss its implication for HCI scholarship in diversity, inclusion, and development.

Research on how lived experiences with technology intersect with home and work are core themes within HCI. Prior work has primarily focused on conventional life and work in Western countries. However, the unconventional is becoming conventional—several rising subcultures are coming into prominence due to socio-economic pressures, aided by social media. One example—#vanlife—is now practised by an estimated three million people in North America. #vanlife combines travel, home, and work by their occupants (vanlifers (vanlifers)) living full-time in cargo vans that they usually convert themselves into living spaces. We present a portrait of vanlifers’ current technology practices gleaned through ~200 hours of fieldwork and interviews. Following a thematic analysis of our data, we identified unique opportunities for integrating technology across culture, design, homesteading, offline organization, and gaming. We have distilled these opportunities into eleven provocations to inspire critical design and informed inquiry for technological interventions for #vanlife.

Multiple methods have been used to study how social values and ethics are implicated in technology design and use, including empirical qualitative studies of technologists’ work. Recently, more experimental approaches such as design fiction explore these themes through fictional worldbuilding. This paper combines these approaches by adapting design fictions as a form of memoing, a qualitative analysis technique. The paper uses design fiction memos to analyze and reflect on ethnographic interviews and observational data about how user experience (UX) professionals at large technology companies engage with values and ethical issues in their work. The design fictions help explore and articulate themes about the values work practices and relationships of power that UX professionals grapple with. Through these fictions, the paper contributes a case study showing how design fiction can be used for qualitative analysis, and provides insights into the role of organizational and power dynamics in UX professionals’ values work.

Ground-truth labeling is an important activity in machine learning. Many studies have examined how crowdworkers apply labels to records in machine learning datasets. However, there have been few studies that have examined the work of domain experts when their knowledge and expertise are needed to apply labels.

We provide a grounded account of the work of labeling teams with domain experts, including the experiences of labeling, collaborative configurations and work-practices, and quality issues. We show three major patterns in the social design of ground truth data: Principled design, Iterative design, and Improvisational design. We interpret our results through theories of from Human Centered Data Science, and particularly work on human interventions in data science work through the design and creation of data.

The promise AI’s proponents have made for decades is one in which our needs are predicted, anticipated, and met - often before we even realize it. Instead, algorithmic systems, particularly AIs trained on large datasets and deployed to massive scales, seem to keep making the wrong decisions, causing harm and rewarding absurd outcomes. Attempts to make sense of why AIs make wrong calls in the moment explain the instances of errors, but how the environment surrounding these systems precipitate those instances remains murky. This paper draws from anthropological work on bureaucracies, states, and power, translating these ideas into a theory describing the structural tendency for powerful algorithmic systems to cause tremendous harm. I show how administrative models and projections of the world create marginalization, just as algorithmic models cause representational and allocative harm. This paper concludes with a recommendation to avoid the absurdity algorithmic systems produce by denying them power.

Dance provides unique opportunities for embodied interdisciplinary learning experiences that can be personally and culturally relevant. danceON is a system that supports learners to leverage their body movement as they engage in artistic practices across data science, computing, and dance. The technology includes a Domain Specific Language (DSL) with declarative syntax and reactive behavior, a media player with pose detection and classification, and a web-based IDE. danceON provides a low-floor allowing users to bind virtual shapes to body positions in under three lines of code, while also enabling complex, dynamic animations that users can design working with conditionals and past position data. We developed danceON to support distance learning and deployed it in two consecutive cohorts of a remote, two-week summer camp for young women of color. We present our findings from an analysis of the experience and the resulting computational performances. The work identifies implications for how design can support learners’ expression across culturally relevant themes and examines challenges from the lens of usability of the computing language and technology.

Directly manipulating the timeline, such as scrubbing for thumbnails, is the standard way of controlling how-to videos. However, when how-to videos involve physical activities, people inconveniently alternate between controlling the video and performing the tasks. Adopting a voice user interface allows people to control the video with voice while performing the tasks with hands. However, naively translating timeline manipulation into voice user interfaces (VUI) results in temporal referencing (e.g. “rewind 20 seconds”), which requires a different mental model for navigation and thereby limiting users’ ability to peek into the content. We present RubySlippers, a system that supports efficient content-based voice navigation through keyword-based queries. Our computational pipeline automatically detects referenceable elements in the video, and finds the video segmentation that minimizes the number of needed navigational commands. Our evaluation (N=12) shows that participants could perform three representative navigation tasks with fewer commands and less frustration using RubySlippers than the conventional voice-enabled video interface.

Learning musical instruments using online instructional videos has become increasingly prevalent. However, pre-recorded videos lack the instantaneous feedback and personal tailoring that human tutors provide. In addition, existing video navigations are not optimized for instrument learning, making the learning experience encumbered. Guided by our formative interviews with guitar players and prior literature, we designed Soloist, a mixed-initiative learning framework that automatically generates customizable curriculums from off-the-shelf guitar video lessons. Soloist takes raw videos as input and leverages deep-learning based audio processing to extract musical information. This back-end processing is used to provide an interactive visualization to support effective video navigation and real-time feedback on the user's performance, creating a guided learning experience. We demonstrate the capabilities and specific use-cases of Soloist within the domain of learning electric guitar solos using instructional YouTube videos. A remote user study, conducted to gather feedback from guitar players, shows encouraging results as the users unanimously preferred learning with Soloist over unconverted instructional videos.

Explicitly alerting users is not always an optimal intervention, especially when they are not motivated to obey. For example, in video-based learning, learners who are distracted from the video would not follow an alert asking them to pay attention. Inspired by the concept of Mindless Computing, we propose a novel intervention approach, Mindless Attractor, that leverages the nature of human speech communication to help learners refocus their attention without relying on their motivation. Specifically, it perturbs the voice in the video to direct their attention without consuming their conscious awareness. Our experiments not only confirmed the validity of the proposed approach but also emphasized its advantages in combination with a machine learning-based sensing module. Namely, it would not frustrate users even though the intervention is activated by false-positive detection of their attentive state. Our intervention approach can be a reliable way to induce behavioral change in human–AI symbiosis.

The need to find or construct tables arises routinely to accomplish many tasks in everyday life, as a table is a common format for organizing data. However, when relevant data is found on the web, it is often scattered across multiple tables on different web pages, requiring tedious manual searching and copy-pasting to collect data. We propose KTabulator, an interactive system to effectively extract, build, or extend ad hoc tables from large corpora, by leveraging their computerized structures in the form of knowledge graphs. We developed and evaluated KTabulator using Wikipedia and its knowledge graph DBpedia as our testbed. Starting from an entity or an existing table, KTabulator allows users to extend their tables by finding relevant entities, their properties, and other relevant tables, while providing meaningful suggestions and guidance. The results of a user study indicate the usefulness and efficiency of KTabulator in ad hoc table creation.

Toolkits for shape-changing interfaces (SCIs) enable designers and researchers to easily explore the broad design space of SCIs. However, despite their utility, existing approaches are often limited in the number of shape-change features they can express. This paper introduces MorpheesPlug , a toolkit for creating SCIs that covers seven of the eleven shape-change features identified in the literature. MorpheesPlug is comprised of (1) a set of six standardized widgets that express the shape-change features with user-definable parameters; (2) software for 3D-modeling the widgets to create 3D-printable pneumatic SCIs; and (3) a hardware platform to control the widgets. To evaluate MorpheesPlug we carried out ten open-ended interviews with novice and expert designers who were asked to design a SCI using our software. Participants highlighted the ease of use and expressivity of the MorpheesPlug.

Embodied conversational agents have changed the ways we can interact with machines. However, these systems often do not meet users’ expectations. A limitation is that the agents are monotonic in behavior and do not adapt to an interlocutor. We present SIVA (a Socially Intelligent Virtual Agent), an expressive, embodied conversational agent that can recognize human behavior during open-ended conversations and automatically align its responses to the conversational and expressive style of the other party. SIVA leverages multimodal inputs to produce rich and perceptually valid responses (lip syncing and facial expressions) during the conversation. We conducted a user study (N=30) in which participants rated SIVA as being more empathetic and believable than the control (agent without style matching). Based on almost 10 hours of interaction, participants who preferred interpersonal involvement evaluated SIVA as significantly more animate than the participants who valued consideration and independence.

Automated vehicles promise a future where drivers can engage in non-driving tasks without hands on the steering wheels for a prolonged period. Nevertheless, automated vehicles may still need to occasionally hand the control back to drivers due to technology limitations and legal requirements. While some systems determine the need for driver takeover using driver context and road condition to initiate a takeover request, studies show that the driver may not react to it. We present DeepTake, a novel deep neural network-based framework that predicts multiple aspects of takeover behavior to ensure that the driver is able to safely take over the control when engaged in non-driving tasks. Using features from vehicle data, driver biometrics, and subjective measurements, DeepTake predicts the driver’s intention, time, and quality of takeover. We evaluate DeepTake performance using multiple evaluation metrics. Results show that DeepTake reliably predicts the takeover intention, time, and quality, with an accuracy of 96%, 93%, and 83%, respectively. Results also indicate that DeepTake outperforms previous state-of-the-art methods on predicting driver takeover time and quality. Our findings have implications for the algorithm development of driver monitoring and state detection.

This paper describes how machine learning training data and symbolic knowledge from curators of conversational systems can be used together to improve the accuracy of those systems and to enable better curatorial tools. This is done in the context of a real-world practice of curators of conversational systems who often embed taxonomically-structured meta-knowledge into their documentation. The paper provides evidence that the practice is quite common among curators, that is used as part of their collaborative practices, and that the embedded knowledge can be mined by algorithms. Further, this meta-knowledge can be integrated, using neuro-symbolic algorithms, to the machine learning-based conversational system, to improve its run-time accuracy and to enable tools to support curatorial tasks. Those results point towards new ways of designing development tools which explore an integrated use of code and documentation by machines.

Program synthesis, which generates programs based on user-provided specifications, can be obscure and brittle: users have few ways to understand and recover from synthesis failures. We propose interpretable program synthesis, a novel approach that unveils the synthesis process and enables users to monitor and guide a synthesizer. We designed three representations that explain the underlying synthesis process with different levels of fidelity. We implemented an interpretable synthesizer for regular expressions and conducted a within-subjects study with eighteen participants on three challenging regex tasks. With interpretable synthesis, participants were able to reason about synthesis failures and provide strategic feedback, achieving a significantly higher success rate compared with a state-of-the-art synthesizer. In particular, participants with a high engagement tendency (as measured by NCS-6) preferred a deductive representation that shows the synthesis process in a search tree, while participants with a relatively low engagement tendency preferred an inductive representation that renders representative samples of programs enumerated during synthesis.

Modern visualization tools aim to allow data analysts to easily create exploratory visualizations. When the input data layout conforms to the visualization design, users can easily specify visualizations by mapping data columns to visual channels of the design. However, when there is a mismatch between data layout and the design, users need to spend significant effort on data transformation.

We propose Falx, a synthesis-powered visualization tool that allows users to specify visualizations in a similarly simple way but without needing to worry about data layout. In Falx, users specify visualizations using examples of how concrete values in the input are mapped to visual channels, and Falx automatically infers the visualization specification and transforms the data to match the design. In a study with 33 data analysts on four visualization tasks involving data transformation, we found that users can effectively adopt Falx to create visualizations they otherwise cannot implement.

There is increased interest in using virtual reality in education, but it often remains an isolated experience that is difficult to integrate into current instructional experiences. In this work, we adapt virtual production techniques from filmmaking to enable mixed reality capture of instructors so that they appear to be standing directly in the virtual scene. We also capitalize on the growing popularity of live streaming software for video conferencing and live production. With XRStudio, we develop a pipeline for giving lectures in VR, enabling live compositing using a variety of presets and real-time output to traditional video and more immersive formats. We present interviews with media designers experienced in film and MOOC production that informed our design. Through walkthrough demonstrations of XRStudio with instructors experienced with VR, we learn how it could be used in a variety of domains. In end-to-end evaluations with students, we analyze and compare differences of traditional video vs. more immersive lectures with XRStudio.

We present a multi-modal approach for automatically generating hierarchical tutorials from instructional makeup videos. Our approach is inspired by prior research in cognitive psychology, which suggests that people mentally segment procedural tasks into event hierarchies, where coarse-grained events focus on objects while fine-grained events focus on actions. In the instructional makeup domain, we find that objects correspond to facial parts while fine-grained steps correspond to actions on those facial parts. Given an input instructional makeup video, we apply a set of heuristics that combine computer vision techniques with transcript text analysis to automatically identify the fine-level action steps and group these steps by facial part to form the coarse-level events. We provide a voice-enabled, mixed-media UI to visualize the resulting hierarchy and allow users to efficiently navigate the tutorial (e.g., skip ahead, return to previous steps) at their own pace. Users can navigate the hierarchy at both the facial-part and action-step levels using click-based interactions and voice commands. We demonstrate the effectiveness of segmentation algorithms and the resulting mixed-media UI on a variety of input makeup videos. A user study shows that users prefer following instructional makeup videos in our mixed-media format to the standard video UI and that they find our format much easier to navigate.

Live streaming is gaining popularity across diverse application domains in recent years. A core part of the experience is streamer-viewer interaction, which has been mainly text-based. Recent systems explored extending viewer interaction to include visual elements with richer expression and increased engagement. However, understanding expressive visual inputs becomes challenging with many viewers, primarily due to the relative lack of structure in visual input. On the other hand, adding rigid structures can limit viewer interactions to narrow use cases or decrease the expressiveness of viewer inputs. To facilitate the sensemaking of many visual inputs while retaining the expressiveness or versatility of viewer interactions, we introduce a visual input management framework (VIMF) and a system, VisPoll, that help streamers specify, aggregate, and visualize many visual inputs. A pilot evaluation indicated that VisPoll can expand the types of viewer interactions. Our framework provides insights for designing scalable and expressive visual communication for live streaming.

Two social motives are distinguished by Motive Disposition Theory: affiliation and power. Motives orient, select and energize our behaviour, suggesting that the choices of power-motivated individuals should be guided by power cues, such as the appearance of strength in a game character or avatar. In study 1 we demonstrate that participants were more likely to pick strong-looking Pokémon for a fight and cute Pokémon as a companion. In addition, we show that even when considering these contexts, the power motive predicts preferences for a powerful appearance, whereas affiliation does not. In study 2 we replicate the study 1 findings and distinguish between two ways to enact the power motive (prosocial and dominant power). We demonstrate that the dominance, but not the prosociality, facet drives the preference for strong-looking Pokémon. Our findings suggest that the need to influence others—the power motive—drives the choice for battle companions who symbolize strength.

Increasingly, modern boardgames incorporate digital apps and tools to deliver content in novel ways. Despite disparate approaches to incorporating digital tools in otherwise non-digital boardgames, there has to date been no detailed classification of the different roles that these tools play in supporting gameplay. In this paper, we present a model for understanding hybrid boardgame play as performing a set of discrete functions. Through a mixed-methods approach incorporating critical play sessions, a survey of 237 boardgame players, and interviews with 18 boardgame designers and publishers, we identified the key functions performed by the digital tools in these games. Using affinity mapping, we grouped these functions into eight core categories, which were tested and refined using a survey of 44 boardgame players and designers. This model defines and classifies the diverse functions that digital tools perform in hybrid digital boardgames, demonstrating their range and potential application for researchers and game designers.

Graphical user authentication (GUA) is a common alternative to text-based user authentication, where people are required to draw graphical passwords on background images. Such schemes are theoretically considered remarkably secure because they offer a large password space. However, people tend to create their passwords on salient image areas introducing high password predictability. Aiming to help people use the password space more effectively, we propose a gameful password creation process. In this paper, we present GamePass, a gamified mechanism that integrates the GUA password creation process. We provide the first evidence that it is possible to nudge people towards better password choices by gamifying the process. GamePass randomly guides participants’ attention to areas other than the salient areas of authentication images, makes the password creation process more fun, and people are more engaged. Gamifying the password creation process enables users to interact better and make less predictable graphical password choices instead of being forced to use a strict password policy.

Many serious games are most effective when played regularly; however, little is known about how individual game elements support player adherence over time. This work draws on evidence from existing frameworks and game design theories as well as from the design of casual games to investigate how individual game mechanics affect player attrition in a serious game. We implemented a math-learning game in which we could individually layer various game mechanics, and over the course of 3 weeks, 99 participants played one of six versions: Baseline, Rewards, Novelty, Completion, Waiting, or Blocking. We compared the game versions by analyzing the players’ performance as well as behaviour. Using survival analysis, we identified that the addition of Completion and Blocking mechanics facilitated the strongest sustained engagement. These findings are congruent with existing theories of player experience and promote the development of guidelines on designing for sustained engagement in serious games.

The landscape of digital games is segregated by player ability. For example, sighted players have a multitude of highly visual games at their disposal, while blind players may choose from a variety of audio games. Attempts at improving cross-ability access to any of those are often limited in the experience they provide, or disregard multiplayer experiences. We explore ability-based asymmetric roles as a design approach to create engaging and challenging mixed-ability play. Our team designed and developed two collaborative testbed games exploring asymmetric interdependent roles. In a remote study with 13 mixed-visual-ability pairs we assessed how roles affected perceptions of engagement, competence, and autonomy, using a mixed-methods approach. The games provided an engaging and challenging experience, in which differences in visual ability were not limiting. Our results underline how experiences unequal by design can give rise to an equitable joint experience.

Videogames’ increasing cultural relevance and suffusion into everyday use contexts suggests they can no longer be considered novelties. Broadly speaking, games research at CHI has concerned two forms of peak experience—historically, research aimed to support flow, or maximise enjoyment and positive emotions; more recently, scholarship engages with more varied experiences of intense emotion, such as emotional challenge. In different ways, both approaches emphasise extra-ordinary player experience (PX). Conversely, videogame play and PX have become more routine—indeed, more ordinary—as the medium’s cultural presence grows. In this paper, we argue that HCI games research is conceptually ill-equipped to investigate these increasingly common and often desirable experiences. We conceptualise “ordinary player experience” – as familiar, emotionally moderate, co-attentive, and abstractly memorable – articulating a phenomenon whose apparent mundanity has seen it elude description to date. We discuss opportunities to productively employ ordinary PX in HCI games research, alongside conceptual implications for PX and player wellbeing.

We present an autobiographical design journey exploring the experience of returning to long-term single player games. Continuing progress from a previously saved game, particularly when substantial time has passed, is an understudied area in games research. To begin our exploration in this domain, we investigated what the return experience is like first-hand. By returning to four long-term single player games played extensively in the past, we revealed a phenomenon we call The Pivot Point, a ‘eureka’ moment in return gameplay. The pivot point anchors our design explorations, where we created prototypes to leverage the pivot point in reconnecting with the experience. These return experiences and subsequent prototyping iterations inform our understanding of how to design better returns to gameplay, which can benefit both producers and consumers of long-term single player games.

Video games are engaging multimedia experiences that require players’ cognitive, emotional, physical (in terms of controllers and exertion), and social faculties. Recent theorizing has suggested that these dimensions of demand can explain processes by which players engage with and respond to gameplay. A relatively new measure—the five-factor, 26-item Video Game Demand Scale (VGDS)—has been tested for dimensionality and measurement validity with English- and German-speaking players, but not for other play populations. Given the popularity of video games among Chinese-speaking players, this brief report demonstrates a successful translation of VGDS into Traditional Chinese (VGDS-C). A sample of N = 863 Chinese speakers in Taiwan were asked to recall and describe a recent gaming experience before completing the VGDS-C along with other gaming-related measures (tests of construct validity). VGDS-C was shown to be a reliable and valid way of assessing players’ perceptions of the myriad demands of video gaming.

Self-report assessment is important for research and game development, e.g., to gather data during play. Games can use dialogues with non-player characters (NPCs) to gather self-report data; however, players might respond differently to dialogues than questionnaires. Without guidance on how in-game assessment affects player perceptions and experiences, designers and researchers are in danger of making decisions that harm data quantity and quality, and perceptions of privacy. We conducted a user study to understand self-report collection from NPC dialogues and traditional in-game overlay questionnaires. Data quality and player experience measures autonomy, curiosity, immersion, and mastery did not differ significantly, although NPC dialogues enhanced meaning. NPC dialogues supported an increase in data quantity through voluntary 5-point scales but not via open responses; however, they also increased the perceived intimacy of shared information despite comparable objective intimacy. NPC dialogues are useful to gather quantitative self-report data. They enable a meaningful play experience but could facilitate negative effects related to privacy.

The current study explores the relationship between perceived cognitive and physical demands of a simple video game, and the balance of reward and effort that results in flow states during gameplay. Cognitive demands and both exertion-based and controller-based physical demands were perceived as lowest in situations where reward was high and effort was low (boredom), moderate when reward and effort were balanced (flow), and highest when the reward was low and effort was high (frustration). Surprisingly, player response times to a secondary task showed the greatest improvement when playing the frustrating video game condition. We interpret this latter finding as evidence of an observed task-switching effect: players initially tried to master the game's over-challenging primary task before giving up and, instead, diverted attention toward a secondary in-game task that required less effort and thus, gave greater attentional rewards to the player. The implications of this cognitive offloading are discussed.

Emotionally impactful game experiences have garnered increasing interest within HCI games research. Yet the perspectives of designers have, to date, remained largely overlooked. We interviewed 14 indie game designers regarding their values and practices in designing emotionally impactful games. Counter to the focus of recent player experience (PX) studies, we find that while designers typically have a clear vision for the intended emotional impact, they aim for their games to provide a space for players to have their own personal experiences and interpretations. Despite this player-centric orientation, players were rarely involved before and during the production to evaluate the emotional experience. Based on these findings, we identify gaps between design practice and PX research, raise open questions around the design and evaluation of emotionally impactful game experiences, and outline opportunities for HCI games research to more productively support game designers.

Player agency is central to interactive narrative and games. While previous work focuses on analyzing player perception of agency through various lenses and phenomena, like meaningful choice and expectations, it is largely theoretical. Few user studies within games explore how players reason about and judge their own agency within interactive narratives. We present an interview study where participants rated their agency experiences within narrative-focused games and described their reasoning. The analysis suggests that agency perception depends on multiple factors beyond meaningful choice, such as social investment and genre-conventions. Participants described varying preferences and value judgements for different factors, indicating that individual differences have a deep impact on agency perception in narrative-focused gameplay. We discuss the implications of these cognitive variables on design, how they can be leveraged with other factors, and how our findings can help future work enhance and measure player agency, within interactive narrative and beyond.

Worlds-in-Miniature (WiMs) are interactive worlds within a world and combine the advantages of an input space, a cartographic map, and an overview+detail interface. They have been used across the extended virtuality spectrum for a variety of applications. Building on an analysis of examples of WiMs from the research literature we contribute a design space for WiMs based on seven design dimensions. Further, we expand upon existing definitions of WiMs to provide a definition that applies across the extended reality spectrum. We identify the design dimensions of size-scope-scale, abstraction, geometry, reference frame, links, multiples, and virtuality. Using our framework we describe existing Worlds-in-Miniature from the research literature and reveal unexplored research areas. Finally, we generate new examples of WiMs using our framework to fill some of these gaps. With our findings, we identify opportunities that can guide future research into WiMs.

In virtual reality (VR) environments, asymmetric bimanual interaction techniques can increase users’ input bandwidth by complementing their perceptual and motor systems (e.g., using the dominant hand to select 3D UI controls anchored around the non-dominant arm). However, it is unclear how to optimize the layout of such 3D UI controls for near-body and mid-air interactions. We evaluate the performance and limitations of non-dominant arm-anchored 3D UIs in VR environments through a bimanual pointing study. Results demonstrated that targets appearing closer to the skin, located around the wrist, or placed on the medial side of the forearm could be selected more quickly than targets farther away from the skin, located around the elbow, or on the lateral side of the forearm. Based on these results, we developed Armstrong guidelines, demonstrated through a Unity plugin to enable designers to create performance-optimized arm-anchored 3D UI layouts.

JetController is a novel haptic technology capable of supporting high-speed and persistent 3-DoF ungrounded force feedback. It uses high-speed pneumatic solenoid valves to modulate compressed air to achieve 20-50Hz of full impulses at 4.0-1.0N, and combines multiple air propulsion jets to generate 3-DoF force feedback. Compared to propeller-based approaches, JetController supports 10-30 times faster impulse frequency, and its handheld device is significantly lighter and more compact. JetController supports a wide range of haptic events in games and VR experiences, from firing automatic weapons in games like Halo (15Hz) to slicing fruits in Fruit Ninja (up to 45Hz). To evaluate JetController, we integrated our prototype with two popular VR games, Half-life: Alyx and Beat Saber, to support a variety of 3D interactions. Study results showed that JetController significantly improved realism, enjoyment, and overall experience compared to commercial vibrating controllers, and was preferred by most participants.

Virtual Reality experiences, such as games and simulations, typically support the usage of bimanual controllers to interact with virtual objects. To recreate the haptic sensation of holding objects of various shapes and behaviors with both hands, previous researchers have used mechanical linkages between the controllers that render adjustable stiffness. However, the linkage cannot quickly adapt to simulate dynamic objects, nor it can be removed to support free movements. This paper introduces GamesBond, a pair of 4-DoF controllers without physical linkage but capable to create the illusion of being connected as a single device, forming a virtual bond. The two controllers work together by dynamically displaying and physically rendering deformations of hand grips, and so allowing users to perceive a single connected object between the hands, such as a jumping rope. With a user study and various applications we show that GamesBond increases the realism, immersion, and enjoyment of bimanual interaction.

Game designers and researchers employ a sophisticated language for producing great player experiences with concepts such as juiciness, which refers to excessive positive feedback. However, much of their discourse excludes the role and value of haptic feedback. In this paper, we adapt terminology from game design to study haptic feedback. Specifically, we define haptic embellishments (HEs) as haptic feedback that reinforce information already provided through other means (e.g., via visual feedback) and juicy haptics as excessive positive haptic feedback with the intention of improving user experience in games and other interactive media. We report two empirical studies of users’ experiences interacting with visuo-haptic content on their phones to 1) study participants’ preferences for ten design principles for HEs and 2) measure the added value of juicy haptics, implemented as HEs, on player experience in a game. Results indicate that juicy haptics can enhance enjoyability, aesthetic appeal, immersion, and meaning.

Current head-mounted displays enable users to explore virtual worlds by simply walking through them (i.e., real-walking VR). This led researchers to create haptic displays that can also simulate different types of elevation shapes. However, existing shape-changing floors are limited by their tabletop scale or the coarse resolution of the terrains they can display due to the limited number of actuators and low vertical resolution. To tackle this challenge, we introduce Elevate, a dynamic and walkable pin-array floor on which users can experience not only large variations in shapes but also the details of the underlying terrain. Our system achieves this by packing 1200 pins arranged on a 1.80 × 0.60m platform, in which each pin can be actuated to one of ten height levels (resolution: 15mm/level). To demonstrate its applicability, we present our haptic floor combined with four walkable applications and a user study that reported increased realism and enjoyment.

Numerous techniques have been proposed for locomotion in virtual reality (VR). Several taxonomies consider a large number of attributes (e.g., hardware, accessibility) to characterize these techniques. However, finding the appropriate locomotion technique (LT) and identifying gaps for future designs in the high-dimensional space of attributes can be quite challenging. To aid analysis and innovation, we devised Locomotion Vault (https://locomotionvault.github.io/), a database and visualization of over 100 LTs from academia and industry. We propose similarity between LTs as a metric to aid navigation and visualization. We show that similarity based on attribute values correlates with expert similarity assessments (a method that does not scale). Our analysis also highlights an inherent trade-off between simulation sickness and accessibility across LTs. As such, Locomotion Vault shows to be a tool that unifies information on LTs and enables their standardization and large-scale comparison to help understand the space of possibilities in VR locomotion.

Smartphone touch screens are potentially attractive for interaction in virtual reality (VR). However, the user cannot see the phone or their hands in a fully immersive VR setting, impeding their ability for precise touch input. We propose mounting a mirror above the phone screen such that the front-facing camera captures the thumbs on or near the screen. This enables the creation of semi-transparent overlays of thumb shadows and inference of fingertip hover points with deep learning, which help the user aim for targets on the phone. A study compares the effect of visual feedback on touch precision in a controlled task and qualitatively evaluates three example applications demonstrating the potential of the technique. The results show that the enabled style of feedback is effective for thumb-size targets, and that the VR experience can be enriched by using smartphones as VR controllers supporting precise touch input.

Selection and manipulation in virtual reality often happen using an avatar’s hands. However, objects outside the immediate reach require effort to select. We develop a target selection technique called Ninja Hands. It maps the movement of a single real hand to many virtual hands, decreasing the distance to targets. We evaluate Ninja Hands in two studies. The first study shows that compared to a single hand, 4 and 8 hands are significantly faster for selecting targets. The second study complements this finding by using a larger target layout with many distractors. We find no decrease in selection time across 8, 27, and 64 hands, but an increase in the time spent deciding which hand to use. Thereby, net movement time still decreases significantly. In both studies, the physical motion exerted also decreases significantly with more hands. We discuss how these findings can inform future implementations of the Ninja Hands technique.

Augmented Reality (AR) is increasingly being used for providing guidance and supporting troubleshooting in industrial settings. While the general application of AR has been shown to provide clear benefits regarding physical tasks, it is important to understand how different visualization types influence user’s performance during the execution of the tasks. Previous studies evaluating AR and user’s performance compared different media types or types of AR hardware as opposed to different types of visualization for the same hardware type. This paper provides details of our comparative study in which we identified the influence of visualization types on the performance of complex machine set-up processes. Although our results show clear advantages to using concrete rather than abstract visualizations, we also find abstract visualizations coupled with videos leads to similar user performance as with concrete visualizations.

Interacting with out of reach or occluded VR objects can be cumbersome. Although users can change their position and orientation, such as via teleporting, to help observe and select, doing so frequently may cause loss of spatial orientation or motion sickness. We present vMirror, an interactive widget leveraging reflection of mirrors to observe and select distant or occluded objects. We first designed interaction techniques for placing mirrors and interacting with objects through mirrors. We then conducted a formative study to explore a semi-automated mirror placement method with manual adjustments. Next, we conducted a target-selection experiment to measure the effect of the mirror’s orientation on users’ performance. Results showed that vMirror can be as efficient as direct target selection for most mirror orientations. We further compared vMirror with teleport technique in a virtual treasure hunt game and measured participants’ task performance and subjective experiences. Finally, we discuss vMirorr user experience and present future directions.

Tactile feedback is widely used to enhance realism in virtual reality (VR). When touching virtual objects, stiffness and roughness are common and obvious factors perceived by the users. Furthermore, when touching a surface with complicated surface structure, differences from not only stiffness and roughness but also surface height are crucial. To integrate these factors, we propose a pin-based handheld device, HairTouch, to provide stiffness differences, roughness differences, surface height differences and their combinations. HairTouch consists of two pins for the two finger segments close to the index fingertip, respectively. By controlling brush hairs’ length and bending direction to change the hairs’ elasticity and hair tip direction, each pin renders various stiffness and roughness, respectively. By further independently controlling the hairs’ configuration and pins’ height, versatile stiffness, roughness and surface height differences are achieved. We conducted a perception study to realize users’ distinguishability of stiffness and roughness on each of the segments. Based on the results, we performed a VR experience study to verify that the tactile feedback from HairTouch enhances VR realism.

For haptic guidance, vibrotactile feedback is a commonly-used mechanism, but requires users to interpret its complicated patterns especially in 3D guidance, which is not intuitive and increases their mental effort. Furthermore, for haptic guidance in virtual reality (VR), not only guidance performance but also realism should be considered. Since vibrotactile feedback interferes with and reduces VR realism, it may not be proper for VR haptic guidance. Therefore, we propose a wearable device, GuideBand, to provide intuitive 3D multilevel force guidance upon the forearm, which reproduces an effect that the forearm is pulled and guided by a virtual guider or telepresent person in VR. GuideBand uses three motors to pull a wristband at different force levels in 3D space. Such feedback usually requires much larger and heavier robotic arms or exoskeletons. We conducted a just-noticeable difference study to understand users’ force level distinguishability. Based on the results, we performed a study to verify that compared with state-of-the-art vibrotactile guidance, GuideBand is more intuitive, needs a lower level of mental effort, and achieves similar guidance performance. We further conducted a VR experience study to observe how users combine and complement visual and force guidance, and prove that GuideBand enhances realism in VR guidance.

The growing online gig economy provides ways for women to participate in a flexible, remote workforce and close the offline gender pay and participation gap. While women in online labor marketplaces earn about as much overall as men, women set lower bill rates suggesting gender differences in pricing strategies. In this study, we surveyed 392 freelancers in the United States (US) on the popular marketplace platform, Upwork, to understand strategies used to set hourly bill rates. We did not find gender differences in pricing strategies that were significantly related to bill rate. Instead, we found that other factors, such as full-time freelancer status and level of self-esteem, may help explain gender differences in bill rates. To better support equity and fairness in the growing gig economy, CHI researchers must identify, assess, and address the complex interaction between societal conditions in online labor markets.

Maintaining an awareness of one’s well-being and making work-related decisions to achieve work-life balance is critical for flexible long-hour workers. In this study, we propose that social sensing could address bottlenecks in worker’s awareness, interpretation of the informatics, and subsequent behavioral change. We conducted a four-week technology probe study by recruiting flexible long-hour professional drivers (Taxi and Uber drivers) and their significant others to use a social sensing prototype which collects data from the drivers and shares it with their partners as well as incorporates partners’ observations. We interviewed them before and after the probe study and found that while technological sensing was able to increase drivers’ awareness of their well-being status and intention to modify behaviors. The “social sensing” design was able to further shape such awareness or intention into action, highlighting the potential of using the sociotechnical approach in promoting work-life balance among long-hour workers.

Live streaming has become increasingly popular, with most streamers presenting their real-life appearance. However, Virtual YouTubers (VTubers), virtual 2D or 3D avatars that are voiced by humans, are emerging as live streamers and attracting a growing viewership in East Asia. Although prior research has found that many viewers seek real-life interpersonal interactions with real-person streamers, it is currently unknown what makes VTuber live streams engaging or how they are perceived differently than real-person streamers. We conducted an interview study to understand how viewers engage with VTubers and perceive the identities of the voice actors behind the avatars (i.e., Nakanohito). The data revealed that Virtual avatars bring unique performative opportunities which result in different viewer expectations and interpretations of VTuber behavior. Viewers intentionally upheld the disembodiment of VTuber avatars from their voice actors. We uncover the nuances in viewer perceptions and attitudes and further discuss the implications of VTuber practices to the understanding of live streaming in general.

Emerging research suggests that people trust algorithmic decisions less than human decisions. However, different populations, particularly in marginalized communities, may have different levels of trust in human decision-makers. Do people who mistrust human decision-makers perceive human decisions to be more trustworthy and fairer than algorithmic decisions? Or do they trust algorithmic decisions as much as or more than human decisions? We examine the role of mistrust in human systems in people’s perceptions of algorithmic decisions. We focus on healthcare Artificial Intelligence (AI), group-based medical mistrust, and Black people in the United States. We conducted a between-subjects online experiment to examine people’s perceptions of skin cancer screening decisions made by an AI versus a human physician depending on their medical mistrust, and we conducted interviews to understand how to cultivate trust in healthcare AI. Our findings highlight that research around human experiences of AI should consider critical differences in social groups.

Crowdsourcing is a new value creation business model. Annual revenue of the Chinese market alone is hundreds of millions of dollars, yet few studies have focused on the practices of the Chinese crowdsourcing workforce, and those that do mainly focus on solo crowdworkers. We have extended our study of solo crowdworker practices to include crowdfarms, a relatively new entry to the gig economy: small companies that carry out crowdwork as a key part of their business. We report here on interviews of people who work in 53 crowdfarms. We describe how crowdfarms procure jobs, carry out macrotasks and microtasks, manage their reputation, and employ different management practices to motivate crowdworkers and customers.

Online video games support the development of social relationships through gameplay, however, gamers often cannot cultivate and maintain relationships based on social factors such as personality when using in-game matchmaking services. To address this, teammate matching sites external to games have emerged and enable gamers to offer to play games with others in exchange for payment. The affordances of these services are different from other existing gamer social sites, e.g., live streaming. Interviews were conducted with 16 dedicated users on Bixin, one of China’s largest paid teammate matching sites, to examine user motivations, practices, and perceptions. The interviews found that gamers selected paid teammates on Bixin using different criteria compared to in-game matchmaking services and emphasized the importance of real-life characteristics such as voice. To maintain connections, paid teammates often also extended communication to external communication services such as WeChat. Although most gamers expected to communicate with paid teammates as if they were friends, very few reported building real friendships with their matched counterparts.

The user study is a fundamental method used in HCI. In designing user studies, we often use compensation strategies to incentivize recruitment. However, compensation can also lead to ethical issues, such as coercion. The CHI community has yet to establish best practices for participant compensation. Through a systematic review of manuscripts at CHI and other associated publication venues, we found high levels of variation in the compensation strategies used within the community and how we report on this aspect of the study methods. A qualitative analysis of justifications offered for compensation sheds light into how some researchers are currently contextualizing this practice. This paper provides a description of current compensation strategies and information that can inform the design of compensation strategies in future studies. The findings may be helpful to generate productive discourse in the HCI community towards the development of best practices for participant compensation in user studies.

A jury of one’s peers is a prominent way to adjudicate disputes and is increasingly used in participatory governance online. The fairness of this approach rests on the assumption that juries are consistent: that the same jury would hand down similar judgments to similar cases. However, prior literature suggests that social influence would instead cause early interactions to cascade into different judgments for similar cases. In this paper, we report an online experiment that changes participants’ pseudonyms as they appear to collaborators, temporarily masking a jury’s awareness that they have deliberated together before. This technique allows us to measure consistency by reconvening the same jury on similar cases. Counter to expectation, juries are equally consistent as individuals, a result that is “good for democracy.” But this consistency arises in part due to group polarization, as consensus develops by hardening initial majority opinions. Furthermore, we find that aggregating groups’ perspectives without deliberation erodes consistency.

Computer technology is often designed in technology hubs in Western countries, invariably making it “WEIRD”, because it is based on the intuition, knowledge, and values of people who are Western, Educated, Industrialized, Rich, and Democratic. Developing technology that is universally useful and engaging requires knowledge about members of WEIRD and non-WEIRD societies alike. In other words, it requires us, the CHI community, to generate this knowledge by studying representative participant samples. To find out to what extent CHI participant samples are from Western societies, we analyzed papers published in the CHI proceedings between 2016-2020. Our findings show that 73% of CHI study findings are based on Western participant samples, representing less than 12% of the world’s population. Furthermore, we show that most participant samples at CHI tend to come from industrialized, rich, and democratic countries with generally highly educated populations. Encouragingly, recent years have seen a slight increase in non-Western samples and those that include several countries. We discuss suggestions for further broadening the international representation of CHI participant samples.

It is widely acknowledged in HCI that culture is embodied in many aspects of an individual's identity and interaction with technology. Whilst existing cultural models have been criticized for providing a deterministic view of culture, alternative methods for incorporating culture in design remain scarce. We introduce the use of Qualitative Secondary Analysis (QSA) as a bottom-up approach to construct a richer and more dynamic understanding of culture to inform our best practices in Cross-Cultural Design. We demonstrate the use of QSA within a culturally specific context, namely Saudi transnationals. We draw upon two case studies (with 55 participants) to investigate the cultural factors underpinning our participants views. We conclude with a reflection on key affordances and challenges of QSA, illustrating how QSA can be leveraged to unravel otherwise overlooked knowledge present in many qualitative HCI studies.

Leaving the field when conducting research in situated contexts, and finding ways to sustain project outcomes beyond academia, is an ongoing struggle in HCI. In our research project, we co-designed technologies with children in classroom contexts for three years. Nearing the projects’ end, we focused on creating resources that enable teachers to continue our work with their pupils. In collaboration with teachers, we developed socio-material tools that support them in empowering neurodiverse children to engage with technology in creative ways and create their own technologies. While the majority of technology design toolkits are stand-alone artefacts, part of our toolkit is an infrastructure to keep guiding and supporting teachers beyond the project’s end. In this paper, we discuss the teachers expectations and requirements for a toolkit and argue that an infrastructure must be part of a toolkit. We present a set of guidelines for researchers planning for a project’s end.

Search and rescue (SAR), a disaster response activity performed to locate and save victims, primarily involves collective sensemaking and planning. SAR responders learn to search and navigate the environment, process information about buildings, and collaboratively plan with maps. We synthesize data from five sources, including field observations and interviews, to understand the informational components of SAR and how information is recorded and communicated. We apply activity theory, uncovering unforeseen factors that are relevant to the design of collaboration systems and training solutions. Through our analysis, we derive design implications to support collaborative information technology and training systems: mixing physical and digital mapping; mixing individual and collective mapping; building for different levels and sources of information; and building for different rules, roles, and activities.

Hackathons are increasingly embraced across diverse sectors as a way of democratizing the design of technology. Several attempts have been made to redefine the format and desired end goal of hackathons in recent years thereby warranting closer methodological scrutiny. In this paper, we apply program theory to analyze the processes and effects of 16 hackathon case studies through published research literature. Building upon existing research on hackathons, our work offers a critical perspective examining the methodological validity of hackathons and exemplifies how specific processes for organizing hackathons are modified for different purposes. Our main contribution is a program theory analysis of hackathon formats that provides an exploration and juxtaposition of 16 case studies in terms of causal relations between the input, process and the effects of hackathons. Our cataloguing of examples can serve as an inspirational planning resource for future organizers of hackathons.

Tabs have become integral to browsing the Web yet have changed little since their introduction nearly 20 years ago. In contrast, the internet has gone through dramatic changes, with users increasingly moving from navigating to websites to exploring information across many sources to support online sensemaking. This paper investigates how tabs today are overloaded with a diverse set of functionalities and issues users face when managing them. We interviewed ten information workers asking about their tab management strategies and walk through each open tab on their work computers four times over two weeks. We uncovered competing pressures pushing for keeping tabs open (ranging from interaction to emotional costs) versus pushing for closing them (such as limited attention and resources). We then surveyed 103 participants to estimate the frequencies of these pressures at scale. Finally, we developed design implications for future browser interfaces that can better support managing these pressures.

Hundreds of millions of speakers of bidirectional (BiDi) languages rely on writing systems that mix the native right-to-left script with left-to-right strings. The global reach of interactive digital technologies requires special attention to these people, whose perception of interfaces is affected by this script mixture. However, empirical research on this topic is scarce. Although leading software vendors provide guidelines for BiDi design, bidirectional interfaces demonstrate inconsistent and incorrect directionality of UI elements, which may cause user confusion and errors.

Through a websites' review, we identified problematic UI items and considered reasons for their existence. In an online survey with 234 BiDi speakers, we observed that in many cases, users' direction preferences were inconsistent with the guidelines. The findings provide potential insights for design rules and empirical evidence for the problem's complexity, suggesting the need for further empirical research and greater attention by the HCI community to the BiDi design problem.

Accurate drone positioning is challenging because pilots only have a limited position and direction perception of a flying drone from their perspective. This makes conventional joystick-based speed control inaccurate and more complicated and significantly degrades piloting performance. We propose PinpointFly, an egocentric drone interface that allows pilots to arbitrarily position and rotate a drone using position-control direct interactions on a see-through mobile AR where the drone position and direction are visualized with a virtual cast shadow (i.e., the drone’s orthogonal projection onto the floor). Pilots can point to the next position or draw the drone’s flight trajectory by manipulating the virtual cast shadow and the direction/height slider bar on the touchscreen. We design and implement a prototype of PinpointFly for indoor and visual line of sight scenarios, which are comprised of real-time and predefined motion-control techniques. We conduct two user studies with simple positioning and inspection tasks. Our results demonstrate that PinpointFly makes the drone positioning and inspection operations faster, more accurate, simpler and fewer workload than a conventional joystick interface with a speed-control method.

AI systems collect our personal information in order to provide personalized services, raising privacy concerns and making users leery. As a result, systems have begun emphasizing overt over covert collection of information by directly asking users. This poses an important question for ethical interaction design, which is dedicated to improving user experience while promoting informed decision-making: Should the interface tout the benefits of information disclosure and frame itself as a help-provider? Or, should it appear as a help-seeker? We decided to find out by creating a mockup of a news recommendation system called Mindz and conducting an online user study (N=293) with the following four variations: AI system as help seeker vs. help provider vs. both vs. neither. Data showed that even though all participants received the same recommendations, power users tended to trust a help-seeking Mindz more whereas non-power users favored one that is both help-seeker and help-provider.

Online symptom checkers (OSC) are widely used intelligent systems in health contexts such as primary care, remote healthcare, and epidemic control. OSCs use algorithms such as machine learning to facilitate self-diagnosis and triage based on symptoms input by healthcare consumers. However, intelligent systems’ lack of transparency and comprehensibility could lead to unintended consequences such as misleading users, especially in high-stakes areas such as healthcare. In this paper, we attempt to enhance diagnostic transparency by augmenting OSCs with explanations. We first conducted an interview study (N=25) to specify user needs for explanations from users of existing OSCs. Then, we designed a COVID-19 OSC that was enhanced with three types of explanations. Our lab-controlled user study (N=20) found that explanations can significantly improve user experience in multiple aspects. We discuss how explanations are interwoven into conversation flow and present implications for future OSC designs.

We investigate the use of a miniaturized Near-Infrared Spectroscopy (NIRS) device in an assisted decision-making task. We consider the real-world scenario of determining whether food contains gluten, and we investigate how end-users interact with our NIRS detection device to ultimately make this judgment. In particular, we explore the effects of different nutrition labels and representations of confidence on participants’ perception and trust. Our results show that participants tend to be conservative in their judgment and are willing to trust the device in the absence of understandable label information. We further identify strategies to increase user trust in the system. Our work contributes to the growing body of knowledge on how NIRS can be mass-appropriated for everyday sensing tasks, and how to enhance the trustworthiness of assisted decision-making systems.

Recently, Artificial Intelligence (AI) has been used to enable efficient decision-making in managerial and organizational contexts, ranging from employment to dismissal. However, to avoid employees’ antipathy toward AI, it is important to understand what aspects of AI employees like and/or dislike. In this paper, we aim to identify how employees perceive current human resource (HR) teams and future algorithmic management. Specifically, we explored what factors negatively influence employees’ perceptions of AI making work performance evaluations. Through in-depth interviews with 21 workers, we found that 1) employees feel six types of burdens (i.e., emotional, mental, bias, manipulation, privacy, and social) toward AI's introduction to human resource management (HRM), and that 2) these burdens could be mitigated by incorporating transparency, interpretability, and human intervention to algorithmic decision-making. Based on our findings, we present design efforts to alleviate employees’ burdens. To leverage AI for HRM in fair and trustworthy ways, we call for the HCI community to design human-AI collaboration systems with various HR stakeholders.

Autonomous vehicles could improve mobility, safety, and inclusion in traffic. While this technology seems within reach, its successful introduction depends on the intended user’s acceptance. A substantial factor for this acceptance is trust in the autonomous vehicle’s capabilities. Visualizing internal information processed by an autonomous vehicle could calibrate this trust by enabling the perception of the vehicle’s detection capabilities (and its failures) while only inducing a low cognitive load. Additionally, the simultaneously raised situation awareness could benefit potential take-overs. We report the results of two comparative online studies on visualizing semantic segmentation information for the human user of autonomous vehicles. Effects on trust, cognitive load, and situation awareness were measured using a simulation (N=32) and state-of-the-art panoptic segmentation on a pre-recorded real-world video (N=41). Results show that the visualization using Augmented Reality increases situation awareness while remaining low cognitive load.

Automated Vehicles (AVs) are expected to radically disrupt our mobility. Whereas much is speculated about how AVs will actually be implemented in the future, we argue that their advent should be taken as an opportunity to enhance all people’s mobility and improve their lives. Thus, it is important to focus on both the environment and the needs of target groups that have not been sufficiently considered in the past. In this paper, we present the findings from a qualitative study (N=11) of public attitude on hypothetical implementation scenarios for AVs. Our results indicate that people are aware of the benefits of shared mobility for the environment and society, and are generally open to using it. However, 1) emotional factors mitigate this openness and 2) security concerns were expressed by female participants. We recommend that identified concerns must be addressed to allow AVs fully exploiting their benefits for society and environment.

Policymakers recommend that automated vehicles (AVs) display their automated driving status using an external human-machine interface (eHMI). However, previous studies suggest that a status eHMI is associated with overtrust, which might be overcome by an additional yielding intent message. We conducted a video-based laboratory study (N = 67) to investigate pedestrians’ trust and crossing behavior in repeated encounters with AVs. In a 2x2 between-subjects design, we investigated (1) the occurrence of a malfunction (AV failing to yield) and (2) system transparency (status eHMI vs. status+intent eHMI). Results show that during initial encounters, trust gradually increases and crossing onset time decreases. After a malfunction, trust declines but recovers quickly. In the status eHMI group, trust was reduced more, and participants showed 7.3 times higher odds of colliding with the AV as compared to the status+intent group. We conclude that a status eHMI can cause pedestrians to overtrust AVs and advocate additional intent messages.

Recent automotive research often focuses on automated driving, including the interaction between automated vehicles (AVs) and so-called “vulnerable road users” (VRUs). While road safety statistics and traffic psychology at least define VRUs as pedestrians, cyclists, and motorcyclists, many publications on human-vehicle interaction use the term without even defining it. The actual target group remains unclear. Since each group already poses a broad spectrum of research challenges, a one-fits-all solution seems unrealistic and inappropriate, and a much clearer differentiation is required. To foster clarity and comprehensibility, we propose a literature-based taxonomy providing a structured separation of (vulnerable) road users, designed to particularly (but not exclusively) support research on the communication between VRUs and AVs. It consists of two conceptual hierarchies and will help practitioners and researchers by providing a uniform and comparable set of terms needed for the design, implementation, and description of HCI applications.

Human-drone interaction is a growing topic of interest within HCI research. Researchers propose many innovative concepts for drone applications, but much of this research does not incorporate knowledge on existing applications already adopted by professionals. This limits the validity of said research. To address this limitation, we present our findings from an in-depth interview study with 10 professional drone pilots. Our participants were armed with significant experience and qualifications – pertinent to both drone operations and a set of applications covering diverse industries. Our findings have resulted in design recommendations that should inform both ends and means of human-drone interaction research. These include, but are not limited to: safety-related protocols, insights from domain-specific use cases, and relevant practices outside of hands-on flight.

In this paper, we investigate the human ability to distinguish political social bots from humans on Twitter. Following motivated reasoning theory from social and cognitive psychology, our central hypothesis is that especially those accounts which are opinion-incongruent are perceived as social bot accounts when the account is ambiguous about its nature. We also hypothesize that credibility ratings mediate this relationship. We asked N = 151 participants to evaluate 24 Twitter accounts and decide whether the accounts were humans or social bots. Findings support our motivated reasoning hypothesis for a sub-group of Twitter users (those who are more familiar with Twitter): Accounts that are opinion-incongruent are evaluated as relatively more bot-like than accounts that are opinion-congruent. Moreover, it does not matter whether the account is clearly social bot or human or ambiguous about its nature. This was mediated by perceived credibility in the sense that congruent profiles were evaluated to be more credible resulting in lower perceptions as bots.

In a fully autonomous driving situation, passengers hand over the steering control to a highly automated system. Autonomous driving behaviour may lead to confusion and negative user experience. When establishing such new technology, the user’s acceptance and understanding are crucial factors regarding success and failure. Using a driving simulator and a mobile application, we evaluated if system transparency during and after the interaction can increase the user experience and subjective feeling of safety and control. We contribute an initial guideline for autonomous driving experience design, bringing together the areas of user experience, explainable artificial intelligence and autonomous driving. The AVAM questionnaire, UEQ-S and interviews show that explanations during or after the ride help turn a negative user experience into a neutral one, which might be due to the increased feeling of control. However, we did not detect an effect for combining explanations during and after the ride.

Designing technologies to support mental health in children is a growing area of research. However less is known about how to design with therapists for children with serious emotional behaviour issues. We conducted a contextual enquiry at an organisation that provides therapy treatment for children of trauma backgrounds. A co-design with therapists of a reflective storytelling activity provided insight into how to integrate design research activities with therapy approaches. Our analysis produced a framework summarising the important elements of a facilitated reflective experience. The framework is intended to guide the design of technologies to support safety, connection, and reflection in scaffolding social emotional learning towards improved emotional behaviour for children. Future directions for applying the framework include: augmenting existing therapy activities with technology, technology to support children learning how to self-regulate their emotions outside the clinic, and technology to help parents in emotion coaching their child.

Expressivity is frequently recurring as a term in HCI, but it is often approached from different perspectives. Affective computing prompted research into emotional expressivity, and with technology becoming more ubiquitous and tangible, the opportunities for expressive behaviors towards systems as well as the expressivity of systems increases. By analyzing exemplar research-through-design cases and a literature survey on the use of expressivity in interaction, we discuss how different perspectives and concepts contribute to understand expressivity in interaction. We integrate these perspectives and make them operational for interaction design by creating a framework including design considerations such as freedom of interaction, action-perception loops, multimodality, subtlety, ambiguity, skill development and temporal form. The framework is a result of a mixed-method approach including a review of existing definitions and scholarly artefacts, and a systematic literature review to identify design cases including an analysis of these design cases. We finally illustrate how the framework has been used to inform the design of a shape-changing soft-robotic interface. As a result, we contribute an integrated framework on how to design for expressivity in interaction.

Although audiobooks are increasingly being used, people tend to perceive audiobook experiences as 'not real reading' due to its intangibility and ephemerality. In this paper, we developed ADIO, a device augmenting audiobook experience through representing personal listening state in the form of an interactive physical bookshelf. ADIO displays a user's listening progress through a pendant's changing length and the user's digital audiobook archive titles. The result of our four-week in-field study with six participants revealed that ADIO provided proof of the user's listening-to, which brought a sense of reading and gave a trigger for recalling the listened-to audiobook content. Additionally, audiobooks' improved visibility reminded participants to listen to them, and ADIO's physical interaction allowed participants to form personal patterns for listening to audiobooks. Our findings proposed new methods for augmenting the audiobook listening experience at three stages and further implications for designing physical curation on users’ digital archives.

Service robots are envisioned to be adaptive to their working environment based on situational knowledge. Recent research focused on designing visual representation of knowledge graphs for expert users. However, how to generate an understandable interface for non-expert users remains to be explored. In this paper, we use knowledge graphs (KGs) as a common ground for knowledge exchange and develop a pattern library for designing KG interfaces for non-expert users. After identifying the types of robotic situational knowledge from the literature, we present a formative study in which participants used cards to communicate the knowledge for given scenarios. We iteratively coded the results and identified patterns for representing various types of situational knowledge. To derive design recommendations for applying the patterns, we prototyped a lab service robot and conducted Wizard-of-Oz testing. The patterns and recommendations could provide useful guidance in designing knowledge-exchange interfaces for robots.

As the internet is increasingly embedded in the everyday things in our homes, we notice a need for greater focus on the role care plays in those relationships—and therefore an opportunity to realize unseen potential in reimagining home Internet of Things (IoT). In this paper we report on our inquiry of home dwellers’ relationships to caring for their everyday things and homes (referred to as thingcare). Findings from our design ethnography reveal four thematic qualities of their relationships to thingcare: Care Spectacle, Care Liminality, Ontological Braiding, and Care Condition. Using these themes as touchstones, we co-speculated to produce four speculative IoT concepts to explore what care as a design ethic might look like for IoT and reflect on nascent opportunities and challenges for domestic IoT design. We conclude by considering structures of power and privilege embedded within care practices that critically open new design imaginaries for IoT.

There is growing attention in the HCI community on how technology could be designed to support experiences of reminiscence on past life experiences. Yet, this research has largely overlooked people with blindness. We present a study that aims to understand everyday experiences of reminiscence for people with blindness. We conducted a qualitative study with 9 participants with blindness to understand their personal routines, wishes and desires, and challenges and tensions regarding the experience of reminiscence. Findings are interpreted to discuss new possibilities that offer starting points for future design initiatives and openings for collaboration aimed at creating technology to better support the practices of capturing, sharing, and reflecting on significant memories of the past.

Peer support through social media has been shown to have significant potential to improve health care outcomes. Despite this, very little is understood about how to design a social media-based peer support system. We use the model of unplatformed design to structure a multi-phase design process of a WhatsApp-based peer support system, with and for participants undergoing extreme weight loss as part of a health care intervention into diabetes management. From a mixed-methods evaluation of a three-month deployment of the system and reflections upon the design process we explore the value of the model in facilitating the expression of authentic peer support, and identify how the unique characteristics of unplatformed design allowed for the creation of a peer support system that was responsive to participants’ existing everyday use of social media technologies.

Digital data has become a key part of everyday life: people manage increasingly large and disparate collections of photos, documents, media, etc. But what happens after death? How can users select and prepare what data to leave behind before their eventual death? To explore how to support users, we first ran an ideation workshop to generate design ideas; then, we created a design workbook with 12 speculative concepts that explore diverging approaches and perspectives. We elicited reactions to the concepts from 20 participants (18-81, varied occupations). We found that participants anticipated different types of motivation at different life stages, wished for tools to feel personal and intimate, and preferred individual control on their post-death self-representation. They also found comprehensive data replicas creepy and saw smart assistants as potential aides for suggesting meaningful data. Based on the results, we discuss key directions for designing more personalized and respectful death-preparation tools.

Facilitation, or the craft and use of specific tools, methods, and practices to influence the way groups gather and converse to reach a goal, requires the nuanced craft of a facilitator to guide complex conversations. Online gathering spaces are often designed with neither the facilitator’s knowledge nor the range of tools their methods require, making online facilitation difficult. In this paper, we describe the design and evaluation of Keeper – an online video call extension environment – informed by the widespread, foundational facilitation method of Circle practice. We define three initial facilitator-driven goals of the platform: 1) enable tone setting, 2) ease the practice and smoothness of online conversation, and 3) improve social presence. After a qualitative study with 72 participants, semi-structured interviews with facilitators, and an RCT with 100 participants, we observe improved tone setting and ease of conversation and suggest several ways to further pursue improved online conversation.

Design research is important for understanding and interrogating how emerging technologies shape human experience. However, design research with Machine Learning (ML) is relatively underdeveloped. Crucially, designers have not found a grasp on ML uncertainty as a design opportunity rather than an obstacle. The technical literature points to data and model uncertainties as two main properties of ML. Through post-phenomenology, we position uncertainty as one defining material attribute of ML processes which mediate human experience. To understand ML uncertainty as a design material, we investigate four design research case studies involving ML. We derive three provocative concepts: thingly uncertainty: ML-driven artefacts have uncertain, variable relations to their environments; pattern leakage: ML uncertainty can lead to patterns shaping the world they are meant to represent; and futures creep: ML technologies texture human relations to time with uncertainty. Finally, we outline design research trajectories and sketch a post-phenomenological approach to human-ML relations.

User engagement with data privacy and security through consent banners has become a ubiquitous part of interacting with internet services. While previous work has addressed consent banners from either interaction design, legal, and ethics-focused perspectives, little research addresses the connections among multiple disciplinary approaches, including tensions and opportunities that transcend disciplinary boundaries. In this paper, we draw together perspectives and commentary from HCI, design, privacy and data protection, and legal research communities, using the language and strategies of “dark patterns” to perform an interaction criticism reading of three different types of consent banners. Our analysis builds upon designer, interface, user, and social context lenses to raise tensions and synergies that arise together in complex, contingent, and conflicting ways in the act of designing consent banners. We conclude with opportunities for transdisciplinary dialogue across legal, ethical, computer science, and interactive systems scholarship to translate matters of ethical concern into public policy.

As the role technology plays in relationships between people and their governments grows, developing a better understanding of how trust can inform designing civic technology with trust is urgent work for human computer-interaction researchers. This paper reports our efforts to design with trust through a two-year design-ethnography with the City of Atlanta Mayor's Office of Immigrant Affairs. We developed a sociotechnical system—Code Enforcer—to help this office guide immigrant residents through successfully engaging the city's code enforcement process. To inform the design process, we adapted our framework of trust-as-distance. While the framework was instrumental for integrating issues of trust throughout our design process, it also introduced tensions between how and by whom trust was enacted and interpreted. By reflecting on these tensions, we tease out the political and moral elements of designing with trust vital for HCI to navigate moving forward.

With AI on the boom, DeepFakes have emerged as a tool with a massive potential for abuse. The hyper-realistic imagery of these manipulated videos coupled with the expedited delivery models of social media platforms gives deception, propaganda, and disinformation an entirely new meaning. Hence, raising awareness about DeepFakes and how to accurately flag them has become imperative. However, given differences in human cognition and perception, this is not straightforward. In this paper, we perform an investigative user study and also analyze existing AI detection algorithms from the literature to demystify the unknowns that are at play behind the scenes when detecting DeepFakes. Based on our findings, we design a customized training program to improve detection and evaluate on a treatment group of low-literate population, which is most vulnerable to DeepFakes. Our results suggest that, while DeepFakes are becoming imperceptible, contextualized education and training can help raise awareness and improve detection.

Although the number of influencers is increasing and being an influencer is one of the most frequently mentioned career aspirations of young people, we still know very little about influencers’ motivations and actual practices from the HCI perspective. Driven by the emerging field of Human-Food Interaction and novel phenomena on social media such as Finstas, ASMR, Mukbang and live streaming, we would like to highlight the significance of food influencers as influential content creators and their social media practices. We have conducted a qualitative interview study and analyzed over 1,500 posts of food content creators on Instagram, focusing on practices of content creation, photography, staging, posting, and use of technology. Based on our findings, we have derived a process model that outlines the practices of this rather small, but influential user group. We contribute to the field of HCI by outlining the practices of food influencers as influential content creators within the social media sphere to open up design spaces for interaction researchers and practitioners.

Within the wider open science reform movement, HCI researchers are actively debating how to foster transparency in their own field. Publication venues play a crucial role in instituting open science practices, especially journals, whose procedures arguably lend themselves better to them than conferences. Yet we know little about how much HCI journals presently support open science practices. We identified the 51 most frequently published-in journals by recent CHI first authors and coded them according to the Transparency and Openness Promotion guidelines, a high-profile standard of evaluating editorial practices. Results indicate that journals in our sample currently do not set or specify clear openness and transparency standards. Out of a maximum of 29, the modal score was 0 (mean = 2.5, SD = 3.6, max = 15). We discuss potential reasons, the aptness of natural science-based guidelines for HCI, and next steps for the HCI community in furthering openness and transparency.

Electrodermal activity data is widely used in HCI to capture rich and unbiased signals. Results from related fields, however, have suggested several methodological issues that can arise when practices do not follow established standards. In this paper, we present a systematic methodological review of CHI papers involving the use of EDA data according to best practices from the field of psychophysiology, where standards are well-established and mature. We found severe issues in our sample at all stages of the research process. To ensure the validity of future research, we highlight pitfalls and offer directions for how to improve community standards.

A user's ownership perception of virtual objects, such as cloud files, is generally uncertain. Is this valid for streaming platforms featuring accounts designed for sharing (DS)? We observe sharing practices within DS accounts of streaming platforms and identify their ownership characteristics and unexpected complications through two mixed-method studies. Casual and Cost-splitting are the two sharing practices identified. The owner is the sole payer for the account in the former, whereas profile holders split the cost in the latter. We distinguish two types of ownership in each practice—Primary and Dual. In Primary ownership, the account owner has the power to allow others to use the account; in Dual ownership, Primary ownership appears in conjunction with joint ownership, notably displaying asymmetric ownership perceptions among users. Conflicts arise when the sharing agreements collapse. Therefore, we propose design recommendations that bridge ownership differences based on sharing practices of DS accounts.

Monitoring advertising around controversial issues is an important step in ensuring accountability and transparency of political processes. To that end, we use the Facebook Ads Library to collect 2312 migration-related advertising campaigns in Italy over one year. Our pro- and anti-immigration classifier (F1=0.85) reveals a partisan divide among the major Italian political parties, with anti-immigration ads accounting for nearly 15M impressions. Although composing 47.6% of all migration-related ads, anti-immigration ones receive 65.2% of impressions. We estimate that about two thirds of all captured campaigns use some kind of demographic targeting by location, gender, or age. We find sharp divides by age and gender: for instance, anti-immigration ads from major parties are 17% more likely to be seen by a male user than a female. Unlike pro-migration parties, we find that anti-immigration ones reach a similar demographic to their own voters. However their audience change with topic: an ad from anti-immigration parties is 24% more likely to be seen by a male user when the ad speaks about migration, than if it does not. Furthermore, the viewership of such campaigns tends to follow the volume of mainstream news around immigration, supporting the theory that political advertisers try to “ride the wave” of current news. We conclude with policy implications for political communication: since the Facebook Ads Library does not allow to distinguish between advertisers intentions and algorithmic targeting, we argue that more details should be shared by platforms regarding the targeting configuration of socio-political campaigns.

Maintenance of industrial equipment is done by fitters, electricians and other maintainers. For safety and quality control, maintainers must follow procedures; historically these have been paper-based. Asset-owning organisations seek to transition maintainers to digital platforms. However, there are limited studies on the potential impact of digitisation on maintenance work and the maintainers that perform it. Our challenge is to identify interface design considerations that support the safe and reliable execution of work. We looked specifically at maintenance procedures and conducted semi-structured interviews with process-plant maintainers. Thematic analysis identified eight factors influencing maintainers’ perceptions towards using digital technologies in their work. We map these factors to three categories, work identity, agency and community. These categories are consistent with concepts from the Job Characteristics Model (JCM). The contribution of this work is the relevance of job characteristics in guiding user interface design for maintainers, for which we make a number of recommendations.

Spreadsheet users routinely read, and misread, others' spreadsheets, but literature offers only a high-level understanding of users’ comprehension behaviors. This limits our ability to support millions of users in spreadsheet comprehension activities. Therefore, we conducted a think-aloud study of 15 spreadsheet users who read others’ spreadsheets as part of their work. With qualitative coding of participants’ comprehension needs, strategies and difficulties at 20-second granularity, our study provides the most detailed understanding of spreadsheet comprehension to date.

Participants comprehending spreadsheets spent around 40% of their time seeking additional information needed to understand the spreadsheet. These information seeking episodes were tedious: around 50% of participants reported feeling overwhelmed. Moreover, participants often failed to obtain the necessary information and worked instead with guesses about the spreadsheet. Eventually, 12 out of 15 participants decided to go back to the spreadsheet's author for clarifications. Our findings have design implications for reading as well as writing spreadsheets.

A prominent approach to combating online misinformation is to debunk false content. Here we investigate downstream consequences of social corrections on users’ subsequent sharing of other content. Being corrected might make users more attentive to accuracy, thus improving their subsequent sharing. Alternatively, corrections might not improve subsequent sharing - or even backfire - by making users feel defensive, or by shifting their attention away from accuracy (e.g., towards various social factors). We identified N=2,000 users who shared false political news on Twitter, and replied to their false tweets with links to fact-checking websites. We find causal evidence that being corrected decreases the quality, and increases the partisan slant and language toxicity, of the users’ subsequent retweets (but has no significant effect on primary tweets). This suggests that being publicly corrected by another user shifts one's attention away from accuracy - presenting an important challenge for social correction approaches.

New parents, defined as parents of children between the infant and preschooler stages, are increasingly turning to online media to exchange support and information to help with their life-changing transition. Understanding parents’ discussion online is crucial to the design and development of technologies that can better support their media interaction. This work studies how new parents use online media using a large-scale parenting corpus. To do so, we first employed a card-sorting methodology to identify a set of parenting topics, with which we trained BERT classifiers to automatically identify the topics of Reddit posts. We then investigate at scale what parenting topics were talked about most by new parents, how topics change over the course of their participation, and how interactions with different topics affect members’ engagement in the community. We conclude with implications of our research in designing future research and online parenting communities.

The amount of autonomy in software engineering tools is increasing as developers build increasingly complex systems. We study factors influencing software engineers’ trust in an autonomous tool situated in a high stakes workplace, because research in other contexts shows that too much or too little trust in autonomous tools can have negative consequences. We present the results of a ten week ethnographic case study of engineers collaborating with an autonomous tool to write control software at the National Aeronautics and Space Administration to support high stakes missions. We find that trust in an autonomous software engineering tool in this setting was influenced by four main factors: the tool’s transparency, usability, its social context, and the organization’s associated processes. Our observations lead us to frame trust as a quality the operator places in their collaboration with the automated system, and we outline implications of this framing and other results for researchers studying trust in autonomous systems, designers of software engineering tools, and organizations conducting high stakes work with these tools.

Vehicle manufacturers and government agencies are considering using vehicle-to-pedestrian (V2P) communication to improve pedestrian safety. However, there are unanswered questions about whether people will heed alerts and warnings presented through a smartphone. We conducted between-subject studies with younger and older adults where they physically crossed a virtual street. They received either permissive alerts (safe to cross), prohibitive warnings (not safe to cross), or no alerts or warnings (control). We found that both older and younger adults were highly likely to heed permissive alerts, even when this meant taking gaps between two vehicles that were smaller than they would typically take on their own. We also found that we could shift participants’ road-crossing behavior toward greater caution when they were only alerted to cross very large gaps between two vehicles. Participants stated that alerts and warnings were useful, but that prohibitive warnings were annoying. These findings give insights into V2P design and pedestrian behavior when smartphone assistance is provided.

There is a growing concern that e-commerce platforms are amplifying vaccine-misinformation. To investigate, we conduct two-sets of algorithmic audits for vaccine misinformation on the search and recommendation algorithms of Amazon—world’s leading e-retailer. First, we systematically audit search-results belonging to vaccine-related search-queries without logging into the platform—unpersonalized audits. We find 10.47% of search-results promote misinformative health products. We also observe ranking-bias, with Amazon ranking misinformative search-results higher than debunking search-results. Next, we analyze the effects of personalization due to account-history, where history is built progressively by performing various real-world user-actions, such as clicking a product. We find evidence of filter-bubble effect in Amazon’s recommendations; accounts performing actions on misinformative products are presented with more misinformation compared to accounts performing actions on neutral and debunking products. Interestingly, once user clicks on a misinformative product, homepage recommendations become more contaminated compared to when user shows an intention to buy that product.

With increasing interest in multisensory experiences in HCI there is a need to consider the potential impact of crossmodal correspondences (CCs) between sensory modalities on perception and interpretation. We investigated CCs between active haptic experiences of tangible 3D objects, visual colour and emotion using the “Bouba/Kiki” paradigm. We asked 30 participants to assign colours and emotional categories to 3D-printed objects with varying degrees of angularity and complexity. We found tendencies to associate high degrees of complexity and angularity with red colours, low brightness and high arousal levels. Less complex round shapes were associated with blue colours, high brightness and positive valence levels. These findings contrast previously reported crossmodal effects triggered by 2D shapes of similar angularity and complexity, suggesting that designers cannot simply extrapolate potential perceptual and interpretive experiences elicited by 2D shapes to seemingly similar 3D tangible objects. Instead, we propose a design space for creating tangible multisensory artefacts that can trigger specific emotional percepts and discuss implications for exploiting CCs in the design of interactive technology.

Static illustrations are ubiquitous means to represent interaction scenarios. Across papers and reports, these visuals demonstrate people’s use of devices, explain systems, or show design spaces. Creating such figures is challenging, and very little is known about the overarching strategies for visually representing interaction scenarios. To mitigate this task, we contribute a unified taxonomy of design elements that compose such figures. In particular, we provide a detailed classification of Structural and Interaction strategies, such as composition, visual techniques, dynamics, representation of users, and many others – all in context of the type of scenarios. This taxonomy can inform researchers’ choices when creating new figures, by providing a concise synthesis of visual strategies, and revealing approaches they were not aware of before. Furthermore, to support the community for creating further taxonomies, we also provide three open-source software facilitating the coding process and visual exploration of the coding scheme.

We present a novel, interactive interface for the integrated cleanup, neatening, structuring and vectorization of sketch imagery. Converting scanned raster drawings into vector illustrations is a well-researched set of problems. Our approach is based on a Delaunay subdivision of the raster drawing. We algorithmically generate a colored grouping of Delaunay regions that users interactively refine by dragging and dropping colors. Sketch strokes defined as marking boundaries of different colored regions are automatically neatened using Bézier curves, and turned into closed regions suitable for fills, textures, layering and animation. We show that minimal user interaction using our technique enables better sketch vectorization than state of art automated approaches. A user study, further shows our interface to be simple, fun and easy to use, yet effectively able to process messy images with a mix of construction lines, noisy and incomplete curves, sketched with arbitrary stroke style.

We present CASSIE, a conceptual modeling system in VR that leverages freehand mid-air sketching, and a novel 3D optimization framework to create connected curve network armatures, predictively surfaced using patches with C0 continuity. Our system provides a judicious balance of interactivity and automation, providing a homogeneous 3D drawing interface for a mix of freehand curves, curve networks, and surface patches. Our system encourages and aids users in drawing consistent networks of curves, easing the transition from freehand ideation to concept modeling. A comprehensive user study with professional designers as well as amateurs (N=12), and a diverse gallery of 3D models, show our armature and patch functionality to offer a user experience and expressivity on par with freehand ideation, while creating sophisticated concept models for downstream applications.

Task-irrelevant distractors affect visuo-motor control for target acquisition and studying such effects has already received much attention in human-computer interaction. However, there has been little research into distractor effects on crossing-based interaction. We thus conducted an empirical study on pen-based interfaces to investigate six crossing tasks with distractor interference in comparison to two tasks without it. The six distractor-related tasks differed in movement precision constraint (directional/amplitude), target size, target distance, distractor location and target-distractor spacing. We also developed and experimentally validated six quantitative models for the six tasks. Our results show that crossing targets with distractors had longer average times and similar accuracy than that without distractors. The effects of distractors varied depending on distractor location, target-distractor spacing and movement precision constraint. When spacing is smaller than 11.27 mm, crossing tasks with distractor interference can be regarded as pointing tasks or a combination of pointing and crossing tasks, which could be better fitted with our proposed models than Fitts’ law. According to these results, we provide practical implications to crossing-based user interface design.

Attentional tunneling describes a phenomenon in Augmented Reality (AR) where users excessively focus on virtual content while neglecting their physical surroundings. This leads to the concern that users could neglect hazardous situations when using AR applications. However, studies have often confounded the role of the virtual content with the role of the associated task in inducing attentional tunneling. In this paper, we disentangle the impact of the associated task and of the virtual content on the attentional tunneling effect by measuring reaction times to events in two user studies. We found that presenting virtual content did not significantly increase user reaction times to events, but adding a task to the content did. This work contributes towards our understanding of the attentional tunneling effect on handheld AR devices, and highlights the need to consider both task and context when evaluating AR application usage.

Abstract: Force feedback devices, such as motor-based exoskeletons or wearables based on electrical muscle stimulation (EMS), have the unique potential to accelerate users’ own reaction time (RT). However, this speedup has only been explored while the device is attached to the user. In fact, very little is known regarding whether this faster reaction time still occurs after the user removes the device from their bodies–this is precisely what we investigated by means of a simple reaction time (RT) experiment, in which participants were asked to tap as soon as they saw an LED flashing. Participants experienced this in three EMS conditions: (1) fast-EMS, the electrical impulses were synced with the LED; (2) agency-EMS, the electrical impulse was delivered 40ms faster than the participant’s own RT, which prior work has shown to preserve one’s sense of agency over this movement; and, (3) late-EMS: the impulse was delivered after the participant’s own RT. Our results revealed that the participants’ RT was significantly reduced by approximately 8ms (up to 20ms) only after training with the agency-EMS condition. This finding suggests that the prioritizing agency during EMS training is key to motor-adaptation, i.e., it enables a faster motor response even after the user has removed the EMS device from their body.

The overall pace of interaction combines the user’s pace and the system’s pace, and a pace mismatch could impair user preferences (e.g., animations or timeouts that are too fast or slow for the user). Motivated by studies of speech rate convergence, we conducted an experiment to examine whether user preferences for system pace are correlated with user pace. Subjects first completed a series of trials to determine their user pace. They then completed a series of hierarchical drag-and-drop trials in which folders automatically expanded when the cursor hovered for longer than a controlled timeout. Results showed that preferences for timeout values correlated with user pace – slow-paced users preferred long timeouts, and fast-paced users preferred short timeouts. Results indicate potential benefits in moving away from fixed or customisable settings for system pace. Instead, systems could improve preferences by automatically adapting their pace to converge towards that of the user.

Back-of-device interaction is a promising approach to interacting on smartphones. In this paper, we create a back-of-device command and text input technique called BackSwipe, which allows a user to hold a smartphone with one hand, and use the index finger of the same hand to draw a word-gesture anywhere at the back of the smartphone to enter commands and text. To support BackSwipe, we propose a back-of-device word-gesture decoding algorithm which infers the keyboard location from back-of-device gestures, and adjusts the keyboard size to suit the gesture scales; the inferred keyboard is then fed back into the system for decoding. Our user study shows BackSwipe is feasible and a promising input method, especially for command input in the one-hand holding posture: users can enter commands at an average accuracy of 92% with a speed of 5.32 seconds/command. The text entry performance varies across users. The average speed is 9.58 WPM with some users at 18.83 WPM; the average word error rate is 11.04% with some users at 2.85%. Overall, BackSwipe complements the extant smartphone interaction by leveraging the back of the device as a gestural input surface.

Decoding on phrase-level may afford more correction accuracy than on word-level according to previous research. However, how phrase-level input affects the user typing behavior, and how to design the interaction to make it practical remain under explored. We present PhraseFlow, a phrase-level input keyboard that is able to correct previous text based on the subsequently input sequences. Computational studies show that phrase-level input reduces the error rate of autocorrection by over 16%. We found that phrase-level input introduced extra cognitive load to the user that hindered their performance. Through an iterative design-implement-research process, we optimized the design of PhraseFlow that alleviated the cognitive load. An in-lab study shows that users could adopt PhraseFlow quickly, resulting in 19% fewer error without losing speed. In real-life settings, we conducted a six-day deployment study with 42 participants, showing that 78.6% of the users would like to have the phrase-level input feature in future keyboards.

Sound presents an invaluable signal source that enables computing systems to perform daily activity recognition. However, microphones are optimized for human speech and hearing ranges: capturing private content, such as speech, while omitting useful, inaudible information that can aid in acoustic recognition tasks. We simulated acoustic recognition tasks using sounds from 127 everyday household/workplace objects, finding that inaudible frequencies can act as a substitute for privacy-sensitive frequencies. To take advantage of these inaudible frequencies, we designed a Raspberry Pi-based device that captures inaudible acoustic frequencies with settings that can remove speech or all audible frequencies entirely. We conducted a perception study, where participants “eavesdropped’’ on PrivacyMic’s filtered audio and found that none of our participants could transcribe speech. Finally, PrivacyMic’s real-world activity recognition performance is comparable to our simulated results, with over 95% classification accuracy across all environments, suggesting immediate viability in performing privacy-preserving daily activity recognition.

When users complete tasks on the computer, the knowledge they leverage and their intent is often lost because it is tedious or challenging to capture. This makes it harder to understand why a colleague designed a component a certain way or to remember requirements for software you wrote a year ago. We introduce think-aloud computing, a novel application of the think-aloud protocol where computer users are encouraged to speak while working to capture rich knowledge with relatively low effort. Through a formative study we find people shared information about design intent, work processes, problems encountered, to-do items, and other useful information. We developed a prototype that supports think-aloud computing by prompting users to speak and contextualizing speech with labels and application context. Our evaluation shows more subtle design decisions and process explanations were captured in think-aloud than via traditional documentation. Participants reported that think-aloud required similar effort as traditional documentation.

Various contact tracing approaches have been applied to help contain the spread of COVID-19, with technology-based tracing and human tracing among the most widely adopted. However, governments and communities worldwide vary in their adoption of digital contact tracing, with many instead choosing the human approach. We investigate how people perceive the respective benefits and risks of human and digital contact tracing through a mixed-methods survey with 291 respondents from the United States. Participants perceived digital contact tracing as more beneficial for protecting privacy, providing convenience, and ensuring data accuracy, and felt that human contact tracing could help provide security, emotional reassurance, advice, and accessibility. We explore the role of self-tracking technologies in public health crisis situations, highlighting how designs must adapt to promote societal benefit rather than just self-understanding. We discuss how future digital contact tracing can better balance the benefits of human tracers and technology amidst the complex contact tracing process and context.

The rise of ridesharing platforms has transformed traditional transportation, making it more accessible for getting to work and accessing grocery stores and healthcare providers, which are essential to physical and mental well-being. However, such technologies are not available everywhere. Additionally, there is a scarcity of HCI work that investigates how vulnerable populations such as rural-dwelling people with HIV face and overcome transportation barriers. To extend past research, we conducted 31 surveys and 18 interviews with people living with HIV (22 surveys, 14 interviews) and their case coordinators (9 surveys, 4 interviews) in rural areas. Contrary to past research, we found that the use of alternative vehicles, extensive support networks, and nonprofit health organizations facilitated transportation. However, distance, the lack of trust and infrastructure, stigma, and other cultural underpinnings made popular forms of urban transportation unappealing. We contextualize our findings with prior research and contribute implications for future research and design.

Polycystic Ovary Syndrome (PCOS) is a condition that causes hormonal imbalance and infertility in women and people with female reproductive organs. PCOS causes different symptoms for different people, with no singular or universal cure. Being a stigmatized and enigmatic condition, it is challenging to discover, diagnose, and manage PCOS. This work aims to inform the design of inclusive health technologies through an understanding of people’s lived experiences and challenges with PCOS. We conducted semi-structured interviews with 10 women diagnosed with PCOS and analyzed a PCOS-specific subreddit forum. We report people’s support-seeking, sense-making, and self-experimentation practices, and find uncertainty and stigma to be key in shaping their unique experiences of the condition. We further identify potential avenues for designing technology to support their diverse needs, such as personalized and contextual tracking, accelerated self-discovery, and co-management, contributing to a growing body of HCI literature on stigmatized topics in women’s health and well-being.

Physical activity (PA) is crucial for reducing the risk of obesity, an epidemic that disproportionately burdens families of low-socioeconomic status (SES). While fitness tracking tools can increase PA awareness, more work is needed to examine (1) how such tools can help people benefit from their social environment, and (2) how reflections can help enhance PA attitudes. We investigated how fitness tracking tools for families can support social modeling and self-modeling (through reflection), two critical processes in Social Cognitive Theory. We developed StoryMap, a novel fitness tracking app for families aimed at supporting both modes of modeling. Then, we conducted a five-week qualitative study evaluating StoryMap with 16 low-SES families. Our findings contribute an understanding of how social and self-modeling can be implemented in fitness tracking tools and how both modes of modeling can enhance key PA attitudes: self-efficacy and outcome expectations. Finally, we propose design recommendations for social personal informatics tools.

Fertility tracking and technology are characterized by logging varied health-related data potentially associated with female fertility cycles. Such data are often seen as private and restricted to the individual level. We conducted an interview study with 21 people (16 in the U.S.) facing challenges to conceive and 5 U.S. healthcare providers specialized in infertility to analyze (in)fertility experiences with data. Our findings suggest that although fertility data are considered personal and private, they are embedded in larger ecological systems of use, influencing and being influenced by different stakeholders, institutional contexts, and sociocultural factors. Leveraging the Ecological Systems Theory, we analyze the relationships and factors shaping individuals’ fertility trajectories, discussing how the different layers influence the work individuals have to engage and the burden imposed on them through various social, institutional, and cultural boundaries. We propose an ecological perspective on fertility data practices and discuss opportunities to counter-influence broader environmental systems through data tracking.

Transgender people face difficulties accessing healthcare from providers, and thus often turn to online sources to seek health information. However, online platforms may not properly support trans health information seeking, and health information found online may be limited in accuracy. To examine how online platforms can best support trans health information seeking, we conducted online focus groups with trans people (n = 26) about their experiences with online health information and their needs and desires for online health information seeking platforms. We found that trans people face both facilitators and barriers to finding accurate, actionable information online. Facilitators include online community discovery, group privacy features, and the dual synchronous and asynchronous nature of online content. Barriers include platform censorship, misinformation, hate speech, and lack of tools to flag inaccurate content. We provide recommendations for how platforms can support trans health information seeking by ensuring that medical information is accurate, accessible, easy to locate, and relevant to a diverse set of trans identities and experiences.

Self-tracking can help personalize self-management interventions for chronic conditions like type 2 diabetes (T2D), but reflecting on personal data requires motivation and literacy. Machine learning (ML) methods can identify patterns, but a key challenge is making actionable suggestions based on personal health data. We introduce GlucoGoalie, which combines ML with an expert system to translate ML output into personalized nutrition goal suggestions for individuals with T2D. In a controlled experiment, participants with T2D found that goal suggestions were understandable and actionable. A 4-week in-the-wild deployment study showed that receiving goal suggestions augmented participants’ self-discovery, choosing goals highlighted the multifaceted nature of personal preferences, and the experience of following goals demonstrated the importance of feedback and context. However, we identified tensions between abstract goals and concrete eating experiences and found static text too ambiguous for complex concepts. We discuss implications for ML-based interventions and the need for systems that offer more interactivity, feedback, and negotiation.

Recently, consumer-facing health technologies such as Artificial Intelligence (AI)-based symptom checkers (AISCs) have sprung up in everyday healthcare practice. AISCs solicit symptom information from users and provide medical suggestions and possible diagnoses, a responsibility that people usually entrust with real-person authorities such as physicians and expert patients. Thus, the advent of AISCs begs a question of whether and how they transform the notion of medical authority in people's everyday healthcare practice. To answer this question, we conducted an interview study with thirty AISC users. We found that users assess the medical authority of AISCs using various factors including AISCs’ automated decisions and interaction design patterns, associations with established medical authorities like hospitals, and comparisons with other health technologies. We reveal how AISCs are used in healthcare delivery, discuss how AI transforms conventional understandings of medical authority, and derive implications for designing AI-enabled health technology.

People with food hypersensitivities experience adverse reactions when eating certain foods and thus need to adapt their diet. When dining out, the challenge is greater as people entrust the care of their allergy, intolerance, or celiac disease, in the hands of staff who might not have enough knowledge to appropriately care for them. This interview study explored how people with food hypersensitivities avoid reactions while eating out, to inspire future digital technology design. Our findings show the social and emotional impact of food hypersensitivities and how people practically cope by investigating restaurants’ safety precautions, correcting orders, or even educating restaurants’ staff. We discuss our findings against the experiences of other people living with chronic conditions and offer design opportunities for digital technologies to enhance dining out experiences of people with food hypersensitivities.

Public controversies around the unethical use of personal data are increasing, spotlighting data ethics as an increasingly important field of study. MyData is a related emerging vision that emphasizes individuals’ control of their personal data. In this paper, we investigate people’s perceptions of various data management scenarios by measuring the perceived ethicality and level of felt concern concerning the scenarios. We deployed a set of 96 unique scenarios to an online crowdsourcing platform for assessment and invited a representative sample of the participants to a second-stage questionnaire about the MyData vision and its potential in the field of healthcare. Our results provide a timely investigation into how topical data-related practices affect the perceived ethicality and the felt concern. The questionnaire analysis reveals great potential in the MyData vision. Through the combined quantitative and qualitative results, we contribute to the field of data ethics.

Personal health informatics continues to grow in both research and practice, revealing many challenges of designing applications that address people’s needs in their health, everyday lives, and collaborations with clinicians. Research suggests strategies to address such challenges, but has struggled to translate these strategies into design practice. This study examines translation of insights from personal health informatics research into resources to support designers. Informed by a review of relevant literature, we present our development of a prototype set of design cards intended to support designers in re-thinking potential assumptions about personal health informatics. We examined our design cards in semi-structured interviews, first with 12 student designers and then with 12 health-focused professional designers and researchers. Our results and discussion reveal tensions and barriers designers encounter, the potential for translational resources to inform the design of health-related technologies, and a need to support designers in addressing challenges of knowledge, advocacy, and evidence in designing for health.

Breastfeeding brings benefits for newborns and parents, but can be a challenging process. In this paper, we leverage a mixed-methods approach that builds on the Integrated Behavioural Model (IBM) to explore parents’ perspectives toward breastfeeding along with their lived experiences, and examine the role of technology in this setting. Results of twelve semi-structured interviews and 175 online survey responses suggest generally positive attitudes toward breastfeeding and good theoretical knowledge. This is combined with a complex lived experience of breastfeeding where main challenges are situated in practical, emotional, and environmental/societal aspects, which are currently not sufficiently recognised by technology that seeks to support breastfeeding. Building upon our findings, we present points for reflection for the design of technology to support breastfeeding, focusing on the importance of drawing from the lived experience of parents, and ensuring that technology not only casts breastfeeding as an individual but also as a collective effort.

Beyond a simple notification of incoming calls or messages, more complex information such as alphabets and digits can be delivered through spatiotemporal tactile patterns (STPs) on a wrist-worn tactile display (WTD) with multiple tactors. However, owing to the limited skin area and spatial acuity of the wrist, frequent confusions occur between closely located tactors, resulting in a low recognition accuracy. Furthermore, the accuracies reported in previous studies have mostly been measured for a specific posture and could further decrease with free arm postures in real life. Herein, we present Heterogeneous Stroke, a design concept for improving the recognition accuracy of STPs on a WTD. By assigning unique vibrotactile stimuli to each tactor, the confusion between tactors can be reduced. Through our implementation of Heterogeneous Stroke, the alphanumeric characters could be delivered with high accuracy (93.8% for 26 alphabets and 92.4% for 10 digits) across different arm postures.

We propose a new type of haptic actuator, which we call MagnetIO, that is comprised of two parts: one battery-powered voice-coil worn on the user's fingernail and any number of interactive soft patches that can be attached onto any surface (everyday objects, user's body, appliances, etc.). When the user's finger wearing our voice-coil contacts any of the interactive patches it detects its magnetic signature via magnetometer and vibrates the patch, adding haptic feedback to otherwise input-only interactions. To allow these passive patches to vibrate, we make them from silicone with regions doped with polarized neodymium powder, resulting in soft and stretchable magnets. This stretchable form-factor allows them to be wrapped to the user's body or everyday objects of various shapes. We demonstrate how these add haptic output to many situations, such as adding haptic buttons to the walls of one's home. In our technical evaluation, we demonstrate that our interactive patches can be excited across a wide range of frequencies (0-500 Hz) and can be tuned to resonate at specific frequencies based on the patch's geometry. Furthermore, we demonstrate that MagnetIO's vibration intensity is as powerful as a typical linear resonant actuator (LRA); yet, unlike these rigid actuators, our passive patches operate as springs with multiple modes of vibration, which enables a wider band around its resonant frequency than an equivalent LRA.

We propose ThermoCaress, a haptic device to create a stroking sensation on the forearm using pressure force and present thermal feedback simultaneously. In our method, based on the phenomenon of thermal referral, by overlapping a stroke of pressure force, users feel as if the thermal stimulation moves although the position of temperature source is static. We designed the device to be compact and soft, using microblowers and inflatable pouches for presenting pressure force and water for presenting thermal feedback. Our user study showed that the device succeeded in generating thermal referrals and creating a moving thermal illusion. The results also suggested that cold temperature enhance the pleasantness of stroking. Our findings contribute to expanding the potential of thermal haptic devices.

In ad-hoc human-robot collaboration (HRC), humans and robots work on a task without pre-planning the robot's actions prior to execution; instead, task allocation occurs in real-time. However, prior research has largely focused on task allocations that are pre-planned - there has not been a comprehensive exploration or evaluation of techniques where task allocation is adjusted in real-time. Inspired by HCI research on territoriality and proxemics, we propose a design space of novel task allocation techniques including both explicit techniques, where the user maintains agency, and implicit techniques, where the efficiency of automation can be leveraged. The techniques were implemented and evaluated using a tabletop HRC simulation in VR. A 16-participant study, which presented variations of a collaborative block stacking task, showed that implicit techniques enable efficient task completion and task parallelization, and should be augmented with explicit mechanisms to provide users with fine-grained control.

We propose a technique that allows an unprecedented level of dexterity in electrical muscle stimulation (EMS), i.e., it allows interactive EMS-based devices to flex the user's fingers independently of each other. EMS is a promising technique for force feedback because of its small form factor when compared to mechanical actuators. However, the current EMS approach to flexing the user's fingers (i.e., attaching electrodes to the base of the forearm, where finger muscles anchor) is limited by its inability to flex a target finger's metacarpophalangeal (MCP) joint independently of the other fingers. In other words, current EMS devices cannot flex one finger alone, they always induce unwanted actuation to adjacent fingers. To tackle the lack of dexterity, we propose and validate a new electrode layout that places the electrodes on the back of the hand, where they stimulate the interossei/lumbricals muscles in the palm, which have never received attention with regards to EMS. In our user study, we found that our technique offers four key benefits when compared to existing EMS electrode layouts: our technique (1) flexes all four fingers around the MCP joint more independently; (2) has less unwanted flexion of other joints (such as the proximal interphalangeal joint); (3) is more robust to wrist rotations; and (4) reduces calibration time. Therefore, our EMS technique enables applications for interactive EMS systems that require a level of flexion dexterity not available until now. We demonstrate the improved dexterity with four example applications: three musical instrumental tutorials (piano, drum, and guitar) and a VR application that renders force feedback in individual fingers while manipulating a yo-yo.

Compared to grounded force feedback, providing tactile feedback via a wearable device can free the user and broaden the potential applications of simulated physical interactions. However, neither the limitations nor the full potential of tactile-only feedback have been precisely examined. Here we investigate how the dimensionality of cutaneous fingertip feedback affects user movements and virtual object recognition. We combine a recently invented 6-DOF fingertip device with motion tracking, a head-mounted display, and novel contact-rendering algorithms to enable a user to tactilely explore immersive virtual environments. We evaluate rudimentary 1-DOF, moderate 3-DOF, and complex 6-DOF tactile feedback during shape discrimination and mass discrimination, also comparing to interactions with real objects. Results from 20 naive study participants show that higher-dimensional tactile feedback may indeed allow completion of a wider range of virtual tasks, but that feedback dimensionality surprisingly does not greatly affect the exploratory techniques employed by the user.

Drone assisted navigation aids for supporting walking activities of visually impaired have been established in related work but fine-point object grasping tasks and the object localization in unknown environments still presents an open and complex challenge. We present a drone-based interface that provides fine-grain haptic feedback and thus physically guides them in hand-object localization tasks in unknown surroundings. Our research is built around community groups of blind or visually impaired (BVI) people, which provide in-depth insights during the development process and serve later as study participants. A pilot study infers users’ sensibility to applied guiding stimuli forces and the different human-drone tether interfacing possibilities. In a comparative follow-up study, we show that our drone-based approach achieves greater accuracy compared to a current audio-based hand guiding system and delivers overall a more intuitive and relatable fine-point guiding experience.

Today’s typical input device is flat, rigid and made of glass. However, advances in sensing technology and interaction design suggest thinking about input on other surface, including soft materials. While touching rigid and soft materials might feel similar, they clearly feel different when pressure is applied to them. Yet, to date, studies only investigated force input on rigid surfaces. We present a first systematic evaluation of the effects of compliance on force input. Results of a visual targeting task for three levels of softness indicate that high force levels appear more demanding for soft surfaces, but that performance is otherwise similar. Performance remained very high (∼ 5% for 20 force levels) regardless of the compliance, suggesting force input was underestimated so far. We infer implications for the design of force input on soft surfaces and conclude that interaction models used on rigid surfaces might be used on soft surfaces.

Providing haptic feedback when manipulating virtual objects is an essential part of immersive virtual reality experiences; however, it is challenging to replicate all of an object's properties and characteristics. We propose the use of visuo-haptic illusions alongside physical proxies to enhance the scope of proxy-based interactions with virtual objects. In this work, we focus on two manipulation techniques, linear translation and stretching across different distances, and investigate how much discrepancy between the physical proxy and the virtual object may be introduced without participants noticing. In a study with 24 participants, we found that manipulation technique and travel distance significantly affect the detection thresholds, and that visuo-haptic illusions impact performance and accuracy. We show that this technique can be used to enable functional proxy objects that act as stand-ins for multiple virtual objects, illustrating the technique through a showcase VR-DJ application.

Different form factors of wearable technology provide unique opportunities for output based on how they are connected to the human body. In this work, we investigate the idea of delivering notifications through devices worn on the underside of a user’s clothing. A wearable worn in such a manner is in direct contact with the user’s skin. We leverage this proximity to test the performance of 10 on-skin sensations (Press, Poke, Pinch, Heat, Cool, Blow, Suck, Vibrate, Moisture and Brush) as methods of notification delivery. We developed prototypes for each stimulus and conducted a user study to evaluate them across 6 locations commonly covered by upper body clothing. Results indicate significant differences in reaction time, error rates and comfort which may influence the design of future under-clothing wearables.

Traditional goal setting simply assumes a binary outcome for goal evaluation. This binary judgment does not consider a user’s effort, which may demotivate the user. This work explores the possibility of mitigating this negative impact with a slight modification on the goal evaluation criterion, by introducing a ‘margin’ that is widely used for quality control in the manufacturing fields. A margin represents a range near the goal where the user’s outcome will be regarded as ‘good enough’ even if the user fails to reach it. We explore users’ perceptions and behaviors through a large-scale survey study and a small-scale field experiment using a coaching system to promote physical activity. Our results provide positive evidence on the margin, such as lowering the burden of goal achievement and increasing motivation to make attempts. We discuss practical design implications on margin-enabled goal setting and evaluation for behavioral change support systems.

We present a qualitative study of a six-month pilot of WhatsApp-based facilitated peer support groups, serving youth living with human immunodeficiency virus (HIV) in an informal settlement in Nairobi, Kenya. Popular chat apps are increasingly being leveraged to make a combination of patient-provider communication and peer support more accessible beyond formal healthcare settings. However, how these interventions are experienced in Global South contexts with phone sharing and intermittent data access is understudied. The context of stigmatized illnesses like HIV further complicates privacy concerns. We draw on chat records and interviews with youth and the facilitator to describe their experience of the intervention. We find that despite tensions in group dynamics, intermittent participation, and contingencies around privacy, youth were motivated by newfound aspirations and community to manage their health. We use our findings to discuss implications for the design of chat-based peer interventions, negotiation of privacy in mobile health applications, and the role of aspirations in health interventions.

Health and well-being are integral parts of the human experience, and yet due to numerous factors are inaccessible for many communities. The transgender community is no exception and faces increased risk for both physical and mental illness. This population faces many unique challenges before, during, and after transition. To gain a deeper understanding of the trans community’s health and well-being needs, we conducted twenty-one interviews with transgender individuals to determine how they navigated their identities and transitions. From our interviews, we examine and highlight the unique needs of the trans population with respect to health, well-being, identity, and transition. We discuss how designers can better understand and accommodate the diversity of this community, give suggestions for the design of technologies for trans health and well-being, and contribute open areas of research.

Interfaces designed to elicit empathy provide an opportunity for HCI with important pro-social outcomes. Recent research has demonstrated that perceiving expressive biosignals can facilitate emotional understanding and connection with others, but this work has been largely limited to visual approaches. We propose that hearing these signals will also elicit empathy, and test this hypothesis with sounding heartbeats. In a lab-based within-subjects study, participants (N = 27) completed an emotion recognition task in different heartbeat conditions. We found that hearing heartbeats changed participants’ emotional perspective and increased their reported ability to “feel what the other was feeling.” From these results, we argue that auditory heartbeats are well-suited as an empathic intervention, and might be particularly useful for certain groups and use-contexts because of its musical and non-visual nature. This work establishes a baseline for empathic auditory interfaces, and offers a method to evaluate the effects of future designs.

Homesickness, which refers to feelings of distress caused by separation from home, is prevalent among university-aged students. Chronic homesickness can exacerbate mood problems, erode academic performance and lead to dropout from school. The present research examines how students use digital technologies to resolve the experience of missing home. Qualitative interviews and diaries with 50 students at major Australian universities revealed that technologies play a significant role in alleviating homesickness. Specifically, students use technologies to acquire social contact, find help and support, build co-presence to recreate their home, connect with culture, experience distant places, and regulate emotions. However, the use of technology sometimes led to increased emotional labour, frequent exposure to homesickness triggers, and heightened perceptions of distance. We conclude by discussing possible design implications for new technologies that allow students to alleviate homesickness by experiencing their home from afar.

The ubiquity of self-tracking devices and smartphone apps has empowered people to collect data about themselves and try to self-improve. However, people with little to no personal analytics experience may not be able to analyze data or run experiments on their own (self-experiments). To lower the barrier to intervention-based self-experimentation, we developed an app called Self-E, which guides users through the experiment. We conducted a 2-week diary study with 16 participants from the local population and a second study with a more advanced group of users to investigate how they perceive and carry out self-experiments with the help of Self-E, and what challenges they face. We find that users are influenced by their preconceived notions of how healthy a given behavior is, making it difficult to follow Self-E’s directions and trusting its results. We present suggestions to overcome this challenge, such as by incorporating empathy and scaffolding in the system.

We propose TiltChair, an actuated office chair that physically manipulates the user’s posture by actively inclining the chair’s seat to address problems associated with prolonged sitting. The system controls the inclination angle and motion speed with the aim of achieving manipulative but unobtrusive posture guidance. To demonstrate its potential, we first built a prototype of TiltChair with a seat that could be tilted by pneumatic control. We then investigated the effects of the seat’s inclination angle and motions on task performance and overall sitting experience through two experiments. The results show that the inclination angle mainly affects the difficulty of maintaining one’s posture, while the motion speed affected the conspicuousness and subjective acceptability of the motion. However, these seating conditions did not affect objective task performance. Based on these results, we propose a design space for facilitating effective seat-inclination behavior using the three dimensions of angle, speed, and continuity. Furthermore, we discuss promising applications.

How effectively do we adhere to nudges and interventions that help us control our online browsing habits? If we have a temporary lapse and disable the behavior change system, do we later resume our adherence, or has the dam broken? In this paper, we investigate these questions through log analyses of 8,000+ users on HabitLab, a behavior change platform that helps users reduce their time online. We find that, while users typically begin with high-challenge interventions, over time they allow themselves to slip into easier and easier interventions. Despite this, many still expect to return to the harder interventions imminently: they repeatedly choose to be asked to change difficulty again on the next visit, declining to have the system save their preference for easy interventions.

A commitment device, an attempt to bind oneself for a successful goal achievement, has been used as an effective strategy to promote behavior change. However, little is known about how commitment devices are used in the wild, and what aspects of commitment devices are related to goal achievements. In this paper, we explore a large-scale dataset from stickK, an online behavior change support system that provides both financial and social commitments. We characterize the patterns of behavior change goals (e.g., topics and commitment setting) and then perform a series of multilevel regression analyses on goal achievements. Our results reveal that successful goal achievements are largely dependent on the configuration of financial and social commitment devices, and a mixed commitment setting is considered beneficial. We discuss how our findings could inform the design of effective commitment devices, and how large-scale data can be leveraged to support data-driven goal elicitation and customization.

In India and other developing countries, Community Health Workers (CHWs) provide the first line of care in delivering necessary maternal and child health services. In this work, we assess the training and skill-building needs of CHWs, through a mobile-based training intervention deployed for six months to 500 CHWs for conducting 144 training sessions in rural India. We qualitatively probed 1178 questions, asked by CHWs, during training sessions, and conducted a content analysis of the learning material provided to CHWs. Further, we interviewed 48 CHWs to understand the rationale of information seeking and perceptions of training needs. We present our understanding of the knowledge gaps of CHWs and how the current learning material and training methods are ineffective in addressing it. Our study presents design implications for HCI4D researchers for mobile learning platforms targeted towards CHWs. We also provide policy-level suggestions to improve the training of CHWs in India or a similar context.

We present findings from a three country study exploring the intersection between female intimate health and religious beliefs. Through a qualitative study with Muslim female populations in Pakistan, Bangladesh and Malaysia, three different Muslim majority contexts, we examine the deep impact Islamic beliefs have on female intimate health and well-being. Our study investigates the perceptions, attitudes and behaviours of Muslim women to their own intimate and sexual bodies through their experiences of menarche, marriage and reproduction and menopause. The intersection of religion and female sexual bodies and health is a neglected area within HCI and we highlight how inextricably specific Islamic values are linked with women’s reproductive health in Muslim communities. We further discuss the opportunities and challenges of designing technologies for religious, non-secular beliefs and values with the aim to improve intimate health practices amongst Muslim women and to broaden the scope of health design within HCI.

Nurses frequently transfer patients as part of their daily work. However, manual patient transfers pose a major risk to nurses’ health. Although the Kinaesthetics care conception can help address this issue, existing support to learn the concept is low. We present KiTT, a tablet-based system, to promote the learning of ergonomic patient transfers based on the Kinaesthetics care conception. KiTT supports the training of Kinaesthetics-based patient transfers by two nurses. The nurses are guided by the phases (i) interactive instructions, (ii) training of transfer conduct, and (iii) feedback and reflection. We evaluated KiTT with 26 nursing-care students in a nursing-care school. Our results indicate that KiTT provides a good subjective support for the learning of Kinaesthetics. Our results also suggest that KiTT can promote the ergonomically correct conduct of patient transfers while providing a good user experience adequate to the nursing-school context, and reveal how KiTT can extend existing practices.

Studies show that users do not reliably click more often on headlines classified as clickbait by automated classifiers. Is this because the linguistic criteria (e.g., use of lists or questions) emphasized by the classifiers are not psychologically relevant in attracting interest, or because their classifications are confounded by other unknown factors associated with assumptions of the classifiers? We address these possibilities with three studies—a quasi-experiment using headlines classified as clickbait by three machine-learning models (Study 1), a controlled experiment varying the headline of an identical news story to contain only one clickbait characteristic (Study 2), and a computational analysis of four classifiers using real-world sharing data (Study 3). Studies 1 and 2 revealed that clickbait did not generate more curiosity than non-clickbait. Study 3 revealed that while some headlines generate more engagement, the detectors agreed on a classification only 47% of the time, raising fundamental questions about their validity.

How to attribute responsibility for autonomous artificial intelligence (AI) systems’ actions has been widely debated across the humanities and social science disciplines. This work presents two experiments (N=200 each) that measure people’s perceptions of eight different notions of moral responsibility concerning AI and human agents in the context of bail decision-making. Using real-life adapted vignettes, our experiments show that AI agents are held causally responsible and blamed similarly to human agents for an identical task. However, there was a meaningful difference in how people perceived these agents’ moral responsibility; human agents were ascribed to a higher degree of present-looking and forward-looking notions of responsibility than AI agents. We also found that people expect both AI and human decision-makers and advisors to justify their decisions regardless of their nature. We discuss policy and HCI implications of these findings, such as the need for explainable AI in high-stakes scenarios.

In order to support fairness-forward thinking by machine learning (ML) practitioners, fairness researchers have created toolkits that aim to transform state-of-the-art research contributions into easily-accessible APIs. Despite these efforts, recent research indicates a disconnect between the needs of practitioners and the tools offered by fairness research. By engaging 20 ML practitioners in a simulated scenario in which they utilize fairness toolkits to make critical decisions, this work aims to utilize practitioner feedback to inform recommendations for the design and creation of fair ML toolkits. Through the use of survey and interview data, our results indicate that though fair ML toolkits are incredibly impactful on users’ decision-making, there is much to be desired in the design and demonstration of fairness results. To support the future development and evaluation of toolkits, this work offers a rubric that can be used to identify critical components of Fair ML toolkits.

With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely follow a model’s predictions when it is beneficial for them to do so or enabling them to detect when a model has made a mistake. We present a sequence of pre-registered experiments (N = 3, 800) in which we showed participants functionally identical models that varied only in two factors commonly thought to make machine learning models more or less interpretable: the number of features and the transparency of the model (i.e., whether the model internals are clear or black box). Predictably, participants who saw a clear model with few features could better simulate the model’s predictions. However, we did not find that participants more closely followed its predictions. Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model’s sizable mistakes, seemingly due to information overload. These counterintuitive findings emphasize the importance of testing over intuition when developing interpretable models.

In Human-AI collaborative settings that are inherently interactive, direction of communication plays a role in how users perceive their AI partners. In an AI-driven cooperative game with partially observable information, players (be it the AI or the human player) require their actions to be interpreted accurately by the other player to yield a successful outcome. In this paper, we investigate social perceptions of AI agents with various directions of communication in a cooperative game setting. We measure subjective social perceptions (rapport, intelligence, and likeability) of participants towards their partners when participants believe they are playing with an AI or with a human and the nature of the communication (responsiveness and leading roles). We ran a large scale study on Mechanical Turk (n=199) of this collaborative game and find significant differences in gameplay outcome and social perception across different AI agents, different directions of communication and when the agent is perceived to be an AI/Human. We find that the bias against the AI that has been demonstrated in prior studies varies with the direction of the communication and with the AI agent.

Artificial Intelligence (AI) education is an increasingly popular topic area for K-12 teachers. However, little research has investigated how AI curriculum and tools can be designed to be more accessible to all teachers and learners. In this study, we take a Value-Sensitive Design approach to understanding the role of teacher values in the design of AI curriculum and tools, and identifying opportunities to integrate AI into core curriculum to leverage learners’ interests. We organized co-design workshops with 15 K-12 teachers, where teachers and researchers co-created lesson plans using AI tools and embedding AI concepts into various core subjects. We found that K-12 teachers need additional scaffolding in AI tools and curriculum to facilitate ethics and data discussions, and value supports for learner evaluation and engagement, peer-to-peer collaboration, and critical reflection. We present an exemplar lesson plan that shows entry points for teaching AI in non-computing subjects and reflect on co-designing with K-12 teachers in a remote setting.

In this paper, we report on perceptual experiments indicating that there are distinct and quantitatively measurable differences in the way we visually perceive genuine versus face-swapped videos.

Recent progress in deep learning has made face-swapping techniques a powerful tool for creative purposes, but also a means for unethical forgeries. Currently, it remains unclear why people are misled, and which indicators they use to recognize potential manipulations. Here, we conduct three perceptual experiments focusing on a wide range of aspects: the conspicuousness of artifacts, the viewing behavior using eye tracking, the recognition accuracy for different video lengths, and the assessment of emotions.

Our experiments show that responses differ distinctly when watching manipulated as opposed to original faces, from which we derive perceptual cues to recognize face swaps. By investigating physiologically measurable signals, our findings yield valuable insights that may also be useful for advanced algorithmic detection.

Domestic robots such as vacuum cleaners or lawnmowers are becoming popular consumer products in private homes, but while current HCI research on domestic robots has highlighted for example personalisation, long-term effects, or design guidelines, little attention has been paid to automation. To address this, we conducted a qualitative study with 24 participants in private households using interviews, contextual technology tours, and robot deployment. Through thematic analysis we identified three themes related to 1) work routines and automation, 2) domestic robot automation and the physical environment, as well as 3) interaction and breakdown intervention. We present an empirical understanding of how task automation using domestic robots can be implemented in the home. Lastly, we discuss our findings in relation to existing literature and highlight three opportunities for improved task automation using domestic robots for future research.

This work investigates the practices and challenges of voice user interface (VUI) designers. Existing VUI design guidelines recommend that designers strive for natural human-agent conversation. However, the literature leaves a critical gap regarding how designers pursue naturalness in VUIs and what their struggles are in doing so. Bridging this gap is necessary for identifying designers’ needs and supporting them. Our interviews with 20 VUI designers identified 12 ways that designers characterize and approach naturalness in VUIs. We categorized these characteristics into three groupings based on the types of conversational context that each characteristic contributes to: Social, Transactional, and Core. Our results contribute new findings on designers’ challenges, such as a design dilemma in augmenting task-oriented VUIs with social conversations, difficulties in writing for spoken language, lack of proper tool support for imbuing synthesized voice with expressivity, and implications for developing design tools and guidelines.

Pair programming has a documented history of benefits, such as increased code quality, productivity, self-efficacy, knowledge transfer, and reduced gender gap. Research uncovered problems with pair programming related to scheduling, collocating, role imbalance, and power dynamics. We investigated the trade-offs of substituting a human with an agent to simultaneously provide benefits and alleviate obstacles in pair programming. We conducted gender-balanced studies with human-human pairs in a remote lab with 18 programmers and Wizard-of-Oz studies with 14 programmers, then analyzed results quantitatively and qualitatively. Our comparative analysis of the two studies showed no significant differences in productivity, code quality, and self-efficacy. Further, agents facilitated knowledge transfer; however, unlike humans, agents were unable to provide logical explanations or discussions. Human partners trusted and showed humility towards agents. Our results demonstrate that agents can act as effective pair programming partners and open the way towards new research on conversational agents for programming.

Perceptions of system competence and communicative ability, termed partner models, play a significant role in speech interface interaction. Yet we do not know what the core dimensions of this concept are. Taking a psycholexical approach, our paper is the first to identify the key dimensions that define partner models in speech agent interaction. Through a repertory grid study (N=21), a review of key subjective questionnaires, an expert review of resulting word pairs and an online study of 356 users of speech interfaces, we identify three key dimensions that make up a users’ partner model: 1) perceptions towards partner competence and dependability; 2) assessment of human-likeness; and 3) a system’s perceived cognitive flexibility. We discuss the implications for partner modelling as a concept, emphasising the importance of salience and the dynamic nature of these perceptions.

The uptake of artificial intelligence-based applications raises concerns about the fairness and transparency of AI behaviour. Consequently, the Computer Science community calls for the involvement of the general public in the design and evaluation of AI systems. Assessing the fairness of individual predictors is an essential step in the development of equitable algorithms. In this study, we evaluate the effect of two common visualisation techniques (text-based and scatterplot) and the display of the outcome information (i.e., ground-truth) on the perceived fairness of predictors. Our results from an online crowdsourcing study (N = 80) show that the chosen visualisation technique significantly alters people’s fairness perception and that the presented scenario, as well as the participant’s gender and past education, influence perceived fairness. Based on these results we draw recommendations for future work that seeks to involve non-experts in AI fairness evaluations.

Current personal informatics models consider reflection as an important stage in users’ journeys with trackers. However, these models describe reflection from a meta perspective and it remains unclear what this stage entails. To design interactive technologies that support reflection, we need a more thorough understanding of how people reflect on their personal data in practice. To that end, we conducted semi-structured interviews with users of fitness trackers and an online survey to study practices in reflecting on fitness data. Our results show that users reported reflecting on data despite lacking reflection support from their tracking technology. Based on our results, we introduce the Technology-Mediated Reflection Model, which describes conditions and barriers for reflection on personal data. Our model consists of the temporal and conceptual cycles of reflection and helps designers identify the possible barriers a user might face when using a system for reflection.

The ability to monitor audience reactions is critical when delivering presentations. However, current videoconferencing platforms offer limited solutions to support this. This work leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals. Using an exploratory survey (N=175), we assessed the most relevant audience responses such as confusion, engagement, and head-nods. We then implemented AffectiveSpotlight, a Microsoft Teams bot that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones. In a within-subjects study with 14 groups (N=117), we observed that the system made presenters significantly more aware of their audience, speak for a longer period of time, and self-assess the quality of their talk more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI). We provide design recommendations for future affective interfaces for online presentations based on feedback from the study.

There are many situations where using personal devices is not socially acceptable, or where nearby people present a privacy risk. For these situations, we explore the concept of hidden interaction techniques through two prototype applications. HiddenHaptics allows users to receive information through vibrotactile cues on a smartphone, and HideWrite allows users to write text messages by drawing on a dimmed smartwatch screen. We conducted three user studies to investigate whether, and how, these techniques can be used without being exposed. Our primary findings are (1) users can effectively hide their interactions while attending to a social situation, (2) users seek to interact when another person is speaking, and they also tend to hide the interaction using their body or furniture, and (3) users can sufficiently focus on the social situation despite their interaction, whereas non-users feel that observing the user hinders their ability to focus on the social activity.

Speech is now common in daily interactions with our devices, thanks to voice user interfaces (VUIs) like Alexa. Despite their seeming ubiquity, designs often do not match users’ expectations. Science fiction, which is known to influence design of new technologies, has included VUIs for decades. Star Trek: The Next Generation is a prime example of how people envisioned ideal VUIs. Understanding how current VUIs live up to Star Trek’s utopian technologies reveals mismatches between current designs and user expectations, as informed by popular fiction. Combining conversational analysis and VUI user analysis, we study voice interactions with the Enterprise’s computer and compare them to current interactions. Independent of futuristic computing power, we find key design-based differences: Star Trek interactions are brief and functional, not conversational, they are highly multimodal and context-driven, and there is often no spoken computer response. From this, we suggest paths to better align VUIs with user expectations.

Voice assistants are fundamentally changing the way we access information. However, voice assistants still leverage little about the web beyond simple search results. We introduce Firefox Voice, a novel voice assistant built on the open web ecosystem with an aim to expand access to information available via voice. Firefox Voice is a browser extension that enables users to use their voice to perform actions such as setting timers, navigating the web, and reading a webpage’s content aloud. Through an iterative development process and use by over 12,000 active users, we find that users see voice as a way to accomplish certain browsing tasks efficiently, but struggle with discovering functionality and frequently discontinue use. We conclude by describing how Firefox Voice enables the development of novel, open web-powered voice-driven experiences.

Silent speech input converts non-acoustic features like tongue and lip movements into text. It has been demonstrated as a promising input method on mobile devices and has been explored for a variety of audiences and contexts where the acoustic signal is unavailable (e.g., people with speech disorders) or unreliable (e.g., noisy environment). Though the method shows promise, very little is known about peoples’ perceptions regarding using it. In this work, first, we conduct two user studies to explore users’ attitudes towards the method with a particular focus on social acceptance and error tolerance. Results show that people perceive silent speech as more socially acceptable than speech input and are willing to tolerate more errors with it to uphold privacy and security. We then conduct a third study to identify a suitable method for providing real-time feedback on silent speech input. Results show users find an abstract feedback method effective and significantly more private and secure than a commonly used video feedback method.

Video-conferencing is essential for many companies, but its limitations in conveying social cues can lead to ineffective meetings. We present MeetingCoach, an intelligent post-meeting feedback dashboard that summarizes contextual and behavioral meeting information. Through an exploratory survey (N=120), we identified important signals (e.g., turn taking, sentiment) and used these insights to create a wireframe dashboard. The design was evaluated with in situ participants (N=16) who helped identify the components they would prefer in a post-meeting dashboard. After recording video-conferencing meetings of eight teams over four weeks, we developed an AI system to quantify the meeting features and created personalized dashboards for each participant. Through interviews and surveys (N=23), we found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity. Based on our findings, we provide suggestions for future feedback system designs of video-conferencing meetings.

Virtual environments (VEs) can create collaborative and social spaces, which are increasingly important in the face of remote work and travel reduction. Recent advances, such as more open and widely available platforms, create new possibilities to observe and analyse interaction in VEs. Using a custom instrumented build of Mozilla Hubs to measure position and orientation, we conducted an academic workshop to facilitate a range of typical workshop activities. We analysed social interactions during a keynote, small group breakouts, and informal networking/hallway conversations. Our mixed-methods approach combined environment logging, observations, and semi-structured interviews. The results demonstrate how small and large spaces influenced group formation, shared attention, and personal space, where smaller rooms facilitated more cohesive groups while larger rooms made small group formation challenging but personal space more flexible. Beyond our findings, we show how the combination of data and insights can fuel collaborative spaces’ design and deliver more effective virtual workshops.

We present a dialogue elicitation study to assess how users envision conversations with a perfect voice assistant (VA). In an online survey, N=205 participants were prompted with everyday scenarios, and wrote the lines of both user and VA in dialogues that they imagined as perfect. We analysed the dialogues with text analytics and qualitative analysis, including number of words and turns, social aspects of conversation, implied VA capabilities, and the influence of user personality. The majority envisioned dialogues with a VA that is interactive and not purely functional; it is smart, proactive, and has knowledge about the user. Attitudes diverged regarding the assistant’s role as well as it expressing humour and opinions. An exploratory analysis suggested a relationship with personality for these aspects, but correlations were low overall. We discuss implications for research and design of future VAs, underlining the vision of enabling conversational UIs, rather than single command “Q&As”.

The use of virtual reality (VR) to simulate confrontational human behaviour has significant potential for use in training, where the recreation of uncomfortable feelings may help users to prepare for challenging real-life situations. In this paper we present a user study (n=68) in which participants experienced simulated confrontational behaviour performed by a virtual character either in immersive VR, or on a 2D display. Participants reported a higher elevation in anxiety in VR, which correlated positively with a perceived sense of physical space. Character believability was influenced negatively by visual elements of the simulation, and positively by behavioural elements, which complements findings from previous work. We recommend the use of VR for simulations of confrontational behaviour, where a realistic emotional response is part of the intended experience. We also discuss incorporation of domain knowledge of human behaviours, and carefully crafted motion-captured sequences, to increase users’ sense of believability.

Bringing positive experiences to users is one of the key goals when designing conversational agents (CAs). Yet we still lack an understanding of users’ underlying needs to achieve positive experiences and how to support them in design. This research first applies Self-Determination Theory in an interview study to explore how users’ needs of competence, autonomy and relatedness could be supported or undermined in CA experiences. Ten guidelines are then derived from the interview findings. The key findings demonstrate that: competence is affected by users’ knowledge of the CA capabilities and effectiveness of the conversation; autonomy is influenced by flexibility of the conversation, personalisation of the experiences, and control over user data; regarding relatedness, users still have concerns over integrating social features into CAs. The guidelines recommend how to inform users about the system capabilities, design effective and socially appropriate conversations, and support increased system intelligence, customisation, and data transparency.

Although social support is important for health and well-being, many young people are hesitant to reach out for support. The emerging uptake of chatbots for social and emotional purposes entails opportunities and concerns regarding non-human agents as sources of social support. To explore this, we invited 16 participants (16–21 years) to use and reflect on chatbots as sources of social support. Our participants first interacted with a chatbot for mental health (Woebot) for two weeks. Next, they participated in individual in-depth interviews. As part of the interview session, they were presented with a chatbot prototype providing information to young people. Two months later, the participants reported on their continued use of Woebot. Our findings provide in-depth knowledge about how young people may experience various types of social support—appraisal, informational, emotional, and instrumental support—from chatbots. We summarize implications for theory, practice, and future research.

This paper traces different conceptions of the body in HCI and identifies a narrative from user to body, body to bodies, and bodies to more-than-human bodies. Firstly, this paper aims to present a broader, updated, survey of work around the body in HCI. The overview shows how bodies are conceptualized as performative, sensing, datafied, intersectional and more-than-human. This paper then diverges from similar surveys of research addressing the body in HCI in that it is more disruptive and offers a critique of these approaches and pointers for where HCI might go next. We end our paper with recommendations drawn from across the different approaches to the body in HCI. In particular, that researchers working with the body have much to gain from the 4th wave HCI approach when designing with and for the body, where our relationships with technologies are understood as entangled and the body is always more-than-human.

Passwords have become a ubiquitous part of our everyday lives, needed for every web-service and system. However, it is challenging to create safe and diverse alphanumeric passwords, and to recall them, imposing a cognitive burden on the user. Through consecutive experiments, we explored the movement space, affordances and interaction, and memorability of a tangible, handheld, embodied password. In this context, we found that: (1) a movement space of 200 mm × 200 mm is preferred; (2) each context has a perceived level of safety, which—together with the affordances and link to familiarity—influences how the password is performed. Furthermore, the artefact’s dimensions should be balanced within the design itself, with the user, and the context, but there is a trade-off between the perceived safety and ergonomics; and (3) the designed embodied passwords can be recalled for at least a week, with participants creating unique passwords which were reproduced consistently.

Starting to menstruate can restrict adolescents’ movements due to physiological changes and societal stigma. We present a participatory soma design project advocating for young adolescents to listen to and care for their newly-menstruating bodies, specifically focusing on participation in sport. We designed Menarche Bits, an open-ended prototyping toolkit consisting of shape-changing actuators and heat pads, and used it in two design workshops with seven participants aged 16-18, as part of collaboration and menstrual advocacy in their sports clubs and high school. The participants designed menstrual technologies that respond to menstrual cramps and depressive, anxious feelings before menstruating. We contribute findings on designing menstrual technologies with adolescents using participatory soma design. We found that a toolkit approach to the design of menstrual technologies can allow for pluralist experiences of menstrual cycles. In addition, we found that participatory design with adolescents benefits from drawing on qualities of embodiment and participants’ own body literacy.

We introduce Azalea: a design to enrich remote dialog by diminishing externalities, created through a process informed by somaesthetics. Azalea is a tactile cushion that envelops a smartphone running a bespoke app. A pair of Azaleas mediate an embodied co-experience between remote interlocutors via a motion-driven soundscape and audio-driven visuals. While most designs for enriching remote communication increase dimensionality and fidelity of modalities, Azalea diminishes distractions and serves an abstract medium for co-experiencing embodied information. We present the theoretical foundations and design tactics of Azalea, and characterize the experience through a qualitative empirical study. Our findings culminated in 12 qualities, supporting 5 themes with design implications that contribute to (1) a design ethos of diminished reality and (2) an expansion of somaesthetic HCI towards expression and communication.

Designing computational support for dance is an emerging area of HCI research, incorporating the cultural, experiential, and embodied characteristics of the third-wave shift. The challenges of recognising the abstract qualities of body movement, and of mediating between the diverse parties involved in the idiosyncratic creative process, present important questions to HCI researchers: how can we effectively integrate computing with dance, to understand and cultivate the felt dimension of creativity, and to aid the dance-making process? In this work, we systematically review the past twenty years of dance literature in HCI. We discuss our findings, propose directions for future HCI works in dance, and distil lessons for related disciplines.

Deep Pressure Therapy relies on exerting firm touch to help individuals with sensory sensitivity. We performed first-person explorations of deep pressure enabled by shape-changing actuation driven by breathing sensing. This revealed a novel design space with rich, evocative, aesthetically interesting interactions that can help increase breathing awareness and appreciation through: (1) applying symmetrical as well as asymmetrical pressure on the torso; (2) using pressure to direct attention to muscles or bone structure involved in different breathing patterns; (3) apply synchronous as well as asynchronous feedback following or opposing the user’s breathing rhythm through applying rhythmic pressure. Taken together these explorations led us to design (4) breathing correspondence interactions – a balance point right between leading and following users’ breathing patterns by first applying deep pressure – almost to the point of being unpleasant – and then releasing in rhythmic flow.

Avatars, the users’ virtual representations, are becoming ubiquitous in virtual reality applications. In this context, the avatar becomes the medium which enables users to manipulate objects in the virtual environment. It also becomes the users’ main spatial reference, which can not only alter their interaction with the virtual environment, but also the perception of themselves. In this paper, we review and analyse the current state-of-the-art for 3D object manipulation and the sense of embodiment. Our analysis is twofold. First, we discuss the impact that the avatar can have on object manipulation. Second, we discuss how the different components of a manipulation technique (i.e. input, control and feedback) can influence the user’s sense of embodiment. Throughout the analysis, we crystallise our discussion with practical guidelines for VR application designers and we propose several research topics towards “avatar-friendly’’ manipulation techniques.

Meditation is a mind-body practice with considerable wellbeing benefits that can take different forms. Novices usually start with focused attention meditation that supports regulation of attention towards an inward focus or internal bodily sensations and away from external stimuli or distractors. Most meditation technologies employ metaphorical mappings of meditative states to visual or soundscape representations to support awareness of mind wandering and attention regulation, although the rationale for such mappings is seldom articulated. Moreover, such external modalities also take the focus attention away from the body. We advance the concept of interoceptive interaction and employed the embodied metaphor theory to explore the design of mappings to the interoceptive sense of thermoception. We illustrate this concept with WarmMind, an on-body interface integrating heat actuators for mapping meditation states. We report on an exploratory study with 10 participants comparing our novel thermal metaphors for mapping meditation states with comparable ones, albeit in aural modality, as provided by Muse meditation app. Findings indicate a tension between the highly discoverable soundscape's metaphors which however hinder attention regulation, and the ambiguous thermal metaphors experienced as coming from the body and supported attention regulation. We discuss the qualities of embodied metaphors underpinning this tension and propose an initial framework to inform the design of metaphorical mappings for meditation technologies.

Though seemingly straightforward and habitual, breathing is a complex bodily function. Problematising the space of designing for breathing as a non-habitual act pertaining to different bodies or situations, we conducted a soma design exploration together with a classical singer. Reflecting on how sensors could capture the impact and somatic experience of being sensed led us to develop a new sensing mechanism using shape-change technologies integrated in the Breathing Shell: a wearable that evokes a reciprocal experience of “feeling the sensor feeling you” when breathing. We contribute with two design implications: 1) Enabling reflections of the somatic impact of being sensed in tandem with the type of data captured, 2) creating a tactile impact of the sensor data on the body. Both implications aim to deepen one’s understanding of how the whole soma relates to or with biosensors and ultimately leading to designing for symbiotic experiences between biosensors and bodies.

The growing interest in wearables for sports and fitness calls for design knowledge and conceptualizations that can help shape future designs. Towards that end, we present and discuss a design space of wearables for these practices, based on a survey of previous work. Through a thematic analysis of 47 research publications in the domain, we surface core design decisions concerning wearability, technology design, and wearable use in practice. Building on these, we show how the design space takes into account the goals of introducing technology, that design decisions can be either directly designed, or left open for appropriation by end-users; and the social organization of the practice. We characterize prior work based on the design space elements, which yields trends and opportunities for design. Our contributions can help designers think about key design decisions, exploit trends and explore new areas in the domain of wearables for sports and fitness practices.

In this paper, we reflect on the applicability of the concept of trajectories to soma design. Soma design is a first-person design method which considers users’ subjective somatic or bodily experiences of a design. Due to bodily changes over time, soma experiences are inherently temporal. Current instruments for articulating soma experiences lack the power to express the effects of experiences on the body over time. To address this, we turn to trajectories, a well-known concept in the HCI community, as a way of mapping this aspect of soma experience. By showing trajectories through a range of dimensions, we can articulate individual experiences and differences in those experiences. Through analysis of a set of soma experience designs and a set of temporal dimensions within the experiences, this paper demonstrates how trajectories can provide a practical conceptual framing for articulating the temporal complexity of soma designs.

We examine a vocabulary of affective language, borrowed from Roland Barthes’ “A Lover’s Discourse: Fragments,” and its applicability to discourse describing smartphone attachment. This vocabulary, adopted from four of Barthes’ terms, waiting, dependency, anxiety, and absence, is used as a discursive lens to illustrate some of the many ways people understand and engage with their relationships to their smartphones. Based on this, a survey is conducted, and a speculative technology probe is created in the form of an instrumented pillow for people to lock away their smartphones during the night. The pillow is deployed in a diary study in which five people sleep with their phone locked away for multiple nights. The self-reported and observed behaviours are presented in a selection of vignettes. The results support the proposed discursive lens and suggest future interdisciplinary strategies to investigate how people relate to interactive technology, using a combined approach of literary theory and a technology probe supported by survey and study data.

Two-dimensional canvases are the core components of many digital productivity and creativity tools, with “artboards” containing objects rather than pixels. Unfortunately, the contents of artboards remain largely inaccessible to blind users relying on screen-readers, but the precise problems are not well understood. This study sought to understand how blind screen-reader users interact with artboards. Specifically, we conducted contextual interviews, observations, and task-based usability studies with 15 blind participants to understand their experiences of artboards found in Microsoft PowerPoint, Apple Keynote, and Google Slides. Participants expressed that the inaccessibility of these artboards contributes to significant educational and professional barriers. We found that the key problems faced were: (1) high cognitive loads from a lack of feedback about artboard contents and object state; (2) difficulty determining relationships among artboard objects; and (3) constant uncertainty about whether object manipulations were successful. We offer design remedies that improve feedback for object state, relationships, and manipulations.

The development of accurate machine learning models for sign languages like American Sign Language (ASL) has the potential to break down communication barriers for deaf signers. However, to date, no such models have been robust enough for real-world use. The primary barrier to enabling real-world applications is the lack of appropriate training data. Existing training sets suffer from several shortcomings: small size, limited signer diversity, lack of real-world settings, and missing or inaccurate labels. In this work, we present ASL Sea Battle, a sign language game designed to collect datasets that overcome these barriers, while also providing fun and education to users. We conduct a user study to explore the data quality that the game collects, and the user experience of playing the game. Our results suggest that ASL Sea Battle can reliably collect and label real-world sign language videos, and provides fun and education at the expense of data throughput.

User-generated videos are an increasingly important source of information online, yet most online videos are inaccessible to blind and visually impaired (BVI) people. To find videos that are accessible, or understandable without additional description of the visual content, BVI people in our formative studies reported that they used a time-consuming trial-and-error approach: clicking on a video, watching a portion, leaving the video, and repeating the process. BVI people also reported video accessibility heuristics that characterize accessible and inaccessible videos. We instantiate 7 of the identified heuristics (2 audio-related, 2 video-related, and 3 audio-visual) as automated metrics to assess video accessibility. We collected a dataset of accessibility ratings of videos by BVI people and found that our automatic video accessibility metrics correlated with the accessibility ratings (Adjusted R2 = 0.642). We augmented a video search interface with our video accessibility metrics and predictions. BVI people using our augmented video search interface selected an accessible video more efficiently than when using the original search interface. By integrating video accessibility metrics, video hosting platforms could help people surface accessible videos and encourage content creators to author more accessible products, improving video accessibility for all.

Despite recent improvements in online accessibility, the Internet remains an inhospitable place for users with photosensitive epilepsy, a chronic condition in which certain light stimuli can trigger seizures and even lead to death in extreme cases. In this paper, we explore how current risk detection systems have allowed attackers to take advantage of design oversights and target vulnerable users with photosensitivity on popular social media platforms. Through interviews with photosensitive individuals and a critical review of existing systems, we constructed design requirements for consumer-driven protective systems and developed a prototype browser extension for actively detecting and disarming potentially seizure-inducing GIFs and videos. We validate our system with a comprehensive dataset of simulated GIFs and GIFs collected from social media. Finally, we conduct a novel quantitative analysis of the prevalence of seizure-inducing GIFs across popular social media platforms and contribute recommendations for improving online accessibility for individuals with photosensitivity. All study materials are available at https://osf.io/5a3dy/.

For 15% of the world population with disabilities, accessibility is arguably the most critical software quality attribute. The ever-growing reliance of users with disability on mobile apps further underscores the need for accessible software in this domain. Existing automated accessibility assessment techniques primarily aim to detect violations of predefined guidelines, thereby produce a massive amount of accessibility warnings that often overlook the way software is actually used by users with disability. This paper presents a novel, high-fidelity form of accessibility testing for Android apps, called Latte, that automatically reuses tests written to evaluate an app’s functional correctness to assess its accessibility as well. Latte first extracts the use case corresponding to each test, and then executes each use case in the way disabled users would, i.e., using assistive services. Our empirical evaluation on real-world Android apps demonstrates Latte’s effectiveness in detecting substantially more useful defects than prior techniques.

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app’s full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

Presenters commonly use slides as visual aids for informative talks. When presenters fail to verbally describe the content on their slides, blind and visually impaired audience members lose access to necessary content, making the presentation difficult to follow. Our analysis of 90 presentation videos revealed that 72% of 610 visual elements (e.g., images, text) were insufficiently described. To help presenters create accessible presentations, we introduce Presentation A11y, a system that provides real-time and post-presentation accessibility feedback. Our system analyzes visual elements on the slide and the transcript of the verbal presentation to provide element-level feedback on what visual content needs to be further described or even removed. Presenters using our system with their own slide-based presentations described more of the content on their slides, and identified  3.26 times more accessibility problems to fix after the talk than when using a traditional slide-based presentation interface. Integrating accessibility feedback into content creation tools will improve the accessibility of informational content for all.

Video accessibility is essential for people with visual impairments. Audio descriptions describe what is happening on-screen, e.g., physical actions, facial expressions, and scene changes. Generating high-quality audio descriptions requires a lot of manual description generation [50]. To address this accessibility obstacle, we built a system that analyzes the audiovisual contents of a video and generates the audio descriptions. The system consisted of three modules: AD insertion time prediction, AD generation, and AD optimization. We evaluated the quality of our system on five types of videos by conducting qualitative studies with 20 sighted users and 12 users who were blind or visually impaired. Our findings revealed how audio description preferences varied with user types and video types. Based on our study’s analysis, we provided recommendations for the development of future audio description generation technologies.

This paper presents a systematic literature review of 292 publications from 97 unique venues on touch-based graphics for people who are blind or have low vision, from 2010 to mid-2020. It is the first review of its kind on touch-based accessible graphics. It is timely because it allows us to assess the impact of new technologies such as commodity 3D printing and low-cost electronics on the production and presentation of accessible graphics. As expected our review shows an increase in publications from 2014 that we can attribute to these developments. It also reveals the need to: broaden application areas, especially to the workplace; broaden end-user participation throughout the full design process; and conduct more in situ evaluation. This work is linked to an online living resource to be shared with the wider community.

Research has explored using Automatic Text Simplification for reading assistance, with prior work identifying benefits and interests from Deaf and Hard-of-Hearing (DHH) adults. While the evaluation of these technologies remains a crucial aspect of research in the area, researchers lack guidance in terms of how to evaluate text complexity with DHH readers. Thus, in this work we conduct methodological research to evaluate metrics identified from prior work (including reading speed, comprehension questions, and subjective judgements of understandability and readability) in terms of their effectiveness for evaluating texts modified to be at various complexity levels with DHH adults at different literacy levels. Subjective metrics and low-linguistic-complexity comprehension questions distinguished certain text complexity levels with participants with lower literacy. Among participants with higher literacy, only subjective judgements of text readability distinguished certain text complexity levels. For all metrics, participants with higher literacy scored higher or provided more positive subjective judgements overall.

The accessibility and affordability of tangible electronic toolkits are significant barriers to their uptake by people with disabilities. We present the design and evaluation of TapeBlocks, a low-cost, low-fidelity toolkit intended to be accessible for people with intellectual disabilities while promoting creativity and engagement. We evaluated TapeBlocks by interviewing makers, special educational needs teachers and support coaches. Analysis of these interviews informed the design of a series of maker workshops using TapeBlocks with young adults living with intellectual disabilities, led by support coaches with support from the research team. Participants were able to engage with TapeBlocks and making, eventually building their own TapeBlocks to make personal creations. Our evaluation reveals how TapeBlocks supports accessible making and playful discovery of electronics for people living with disabilities, and addresses a gap in existing toolkits by being tinkerable, affordable and having a low threshold for engagement.

The recently advanced robotics technology enables robots to assist users in their daily lives. Haptic guidance (HG) improves users’ task performance through physical interaction between robots and users. It can be classified into optimal action-based HG (OAHG), which assists users with an optimal action, and user prediction-based HG (UPHG), which assists users with their next predicted action. This study aims to understand the difference between OAHG and UPHG and propose a combined HG (CombHG) that achieves optimal performance by complementing each HG type, which has important implications for HG design. We propose implementation methods for each HG type using deep learning-based approaches. A user study (n=20) in a haptic task environment indicated that UPHG induces better subjective evaluations, such as naturalness and comfort, than OAHG. In addition, the CombHG that we proposed further decreases the disagreement between the user intention and HG, without reducing the objective and subjective scores.

To make off-screen interaction without specialized hardware practical, we investigate using deep learning methods to process the common built-in IMU sensor (accelerometers and gyroscopes) on mobile phones into a useful set of one-handed interaction events. We present the design, training, implementation and applications of TapNet, a multi-task network that detects tapping on the smartphone. With phone form factor as auxiliary information, TapNet can jointly learn from data across devices and simultaneously recognize multiple tap properties, including tap direction and tap location. We developed two datasets consisting of over 135K training samples, 38K testing samples, and 32 participants in total. Experimental evaluation demonstrated the effectiveness of the TapNet design and its significant improvement over the state of the art. Along with the datasets, codebase1, and extensive experiments, TapNet establishes a new technical foundation for off-screen mobile input.

Typing on wearables while situationally impaired, such as while walking, is challenging. However, while HCI research on wearable typing is diverse, existing work focuses on stationary scenarios and fine-grained input that will likely perform poorly when users are on-the-go. To address this issue we explore single-handed wearable typing using intra-hand touches between the thumb and fingers, a modality we argue will be robust to the physical disturbances inherent to input while mobile. We first examine the impact of walking on performance of these touches, noting no significant differences in accuracy or speed, then feed our study data into a multi-objective optimization process in order to design keyboard layouts (for both five and ten keys) capable of supporting rapid, accurate, comfortable, and unambiguous typing. A final study tests these layouts against QWERTY baselines and reports performance improvements of up to 10.45% WPM and 39.44% WER when users type while walking.

We present a computational approach to haptic design embedded in everyday tangible interaction with digital fabrication. To generate haptic feedback, the use of permanent magnets as the mechanism potentially contributes to simpleness and robustness; however, it is difficult to manually design how magnets should be embedded in the objects. Our approach enables the inverse design of magnetic force feedback; that is, we computationally solve an inverse problem to obtain an optimal arrangement of permanent magnets that renders the user-specified haptic sensation. To solve the inverse problem in a practical manner, we also present techniques on magnetic simulation and optimization. We demonstrate applications to explore the design possibility of augmenting digital fabrication for everyday use.

Motion correlation interfaces are those that present targets moving in different patterns, which the user can select by matching their motion. In this paper, we re-formulate the task of target selection as a probabilistic inference problem. We demonstrate that previous interaction techniques can be modelled using a Bayesian approach and that how modelling the selection task as transmission of information can help us make explicit the assumptions behind similarity measures. We propose ways of incorporating uncertainty into the decision-making process and demonstrate how the concept of entropy can illuminate the measurement of the quality of a design. We apply these techniques in a case study and suggest guidelines for future work.

We present a novel simulation model of point-and-click behaviour that is applicable both when a target is stationary or moving. To enable more realistic simulation than existing models, the model proposed in this study takes into account key features of the user and the external environment, such as intermittent motor control, click decision-making, visual perception, upper limb kinematics and the effect of input device. The simulated user’s point-and-click behaviour is formulated as a Markov decision process (MDP), and the user’s policy of action is optimised through deep reinforcement learning. As a result, our model successfully and accurately reproduced the trial completion time, distribution of click endpoints, and cursor trajectories of real users. Through an ablation study, we showed how the simulation results change when the model’s sub-modules are individually removed. The implemented model and dataset are publicly available.

Typing with ten fingers on a virtual keyboard in virtual or augmented reality exposes a challenging input interpretation problem. There are many sources of noise in this interaction context and these exacerbate the challenge of accurately translating human actions into text. A particularly challenging input noise source arises from the physiology of the hand. Intentional finger movements can produce unintentional coactivations in other fingers. On a physical keyboard, the resistance of the keys alleviates this issue. On a virtual keyboard, coactivations are likely to introduce spurious input events under a naïve solution to input detection. In this paper we examine the features that discriminate intentional activations from coactivations. Based on this analysis, we demonstrate three alternative coactivation detection strategies with high discrimination power. Finally, we integrate coactivation detection into a probabilistic decoder and demonstrate its ability to further reduce uncorrected character error rates by approximately 10% relative and 0.9% absolute.

Gaze-based selection has received significant academic attention over a number of years. While advances have been made, it is possible that further progress could be made if there were a deeper understanding of the adaptive nature of the mechanisms that guide eye movement and vision. Control of eye movement typically results in a sequence of movements (saccades) and fixations followed by a ‘dwell’ at a target and a selection. To shed light on how these sequences are planned, this paper presents a computational model of the control of eye movements in gaze-based selection. We formulate the model as an optimal sequential planning problem bounded by the limits of the human visual and motor systems and use reinforcement learning to approximate optimal solutions. The model accurately replicates earlier results on the effects of target size and distance and captures a number of other aspects of performance. The model can be used to predict number of fixations and duration required to make a gaze-based selection. The future development of the model is discussed.

Touch input is dominantly detected using mutual-capacitance sensing, which measures the proximity of close-by objects that change the electric field between the sensor lines. The exponential drop-off in intensities with growing distance enables software to detect touch events, but does not reveal true contact areas. In this paper, we introduce CapContact, a novel method to precisely infer the contact area between the user’s finger and the surface from a single capacitive image. At 8 × super-resolution, our convolutional neural network generates refined touch masks from 16-bit capacitive images as input, which can even discriminate adjacent touches that are not distinguishable with existing methods. We trained and evaluated our method using supervised learning on data from 10 participants who performed touch gestures. Our capture apparatus integrates optical touch sensing to obtain ground-truth contact through high-resolution frustrated total internal reflection. We compare our method with a baseline using bicubic upsampling as well as the ground truth from FTIR images. We separately evaluate our method’s performance in discriminating adjacent touches. CapContact successfully separated closely adjacent touch contacts in 494 of 570 cases (87%) compared to the baseline’s 43 of 570 cases (8%). Importantly, we demonstrate that our method accurately performs even at half of the sensing resolution at twice the grid-line pitch across the same surface area, challenging the current industry-wide standard of a ∼ 4 mm sensing pitch. We conclude this paper with implications for capacitive touch sensing in general and for touch-input accuracy in particular.

Arm discomfort is a common issue in Cross Reality applications involving prolonged mid-air interaction. Solving this problem is difficult because of the lack of tools and guidelines for 3D user interface design. Therefore, we propose a method to make existing ergonomic metrics available to creators during design by estimating the interaction cost at each reachable position in the user’s environment. We present XRgonomics, a toolkit to visualize the interaction cost and make it available at runtime, allowing creators to identify UI positions that optimize users’ comfort. Two scenarios show how the toolkit can support 3D UI design and dynamic adaptation of UIs based on spatial constraints. We present results from a walkthrough demonstration, which highlight the potential of XRgonomics to make ergonomics metrics accessible during the design and development of 3D UIs. Finally, we discuss how the toolkit may address design goals beyond ergonomics.

Hand gestures are a natural and expressive input method enabled by modern mixed reality headsets. However, it remains challenging for developers to create custom gestures for their applications. Conventional strategies to bespoke gesture recognition involve either hand-crafting or data-intensive deep-learning. Neither approach is well suited for rapid prototyping of new interactions. This paper introduces a flexible and efficient alternative approach for constructing hand gestures. We present Gesture Knitter: a design tool for creating custom gesture recognizers with minimal training data. Gesture Knitter allows the specification of gesture primitives that can then be combined to create more complex gestures using a visual declarative script. Designers can build custom recognizers by declaring them from scratch or by providing a demonstration that is automatically decoded into its primitive components. Our developer study shows that Gesture Knitter achieves high recognition accuracy despite minimal training data and delivers an expressive and creative design experience.

Millimeter wave (mmWave) Doppler radar is a new and promising sensing approach for human activity recognition, offering signal richness approaching that of microphones and cameras, but without many of the privacy-invading downsides. However, unlike audio and computer vision approaches that can draw from huge libraries of videos for training deep learning models, Doppler radar has no existing large datasets, holding back this otherwise promising sensing modality. In response, we set out to create a software pipeline that converts videos of human activities into realistic, synthetic Doppler radar data. We show how this cross-domain translation can be successful through a series of experimental results. Overall, we believe our approach is an important stepping stone towards significantly reducing the burden of training such as human sensing systems, and could help bootstrap uses in human-computer interaction.

Design research has recently turned to theoretical perspectives, including care ethics and posthumanism, to counter the industrial processes that have led to climate crisis. As design theorists and ethnographers of interaction, we researched experimental eco-farming in a community that shared many of these theoretical and ideological commitments. Our goal was not to offer an account of use and provide design implications in support of it. Instead, we chose to identify concrete practices and artifacts that embody the sorts of industrial transformations that we are seeking—even if they are manifest in an imperfect or partial form. We encountered practices focused on community building, local resilience to climate disruptions, experiments in eco-farming, economic survival, and attracting the next generation. One interlocutor translated these concerns into a simple binary, asking, “do we want to live here?” This paper contributes to a design research agenda that might (eventually) provide an affirmative answer.

Guqin is a plucked seven-string traditional Chinese musical instrument that exists for over 3,000 years. However, as an Intangible World Cultural Heritage, the inheritance of Guqin and its culture in modern society is in deep danger. According to our study with 1,006 Chinese worldwide, Guqin as an instrument is not well-known and barely accessible. To better promote Guqin, we developed two interactive systems: VirGuqin and MRGuqin. VirGuqin was developed using a low-cost motion tracking device and was tested in a museum. 89% of 308 participants expressed an increase in interest in learning Guqin after using our system. MRGuqin was developed as a mixed reality learning environment to reduce the entry barrier to Guqin, and was tested by 16 participants, allowing them to learn Guqin significantly faster and perform better than the current practice. Our study demonstrates how technology can be used to help the inheritance of this dying art.

HCI and STS researchers have previously described the ethical complexity of practice, drawing together aspects of organizational complexity, design knowledge, and ethical frameworks. Building on this work, we investigate the identity claims and beliefs that impact practitioners’ ability to recognize and act upon ethical concerns in a range of technology-focused disciplines. In this paper, we report results from an interview study with 12 practitioners, identifying and describing their identity claims related to ethical awareness and action. We conducted a critically-focused thematic analysis to identify eight distinct claims representing roles relating to learning, educating, following policies, feeling a sense of responsibility, being a member of a profession, a translator, an activist, and deliberative. Based on our findings, we demonstrate how the claims foreground building competence in relation to ethical practice. We highlight the dynamic interplay among these claims and point towards implications for identity work in socio-technical contexts.

As Extended Reality (XR) devices and applications become more mainstream, so too will XR advertising — advertising that takes place in XR mediums. Due to the defining features of XR devices, such as the immersivity of the medium and the ability of XR devices to simulate reality, there are fears that these features could be exploited to create manipulative XR ads that trick consumers into buying products they do not need or might harm them. Using scenario construction, we investigate potential future incarnations of manipulative XR advertising and their harms. We identify five key mechanisms of manipulative XR advertising: misleading experience marketing; inducing artificial emotions in consumers; sensing and targeting people when they are vulnerable; emotional manipulation through hyperpersonalization; and distortion of reality. We discuss research challenges and questions in order to address and mitigate manipulative XR advertising risks.

As crucial public functions are transferred to computer systems, emerging technologies have public implications that are often shaped beyond public influence and oversight. “Smart city” and “modernization” projects are just some examples of such transformations. This paper focuses on struggles over the acquisition, control, and maintenance of these public, digital infrastructures. We focus on the forms of HCI knowledge and practice that proved useful to a coalition of community organizations claiming rights of input into and political oversight over surveillance technology. Their claims were a response to their exclusion from decision-making about smart city implementation in San Diego. We offer tactics “from below” as a way to attune HCI to the needs and practices of those excluded from power over widespread technology infrastructures. Ultimately, we argue that HCI cultivates a variety of capacities beyond design and redesign that can strengthen struggles to shape real-world technologies from below.

We present a vision for conversational user interfaces (CUIs) as probes for speculating with, rather than as objects to speculate about. Popular CUIs, e.g., Alexa, are changing the way we converse, narrate, and imagine the world(s) to come. Yet, current conversational interactions normatively may promote non-desirable ends, delivering a restricted range of request-response interactions with sexist and digital colonialist tendencies. Our critical design approach envisions alternatives by considering how future voices can reside in CUIs as enabling probes. We present novel explorations that illustrate the potential of CUIs as critical design material, by critiquing present norms and conversing with imaginary species. As micro-level interventions, we show that conversations with diverse futures through CUIs can persuade us to critically shape our discourse on macro-scale concerns of the present, e.g., sustainability. We reflect on how conversational interactions with pluralistic, imagined futures can contribute to how being human stands to change.

There is a developing recognition of the social and economic costs entailed in global supply chains. In this paper, we report on efforts to provide alternative, more sustainable and resilient models of production. Community Supported Agricultures (CSAs) address this problem but require new means of exchange which, we suggest, offer a design opportunity for sustainable HCI research. This paper presents a two months participatory observation in a food movement, a German CSA which developed a distribution system involving their own currency. Based on our ethnographic observations, we focus our discussion on (1) the solidaristic principles upon which the movement is based and (2) techniques of mediating between consumers’ wishes and the constraints of local agricultural production. By relating to the continued development of CSAs, we identify three interrelated innovation gaps and discuss new software architectures aimed at resolving the problems which arise as the movement grows.

Sustainable HCI (SHCI) constitutes a relatively new research field within HCI. We have identified four literature reviews of the field conducted between 2009-2014. In this paper, we present and discuss the results of a systematic literature review of peer-reviewed conference and journal articles that have been published in the field during the last ten years (2010-2019). To this end, we apply the United Nations’ Sustainable Development Goals (SDGs) as a framework to classify and discern high-level goals SHCI researchers have worked towards during this period. This paper contributes to HCI by 1) identifying Sustainable Development Goals that SHCI researchers have worked towards, 2) discerning main research trends in the field during the last decade, 3) using the SDG framework generatively to enumerate and reflect on areas that this far have not been covered by SHCI research and 4) presenting takeaways and opportunities for further research by the larger HCI community.

This paper investigates the practices of organising face-to-face events of a volunteer-run food-sharing community in Denmark. The ethnographic fieldwork draws attention to the core values underlying the ways sharing events are organised, and how - through the work of volunteers - surplus food is transformed from a commodity to a gift. The findings illustrate the community’s activist agenda of food waste reduction, along with the volunteers’ concerns and practical labour of running events and organising the flow of attendees through various queuing mechanisms. The paper contributes to the area of Food and HCI by: i) outlining the role of queuing in organising activism and ii) reflecting on the role that values, such as collective care and commons, can play in structuring queuing at face-to-face events.

Within Participatory- and Co-Design projects, the issue of sustainability and maintenance of the co-designed artefacts is a crucial yet largely unresolved issue. In this paper, we look back on four years of work on co-designing tools that assist refugees and migrants in their efforts to settle in Germany, the last of which the project has been independently maintained by our community collaborators. We reflect on the role of pre-existing care practices amongst our community collaborators, and a continued openness throughout the project, that allowed a complex constellation of actors to be involved in its ongoing maintenance and our own, often mundane activities which have contributed to the sustainability of the results. Situating our account within an HCI for Social Justice agenda, we thereby contribute to an ongoing discussion about the sustainability of such activities.

In recent years HCI and CSCW work has increasingly begun to address complex social problems and issues of social justice worldwide. Such activist-leaning work is not without problems. Through the experiences and reflections of an activist becoming academic and an academic becoming an activist, we outline these difficulties such as (1) the risk of perpetuating violence, oppression and exploitation when working with marginalised communities, (2) the reception of activist-academic work within our academic communities, and (3) problems of social justice that exist within our academic communities. Building on our own experiences, practices and existing literature from a variety of disciplines we advocate for the possibility of an activist-academic practice, outline possible ways forward and formulate questions we need to answer for HCI to contribute to a more just world.

Working with emergent users in two of Mumbai’s slums, we explored the value and uses of photovoltaic (PV) self-powering digital materials. Through a series of co-design workshops, a diary study and responses by artists and craftspeople, we developed the PV-Pix concept for inter-home connections. Each PV-Pix element consists of a deformable energy harvesting material that, when actuated by a person in one home, changes its physical state both there and in a connected home. To explore the concept we considered two forms of PV-Pix: one uses rigid materials and the other flexible ones. We deployed two low-fidelity prototypes, each constructed of a grid of one PV-Pix type, in four slum homes over a four week period to further understand the usability and uses of the materials, eliciting interesting inter-family communication practices. Encouraged by these results we report on a first-step towards working prototypes and demonstrate the technical viability of the approach.

The relationships that constitute the global industrial food system tend towards two dominant values that are creating unsustainable social and environmental inequalities. The first is a human-centered perspective on food that privileges humans over all other species. The second is a view of food as a commodity to be traded for maximum economic value, rewarding a small number of shareholders. We present work that explores the unique algorithmic affordances of blockchain to create new types of value exchange and governance in the food system. We describe a project that used roleplay with urban agricultural communities to co-design blockchain-based food futures and explore the conditions for creating a thriving multispecies food commons. We discuss how the project helped rethink algorithmic food justice by reconfiguring more-than-human values and reconfiguring food as more-than-human commons. We also discuss some of the challenges and tensions arising from these explorations.

Non-expert annotators (who lack sufficient domain knowledge) are often recruited for manual image labeling tasks owing to the lack of expert annotators. In such a case, label quality may be relatively low. We propose leveraging the spatial layout for improving label quality in non-expert image annotation. In the proposed system, an annotator first spatially lays out the incoming images and labels them on an open space, placing related items together. This serves as a working space (spatial organization) for tentative labeling. During the process, the annotator observes and organizes the similarities and differences between the items. Finally, the annotator provides definitive labels to the images based on the results of the spatial layout. We ran a user study comparing the proposed method and a traditional non-spatial layout in an image labeling task. The results demonstrated that annotators can complete the labeling tasks more accurately using the spatial layout interface than the non-spatial layout interface.

Computational notebooks, which seamlessly interleave code with results, have become a popular tool for data scientists due to the iterative nature of exploratory tasks. However, notebooks provide a single execution state for users to manipulate through creating and manipulating variables. When exploring alternatives, data scientists must carefully create many-step manipulations in visually distant cells.

We conducted formative interviews with 6 professional data scientists, motivating design principles behind exposing multiple states. We introduce forking — creating a new interpreter session — and backtracking — navigating through previous states. We implement these interactions as an extension to notebooks that help data scientists more directly express and navigate through decision points a single notebook. In a qualitative evaluation, 11 professional data scientists found the tool would be useful for exploring alternatives and debugging code to create a predictive model. Their insights highlight further challenges to scaling this functionality.

Code search is an important and frequent activity for developers using computational notebooks (e.g., Jupyter). The flexibility of notebooks brings challenges for effective code search, where classic search interfaces for traditional software code may be limited. In this paper, we propose, NBSearch, a novel system that supports semantic code search in notebook collections and interactive visual exploration of search results. NBSearch leverages advanced machine learning models to enable natural language search queries and intuitive visualizations to present complicated intra- and inter-notebook relationships in the returned results. We developed NBSearch through an iterative participatory design process with two experts from a large software company. We evaluated the models with a series of experiments and the whole system with a controlled user study. The results indicate the feasibility of our analytical pipeline and the effectiveness of NBSearch to support code search in large notebook collections.

Researchers have explored several avenues to mitigate data scientists’ frustrations with computational notebooks, including: (1) live programming, to keep notebook results consistent and up to date; (2) supplementing scripting with graphical user interfaces (GUIs), to improve ease of use; and (3) providing domain-specific languages (DSLs), to raise a script’s level of abstraction. This paper introduces Glinda, which combines these three approaches by providing a live programming experience, with interactive results, for a domain-specific language for data science. The language’s compiler uses an open-ended set of “recipes” to execute steps in the user’s data science workflow. Each recipe is intended to combine the expressiveness of a written notation with the ease-of-use of a GUI. Live programming provides immediate feedback to a user’s input, whether in the form of program edits or GUI gestures. In a qualitative evaluation with 12 professional data scientists, participants highly rated the live programming and interactive results. They found the language productive and sufficiently expressive and suggested opportunities to extend it.

Training deep neural networks can generate non-descriptive error messages or produce unusual output without any explicit errors at all. While experts rely on tacit knowledge to apply debugging strategies, non-experts lack the experience required to interpret model output and correct Deep Learning (DL) programs. In this work, we identify DL debugging heuristics and strategies used by experts, andIn this work, we categorize the types of errors novices run into when writing ML code, and map them onto opportunities where tools could help. We use them to guide the design of Umlaut. Umlaut checks DL program structure and model behavior against these heuristics; provides human-readable error messages to users; and annotates erroneous model output to facilitate error correction. Umlaut links code, model output, and tutorial-driven error messages in a single interface. We evaluated Umlaut in a study with 15 participants to determine its effectiveness in helping developers find and fix errors in their DL programs. Participants using Umlaut found and fixed significantly more bugs and were able to implement fixes for more bugs compared to a baseline condition.

End-user programmers opportunistically copy-and-paste code snippets from colleagues or the web to accomplish their tasks. Unfortunately, these snippets often don’t work verbatim, so these people—who are non-specialists in the programming language—make guesses and tweak the code to understand and apply it successfully. To support their desired workflow and facilitate tweaking and understanding, we built a prototype tool, TweakIt, that provides users with a familiar live interaction to help them understand, introspect, and reify how different code snippets would transform their data. Through a usability study with 14 data analysts, participants found the tool to be useful to understand the function of otherwise unfamiliar code, to increase their confidence about what the code does, to identify relevant parts of code specific to their task, and to proactively explore and evaluate code. Overall, our participants were enthusiastic about incorporating TweakIt in their own day-to-day work.

Trigger-action programming (if-this-then-that rules) empowers non-technical users to automate services and smart devices. As a user’s set of trigger-action programs evolves, the user must reason about behavior differences between similar programs, such as between an original program and several modification candidates, to select programs that meet their goals. To facilitate this process, we co-designed user interfaces and underlying algorithms to highlight differences between trigger-action programs. Our novel approaches leverage formal methods to efficiently identify and visualize differences in program outcomes or abstract properties. We also implemented a traditional interface that shows only syntax differences in the rules themselves. In a between-subjects online experiment with 107 participants, the novel interfaces better enabled participants to select trigger-action programs matching intended goals in complex, yet realistic, situations that proved very difficult when using traditional interfaces showing syntax differences.

Many programmers want to use deep learning due to its superior accuracy in many challenging domains. Yet our formative study with ten programmers indicated that, when constructing their own deep neural networks (DNNs), they often had a difficult time choosing appropriate model structures and hyperparameter values. This paper presents ExampleNet—a novel interactive visualization system for exploring common and uncommon design choices in a large collection of open-source DNN projects. ExampleNet provides a holistic view of the distribution over model structures and hyperparameter settings in the corpus of DNNs, so users can easily filter the corpus down to projects tackling similar tasks and compare design choices made by others. We evaluated ExampleNet in a within-subjects study with sixteen participants. Compared with the control condition (i.e., online search), participants using ExampleNet were able to inspect more online examples, make more data-driven design decisions, and make fewer design mistakes.

This paper explores software’s role in visual art production by examining how artists use and develop software. We conducted interviews with professional artists who were collaborating with software developers, learning software development, and building and maintaining software. We found artists were motivated to learn software development for intellectual growth and access to technical communities. Artists valued efficient workflows through skilled manual execution and personal software development, but avoided high-level forms of software automation. Artists identified conflicts between their priorities and those of professional developers and computational art communities, which influenced how they used computational aesthetics in their work. These findings contribute to efforts in systems engineering research to integrate end-user programming and creativity support across software and physical media, suggesting opportunities for artists as collaborators. Artists’ experiences writing software can guide technical implementations of domain-specific representations, and their experiences in interdisciplinary production can aid inclusive community building around computational tools.

User interface design is a complex task that involves designers examining a wide range of options. We present Spacewalker, a tool that allows designers to rapidly search a large design space for an optimal web UI with integrated support. Designers first annotate each attribute they want to explore in a typical HTML page, using a simple markup extension we designed. Spacewalker then parses the annotated HTML specification, and intelligently generates and distributes various configurations of the web UI to crowd workers for evaluation. We enhanced a genetic algorithm to accommodate crowd worker responses from pairwise comparison of UI designs, which is crucial for obtaining reliable feedback. Based on our experiments, Spacewalker allows designers to effectively search a large design space of a UI, using the language they are familiar with, and improve their design rapidly at a minimal cost.

Reverse engineering (RE) of user interfaces (UIs) plays an important role in software evolution. However, the large diversity of UI technologies and the need for UIs to be resizable make this challenging. We propose ReverseORC, a novel RE approach able to discover diverse layout types and their dynamic resizing behaviours independently of their implementation, and to specify them by using OR constraints. Unlike previous RE approaches, ReverseORC infers flexible layout constraint specifications by sampling UIs at different sizes and analyzing the differences between them. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. We demonstrate that ReverseORC works across different platforms with very different layout approaches, e.g., for GUIs as well as for the Web. Furthermore, it can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.

This paper presents GestureMap, a visual analytics tool for gesture elicitation which directly visualises the space of gestures. Concretely, a Variational Autoencoder embeds gestures recorded as 3D skeletons on an interactive 2D map. GestureMap further integrates three computational capabilities to connect exploration to quantitative measures: Leveraging DTW Barycenter Averaging (DBA), we compute average gestures to 1) represent gesture groups at a glance; 2) compute a new consensus measure (variance around average gesture); and 3) cluster gestures with k-means. We evaluate GestureMap and its concepts with eight experts and an in-depth analysis of published data. Our findings show how GestureMap facilitates exploring large datasets and helps researchers to gain a visual understanding of elicited gesture spaces. It further opens new directions, such as comparing elicitations across studies. We discuss implications for elicitation studies and research, and opportunities to extend our approach to additional tasks in gesture elicitation.

Due to advances in deep learning, gestures have become a more common tool for human-computer interaction. When implementing a large amount of training data, deep learning models show remarkable performance in gesture recognition. Since it is expensive and time consuming to collect gesture data from people, we are often confronted with a practicality issue when managing the quantity and quality of training data. It is a well-known fact that increasing training data variability can help to improve the generalization performance of machine learning models. Thus, we directly intervene in the collection of gesture data to increase human gesture variability by adding some words (called styling words) into the data collection instructions, e.g., giving the instruction "perform gesture #1 faster" as opposed to "perform gesture #1." Through an in-depth analysis of gesture features and video-based gesture recognition, we have confirmed the advantageous use of styling words in gesture training data collection.

This paper contributes the first large-scale dataset of 17,979 hand-drawn sketches of 21 UI element categories collected from 967 participants, including UI/UX designers, front-end developers, HCI, and CS grad students, from 10 different countries. We performed a perceptual study with this dataset and found out that UI/UX designers can recognize the UI element sketches with ~96% accuracy. To compare human performance against computational recognition methods, we trained the state-of-the-art DNN-based image classification models to recognize the UI elements sketches. This study revealed that the ResNet-152 model outperforms other classification networks and detects unknown UI element sketches with 91.77% accuracy (chance is 4.76%). We have open-sourced the entire dataset of UI element sketches to the community intending to pave the way for further research in utilizing AI to assist the conversion of lo-fi UI sketches to higher fidelities.

In recent times, millions of people enjoy watching video gameplays at an eSports stadium or home. We seek a method that improves gameplay spectator or viewer experiences by presenting multisensory stimuli. Using a motion chair, we provide the motion effects automatically generated from the audiovisual stream to the viewers watching a first-person shooter (FPS) gameplay. The motion effects express the game character’s movement and gunfire action. We describe algorithms for the computation of such motion effects developed using computer vision techniques and deep learning. By a user study, we demonstrate that our method of providing motion effects significantly improves the viewing experiences of FPS gameplay. The contributions of this paper are with the motion synthesis algorithms integrated for FPS games and the empirical evidence for the benefits of experiencing multisensory gameplays.

Tilt as a gaming term is associated with frustration, rage, and deterioration of gameplay ability. In this exploratory study, we surveyed 95 esports players in a high school esports league on their definitions of tilt, triggers for tilting, responses to tilt, and perception of its malleability. We found that players are tilted most commonly by their own teammates rather than opponents, with their most negative tilt responses reserved for themselves. The majority surveyed believe that they can change how easily they are tilted and believing so was found to lead players to choose more positive responses to tilt. In contrast, perceiving tilt as malleable was found to increase the probability that participants respond with positive strategies. Implications for efforts to improve esports culture and community are discussed.

Competitive VR gaming has emerged as a new trend in recent years, due to the availability of consumer grade VR technologies and the rise of esports as a billion dollar industry. Despite the considerable attention to competitive VR gaming, there is a lack of research on attitudes and experiences that players have with these games. In this qualitative study with a pre-post interview design, we recruited eight competitive Counter-Strike: Global Offensive players from a university esports club. We aimed to understand their attitudes towards VR esports and their experiences playing a representative location based VR esports game. Findings showed that players had visceral and positive affective experiences in the game, such as how players map physical movements to the game. These findings can help design future competitive VR esports, while also further contributing to HCI as the first exploration on player experiences with VR esports, laying groundwork for future studies.

Esports are rapidly growing within academia. In HCI, earlier work explored how problematic behaviors emerging from gender biases (e.g., toxicity) negatively impact female participation in esports. Here, we further explore gender biases in esports by interviewing 19 self-identified female and male professional gamers and event organizers. We inquire our interviewees about personal experiences with gender biases in esports and their perspective on how these biases impact participation, inclusivity, and career prospects. Our interviewees see gender biases in esports as a consequence of stereotypical gender roles in gaming tout-court (e.g., girls do not like violence, boys are competitive by nature). The rationale for separating male and female in esports, however, seems rooted in the need for female gamers to create role-models and grow in self-confidence. We scrutinize the considerations emerging from our interviews under a Feminist HCI lens and discuss how HCI research can help design equitable environments in esports.

Esports is a rapidly growing industry, generating interest from research disciplines including marketing, social sciences, and human-computer interaction. Despite its continued growth, there is a lack of studies surrounding the health of esports players. Previous work on the subject is limited, as research has only recently begun to explore the potential factors affecting physical and psychological wellness. Using an exploratory mixed-methods approach, a series of semi-structured interviews (n = 10) and an online survey (n = 68) were used to identify the biggest health concerns among esports players. The results demonstrate a better understanding of issues regarding physical and psychological wellness according to the players. Moving forward, we suggest the HCI community adapts mindfulness, ergonomics, and social-emotional learning as methods for supporting player’s health concerns.

Game industry professionals are frequently implementing new methods of addressing ethical issues related to in-game toxicity and disruptive player behaviours associated with online multiplayer games. However, academic work on these behaviours tends to focus on the perspectives of players rather than the industry. To fully understand the ethics of multiplayer games and promote ethical design, we must examine the challenges facing those designing multiplayer games through an ethical lens. To this end, this paper presents a reflexive thematic analysis of 21 in-depth interviews with games industry professionals on their ethical views and experiences in game design and community management. We identify a number of tensions involved in making ethics-related design decisions for divided player communities alongside current game design practices that are concerned with functionality, revenue and entertainment. We then put forward a set of design considerations for integrating ethics into multiplayer game design.

Video game play is among the most popular forms of entertainment in the world and eSports is a multi-billion dollar industry. Esports gamers, and competitive gamers more broadly, want fast game systems to maximize their chances of winning. In general, the faster the game system, the lower the latency between a player’s action and the intended outcome. But how much small reductions in local latencies benefit competitive players is not known. This paper presents results from a 43-person user study that evaluates the impact of system latencies for high-end gaming systems (below 125 ms) on experienced Counter-strike: Global Offensive (CS:GO) players. Analysis of the results show pronounced benefits to CS:GO player performance (accuracy and score) for even small reductions in latency, with subjective opinions on Quality of Experience following suit.

In 2020, the rapid spread of COVID-19 forced many people to self-isolate, resulting in struggles with mental health [60], and an increase in gaming [3]. In this paper, we seek to examine how individuals used digital games during the quarantine. We conducted a two-tier qualitative study where we used a thematic analysis of tweets to derive questions for an online survey that we distributed. Results of thematic analysis of survey responses identified 15 themes. Some themes confirm previous works’ findings, particularly how games are used to increase social connection or distract oneself from unpleasant situations. We also found new themes unique to the quarantine, such as interactions with non-player characters used as a surrogate for real-world interaction and using in-game routines as a substitute to real-world routines lost due to the pandemic. This work discusses the use of games during the pandemic and can be seeds for future studies.

Whereas past CHI-related work has paid attention to various elements of the esports industry, there exists a scarcity of such research on the increasing convergence between esports and gambling. This study, therefore, aimed to shed light on the emerging phenomenon of esports betting in two ways. First, we theorized about its characteristics according to the 5W1H framework. Second, using semi-structured laddering interviews and Means-End Chain Theory, we assessed the esports betting motivations of young, male and recreational gamblers who interact with monetary and skin betting websites. Our results show that gamblers follow various motivational pathways when using an esports betting platform and construct their motives in relation to platform-specific properties and personal values. As such, we demonstrate that a holistic understanding of esports betting is of utmost importance if we want to explain and investigate its appeal to a worldwide audience.

Live streaming sites such as Twitch offer new ways for remote audiences to engage with and affect gameplay. While research has considered how audiences interact with games, HCI lacks clear demarcations of the potential design spaces for audience participation. This paper introduces and validates a theme map of audience participation in game live streaming for student designers. This map is a lens that reveals relationships among themes and sub-themes of Agency, Pacing, and Community—to explore, reflect upon, describe, and make sense of emerging, complex design spaces. We are the first to articulate such a lens, and to provide a reflective tool to support future research and education. To create the map, we perform a thematic analysis of design process documents of a course on audience participation for Twitch, using this analysis to visually coordinate relationships between important themes. To help student designers analyze and reflect on existing experiences, we supplement the theme map with a set of mapping procedures. We validate the applicability of our map with a second set of student designers, who found the map useful as a comparative and reflective tool.

With the rise of competitive online gaming and esports, players’ ability to review, reflect upon, and improve their in-game performance has become important. Post-play visualizations are key for such improvements. Despite the increased interest in visualizations of gameplay, research specifically informing the design of player-centric visualizations is currently limited. As with all visualizations, their design should, however, be guided by a thorough understanding of the goals to be achieved and which information is important and why.

This paper reports on a mixed-methods study exploring the information demands posed by players on post-play visualizations and the goals they pursue with such visualizations. We focused on three genres that enjoy great popularity within the competitive gaming scene. Our results provide useful guideposts on which data to focus on by offering an overview of the relevance of different in-game metrics across genres. Lastly, we outline high-level implications for the design of post-play visualizations.

In first-person shooters (FPS), professional players (a.k.a., Gosu) outperform amateur players. The secrets behind the performance of professional FPS players have been debated in online communities with many conjectures; however, attempts of scientific verification have been limited. We addressed this conundrum through a data-collection study of the gameplay of eight professional and eight amateur players in the commercial FPS Counter-Strike: Global Offensive. The collected data cover behavioral data from six sensors (motion capture, eye tracker, mouse, keyboard, electromyography armband, and pulse sensor) and in-game data (player data and event logs). We examined conjectures in four categories: aiming, character movement, physicality, and device and settings. Only 6 out of 13 conjectures were supported with statistically sufficient evidence.

We present a system for connecting partners in long-distance relationships in bedrooms and at bedtime, a space and time that most couples share. Unlike communications explicitly initiated by users, our system is always-on, staying in the background and enabling remote presence without constant use. We present findings from a field study in which the system was deployed into participants’ bedrooms. The system includes an automated photo-stream (rather than video), which was found to provide a balance between the feeling of presence and privacy, and to remove the pressure to communicate. The system also includes a real-time shared inking canvas with disappearing ink, which was found to provide a rich versatile medium allowing for new patterns of communication, live interventions, and collaborative drawing. Learnings from how our system balances privacy and remote connectedness may also have relevance for other domains such as remote healthcare and education.

With the growing ubiquity of wearable devices, sensed physiological responses provide new means to connect with others. While recent research demonstrates the expressive potential for biosignals, the value of sharing these personal data remains unclear. To understand their role in communication, we created Significant Otter, an Apple Watch/iPhone app that enables romantic partners to share and respond to each other’s biosignals in the form of animated otter avatars. In a one-month study with 20 couples, participants used Significant Otter with biosignals sensing OFF and ON. We found that while sensing OFF enabled couples to keep in touch, sensing ON enabled easier and more authentic communication that fostered social connection. However, the addition of biosignals introduced concerns about autonomy and agency over the messages they sent. We discuss design implications and future directions for communication systems that recommend messages based on biosignals.

Social media platforms continue to evolve as archival platforms, where important milestones in an individual’s life are socially disclosed for support, solidarity, maintaining and gaining social capital, or to meet therapeutic needs. However, a limited understanding of how and what life events are disclosed (or not) prevents designing platforms to be sensitive to life events. We ask what life events individuals disclose on a 256 participants’ year-long Facebook dataset of 14K posts against their self-reported life events. We contribute a codebook to identify life event disclosures and build regression models on factors explaining life events’ disclosures. Positive and anticipated events are more likely, whereas significant, recent, and intimate events are less likely to be disclosed on social media. While all life events may not be disclosed, online disclosures can reflect complementary information to self-reports. Our work bears practical and platform design implications in providing support and sensitivity to life events.

Music fans strategically support their artists. Their collective efforts can extend to social causes as well: In 2020 for example, ARMY—the fandom of the music group BTS—successfully organized the #MatchAMillion campaign to raise over one million USD to support Black Lives Matter. To better understand factors of fandoms’ collaborative success for arguably unrelated social goals, we conducted a survey focusing on ARMYs’ perceptions of their fandom and their social effort. Most ARMYs viewed the fandom as a community, loosely structured with pillar accounts. They reported trust in each other as well as high team composition, which mediated the relationship between their neutral psychological safety and high efficacy. Respondents attributed their success in #MatchAMillion to shared values, good teamwork, and established infrastructure. Our findings elucidate contextual factors that contribute to ARMY’s collaborative success and highlight themes that may be applied to studying other fandoms and their collaborative efforts.

Live streaming is a rapidly growing industry, with millions of content creators using platforms like Twitch to share games, art, and other activities. However, with this rise in popularity, most streamers often fail to attract viewers and grow their platforms. Analytic tools—which have shown success in other business and learning contexts—may be one potential solution, but their use in streaming settings remains unexplored. In this study, we focused on game streaming and interviewed 18 game streamers on Twitch and Mixer about their information needs and current use of tools, supplemented by explorations into their Discord communities. We find that streamers have a range of content, marketing, and community information needs, many of which are not being met by available tools. We conclude with design implications for developing more streamer-centered analytics for video game streamers.

Loneliness threatens public mental wellbeing during COVID-19. In response, YouTube creators participated in the #StayHome #WithMe movement (SHWM) and made myriad videos for people experiencing loneliness or boredom at home. User-shared videos generate parasocial attachment and virtual connectedness. However, there is limited knowledge of how creators contributed videos during disasters to provide social provisions as disaster-relief. Grounded on Weiss’s loneliness theory, this work analyzed 1488 SHWM videos to examine video sharing as a pathway to social provisions. Findings suggested that skill and knowledge sharing, entertaining arts, homelife activities, live chatting, and gameplay were the most popular video styles. YouTubers utilized parasocial relationships to form a space for staying away from the disaster. SHWM YouTubers provided friend-like, mentor-like, and family-like provisions through videos in different styles. Family-like provisions led to the highest overall viewer engagement. Based on the findings, design implications for supporting viewers’ mental wellbeing in disasters are discussed.

The growing number of video chat users includes socially anxious people, but it is not known how video chat interfaces affect their interpersonal interactions. In our first study, we use a get-to-know-you task to show that when video feedback of oneself is disabled, higher social anxiety is associated with more public self-awareness, use of 2nd person pronouns, and experienced anxiety. Higher social anxiety was linked to discussing more topics, but discussing more topics only elicited higher self-disclosure and trust when social anxiety was low. In our second study, we assess these same effects using a presentation layout video chat interface and observe no effects of social anxiety on public self-awareness, 2nd person pronoun use, or number of topics discussed; no effect of feedback on experienced anxiety; and no link between number of topics and self-disclosure. Video chat adopters and designers should consider how feedback and interface layout affect conversations.

Even though today’s videoconferencing systems are often very useful, these systems do not provide support for one of the most important aspects of in-person meetings: the ad hoc, private conversations that happen before, after, and during the breaks of scheduled events–the proverbial hallway conversations. Here we describe our design of a simple system, called Minglr, which supports this kind of interaction by facilitating the matching of conversational partners. We describe two studies of this system’s use at two virtual conferences with over 450 total participants. Our results provide evidence for the usefulness of this capability, showing that, for example, 81% of people who used the system successfully thought that future virtual conferences should include a tool with similar functionality. We believe that similar functionality is likely to be widely implemented in many videoconferencing systems and to increase the feasibility and desirability of many kinds of remote work and socializing.

An estimated 100,000 people work today as commercial content moderators. These moderators are often exposed to disturbing content, which can lead to lasting psychological and emotional distress. This literature review investigates moderators’ psychological symptomatology, drawing on other occupations involving trauma exposure to further guide understanding of both symptoms and support mechanisms. We then introduce wellness interventions and review both programmatic and technological approaches to improving wellness. Additionally, we review methods for evaluating intervention efficacy. Finally, we recommend best practices and important directions for future research. Content Warning: we discuss the intense labor and psychological effects of CCM, including graphic descriptions of mental distress and illness.

Co-customizations are collaborative customizations in messaging apps that all conversation members can view and change, e.g. the color of chat bubbles on Facebook Messenger. Co-customizations grant new opportunities for expressing intimacy; however, most apps offer private customizations only. To investigate how people in close relationships integrate co-customizations into their established communication app ecosystems, we built DearBoard: an Android keyboard that allows two people to co-customize its color theme and a toolbar of expression shortcuts (emojis and GIFs). In a 5-week field study with 18 pairs of couples, friends, and relatives, participants expressed their shared interests, history, and knowledge of each other through co-customizations that served as meaningful decorations, interface optimizations, conversation themes, and non-verbal channels for playful, affectionate interactions. The co-ownership of the co-customizations invited participants to negotiate who customizes what and for whom they customize. We discuss how co-customizations mediate intimacy through place-making efforts and suggest design opportunities.

Message deletion in mobile messaging apps allows people to “unsay” things they have said. This paper explores how and why people use (or do not use) this feature within remediation strategies after a communication error is identified. We present findings from a multi-stage survey designed to explore people’s general experiences of the message deletion feature (N = 401), peoples’ experiences of using this feature during the remediation of an error (N = 70), and receivers’ perceptions around recent message deletions (N = 68). While people are typically aware of the deletion feature, it is infrequently used. When used, it is primarily done so to improve conversations by reducing confusion between conversation partners. We found people being aware of message deletions creating information-gaps which can provoke curiosity in recipients, causing them to develop narratives to help address the uncertainty. We found concerns amongst senders that these narratives would be of a negative nature, having an undesirable impact on how others perceive them. We use our findings to suggest ways in which mobile messaging apps could improve conversational experiences around erroneous and regrettable messages.

We present a video-analysis study of museum visitors’ interactions at two tangible interactive exhibits in a transport museum. Our focus is on groups’ social and shared interactions, in particular how exhibit setup and structure influence collaboration patterns. Behaviors at the exhibits included individuals focusing beyond their personal activity towards companions’ interaction, adults participating via physical interaction, and visitors taking opportunities to interact when companions moved between sections of the exhibit or stepped back from interaction. We demonstrate how exhibits’ physical configuration and interactive control engendered behavioral patterns. Systematic analysis reveals how different configurations (concerning physical-spatial hardware and interactive software) distribute control differently amongst visitors. We present four mechanisms for how control can be distributed at an interactive installation: functional, temporal, physical and indirect verbal. In summary, our work explores how mechanisms that distribute control influence patterns of shared interaction with the exhibits and social interaction between museum visitor companions.

Co-designing with children in an online environment is increasingly important due to external factors, such as the COVID-19 pandemic, and the diversification and inclusion of youth participants. Many prior studies about co-design with youth focus on co-located or asynchronous online sessions. However, conducting synchronous online co-design sessions adds layers of complexity and uncertainty to collaboration. This paper introduces a model explicating factors to consider when co-designing with children synchronously in an online space. We examined ten consecutive intergenerational participatory design sessions online where children (ages 7-11) and adults designed new technologies. Along with highlighting unexpected moments and interactions, we use theories of improvisation to guide our understanding of dynamic situations that are out of the control of researchers. This work contributes to improving theoretical understanding of improvisation as a method of inquiry for co-designing with youth, and offers practical suggestions for suitable online co-design techniques and implementation.

When schools and families form a good partnership, children benefit. With the recent flourishing of communication apps, families and schools in China have shifted their primary communication channels to chat groups hosted on popular instant-messenger(IM) tools such as WeChat and QQ. With an interview study consisting of 18 parents and 9 teachers, followed by a survey study with 210 teachers, we found that IM group chat has become the most popular way that the majority of parents and teachers communicate, from among the many different channels available. While there are definite advantages to this kind of group chat, we also found a number of problematic issues, including a lack of privacy and repeated negative feedback shared by both parents and teachers. We discuss our results on how IM-based group chat could affect Chinese teachers’ authoritative figures, affect Chinese teacher’s work-life balance and potentially compromise Chinese students’ privacy.

The COVID-19 pandemic and subsequent closure of schools forced families across the globe to transition to school at home. This unprecedented context is likely to have a lasting impact on the practice of schooling and the role of online, digital platforms within school contexts. In this paper we present a contextual inquiry of an ‘emergency home school context’, detailing how nine young families in Melbourne, Australia adapted to the unexpected introduction of school to the home following the government-directed closure of schools. Through an online interview and photo-journal study, we develop an emplaced understanding of the context detailing how the relations between people and places around the home evolved over time. We present five design considerations for digital platforms to support the emergency home school context, placing focus on the fluid roles, relationships and evolving sense of place.

The COVID-19 global pandemic and resulted lockdown policies have forced education in nearly every country to switch from a traditional co-located paradigm to a pure online “distance learning from home” paradigm. Lying in the center of this learning paradigm shift is the emergence and wide adoption of distance communication tools and live streaming platforms for education. Here, we present a mixed-methods study on live streaming based education experience during the COVID-19 pandemic. We focus our analysis on Chinese higher education, carried out semi-structured interviews on 30 students, and 7 instructors from diverse colleges and disciplines, meanwhile launched a large-scale survey covering 6291 students and 1160 instructors in one leading Chinese university. Our study not only reveals important design guidelines and insights to better support current remote learning experience during the pandemic, but also provides valuable implications towards constructing future collaborative education supporting systems and experience after pandemic.

Today, preschool-aged children have an abundance of digital content to choose from, with some more desirable than others from a developmental perspective. We aim to describe and better understand the interplay of factors that influence children’s selection of media content using a year-long, multi-case ethnography of 13 diverse families in Southern California. We found that young children’s media content selection may be best understood as an ecologically situated process involving the interplay between the content, the child, their family, community, and societal spheres. Children do not make media selections on their own. Rather, these choices are supported or constrained by a range of resource, culture, and policy factors specific to family and community background. We argue that policy makers and technology designers are better served by an ecological perspective if they wish to understand how digital content is selected and used by children in sociocultural context.

Digital fabrication courses that relied on physical makerspaces were severely disrupted by COVID-19. As universities shut down in Spring 2020, instructors developed new models for digital fabrication at a distance. Through interviews with faculty and students and examination of course materials, we recount the experiences of eight remote digital fabrication courses. We found that learning with hobbyist equipment and online social networks could emulate using industrial equipment in shared workshops. Furthermore, at-home digital fabrication offered unique learning opportunities including more iteration, machine tuning, and maintenance. These opportunities depended on new forms of labor and varied based on student living situations. Our findings have implications for remote and in-person digital fabrication instruction. They indicate how access to tools was important, but not as critical as providing opportunities for iteration; they show how remote fabrication exacerbated student inequities; and they suggest strategies for evaluating trade-offs in remote fabrication models with respect to learning objectives.

This paper investigates in-class interactions in synchronous online classrooms when the choice of modality is discretionary, such that students choose when and if they turn on their cameras and microphones. Instructor interviews (N = 7) revealed that most students preferred not to share videos and verbally participate. This hindered instructors’ ability to read their classrooms and make deeper connections with students. Survey results (N = 102) suggested that students felt a lacking sense of community in online vs. in-person lectures. Some students felt uncomfortable broadcasting their appearances to everyone in the class, and some were unaware of the benefits for instructors. Most students favored using the text chat to participate. Considering the needs of both instructors and students, we propose recommendations to mitigate the loss of classroom interactions by collecting and presenting less invasive social cues in an aggregated format, and incorporating opportunities for informal exchanges and individual control to spark peer bonding.

We investigated changes in and factors affecting American adolescents’ subjective wellbeing during the early months (April – August 2020) of the coronavirus pandemic in the United States. Twenty-one teens (14-19 years) participated in interviews at the start and end of the study and completed ecological momentary assessments three times per week between the interviews. There was an aggregate trend toward increased wellbeing, with considerable variation within and across participants. Teens reported greater reliance on networked technologies as their unstructured time increased during lockdown. Using multilevel growth modeling, we found that how much total time teens spent with technology had less bearing on daily fluctuations in wellbeing than the satisfaction and meaning they derived from their technology use. Ultimately, teens felt online communication could not replace face-to-face interactions. We conducted two follow-up participatory design sessions with nine teens to explore these insights in greater depth and reflect on general implications for design to support teens’ meaningful technology experiences and wellbeing during disruptive life events.

Social class contexts shape parents’ guiding principles around teens’ smartphone use. These contexts can affect how parents coach and censor their teens’ smartphone use and can create tensions in the home. Through 174 interviews (87 parent-teen dyads), we find that upper-middle-class families generally adopt an orientation toward scaffolded achievements and working-class families tend to embrace an orientation toward empowered self-sufficiency. We further find that these class-based orientations contribute to parent-teen tensions. For upper-middle-class families, tensions arise when parents insist that teens should use smartphones to get help with academic and enrichment activities and teens disagree about whether their phone-related activities align with this goal. In contrast, we find that conflict can occur in working-class families when teens use their smartphones to get assistance and parents interpret such activity as teens being lazy or not self-sufficient. These findings highlight the role of social class contexts in shaping families’ orientations toward teens’ smartphone use and phone-related tensions.

Researchers and designers have incorporated social media affordances into learning technologies to engage young people and support personally relevant learning, but youth may reject these attempts because they do not meet user expectations. Through in-depth case studies, we explore the sociotechnical ecosystems of six teens (ages 15-18) working at a science center that had recently introduced a digital badge system to track and recognize their learning. By analyzing interviews, observations, ecological momentary assessments, and system data, we examined tensions in how badges as connected learning technologies operate in teens' sociotechnical ecosystems. We found that, due to issues of unwanted context collapse and incongruent identity representations, youth only used certain affordances of the system and did so sporadically. Additionally, we noted that some features seemed to prioritize values of adult stakeholders over youth. Using badges as a lens, we reveal critical tensions and offer design recommendations for networked learning technologies.

“Study with me” videos contain footage of people studying for hours, in which social components like conversations or informational content like instructions are absent. Recently, they became increasingly popular on video-sharing platforms. This paper provides the first broad look into what “study with me” videos are and how people use them. We analyzed 30 “study with me” videos and conducted 12 interviews with their viewers to understand their motivation and viewing practices. We identified a three-factor model that explains the mechanism for shaping a satisfactory studying experience in general. One of the factors, a well-suited ambience, was difficult to achieve because of two common challenges: external conditions that prevent studying in study-friendly places and extra cost needed to create a personally desired ambience. We found that the viewers used “study with me” videos to create a personalized ambience at a lower cost, to find controllable peer pressure, and to get emotional support. These findings suggest that the viewers self-regulate their learning through watching “study with me” videos to improve efficiency even when studying alone at home.

Personal informatics (PI) technologies allow users to collect data about aspects of their lifestyle like mood or step count. Though teens increasingly encounter and use such technologies, little is known about how they ascribe meaning to their own PI activities. We report a qualitative study of the PI experiences of eighteen teens (aged 14 – 17). Following a learning phase focused on interpreting PI data, participants chose a personal goal that interested them and a PI tool to track it for 4-8 weeks in everyday contexts. Participants proved to be competent, flexible users of PI tools, tracking a range of meaningful life factors, from ‘worries’ to ‘exercise’; they valued learning about ‘natural patterns’ in their lives and were motivated to manage their emotions and evaluate whether they were doing the right thing. Our findings contribute to understanding how young people can engage in appropriation and interpretation of PI data – suggesting opportunities for educational interventions and design.

Bullying is a challenge concerning us all, and particularly our children. This has already been acknowledged by CHI, among others. Despite the interest, there is a lack of comprehensive understanding of the state of the art – a critical review is needed, addressing bullying in the lives of children, in the context of and/or by the means of design and technology, covering CHI as well as related computing fields, being inspired by the strong body of knowledge within human sciences. We report on a comprehensive literature review on the topic, with the aim to understand what and how has been done so far to handle this troublesome and widespread phenomenon as well as to indicate how to move the field forward. We report how the topic has been examined and with what kind of means tackled, revealing interesting underlying assumptions about design, technology and human agency.

Subtle patterns in users’ think-aloud (TA) verbalizations and speech features are shown to be telltale signs of User Experience (UX) problems. However, such patterns were uncovered among young adults. Whether such patterns apply for older adults remains unknown. We conducted TA usability testing with older adults using physical and digital products. We analyzed their verbalizations, extracted speech features, identified UX problems, and uncovered the patterns that indicate UX problems. Our results show that when older adults encounter problems, their verbalizations tend to include observations (remarks), negations, question words and words with negative sentiments; and their voices tend to include high loudness, high pitch and high speech rate. We compare these subtle patterns with those of young adults uncovered in recent studies and discuss the implications of these patterns for the design of Human-AI collaborative UX analysis tools to better pinpoint UX problems.

Priming is used as a way of increasing the diversity of proposals in end-user elicitation studies, but priming has not been investigated thoroughly in this context. We conduct a distributed end-user elicitation study with 167 participants, which had three priming groups: a no-priming control group, sci-fi priming, and a creative mindset group. We evaluated the gestures proposed by these groups in a distributed learnability and memorability study with 18 participants. We found that the user-elicited gestures from the sci-fi group were significantly faster to learn, requiring an average of 1.22 viewings to learn compared to 1.60 viewings required to learn the control gestures, and 1.56 viewings to learn the gestures elicited from the creative mindset group. In addition, both primed gesture groups had higher memorability with 80% of the sci-fi-primed gestures and 73% of the creative mindset group gestures were recalled correctly after one week without practice compared to 43% of the control group gestures.

There is a rapidly growing literature on dark patterns, user interface designs—typically related to shopping or privacy—that researchers deem problematic. Recent work has been predominantly descriptive, documenting and categorizing objectionable user interfaces. These contributions have been invaluable in highlighting specific designs for researchers and policymakers. But the current literature lacks a conceptual foundation: What makes a user interface a dark pattern? Why are certain designs problematic for users or society?

We review recent work on dark patterns and demonstrate that the literature does not reflect a singular concern or consistent definition, but rather, a set of thematically related considerations. Drawing from scholarship in psychology, economics, ethics, philosophy, and law, we articulate a set of normative perspectives for analyzing dark patterns and their effects on individuals and society. We then show how future research on dark patterns can go beyond subjective criticism of user interface designs and apply empirical methods grounded in normative perspectives.

Online display advertising on websites is widely disliked by users, with many turning to ad blockers to avoid “bad” ads. Recent evidence suggests that today’s ads contain potentially problematic content, in addition to well-studied concerns about the privacy and intrusiveness of ads. However, we lack knowledge of which types of ad content users consider problematic and detrimental to their browsing experience. Our work bridges this gap: first, we create a taxonomy of 15 positive and negative user reactions to online advertising from a survey of 60 participants. Second, we characterize classes of online ad content that users dislike or find problematic, using a dataset of 500 ads crawled from popular websites, labeled by 1000 participants using our taxonomy. Among our findings, we report that users consider a substantial amount of ads on the web today to be clickbait, untrustworthy, or distasteful, including ads for software downloads, listicles, and health & supplements.

Prior work in cross-cultural psychology and neuroscience has shown robust variations in visual attention patterns. People from East Asian societies, in which a holistic thinking style predominates, have been found to attend to contextual information in scenes more than Westerners, whose tendency to think analytically expresses itself in greater attention to foreground objects. This paper applies these findings to website design, using an online study to evaluate whether Japanese (N=65) remember more and are faster at finding contextual website information than US Americans (N=84). Our results do not support this hypothesis. Instead, Japanese overall took significantly longer to find information than US participants—a difference that was exacerbated by an increase in website complexity—suggesting that Japanese may holistically take in a website before engaging with detailed information. We discuss implications of these findings for website design and cross-cultural research.

Spatial computing devices that blend virtual and real worlds have the potential to soon become ubiquitous. Yet, creating experiences for spatial computing is non-trivial and needs skills in programming and 3D content creation, rendering them inaccessible to a wider group of users. We present SpatialProto, an in-situ spatial prototyping system for lowering the barrier to engage in spatial prototyping. With a depth-sensing capable mixed reality headset, SpatialProto lets users record animated objects of the real-world environment (e.g. paper, clay, people, or any other prop), extract only the relevant parts, and directly place and transform these recordings in their physical environment. We describe the design and implementation of SpatialProto, a user study evaluating the system’s prototype with non-expert users (n = 9), and demonstrate applications where multiple captures are fused for compelling augmented reality experiences.

Interactive machine learning (iML) tools help to make ML accessible to users with limited ML expertise. However, gathering necessary training data and expertise for model-building remains challenging. Transfer learning, a process where learned representations from a model trained on potentially terabytes of data can be transferred to a new, related task, offers the possibility of providing ”building blocks” for non-expert users to quickly and effectively apply ML in their work. However, transfer learning largely remains an expert tool due to its high complexity. In this paper, we design a prototype to understand non-expert user behavior in an interactive environment that supports transfer learning. Our findings reveal a series of data- and perception-driven decision-making strategies non-expert users employ, to (in)effectively transfer elements using their domain expertise. Finally, we synthesize design implications which might inform future interactive transfer learning environments.

The advancement of computational design and fabrication technologies has allowed combining physical and digital processes in architecture. Existing methods for physical-digital integration offer limited support for explorations with folded non-linear surfaces. This paper introduces a feedback-oriented design approach linking physical models with digital tools to enhance ideation processes in architecture. We employ paper as a medium for translating simple mock-up ideas to more elaborate digital design models. We explain the physical exploration, 3D scanning, digital simulation, and fabrication processes. Then, we discuss the results, observations, and limitations of this design approach.

Reflection is an often addressed design goal in Human-Computer Interaction (HCI) research. An increasing number of artefacts for reflection have been developed in recent years. However, evaluating if and how an interactive technology helps a user reflect is still complex. This makes it difficult to compare artefacts (or prototypes) for reflection, impeding future design efforts. To address this issue, we developed the Technology-Supported Reflection Inventory (TSRI), which is a scale that evaluates how effectively a system supports reflection. We first created a list of possible scale items based on past work in defining reflection. The items were then reviewed by experts. Next, we performed exploratory factor analysis to reduce the scale to its final length of nine items. Subsequently, we confirmed test-retest validity of our instrument, as well as its construct validity. The TSRI enables researchers and practitioners to compare prototypes designed to support reflection.

Visual and physical representations of historical personal data have been discussed as artifacts that can lead to self-reflection through meaning-making. However, it is yet unclear how those two concepts relate to each other. We focus on meaningfulness, a part of meaning-making that relates to feelings. In this paper, we present three projects where mundane objects, our data agents, are combined in meaningful ways with personal data with the aim to trigger reflection by placing a person’s individual experience of data in relation to others’. To identify relationships between self-reflection and meaningfulness we use Fleck and Fitzpatrick’s framework to describe the levels of reflection that we found in our projects and Mekler and Hornbæk’s meaning framework to define the depth of reflection. We conclude with a discussion on four themes in which we outline how data agents informed the intersections between our central concepts. This paper, constitutes a first step towards unpacking those relationships and invites for further explorations by the HCI community.

In the attention economy, video apps employ design mechanisms like autoplay that exploit psychological vulnerabilities to maximize watch time. Consequently, many people feel a lack of agency over their app use, which is linked to negative life effects such as loss of sleep. Prior design research has innovated external mechanisms that police multiple apps, such as lockout timers. In this work, we shift the focus to how the internal mechanisms of an app can support user agency, taking the popular YouTube mobile app as a test case. From a survey of 120 U.S. users, we find that autoplay and recommendations primarily undermine sense of agency, while playlists and search support it. From 13 co-design sessions, we find that when users have a specific intention for how they want to use YouTube they prefer interfaces that support greater agency. We discuss implications for how designers can help users reclaim a sense of agency over their media use.

This paper presents and evaluates Aiki, a simple browser extension designed to redirect a user to a learning platform for a fixed amount of time before accessing websites defined as ’procrastination’ or ’time-wasting’ websites. The goal of the extension is to enable the user to exchange time spent on pages they believe contribute less to their own productivity for microlearning activities, defined as small or short-term learning activities. The paper describes the design and development of Aiki and evaluates the extension with a group of n = 10 participants studying the Danish language. Based on a two-week study, we conclude that this type of extension, even in its preliminary version, has the potential to improve language skills in a lightweight manner and that redirection is an important alternative to blocking for procrastination management.

The rise of autonomous systems in cities, such as automated vehicles (AVs), requires new approaches for prototyping and evaluating how people interact with those systems through context-based user interfaces, such as external human-machine interfaces (eHMIs). In this paper, we present a comparative study of three prototype representations (real-world VR, computer-generated VR, real-world video) of an eHMI in a mixed-methods study with 42 participants. Quantitative results show that while the real-world VR representation results in higher sense of presence, no significant differences in user experience and trust towards the AV itself were found. However, interview data shows that participants focused on different experiential and perceptual aspects in each of the prototype representations. These differences are linked to spatial awareness and perceived realism of the AV behaviour and its context, affecting in turn how participants assess trust and the eHMI. The paper offers guidelines for prototyping and evaluating context-based interfaces through simulations.

Accessibility research has grown substantially in the past few decades, yet there has been no literature review of the field. To understand current and historical trends, we created and analyzed a dataset of accessibility papers appearing at CHI and ASSETS since ASSETS’ founding in 1994. We qualitatively coded areas of focus and methodological decisions for the past 10 years (2010-2019, N=506 papers), and analyzed paper counts and keywords over the full 26 years (N=836 papers). Our findings highlight areas that have received disproportionate attention and those that are underserved—for example, over 43% of papers in the past 10 years are on accessibility for blind and low vision people. We also capture common study characteristics, such as the roles of disabled and nondisabled participants as well as sample sizes (e.g., a median of 13 for participant groups with disabilities and older adults). We close by critically reflecting on gaps in the literature and offering guidance for future work in the field.

Despite the potential benefits of assistive technologies (ATs) for people with various disabilities, only around 7% of Chinese with disabilities have had an opportunity to use ATs. Even for those who have used ATs, the abandonment rate was high. Although China has the world’s largest population with disabilities, prior research exploring how ATs are used and perceived, and why ATs are abandoned have been conducted primarily in North America and Europe. In this paper, we present an interview study conducted in China with 26 people with various disabilities to understand their practices, challenges, perceptions, and misperceptions of using ATs. From the study, we learned about factors that influence AT adoption practices (e.g., misuse of accessible infrastructure, issues with replicating existing commercial ATs), challenges using ATs in social interactions (e.g., Chinese stigma), and misperceptions about ATs (e.g., ATs should overcome inaccessible social infrastructures). Informed by the findings, we derive a set of design considerations to bridge the existing gaps in AT design (e.g., manual vs. electronic ATs) and to improve ATs’ social acceptability in China.

In the US, abuse of individuals with intellectual and developmental disabilities (I/DD) is at epidemic proportions. Further, abuse incidents of individuals with I/DD are woefully under-reported. We surveyed practitioners who help individuals with I/DD post-abuse to get a broader context on the problem. We found that abuse of individuals with I/DD was often reported by someone other than the survivor, as survivors faced impediments in reporting. Consequently, we argue for developing a mobile-computing-based reporting tool for empowering individuals with I/DD to self-report abuse. Next, we conducted focus groups of individuals with I/DD to evaluate the tool’s viability, with respect to their ability to recognize/report abuse and use mobile-computing devices. We found individuals with I/DD could recognize/report abuse well when they received appropriate training. We also found individuals with I/DD could independently use their devices, though they shared access to them with family. Based on these findings, we call for several lines of accessibility research in designing an abuse self-reporting tool for individuals with I/DD.

Best practices for conducting HCI research on dementia care increasingly involve multiple stakeholders and incorporate diverse viewpoints. When done effectively, involving proxy stakeholders such as family members and professionals can help bring forward the voices of people with dementia. However, concrete practical guidance for navigating the challenges of integrating different perspectives is lacking. We critically reflect on our own recent qualitative fieldwork involving participants with dementia, family caregivers, and facilitators at a local social program for people with dementia, re-examining our interview transcripts and observation notes through content analysis. We illustrate practical approaches to prioritizing participants’ voices through concrete excerpts that demonstrate strategies for better managing dynamics, intervening effectively, and engaging all stakeholders in the research process. Our reflections and proposed guidelines can benefit HCI researchers and practitioners working with vulnerable populations. We hope this work will spur further discussion and critique to strengthen and improve research practices in this domain.

Content creators are instructed to write textual descriptions of visual content to make it accessible; yet existing guidelines lack specifics on how to write about people’s appearance, particularly while remaining mindful of consequences of (mis)representation. In this paper, we report on interviews with screen reader users who were also Black, Indigenous, People of Color, Non-binary, and/or Transgender on their current image description practices and preferences, and experiences negotiating theirs and others’ appearances non-visually. We discuss these perspectives, and the ethics of humans and AI describing appearance characteristics that may convey the race, gender, and disabilities of those photographed. In turn, we share considerations for more carefully describing appearance, and contexts in which such information is perceived salient. Finally, we offer tensions and questions for accessibility research to equitably consider politics and ecosystems in which technologies will embed, such as potential risks of human and AI biases amplifying through image descriptions.

Despite efforts to support students with disabilities in higher education, few continue to pursue doctoral degrees in computing. We conducted an interview study with 12 blind and low vision, and 7 deaf and hard of hearing current and former doctoral students in computing to understand how graduate students adjust to inaccessibility and ineffective accommodations. We asked participants how they worked around inaccessibility, managed ineffective accommodations, and advocated for tools and services. Employing a lens of ableism in our analysis, we found that participants’ extra effort to address accessibility gaps gave rise to a burden of survival, which they sustained to meet expectations of graduate-level productivity. We recommend equitable solutions that acknowledge taken-for-granted workarounds and that actively address inaccessibility in the graduate school context.

There is considerable effort within the HCI community to explore, document, and advocate for the lived experiences of persons with disabilities (PWDs). However, PWDs from the Global South, particularly Africa, are underrepresented in this scholarship. We contribute to closing this gap by investigating the unmet transit needs and characterization of technology within the disability community in Kampala, Uganda. We investigated transportation due to the increase in ride-share solutions created by widespread mobile computing and the resulting disruption of transportation worldwide. We hosted co-design sessions with disability advocates and adapted the stakeholder tokens method from the value-sensitive design framework to map the stakeholder ecosystem. Our key insight is the identification of a new group of non-traditional core stakeholders who highlight the values of inclusion, mobility, and safety within the ecosystem. Finally, we discuss how our findings engage with concepts of disability justice and perceptions of power.

Designers of digital content have access to various resources that they use to help them meet disabled people’s accessibility needs. Disability simulations are one resource, but often criticized for failing to guide digital designers appropriately, and it is unclear if digital designers are aware of the issues surrounding disability simulations. I surveyed 92 digital designers to understand their perspectives toward disability simulations (both perceived advantages and disadvantages). I then shared work process challenges faced by digital designers and their reasons for using disability simulations with 17 people with vision impairments to facilitate a discussion on this topic. The interviewees discussed ideas that suggest many paths can be explored to connect digital designers and disabled people, in general, to reduce reliance on simulations, and a change is needed within workplace processes, culture, and staffing to further support positive change. There are research opportunities to investigate establishing avenues for connecting digital designers and disabled people in a way that is beneficial to both groups.

Authentication has become increasingly ubiquitous for controlling access to personal computing devices (e.g., laptops, tablets, and smartphones). In this paper, we aim to understand the authentication process used by people with upper extremity impairment (UEI). A person with UEI lacks range of motion, strength, endurance, speed, and/or accuracy associated with arms, hands, or fingers. To this end, we conducted semi-structured interviews with eight (8) adults with UEI about their use of authentication for their personal computing devices. We found that our participants primarily use passwords and PINs as a verification credential during authentication. We found the process of authentication to have several accessibility issues for our participants. Consequently, our participants implemented a variety of workarounds that prioritized usability over security throughout the authentication process. Based on these findings, we present six broad subareas of research that should be explored in order to create more accessible authentication for people with UEI.

We conceptualize Digital Design Marginalization (DDM) as the process in which a digital interface design excludes certain users and contributes to marginalization in other areas of their lives. Due to non-inclusive designs, many underrepresented users face barriers in accessing essential services that are moving increasingly, sometimes exclusively, online – services such as personal finance, healthcare, social connectivity, and shopping. This can further perpetuate the “digital divide,” a technology-based form of social inequality that has offline consequences. We introduce the term Marginalizing Design to describe designs that contribute to DDM. In this paper, we focus on the impact of Marginalizing Design on older adults through examples from our research and discussions of services that may have marginalizing designs for older adults. Our aim is to provide a conceptual lens for designers, service providers, and policy makers through which they can use to purposely lessen or avoid digitally marginalizing groups of users.

Crowdwork can enable invaluable opportunities for people with disabilities, not least the work flexibility and the ability to work from home, especially during the current Covid-19 pandemic. This paper investigates how engagement in crowdwork tasks is affected by individual disabilities and the resulting implications for HCI. We first surveyed 1,000 Amazon Mechanical Turk (AMT) workers to identify demographics of crowdworkers who identify as having various disabilities within the AMT ecosystem—including vision, hearing, cognition/mental, mobility, reading and motor impairments. Through a second focused survey and follow-up interviews, we provide insights into how respondents cope with crowdwork tasks. We found that standard task factors, such as task completion time and presentation, often do not account for the needs of users with disabilities, resulting in anxiety and a feeling of depression on occasion. We discuss how to alleviate barriers to enable effective interaction for crowdworkers with disabilities.

The Trail Making Test (TMT) is a frequently used neuropsychological test for assessing cognitive performance. The subject connects a sequence of numbered nodes by using a pen on normal paper. We present an automatic cognitive assessment tool that analyzes samples of the TMT which we record using a digital pen. This enables us to analyze digital pen features that are difficult or impossible to evaluate manually. Our system automatically measures several pen features, including the completion time which is the main performance indicator used by clinicians to score the TMT in practice. In addition, our system provides a structured report of the analysis of the test, for example indicating missed or erroneously connected nodes, thereby offering more objective, transparent and explainable results to the clinician. We evaluate our system with 40 elderly subjects from a geriatrics daycare clinic of a large hospital.

Alerting customers on suspected online-payment fraud and persuade them to terminate transactions is increasingly requested with the rapid growth of digital finance worldwide. We explored the feasibility of using a conversational agent (CA) to fulfill this request. Shing, a voice-based CA, proactively initializes and repairs the conversation with empathetical communication skills in order to alert customers when a suspected online-payment fraud is detected, collects important information for fraud scrutiny and persuades customers to terminate the transaction once the fraud is confirmed. We evaluated our system by comparing it with a rule-based CA with regards to customer response and perceptions in a real-world context where our systems took 144,795 phone calls in total in which 83,019 (57.3%) natural breakdowns happened. Results showed that more customers stopped risky transactions after conversing with Shing. They seemed more willing to converse with Shing for more dialogue turns and provide transaction details. Our work presents practical implications for the design of proactive CA.

Building conversational agents that can conduct natural and prolonged conversations has been a major technical and design challenge, especially for community-facing conversational agents. We posit Mutual Theory of Mind as a theoretical framework to design for natural long-term human-AI interactions. From this perspective, we explore a community’s perception of a question-answering conversational agent through self-reported surveys and computational linguistic approach in the context of online education. We first examine long-term temporal changes in students’ perception of Jill Watson (JW), a virtual teaching assistant deployed in an online class discussion forum. We then explore the feasibility of inferring students’ perceptions of JW through linguistic features extracted from student-JW dialogues. We find that students’ perception of JW’s anthropomorphism and intelligence changed significantly over time. Regression analyses reveal that linguistic verbosity, readability, sentiment, diversity, and adaptability reflect student perception of JW. We discuss implications for building adaptive community-facing conversational agents as long-term companions and designing towards Mutual Theory of Mind in human-AI interaction.

Designing reliable Speech Emotion Recognition systems is a complex task that inevitably requires sufficient data for training purposes. Such extensive datasets are currently available in only a few languages, including English, German, and Italian. In this paper, we present SEMOUR, the first scripted database of emotion-tagged speech in the Urdu language, to design an Urdu Speech Recognition System. Our gender-balanced dataset contains 15,040 unique instances recorded by eight professional actors eliciting a syntactically complex script. The dataset is phonetically balanced, and reliably exhibits a varied set of emotions as marked by the high agreement scores among human raters in experiments. We also provide various baseline speech emotion prediction scores on the database, which could be used for various applications like personalized robot assistants, diagnosis of psychological disorders, and getting feedback from a low-tech-enabled population, etc. On a random test sample, our model correctly predicts an emotion with a state-of-the-art 92% accuracy.

Prototyping AI user experiences is challenging due in part to probabilistic AI models making it difficult to anticipate, test, and mitigate AI failures before deployment. In this work, we set out to support practitioners with early AI prototyping, with a focus on natural language (NL)-based technologies. Our interviews with 12 NL practitioners from a large technology company revealed that, in addition to challenges prototyping AI, prototyping was often not happening at all or focused only on idealized scenarios due to a lack of tools and tight timelines. These findings informed our design of the AI Playbook, an interactive and low-cost tool we developed to encourage proactive and systematic consideration of AI errors before deployment. Our evaluation of the AI Playbook demonstrates its potential to 1) encourage product teams to prioritize both ideal and failure scenarios, 2) standardize the articulation of AI failures from a user experience perspective, and 3) act as a boundary object between user experience designers, data scientists, and engineers.

In recent years, mobile accessibility has become an important trend with the goal of allowing all users the possibility of using any app without many limitations. User reviews include insights that are useful for app evolution. However, with the increase in the amount of received reviews, manually analyzing them is tedious and time-consuming, especially when searching for accessibility reviews. The goal of this paper is to support the automated identification of accessibility in user reviews, to help technology professionals in prioritizing their handling, and thus, creating more inclusive apps. Particularly, we design a model that takes as input accessibility user reviews, learns their keyword-based features, in order to make a binary decision, for a given review, on whether it is about accessibility or not. The model is evaluated using a total of 5,326 mobile app reviews. The findings show that (1) our model can accurately identify accessibility reviews, outperforming two baselines, namely keyword-based detector and a random classifier; (2) our model achieves an accuracy of 85% with relatively small training dataset; however, the accuracy improves as we increase the size of the training dataset.

Machine learning classifiers for human-facing tasks such as comment toxicity and misinformation often score highly on metrics such as ROC AUC but are received poorly in practice. Why this gap? Today, metrics such as ROC AUC, precision, and recall are used to measure technical performance; however, human-computer interaction observes that evaluation of human-facing systems should account for people’s reactions to the system. In this paper, we introduce a transformation that more closely aligns machine learning classification metrics with the values and methods of user-facing performance measures. The disagreement deconvolution takes in any multi-annotator (e.g., crowdsourced) dataset, disentangles stable opinions from noise by estimating intra-annotator consistency, and compares each test set prediction to the individual stable opinions from each annotator. Applying the disagreement deconvolution to existing social computing datasets, we find that current metrics dramatically overstate the performance of many human-facing machine learning tasks: for example, performance on a comment toxicity task is corrected from .95 to .73 ROC AUC.

Recent studies show the effectiveness of interview chatbots for information elicitation. However, designing an effective interview chatbot is non-trivial. Few tools exist to help designers design, evaluate, and improve an interview chatbot iteratively. Based on a formative study and literature reviews, we propose a computational framework for quantifying the performance of interview chatbots. Incorporating the framework, we have developed iChatProfile, an assistive chatbot design tool that can automatically generate a profile of an interview chatbot with quantified performance metrics and offer design suggestions for improving the chatbot based on such metrics. To validate the effectiveness of iChatProfile, we designed and conducted a between-subject study that compared the performance of 10 interview chatbots designed with or without using iChatProfile. Based on the live chats between the 10 chatbots and 1349 users, our results show that iChatProfile helped the designers build significantly more effective interview chatbots, improving both interview quality and user experience.

Recent work in fair machine learning has proposed dozens of technical definitions of algorithmic fairness and methods for enforcing these definitions. However, we still lack an understanding of how to develop machine learning systems with fairness criteria that reflect relevant stakeholders’ nuanced viewpoints in real-world contexts. To address this gap, we propose a framework for eliciting stakeholders’ subjective fairness notions. Combining a user interface that allows stakeholders to examine the data and the algorithm’s predictions with an interview protocol to probe stakeholders’ thoughts while they are interacting with the interface, we can identify stakeholders’ fairness beliefs and principles. We conduct a user study to evaluate our framework in the setting of a child maltreatment predictive system. Our evaluations show that the framework allows stakeholders to comprehensively convey their fairness viewpoints. We also discuss how our results can inform the design of predictive systems.

In single-target video object tracking, an initial bounding box is drawn around a target object and propagated through a video. When this bounding box is provided by a careful human expert, it is expected to yield strong overall tracking performance that can be mimicked at scale by novice crowd workers with the help of advanced quality control methods. However, we show through an investigation of 900 crowdsourced initializations that such quality control strategies are inadequate for this task in two major ways: first, the high level of redundancy in these methods (e.g., averaging multiple responses to reduce error) is unnecessary, as 23% of crowdsourced initializations perform just as well as the gold-standard initialization. Second, even nearly perfect initializations can lead to degraded long-term performance due to the complexity of object tracking. Considering these findings, we evaluate novel approaches for automatically selecting bounding boxes to re-query, and introduce Smart Replacement, an efficient method that decides whether to use the crowdsourced replacement initialization.

Advances in artificial intelligence (AI) have made it increasingly applicable to supplement expert’s decision-making in the form of a decision support system on various tasks. For instance, an AI-based system can provide therapists quantitative analysis on patient’s status to improve practices of rehabilitation assessment. However, there is limited knowledge on the potential of these systems. In this paper, we present the development and evaluation of an interactive AI-based system that supports collaborative decision making with therapists for rehabilitation assessment. This system automatically identifies salient features of assessment to generate patient-specific analysis for therapists, and tunes with their feedback. In two evaluations with therapists, we found that our system supports therapists significantly higher agreement on assessment (0.71 average F1-score) than a traditional system without analysis (0.66 average F1-score, p < 0.05). After tuning with therapist’s feedback, our system significantly improves its performance from 0.8377 to 0.9116 average F1-scores (p < 0.01). This work discusses the potential of a human-AI collaborative system to support more accurate decision making while learning from each other’s strengths.

Crowdsourcing can collect many diverse ideas by prompting ideators individually, but this can generate redundant ideas. Prior methods reduce redundancy by presenting peers’ ideas or peer-proposed prompts, but these require much human coordination. We introduce Directed Diversity, an automatic prompt selection approach that leverages language model embedding distances to maximize diversity. Ideators can be directed towards diverse prompts and away from prior ideas, thus improving their collective creativity. Since there are diverse metrics of diversity, we present a Diversity Prompting Evaluation Framework consolidating metrics from several research disciplines to analyze along the ideation chain — prompt selection, prompt creativity, prompt-ideation mediation, and ideation creativity. Using this framework, we evaluated Directed Diversity in a series of a simulation study and four user studies for the use case of crowdsourcing motivational messages to encourage physical activity. We show that automated diverse prompting can variously improve collective creativity across many nuanced metrics of diversity.

Qualitative research can produce a rich understanding of a phenomenon but requires an essential and strenuous data annotation process known as coding. Coding can be repetitive and time-consuming, particularly for large datasets. Existing AI-based approaches for partially automating coding, like supervised machine learning (ML) or explicit knowledge represented in code rules, require high technical literacy and lack transparency. Further, little is known about the interaction of researchers with AI-based coding assistance. We introduce Cody, an AI-based system that semi-automates coding through code rules and supervised ML. Cody supports researchers with interactively (re)defining code rules and uses ML to extend coding to unseen data. In two studies with qualitative researchers, we found that (1) code rules provide structure and transparency, (2) explanations are commonly desired but rarely used, (3) suggestions benefit coding quality rather than coding speed, increasing the intercoder reliability, calculated with Krippendorff’s Alpha, from 0.085 (MAXQDA) to 0.33 (Cody).

Developing human-like conversational agents is a prime area in HCI research and subsumes many tasks. Predicting listener backchannels is one such actively-researched task. While many studies have used different approaches for backchannel prediction, they all have depended on manual annotations for a large dataset. This is a bottleneck impacting the scalability of development. To this end, we propose using semi-supervised techniques to automate the process of identifying backchannels, thereby easing the annotation process. To analyze our identification module’s feasibility, we compared the backchannel prediction models trained on (a) manually-annotated and (b) semi-supervised labels. Quantitative analysis revealed that the proposed semi-supervised approach could attain 95% of the former’s performance. Our user-study findings revealed that almost 60% of the participants found the backchannel responses predicted by the proposed model more natural. Finally, we also analyzed the impact of personality on the type of backchannel signals and validated our findings in the user-study.

AI technologies are often used to aid people in performing discrete tasks with well-defined goals (e.g., recognising faces in images). Emerging technologies that provide continuous, real-time information enable more open-ended AI experiences. In partnership with a blind child, we explore the challenges and opportunities of designing human-AI interaction for a system intended to support social sensemaking. Adopting a research-through-design perspective, we reflect upon working with the uncertain capabilities of AI systems in the design of this experience. We contribute: (i) a concrete example of an open-ended AI system that enabled a blind child to extend his own capabilities; (ii) an illustration of the delta between imagined and actual use, highlighting how capabilities derive from the human-AI interaction and not the AI system alone; and (iii) a discussion of design choices to craft an ongoing human-AI interaction that addresses the challenge of uncertain outputs of AI systems.

The question of who gets to contribute to design futures and technology innovation has become a topic of conversation across HCI, CSCW, and other computing communities. This conversation has grave implications for communities that often find themselves an afterthought in technology design, and who coincidentally could benefit most from technological interventions in response to societal oppression. To explore this topic, we examined “futuring” through co-designed speculative design fictions as methods to envision utopian and dystopian futures. In a case study, we examined technology’s role in the imagined futures of youth participants of a Chicago summer design program. We highlight emerging themes and contribute an analysis of remote co-design through an Afrofuturism lens. Our analysis shows that concepts of utopian futures and technologies to support those futures are still heavily laden with dystopian realities of racism and poverty. We discuss ways that speculative design fictions and futuring can serve to address inclusivity in concept generation for new technologies, and we provide recommendations for conducting design techniques remotely with historically excluded populations.

Ranging from subtle to overt, unintentional to systemic, navigating racism is additional everyday work for many people. Yet the needs of people who experience racism have been overlooked as a fertile ground for better technology. Through a series of workshops we call Foundational Fiction, we engaged BIPOC (Black, Indigenous, People of Color) in participatory design to identify qualities of technology that can support people coping before, during, and after a racist interaction. Participants developed storyboards for digital tools that offer advice, predict consequences, identify racist remarks and intervene, educate both targets and perpetrators about interpersonal and systemic racism, and more. In the paper we present our workshop method utilizing interactive fiction, participants’ design concepts, prevalent themes (reducing uncertainty and offering comfort), and we provide critical analysis of the complexity of technology in these contexts. This work identifies specific opportunities for exploring anti-racist social tools.

In human-computer interaction (HCI) studies, bias in the gender representation of participants can jeopardize the generalizability of findings, perpetuate bias in data driven practices, and make new technologies dangerous for underrepresented groups. Key to progress towards inclusive and equitable gender practices is diagnosing the current status of bias and identifying where it comes from. In this mixed-methods study, we interviewed 13 HCI researchers to identify the potential bias factors, defined a systematic data collection procedure for meta-analysis of participant gender data, and created a participant gender dataset from 1,147 CHI papers. Our analysis provided empirical evidence for the underrepresentation of women, the invisibility of non-binary participants, deteriorating representation of women in MTurk studies, and characteristics of research topics prone to bias. Based on these findings, we make concrete suggestions for promoting inclusive community culture and equitable research practices in HCI.

Gender input forms act as gates to accessing information, websites, and services online. Non-binary people regularly have to interact with them, though many do not offer non-binary gender options. This results in non-binary individuals having to either choose an incorrect gender category or refrain from using a site or service—which is occasionally infeasible (e.g., when accessing health services). We tested five different forms through a survey with binary and non-binary participants (n = 350) in three contexts—a digital health form, a social media website, and a dating app. Our results indicate that the majority of participants found binary “male or female” forms exclusive and uncomfortable to fill out across all contexts. We conclude with design considerations for improving gender input forms and consequently their underlying gender model in databases. Our work aims to sensitize designers of (online) gender web forms to the needs and desires of non-binary people.

While multiple studies suggest that female-identified participants are more likely to experience cybersickness in virtual reality (VR), our systematic review of 71 eligible VR publications (59 studies and 12 surveys) pertaining to gender and cybersickness reveals a number of confounding factors in study design (e.g., a variety of technical specifications, tasks, content), a lack of demographic data, and a bias in participant recruitment. Our review shows an ongoing need within VR research to more consistently include and report on women’s experiences in VR to better understand the gendered possibility of cybersickness. Based on the gaps identified in our systematic review, we contribute study design recommendations for future work, arguing that gender considerations are necessary at every stage of VR study design, even when the study is not ‘about’ gender.

While makerspaces are often discussed in terms of a utopian vision of democratization and empowerment, many have shown how these narratives are problematic. There remains optimism for the future of makerspaces, but there is a gap in knowledge of how to articulate their promise and how to pursue it. We present a reflexive and critical reflection of our efforts as leaders of a university makerspace to articulate a vision, as well as our experience running a maker fashion show that aimed to address some specific critiques. We analyze interviews of participants from the fashion show using feminist utopianism as a lens to help us understand an alternate utopian narrative for making. Our contributions include insights about how a particular making context embodies feminist utopianism, insights about the applicability of feminist utopianism to makerspace research and visioning efforts, and a discussion about how our results can guide makerspace leaders and HCI researchers.

Affirmative consent is the idea that someone must ask for, and earn, enthusiastic approval before interacting with someone else. For decades, feminist activists and scholars have used affirmative consent to theorize and prevent sexual assault. In this paper, we ask: Can affirmative consent help to theorize online interaction? Drawing from feminist, legal, and HCI literature, we introduce the feminist theory of affirmative consent and use it to analyze social computing systems. We present affirmative consent’s five core concepts: it is voluntary, informed, revertible, specific, and unburdensome. Using these principles, this paper argues that affirmative consent is both an explanatory and generative theoretical framework. First, affirmative consent is a theoretical abstraction for explaining various problematic phenomena in social platforms—including mass online harassment, revenge porn, and problems with content feeds. Finally, we argue that affirmative consent is a generative theoretical foundation from which to imagine new design ideas for consentful socio-technical systems.

Self-care apps offer a wide variety of different therapy paradigms, pedagogies and concepts for people to maintain and make sense of their mental health. However, as human-made artefacts, these apps are being imbued with their designers’ interests, opinions, biases and assumptions about self-care. This paper is interested in making these (often) implicit notions visible. After selecting 69 apps from the Google Play Store, we use Feminist Content Analysis to investigate the store descriptions of these apps: Inductively through thematic analysis and deductively through charting concepts found within the descriptions. Our findings indicate that commercial self-care apps portray themselves as “future creating” tools for individual self-discovery, but they also create narratives that propagate an overly simplistic, individualistic and potentially harmful view of mental distress. We conclude this paper by sketching out alternative design considerations for how self-care apps can portray themselves and communicate in a more transparent, plurality-embracing fashion.

Sexual consent has undergone a transformation toward an “enthusiastic” feminist model that emphasizes consent as an ongoing and voluntary process of negotiation and affirmation. This paper considers how such a model can advance understandings of consent in HCI research and design in relation to embodied interactions with emerging technologies that also occur outside of sexual interactions. We apply the popular “FRIES” model of sexual consent (Freely given, Reversible, Informed, Enthusiastic and Specific) to three areas of embodied interaction: 1) bodily-play interactions, 2) persuasive interactions with smart technologies, and 3) intimate interactions with anthropomorphized devices. Based on erotic play practices, we contribute a “TEASE” process guideline (Traffic lights, Establish ongoing dialogue, Aftercare, Safewords, and Explicate soft/hard limits) to advance consensual practice in HCI and develop implementation scenarios.

Data-centered participatory design research projects—wherein researchers collaborate with community members for the purpose of gathering, generating, or communicating data about the community or their causes—can place epistemic burdens on minoritized or racialized groups, even in projects focused on social justice outcomes. Analysis of epistemic burden encourages researchers to rethink the purpose and value of data in community organizing and activism more generally. This paper describes three varieties of epistemic burden drawn from two case studies based on the authors’ previous work with anti-police brutality community organizations. The authors conclude with a discussion of ways to alleviate and avoid these issues through a series of questions about participatory research design. Ultimately, we call for a reorientation of knowledge production away from putative design solutions to community problems and toward a more robust interrogation of the power dynamics of research itself.

Smart technology turns the home into an active agent, which shifts the power structure within the household. This paper examines how initiators of smart technology insert their vision of the good life into households, and how these technologies exert power over the residents. Through a thematic analysis of interviews with five households, we consider Foucault’s theory on disciplinary power to examine how smart home technologies shape the experience of the home by shifting the flow of information and thereby reify power structures. Results indicate that the implementation of smart technology can affect access to shared spaces, constrain interactions, and predefine practices thereby establishing hierarchies within the household. We turn the discussion towards ethical challenges concerning control, whose problems the smart home is concerned with, and how the smart home embeds itself in the household. We conclude with design considerations and future work.

The menopause transition involves bodily-rooted, socially-shaped changes, often in a context of medicalisation that marginalises people based on their age and gender. With the goal of addressing this social justice matter with a participatory design approach, we started to cultivate partnerships with people going through menopause. This paper reports on interviews with 12 women and a design workshop with three. Our data analysis highlights their experiences from a holistic perspective that reclaims the primacy of the body and acknowledges the entanglement of the physical and the psychosocial. Participants’ design concepts show how design can come close the body to make space for menopause experiences, recognising and transforming them. We discuss how HCI can actively engage with the body to promote appreciation for it during menopause, and call for design that accompanies people in resisting the medicalisation of menopause as an enactment of social justice in everyday life.

Fertility tracking applications are technologies that collect sensitive information about their users i.e. reproductive potential. For many, these apps are an affordable solution when trying to conceive or managing their pregnancy. However, intimate data are not only collected but also shared beyond users knowledge or consent. In this paper, we explore the privacy risks that can originate from the mismanagement, misuse, and misappropriation of intimate data, which are entwined in individual life events and in public health issues such as abortion and (in)fertility. We look at differential vulnerabilities to enquire data’s vulnerability and that of ‘data subjects’. We introduce the General Data Protection Regulation (GDPR) and how it addresses fertility data. We evaluate the privacy of 30 top ‘fertility apps’ through their privacy notices and tracking practices. Lastly, we discuss the regulations and fertility data as critical to the future design of tracking technologies and privacy rights.

Today’s smart cities use thousands of physical sensors distributed across the urban landscape to support decision making in areas such as infrastructure monitoring, public health, and resource management. These weather-hardened devices require power and connectivity, and often cost thousands just to install, let alone maintain. In this paper, we show how long-range laser vibrometry can be used for low-cost, city-scale sensing. Although typically limited to just a few meters of sensing range, the use of retroreflective markers can boost this to 1km or more. Fortuitously, cities already make extensive use of retroreflective materials for street signs, construction barriers, road studs, license plates, and many other markings. We describe how our prototype system can co-opt these existing markers at very long ranges and use them as unpowered accelerometers for use in a wide variety of sensing applications.

Most augmented reality (AR) authoring tools only support the author’s current environment, but designers often need to create site-specific experiences for a different environment. We propose DistanciAR, a novel tablet-based workflow for remote AR authoring. Our baseline solution involves three steps. A remote environment is captured by a camera with LiDAR; then, the author creates an AR experience from a different location using AR interactions; finally, a remote viewer consumes the AR content on site. A formative study revealed understanding and navigating the remote space as key challenges with this solution. We improved the authoring interface by adding two novel modes: Dollhouse, which renders a bird’s-eye view, and Peek, which creates photorealistic composite images using captured images. A second study compared this improved system with the baseline, and participants reported that the new modes made it easier to understand and navigate the remote scene.

3D sketching in virtual reality (VR) enables users to create 3D virtual objects intuitively and immersively. However, previous studies showed that mid-air drawing may lead to inaccurate sketches. To address this issue, we propose to use one hand as a canvas proxy and the index finger of the other hand as a 3D pen. To this end, we first perform a formative study to compare two-handed interaction with tablet-pen interaction for VR sketching. Based on the findings of this study, we design HandPainter, a VR sketching system which focuses on the direct use of two hands for 3D sketching without requesting any tablet, pen, or VR controller. Our implementation is based on a pair of VR gloves, which provide hand tracking and gesture capture. We devise a set of intuitive gestures to control various functionalities required during 3D sketching, such as canvas panning and drawing positioning. We show the effectiveness of HandPainter by presenting a number of sketching results and discussing the outcomes of a user study-based comparison with mid-air drawing and tablet-based sketching tools.

Despite the central importance of research papers to scientific progress, they can be difficult to read. Comprehension is often stymied when the information needed to understand a passage resides somewhere else—in another section, or in another paper. In this work, we envision how interfaces can bring definitions of technical terms and symbols to readers when and where they need them most. We introduce ScholarPhi, an augmented reading interface with four novel features: (1) tooltips that surface position-sensitive definitions from elsewhere in a paper, (2) a filter over the paper that “declutters” it to reveal how the term or symbol is used across the paper, (3) automatic equation diagrams that expose multiple definitions in parallel, and (4) an automatically generated glossary of important terms and symbols. A usability study showed that the tool helps researchers of all experience levels read papers. Furthermore, researchers were eager to have ScholarPhi’s definitions available to support their everyday reading.

Human-robot interaction designers and developers navigate a complex design space, which creates a need for tools that support intuitive design processes and harness the programming capacity of state-of-the-art authoring environments. We introduce Figaro, an expressive tabletop authoring environment for mobile robots, inspired by shadow puppetry, that provides designers with a natural, situated representation of human-robot interactions while exploiting the intuitiveness of tabletop and tangible programming interfaces. On the tabletop, Figaro projects a representation of an environment. Users demonstrate sequences of behaviors, or scenes, of an interaction by manipulating instrumented figurines that represent the robot and the human. During a scene, Figaro records the movement of figurines on the tabletop and narrations uttered by users. Subsequently, Figaro employs real-time program synthesis to assemble a complete robot program from all scenes provided. Through a user study, we demonstrate the ability of Figaro to support design exploration and development for human-robot interaction.

Prototyping electronic devices that meet today’s consumer standards is a time-consuming task that requires multi-domain expertise. Consumers expect electronic devices to have visually appealing interfaces with both tactile and screen-based interfaces. Appliancizer, our interactive computational design tool, exploits the similarities between graphical and tangible interfaces, allowing web pages to be rapidly transformed into physical electronic devices. Using a novel technique we call essential interface mapping, our tool converts graphical user interface elements (e.g., an HTML button) into tangible interface components (e.g., a physical button) without changing the application source code. Appliancizer automatically generates the PCB and low-level code from web-based prototypes and HTML mock-ups. This makes the prototyping of mixed graphical-tangible interactions as easy as modifying a web page and allows designers to leverage the well-developed ecosystem of web technologies. We demonstrate how our technique simplifies and accelerates prototyping by developing two devices with Appliancizer.

User Interface (UI) mockups are commonly used as shared context during interface development collaboration. In practice, UI designers often use screenshots and sketches to create mockups of desired UI behaviors for communication. However, in the later stages of UI development, interfaces can be arbitrarily complex, making it labor-intensive to sketch, and static screenshots are limited in the types of interactive and dynamic behaviors they can express. We introduce CoCapture, a system that allows designers to easily create UI behavior mockups on existing web interfaces by demonstrating and remixing, and to accurately describe their requests to helpers by referencing the resulting mockups using hypertext. We showed that participants could more accurately describe UI behaviors with CoCapture than with existing sketch and communication tools and that the resulting descriptions were clear and easy to follow. Our approach can help teams develop UIs efficiently by bridging communication gaps with more accurate visual context.

Modern manufacturing processes are in a state of flux, as they adapt to increasing demand for flexible and self-configuring production. This poses challenges for training workers to rapidly master new machine operations and processes, i.e. machine tasks. Conventional in-person training is effective but requires time and effort of experts for each worker trained and not scalable. Recorded tutorials, such as video-based or augmented reality (AR), permit more efficient scaling. However, unlike in-person tutoring, existing recorded tutorials lack the ability to adapt to workers’ diverse experiences and learning behaviors. We present AdapTutAR, an adaptive task tutoring system that enables experts to record machine task tutorials via embodied demonstration and train learners with different AR tutoring contents adapting to each user’s characteristics. The adaptation is achieved by continually monitoring learners’ tutorial-following status and adjusting the tutoring content on-the-fly and in-situ. The results of our user study evaluation have demonstrated that our adaptive system is more effective and preferable than the non-adaptive one.

Haptic feedback significantly enhances virtual experiences. However, supporting haptics currently requires modifying the codebase, making it impractical to add haptics to popular, high-quality experiences such as best selling games, which are typically closed-source. We present HapticSeer, a multi-channel, black-box, platform-agnostic approach to detecting game events for real-time haptic feedback. The approach is based on two key insights: 1) all games have 3 types of data streams: video, audio, and controller I/O, that can be analyzed in real-time to detect game events, and 2) a small number of user interface design patterns are reused across most games, so that event detectors can be reused effectively. We developed an open-source HapticSeer framework and implemented several real-time event detectors for commercial PC and VR games. We validated system correctness and real-time performance, and discuss feedback from several haptics developers that used the HapticSeer framework to integrate research and commercial haptic devices.

Tangibles on capacitive touchscreens are a promising approach to overcome the limited expressiveness of touch input. While research has suggested many approaches to detect tangibles, the corresponding tangibles are either costly or have a considerable minimal size. This makes them bulky and unattractive for many applications. At the same time, they obscure valuable display space for interaction.

To address these shortcomings, we contribute Itsy-Bits: a fabrication pipeline for 3D printing and recognition of tangibles on capacitive touchscreens with a footprint as small as a fingertip. Each Itsy-Bit consists of an enclosing 3D object and a unique conductive 2D shape on its bottom. Using only raw data of commodity capacitive touchscreens, Itsy-Bits reliably identifies and locates a variety of shapes in different sizes and estimates their orientation. Through example applications and a technical evaluation, we demonstrate the feasibility and applicability of Itsy-Bits for tangibles with small footprints.

3D printing has revolutionized rapid prototyping by speeding up the creation of custom-shaped objects. With the rise of multi-material 3D printers, these custom-shaped objects can now be made interactive in a single pass through passive conductive structures. However, connecting conventional electronics to these conductive structures often still requires time-consuming manual assembly involving many wires, soldering or gluing.

To alleviate these shortcomings, we propose : a fabrication pipeline and interfacing concept to magnetically connect a 3D-printed object equipped with passive sensing structures to conventional sensing electronics. To this end, utilizes ferromagnetic and conductive 3D-printed structures, printable in a single pass on standard printers. We further present a proof-of-concept capacitive sensing board that enables easy and robust magnetic assembly to quickly create interactive 3D-printed objects. We evaluate by assessing the robustness and quality of the connection and demonstrate its broad applicability by a series of example applications.

Tangibles can enrich interaction with digital surfaces. Among others, they support eyes-free control or increase awareness of other users’ actions. Tangibles have been studied in combination with horizontal surfaces such as tabletops, but not with vertical screens such as wall displays. The obvious obstacle is gravity: tangibles cannot be placed on such surfaces without falling. We present WallTokens, easy-to-fabricate tangibles to interact with a vertical surface. A WallToken is a passive token whose footprint is recognized on a tactile surface. It is equipped with a push-handle that controls a suction cup. This makes it easy for users to switch between sliding the token or attaching it to the wall. We describe how to build such tokens and how to recognize them on a tactile surface. We report on a study showing the benefits of WallTokens for manipulating virtual objects over multi-touch gestures. This project is a step towards enabling tangible interaction in a wall display context.

Acoustic levitation is gaining popularity as an approach to create physicalized mid-air content by levitating different types of levitation primitives. Such primitives can be independent particles or particles that are physically connected via threads or pieces of cloth to form shapes in mid-air. However, initialization (i.e., placement of such primitives in their mid-air target locations) currently relies on either manual placement or specialized ad-hoc implementations, which limits their practical usage. We present ArticuLev, an integrated pipeline that deals with the identification, assembly and mid-air placement of levitated shape primitives. We designed ArticuLev with the physical properties of commonly used levitation primitives in mind. It enables experiences that seamlessly combine different primitives into meaningful structures (including fully articulated animated shapes) and supports various levitation display approaches (e.g., particles moving at high speed). In this paper, we describe our pipeline and demonstrate it with heterogeneous combinations of levitation primitives.

Searching for relative mobile user interface (UI) design examples can aid interface designers in gaining inspiration and comparing design alternatives. However, finding such design examples is challenging, especially as current search systems rely on only text-based queries and do not consider the UI structure and content into account. This paper introduces VINS, a visual search framework, that takes as input a UI image (wireframe, high-fidelity) and retrieves visually similar design examples. We first survey interface designers to better understand their example finding process. We then develop a large-scale UI dataset that provides an accurate specification of the interface’s view hierarchy (i.e., all the UI components and their specific location). By utilizing this dataset, we propose an object-detection based image retrieval framework that models the UI context and hierarchical structure. The framework achieves a mean Average Precision of 76.39% for the UI detection and high performance in querying similar UI designs.

Documentation for DIY tasks serve as codified project knowledge and help makers reach new understandings and appreciations for the artifact. Engaging in reflective processes using the documentation can be challenging when it comes to physical objects as the documentation and the artifact exist separately. We hypothesize that spatially associating the documentation information with the artifact can provide richer contextualization to reflect upon the artifact and design process. We implemented and evaluated Documented, a web application that helps makers associate documentation to 3D printed objects. Information can be embedded using printed tags spatially placed on the model and accessed using mobile AR. Our study highlights the different strategies participants had for organizing, embedding, and retrieving information. Informed by our results, we discuss how the coupling of the documentation and the artifact can support reflection and identify potential barriers that need further investigation.

Flower jellies, a delicate dessert in which a flower-shaped jelly floats inside another clear jelly, fascinate people with both their beauty and elaborate construction. In efforts to simplify the challenging fabrication and enrich the design space of this dessert, we present Flower Jelly Printer: a printing device and design software for digitally fabricating flower jellies. Our design software lets users play with parameters and preview the resulting forms until achieving their desired shapes. We also developed slit injection printing that directly injects colored jelly into a base jelly, and shared several design examples to show the breadth of design possibilities. Finally, the user study with novice and experienced users demonstrates that our system benefits creators of all experience levels by iterative design and precise fabrication. We hope to enable more people to design and create their own flower jellies while expanding access and the design space for digitally fabricated foods.

Unprecedented maker efforts arose in response to COVID-19 medical supply gaps worldwide. Makers in the U.S., participated in peer-production activities to manufacture personal protective equipment (PPE). Whereas, medical makers, who innovate exclusively for points of care, pivoted towards safer, reliable PPE. What were their efforts to pivot medical maker infrastructure towards reliable production of safe equipment at higher volumes? We interviewed 13 medical makers as links between institutions, maker communities, and wider regional industry networks. These medical makers organized stopgap manufacturing in institutional spaces to resolve acute shortages (March–May) and chronic shortages (May–July). They act as intermediaries in efforts to prototype and produce devices under regulatory, material, and human constraints of a pandemic. We re-frame their making efforts as repair work to offer an alternate critical view of optimism around making for crisis. We contribute an understanding of these efforts to inform infrastructure design for making with purpose and safety leading to opportunities for community production of safe devices at scale.

Visual metaphors, which are widely used in graphic design, can deliver messages in creative ways by fusing different objects. The keys to creating visual metaphors are diverse exploration and creative combinations, which is challenging with conventional methods like image searching. To streamline this ideation process, we propose to use a mind-map-like structure to recommend and assist users to explore materials. We present MetaMap, a supporting tool which inspires visual metaphor ideation through multi-dimensional example-based exploration. To facilitate the divergence and convergence of the ideation process, MetaMap provides 1) sample images based on keyword association and color filtering; 2) example-based exploration in semantics, color, and shape dimensions; and 3) thinking path tracking and idea recording. We conduct a within-subject study with 24 design enthusiasts by taking a Pinterest-like interface as the baseline. Our evaluation results suggest that MetaMap provides an engaging ideation process and helps participants create diverse and creative ideas.

Mathematical models and expressions traditionally evolved as symbolic representations, with cognitively arbitrary rules of symbol manipulation. The embodied mathematics philosophy posits that abstract math concepts are layers of metaphors grounded in our intuitive arithmetic capabilities, such as categorizing objects and part-whole analysis. We introduce a design framework that facilitates the construction and exploration of embodied representations for algebraic expressions, using interactions inspired by innate arithmetic capabilities. We instantiated our design in a sketch interface that enables construction of visually interpretable compositions that are directly mappable to algebraic expressions and explorable through a ladder of abstraction [47]. The emphasis is on bottom-up construction, with the user sketching pictures while the system generates corresponding algebra. We present diverse examples created by our prototype system. A coverage of the US Common Core curriculum and playtesting studies with children point to the future direction and potential for a sketch-based design paradigm for mathematics.

The access and growing ubiquity of digital fabrication has ushered in a celebration of creativity and “making.” However, the focus is often on the resulting static artifact or the creative process and tools to design it. We envision a post-making process that extends past these final static objects — not just in their making but in their “unmaking.” By drawing from artistic movements such as Auto-Destructive Art, intentionally inverting well-established engineering principles of structurally sound designs, and safely misusing unstable materials, we demonstrate an important extension to making — unmaking. In this paper, we provide designers with a new vocabulary of unmaking operations within standard 3D modeling tools. We demonstrate how such designs can be realized using a novel multi-material 3D printing process. Finally, we detail how unmaking allows designs to change over time, is an ally to sustainability and re-usability, and captures themes of “aura,” emotionality, and personalization.

Though photographs of real people are typically used to portray personas, there is little research into the potential advantages or disadvantages of using such images, relative to other image styles. We conducted an experiment with 149 participants, testing the effects of six different image styles on user perceptions and personality traits that are attributed to personas by the participants. Results show that perceptions of clarity, completeness, consistency, credibility, and empathy for a persona increase with picture realism. Personas with more realistic pictures are also perceived as more agreeable, open, and emotionally stable, with higher confidence in these assessments. We also find evidence of the uncanny valley effect, with realistic cartoon personas experiencing a decrease in the user perception scores.

Digital tools that support creative activities are ubiquitous in the design industry, yet practitioners appear to prefer pen and paper for design ideation. To better understand this exception, we conducted a comparative study between analog and digital tools and their impact on the divergent and convergent thinking patterns of groups of designers. We analysed how 24 participants solved comparable design ideation tasks in two conditions using linkographic protocol analysis – a notation method that focuses on identifying and linking small steps in the design process called moves. Our findings suggest that digital ideation tools yield more convergent thinking compared to analog tools, with no discernible impact on general productivity or divergent thinking.

3D printing, as a rapid prototyping technique, usually fabricates objects that are difficult to modify physically. This paper presents FlexTruss, a design and construction pipeline based on the assembly of modularized truss-shaped objects fabricated with conventional 3D printers and assembled by threading. To create an end-to-end system, a parametric design tool with an optimal Euler path calculation method is developed, which can support both inverse and forward design workflow and multi-material construction of modular parts. In addition, the assembly of truss modules by threading is evaluated with a series of application cases to demonstrate the affordance of FlexTruss. We believe that FlexTruss extends the design space of 3D printing beyond typically hard and fixed forms, and it will provide new capabilities for designers and researchers to explore the use of such flexible truss structures in human-object interaction.

3D printing technology makes Do-It-Yourself and reforming everyday objects a reality. However, designing and fabricating attachments that can seamlessly adapt existing objects to extended functionality is a laborious process, which requires accurate measuring, modeling, manufacturing, and assembly. This paper presents ShrinCage, a 4D printing system that allows novices to easily create shrinkable adaptations to fit and fasten existing objects. Specifically, the design tool presented in this work aid in the design of attachment that adapts to irregular morphologies, which accommodates the variations in measurements and fabrication, subsequently simplifying the modeling and assembly processes. We further conduct mechanical tests and user studies to evaluate the availability and feasibility of this method. Numerous application examples created by ShrinCage prove that it can be adopted by aesthetic modification, assistive technology, repair, upcycling, and augmented 3D printing.

Digital fabrication and craftsmanship is entering into a new phase with increasing levels of complexity and a renewed desire for composites and cross-material experimentation. However, allowing work to travel from machine to machine, remains a challenge in terms of workflow, communication, orientation and material. Based on an exploration to combine embroidery and 3D printing in the pursuit of inflatable solutions, we propose the metaphor of the drawing game Exquisite Corpse to outline the three emerging concerns: turn taking, orientation and trade-offs. We propose a set of guidelines that suggest ways in which, we may allow different digital fabrication machines to be used in sequence, as a method for adding complexity to the things we make and the ways our machines may talk to one another.

Interest in combining interactive play and the human body, using “bodily play” systems, is increasing. While these systems primarily prioritize a player's control over their bodily actions, we see intriguing possibilities in the pursuit of “limited control over the body” as an intriguing design resource for bodily play systems. In this paper, we use three of our bodily play systems to illustrate how designers can engage with limited control over the body by varying the player's degree of indirect control (for instance, via other bodily activity and external triggers). We also propose four strategies for employing limited control over the body: Exploration, Reflection, Learning and Embracement. We hope our own work and the strategies developed from it will assist designers to employ limited control over the body, ultimately helping people benefit from engaging their bodies through play.

Prior research has highlighted opportunities for technology to better support the tabletop game experience in offline and online settings, but little work has focused on the social aspect of tabletop gaming. We investigated the social and collaborative aspects of tabletop gaming in the unique context of “social distancing” during the 2020 COVID-19 pandemic to shed light on the experience of remote tabletop gaming. With a multi-method qualitative approach (including digital ethnography and in-depth interviews), we empirically studied how people appropriate existing technologies and adapt their offline practices to play tabletop games remotely. We identify three themes that describe people's game and social experience during remote play: creating a shared tabletop environment (shared space), enabling a collective understanding (shared information and awareness), and facilitating a communal temporal experience (shared time). We reflect on challenges and design opportunities for a better experience in the age of remote collaboration.

Online platforms rely upon users or automated tools to flag toxic behaviors, the very first step in online moderation. While much recent research has examined online moderation, the role of flag remains poorly understood. This question becomes even more urgent in automated moderation, where flagging becomes a primary source of human judgment. We conducted a qualitative study of flagging practices in League of Legends (LoL), a popular eSports game. We found stark differences between how flag is designed to identify toxicity, and flaggability, or how players use and appropriate flag. Players distrust flag, but also appropriate flag for instrumental purposes. Thus, flaggability diverges decidedly from the conception of toxicity, and must be understood within the highly competitive gaming context of LoL. These findings help shed light on the situated nature of flaggability, the role of flag in online moderation, as well as implications for designing flag and moderation.

Video game toxicity, endemic to online play, represents a pervasive and complex problem. Antisocial behaviours in online play directly harm player wellbeing, enjoyment, and retention—but research has also revealed that some players normalize toxicity as an inextricable and acceptable element of the competitive video game experience. In this work, we explore perceptions of toxicity and how they are predicted by player traits, demonstrating that participants reporting a higher tendency towards Conduct Reconstrual, Distorting Consequences, Dehumanization, and Toxic Online Disinhibition perceive online game interactions as less toxic. Through a thematic analysis on willingness to report, we also demonstrate that players abstain from reporting toxic content because they view it as acceptable, typical of games, as banter, or as not their concern. We propose that these traits and themes represent contributing factors to the cyclical normalization of toxicity. These findings further highlight the multifaceted nature of toxicity in online video games.

Uncertainty is widely acknowledged as an engaging gameplay element but rarely used in exergames. In this research, we explore the role of uncertainty in exergames and introduce three uncertain elements (false-attacks, misses, and critical hits) to an exergame. We conducted a study under two conditions (uncertain and certain), with two display types (virtual reality and large display) and across young and middle-aged adults to measure their effect on game performance, experience, and exertion. Results show that (1) our designed uncertain elements are instrumental in increasing exertion levels; (2) when playing a motion-based first-person perspective exergame, virtual reality can improve performance, while maintaining the same motion sickness level as a large display; and (3) exergames for middle-aged adults should be designed with age-related declines in mind, similar to designing for elderly adults. We also framed two design guidelines for exergames that have similar features to the game used in this research.

Warm-up games are widespread practices in multiple activities across domains, yet little scholarly work can be found about their role in physical training. Here, we study potential goals and benefits of warm-up games, and explore opportunities for technology inclusion through investigating a collection of warm-up games gathered: online, from a survey of online warm-up games curated, described, and used by Physical Education teachers; and in person, from an ongoing design research work as part of a technology-supported circus training course. Further, in the context of the latter, we conducted explorative design interventions, augmenting a range of the warm-up games with wearable technology. Our work surfaces major goals and benefits of warm-up games, which can be broadly classified as preparing participants physically, socially, and mentally. We also show how the inclusion of open-ended technology can support these goals and discuss broader opportunities for technology inclusion in warm-up games.

This paper presents restraints - directly imposed restrictions on players' bodily movements, as a mechanic for bodily play in HCI. While this is a familiar mechanic in non-digital movement-based games, its potential in designing bodily play experiences in HCI has been scarcely explored. Three types of restraints observed in non-digital movement-based games, are explored here: fixating body parts, excluding body parts and depriving/manipulating bodily senses. Then, we investigate the experiential dynamics of restraints as a bodily play mechanic bridging a phenomenological perspective on bodily movement with theories on play. These investigations form the theoretical framework for the subsequent analysis of five digital body game examples. Building on this analysis and theoretical framework, we formulate five design strategies for implementing restraints as a mechanic for bodily play in HCI. We propose restraints as a generative resource for researchers and designers interested in understanding and designing bodily play experiences in HCI.

The design space of social exergames remains narrow despite the many benefits of playing and exercising together. Towards opening this design space, we followed a Research through Design (RtD) approach focused on exergames that can be fun and immersive social training experiences. Through embodied sketching activities with designers and 10 pairs of players, we explored future games for the ExerCube, an immersive exergame platform. Our work contributes with forms of intermediate-level knowledge: a design space model (the Immersive Social Fitness—ImSoFit—Games model); and a novel design vocabulary including new bodily orientations in co-located physical interaction. We illustrate their use and value scrutinizing three of our games and applying three analytical lenses to 1) understand how design choices impact how players move together; 2) evaluate design expectations and analyze players’ behavior in relation to design choices; and 3) potentially extend the design space of immersive co-located social fitness games.

Virtual reality (VR) multiplayer games increasingly use asymmetry (e.g., differences in a person’s capability or the user interface) and resulting interdependence between players to create engagement even when one player has no access to a head-mounted display (HMD). Previous work shows this enhances player experience (PX). Until now, it remains unclear whether and how an asymmetric game design with interdependences creates comparably enjoyable PX for both an HMD and a non-HMD player. In this work, we designed and implemented an asymmetric VR game (different in its user interface) with two types of interdependence: strategic (difference in game information/player capability) and biometric (difference in player’s biometric influence). Our mixed-methods user study (N=30) shows that asymmetries positively impact PX for both player roles, that interdependence strongly affects players’ perception of agency, and that biometric feedback—while subjective—is a valuable game mechanic.

Recent studies on gaming wearables show that wearables can contribute to the gaming experience by bolstering performativity, facilitating social interaction, and accommodating distinct interaction modalities. Still, these studies focused on contexts such as role-playing, casual, or festival games. Stakeholder-oriented research that explores the integration of wearables for mainstream gaming platforms such as game consoles is scarce. To fill this gap, we have conducted an exploratory study through 6 participatory design workshops focusing on different aspects of wearables with 33 participants from different stakeholders. As a result, we have created fifteen design themes and three gaming wearable concepts that led to seven actionable design implications which can be adopted by designers and researchers for designing gaming wearables.

The spectatorship experience for virtual reality (VR) games differs strongly from its non-VR precursor. When watching non-VR games on platforms such as Twitch, spectators just see what the player sees, as the physical interaction is mostly unimportant for the overall impression. In VR, the immersive full-body interaction is a crucial part of the player experience. Hence, content creators, such as streamers, often rely on green screens or similar solutions to offer a mixed-reality third-person view to disclose their full-body actions. Our work compares the most popular realizations of the first-person and the third-person perspective in an online survey (N = 217) with three different VR games. Contrary to the current trend to stream in third-person, our key result is that most viewers prefer the first-person version, which they attribute mostly to the better focus on in-game actions and higher involvement. Based on the study insights, we provide design recommendations for both perspectives.

While virtual character embodiment has been studied as a mitigator of singular societal biases in fully immersive VR and empathy games, there have been no major studies on representation featuring standard game play or intersectional identities. In our study, participants played a short 2D video game with racial and gender character manipulations. They then rated a LinkedIn profile application to examine interactions of racial and gender biases. White male participants showed bias against black and female applicants, with the black female applicant experiencing both racial and gender bias. However, participants who embodied certain underrepresented characters in the game displayed reduced biases. Participants’ perceived identification with the characters moderated this effect. The study highlights a lack of homogeneity in the prevalence and potential reduction of different societal biases and incorporates intersectionality to illustrate how multiple parts of a player character’s identity can be used to combat biases.

Many journalists and newsrooms now incorporate audience contributions in their sourcing practices by leveraging user-generated content (UGC). However, their sourcing needs and practices as they seek information from UGCs are still not deeply understood by researchers or well-supported in tools. This paper first reports the results of a qualitative interview study with nine professional journalists about their UGC sourcing practices, detailing what journalists typically look for in UGCs and elaborating on two UGC sourcing approaches: deep reporting and wide reporting. These findings then inform a human-centered design approach to prototype a UGC sourcing tool for journalists, which enables journalists to interactively filter and rank UGCs based on users’ example content. We evaluate the prototype with nine professional journalists who source UGCs in their daily routines to understand how UGC sourcing practices are enabled and transformed, while also uncovering opportunities for future research and design to support journalistic sourcing practices and sensemaking processes.

Virtual meetings are critical for remote work because of the need for synchronous collaboration in the absence of in-person interactions. In-meeting multitasking is closely linked to people’s productivity and wellbeing. However, we currently have limited understanding of multitasking in remote meetings and its potential impact. In this paper, we present what we believe is the most comprehensive study of remote meeting multitasking behavior through an analysis of a large-scale telemetry dataset collected from February to May 2020 of U.S. Microsoft employees and a 715-person diary study. Our results demonstrate that intrinsic meeting characteristics such as size, length, time, and type, significantly correlate with the extent to which people multitask, and multitasking can lead to both positive and negative outcomes. Our findings suggest important best-practice guidelines for remote meetings (e.g., avoid important meetings in the morning) and design implications for productivity tools (e.g., support positive remote multitasking).

Targeting the right group of workers for crowdsourcing often achieves better quality results. One unique example of targeted crowdsourcing is seeking community-situated workers whose familiarity with the background and the norms of a particular group can help produce better outcome or accuracy. These community-situated crowd workers can be recruited in different ways from generic online crowdsourcing platforms or from online recovery communities. We evaluate three different approaches to recruit generic and community-situated crowd in terms of the time and the cost of recruitment, and the accuracy of task completion. We consider the context of Alcoholics Anonymous (AA), the largest peer support group for recovering alcoholics, and the task of identifying and validating AA meeting information. We discuss the benefits and trade-offs of recruiting paid vs. unpaid community-situated workers and provide implications for future research in the recovery context and relevant domains of HCI, and for the design of crowdsourcing ICT systems.

Drones are increasingly used to support humanitarian crises and events that involve dangerous or costly tasks. While drones have great potential for remote collaborative work and aerial telepresence, existing drone technology is limited in its support for synchronous collaboration among multiple remote users. Through three design iterations and evaluations, we prototyped Squadrone, a novel aerial telepresence platform that supports synchronous mid-air collaboration among multiple remote users. We present our design and report results from evaluating our iterations with 13 participants in 3 different collaboration configurations. Our first design iteration validates the basic functionality of the platform. Then, we establish the effectiveness of collaboration using a 360-degree shared aerial display. Finally, we simulate a type of search task in an open environment to see if collaborative telepresence impacts members’ participation. The results validate some initial goals for Squadrone and are used to reflect back on a recent telepresence design framework.

In hybrid meetings, multiple co-located participants communicate with remote participants through video. But video communication inhibits non-verbal cues, and this often causes remote participants to feel excluded. To address this issue, we built MirrorBlender: a What-You-See-Is-What-I-See video-conferencing system for blending, repositioning, and resizing mirrors. Mirrors here denote shared video feeds of people and screens. In a qualitative study of MirrorBlender with three hybrid meeting sessions, we found that the shared control of mirrors supported users in negotiating a blended interpersonal space. Moreover, it enabled diverse acts of inclusion of remote participants. In particular, remote participants brought attention to themselves by manipulating the position, scale, and translucency of their camera and screen feeds. Participants also embodied and leveraged their mirror images for deictic gestures and playful interactions. Based on these findings, we discuss new opportunities for supporting video-mediated collaboration.

Social media has become an effective recruitment tool for higher-waged and white-collar professionals. Yet, past studies have questioned its effectiveness for the recruitment of lower-waged workers. It is also unclear whether or how employers leverage social media in their recruitment of low-wage job seekers, and how social media could better support the needs of both stakeholders. Therefore, we conducted 15 semi-structured interviews with employers of low-wage workers in the U.S. We found that employers: use social media, primarily Facebook, to access large pools of active low-wage job seekers; and recognize indirect signals about low-wage job seekers’ commitment and job readiness. Our work suggests that there remains a visible, yet unaddressed power imbalance between low-wage workers and employers in the use of social media, which risks further destabilizing the precarious labor market.

Team members commonly collaborate on visual documents remotely and asynchronously. Particularly, students are frequently restricted to this setting as they often do not share work schedules or physical workspaces. As communication in this setting has delays and limits the main modality to text, members exert more effort to reference document objects and understand others’ intentions. We propose Winder, a Figma plugin that addresses these challenges through linked tapes—multimodal comments of clicks and voice. Bidirectional links between the clicked-on objects and voice recordings facilitate understanding tapes: selecting objects retrieves relevant recordings, and playing recordings highlights related objects. By periodically prompting users to produce tapes, Winder preemptively obtains information to satisfy potential communication needs. Through a five-day study with eight teams of three, we evaluated the system’s impact on teams asynchronously designing graphical user interfaces. Our findings revealed that producing linked tapes could be as lightweight as face-to-face (F2F) interactions while transmitting intentions more precisely than text. Furthermore, with preempted tapes, teammates coordinated tasks and invited members to build on each others’ work.

Augmented/Virtual Reality (AR/VR) is still a fragmented space to design for due to the rapidly evolving hardware, the interdisciplinarity of teams, and a lack of standards and best practices. We interviewed 26 professional AR/VR designers and developers to shed light on their tasks, approaches, tools, and challenges. Based on their work and the artifacts they generated, we found that AR/VR application creators fulfill four roles: concept developers, interaction designers, content authors, and technical developers. One person often incorporates multiple roles and faces a variety of challenges during the design process from the initial contextual analysis to the deployment. From analysis of their tool sets, methods, and artifacts, we describe critical key challenges. Finally, we discuss the importance of prototyping for the communication in AR/VR development teams and highlight design implications for future tools to create a more usable AR/VR tool chain.

Freelancing platforms, such as Upwork, represent an expansion of the gig economy to encompass knowledge-based work. Prior research in HCI has primarily focused on forms of gig work such as ride-sharing and microwork but has not addressed how freelancing platforms are disrupting high-skilled knowledge work. To understand freelancers’ perspectives on how these platforms are disrupting their work we have collected and thematically analysed 528 posts with 7499 comments from four relevant subforums on Reddit. The qualitative findings reveal tensions between wanting autonomy and control and the necessity of opportunities and convenience. Freelancing platforms are perceived as systems that present advantages to find clients, gain experience and mitigate precarity. However, these platforms constrain the control over their work that freelancers value. The paper contributes an improved understanding of freelance work, the role and potential for freelancing platforms in the knowledge-based gig economy, and directions for worker-centred design.

Manual schedule creation often involves satisfying numerous unique and conflicting constraints, which becomes more cognitively demanding when creating a common academic schedule with other individuals. Poor decision making caused by cognitive overload can result in unsuitable schedules. This study proposes the use of Boolean satisfiability (SAT) solvers in an academic scheduling system to help students balance scheduling preferences and satisfy necessary constraints. Based on the availability of courses and the scheduling preferences of users, the system automatically resolves conflicts and presents possible schedules. In a controlled experiment with 42 undergraduate students, cognitive demand was reduced by eliminating menial decisions, which significantly optimized the creation of a common schedule among peers. We found that human errors and emotional stress were diminished, and schedules created using the system were more satisfactory to participants. Finally, we present recommendations and design implications for future academic scheduling systems.

With growing interest regarding cooperation support using Augmented Reality (AR), social presence has become a popular measure of its quality. While this concept is established throughout cooperation research, its role in AR is still unclear: Some work uses social presence as an indicator for support quality, while others found no impact at all. To clarify this role, we conducted a literature review of recent publications that empirically investigated social presence in cooperative AR. After a thorough selection procedure, we analyzed 19 publications according to factors influencing social presence and the impact of social presence on cooperation support. We found that certain interventions support social presence better than others, that social presence has an influence on user's preferences and that the relation between social presence and cooperation quality may depend on the symmetry of the cooperation task. This contributes to existing research by clarifying the role of social presence for cooperative AR and deriving corresponding design recommendations.

Performers acutely need but lack tools to remotely rehearse and create live theatre, particularly due to global restrictions on social interactions during the Covid-19 pandemic. No studies, however, have heretofore examined how remote video-collaboration affects performance. This paper presents the findings of a field study with 16 domain experts over six weeks investigating how tele-immersion affects the rehearsal and performance of improvisational theatre. To conduct the study, an original media server was developed for co-locating remote performers into shared virtual 3D environments which were accessed through popular video conferencing software. The results of this qualitative study indicate that tele-immersive environments uniquely provide performers with a strong sense of co- presence, feelings of physical connection, and an increased ability to enter the social-flow states required for improvisational theatre. Based on our observations, we put forward design recommendations for video collaboration tools tailored to the unique demands of live performance.

Immersive Analytics is a quickly evolving field that unites several areas such as visualisation, immersive environments, and human-computer interaction to support human data analysis with emerging technologies. This research has thrived over the past years with multiple workshops, seminars, and a growing body of publications, spanning several conferences. Given the rapid advancement of interaction technologies and novel application domains, this paper aims toward a broader research agenda to enable widespread adoption. We present 17 key research challenges developed over multiple sessions by a diverse group of 24 international experts, initiated from a virtual scientific workshop at ACM CHI 2020. These challenges aim to coordinate future work by providing a systematic roadmap of current directions and impending hurdles to facilitate productive and effective applications for Immersive Analytics.

Geographic data visualisation on virtual globes is intuitive and widespread, but has not been thoroughly investigated. We explore two main design factors for quantitative data visualisation on virtual globes: i) commonly used primitives (2D bar, 3D bar, circle) and ii) the orientation of these primitives (tangential, normal, billboarded). We evaluate five distinctive visualisation idioms in a user study with 50 participants. The results show that aligning primitives tangentially on the globe’s surface decreases the accuracy of area-proportional circle visualisations, while the orientation does not have a significant effect on the accuracy of length-proportional bar visualisations. We also find that tangential primitives induce higher perceived mental load than other orientations. Guided by these results we design a novel globe visualisation idiom, Geoburst, that combines a virtual globe and a radial bar chart. A preliminary evaluation reports potential benefits and drawbacks of the Geoburst visualisation.

We present an observational study to compare co-located and situated real-time visualizations in basketball free-throw training. Our goal is to understand the advantages and concerns of applying immersive visualization to real-world skill-based sports training and to provide insights for designing AR sports training systems. We design both a situated 3D visualization on a head-mounted display and a 2D visualization on a co-located display to provide immediate visual feedback on a player’s shot performance. Using a within-subject study design with experienced basketball shooters, we characterize user goals, report on qualitative training experiences, and compare the quantitative training results. Our results show that real-time visual feedback helps athletes refine subsequent shots. Shooters in our study achieve greater angle consistency with our visual feedback. Furthermore, AR visualization promotes an increased focus on body form in athletes. Finally, we present suggestions for the design of future sports AR studies.

Most mobile health apps employ data visualization to help people view their health and activity data, but these apps provide limited support for visual data exploration. Furthermore, despite its huge potential benefits, mobile visualization research in the personal data context is sparse. This work aims to empower people to easily navigate and compare their personal health data on smartphones by enabling flexible time manipulation with speech. We designed and developed Data@Hand, a mobile app that leverages the synergy of two complementary modalities: speech and touch. Through an exploratory study with 13 long-term Fitbit users, we examined how multimodal interaction helps participants explore their own health data. Participants successfully adopted multimodal interaction (i.e., speech and touch) for convenient and fluid data exploration. Based on the quantitative and qualitative findings, we discuss design implications and opportunities with multimodal interaction for better supporting visual data exploration on mobile devices.

Data physicalisations afford people the ability to directly interact with data using their hands, potentially achieving a more comprehensive understanding of a dataset. Due to their complex nature, the representation of graphs and networks could benefit from physicalisation, bringing the dataset from the digital world into the physical one. However, no empirical work exists investigating the effects physicalisations have upon comprehension as they relate to graph representations. In this work, we present initial design considerations for graph physicalisations, as well as an empirical study investigating differences in comprehension between virtual and physical representations. We found that participants perceived themselves as being more accurate via touch and sight (visual-haptic) than the graphical-only modality, and perceived a triangle count task as less difficult in visual-haptic than in the graphical-only modality. Additionally, we found that participants significantly preferred interacting with visual-haptic over other conditions, despite no significant effect on task time or error.

Natural language interfaces (NLIs) for data visualization are becoming increasingly popular both in academic research and in commercial software. Yet, there is a lack of empirical understanding of how people specify visualizations through natural language. We conducted an online study (N = 102), showing participants a series of visualizations and asking them to provide utterances they would pose to generate the displayed charts. From the responses, we curated a dataset of 893 utterances and characterized the utterances according to (1) their phrasing (e.g., commands, queries, questions) and (2) the information they contained (e.g., chart types, data aggregations). To help guide future research and development, we contribute this utterance dataset and discuss its applications toward the creation and benchmarking of NLIs for visualization.

We explore network visualisation on a two-dimensional torus topology that continuously wraps when the viewport is panned. That is, links may be “wrapped” across the boundary, allowing additional spreading of node positions to reduce visual clutter. Recent work has investigated such pannable wrapped visualisations, finding them not worse than unwrapped drawings for small networks for path-following tasks. However, they did not evaluate larger networks nor did they consider whether torus-based layout might also better display high-level network structure like clusters. We offer two algorithms for improving toroidal layout that is completely autonomous and automatic panning of the viewport to minimiswe wrapping links. The resulting layouts afford fewer crossings, less stress, and greater cluster separation. In a study of 32 participants comparing performance in cluster understanding tasks, we find that toroidal visualisation offers significant benefits over standard unwrapped visualisation in terms of improvement in error by 62.7% and time by 32.3%.

Sketchnoting is a form of visual note taking where people listen to, synthesize, and visualize ideas from a talk or other event using a combination of pictures, diagrams, and text. Little is known about the design space of this kind of visual note taking. With an eye towards informing the implementation of digital equivalents of sketchnoting, inking, and note taking, we introduce a classification of sketchnote styles and techniques, with a qualitative analysis of 103 sketchnotes, and situated in context with six semi-structured follow up interviews. Our findings distill core sketchnote components (content, layout, structuring elements, and visual styling) and dimensions of the sketchnote design space, classifying levels of conciseness, illustration, structure, personification, cohesion, and craftsmanship. We unpack strategies to address particular note taking challenges, for example dealing with constraints of live drawings, and discuss relevance for future digital inking tools, such as recomposition, styling, and design suggestions.

Plots and tables are commonplace in today’s data-driven world, and much research has been done on how to make these figures easy to read and understand. Often times, however, the information they contain conveys only the end result of a complex and subtle data analysis pipeline. This can leave the reader struggling to understand what steps were taken to arrive at a figure, and what implications this has for the underlying results. In this paper, we introduce datamations, which are animations designed to explain the steps that led to a given plot or table. We present the motivation and concept behind datamations, discuss how to programmatically generate them, and provide the results of two large-scale randomized experiments investigating how datamations affect people’s abilities to understand potentially puzzling results compared to seeing only final plots and tables containing those results.

We present Marvis, a conceptual framework that combines mobile devices and head-mounted Augmented Reality (AR) for visual data analysis. We propose novel concepts and techniques addressing visualization-specific challenges. By showing additional 2D and 3D information around and above displays, we extend their limited screen space. AR views between displays as well as linking and brushing are also supported, making relationships between separated visualizations plausible. We introduce the design process and rationale for our techniques. To validate Marvis’ concepts and show their versatility and widespread applicability, we describe six implemented example use cases. Finally, we discuss insights from expert hands-on reviews. As a result, we contribute to a better understanding of how the combination of one or more mobile devices with AR can benefit visual data analysis. By exploring this new type of visualization environment, we hope to provide a foundation and inspiration for future mobile data visualizations.

Recent research in the area of immersive analytics demonstrated the utility of head-mounted augmented reality devices for visual data analysis. However, it can be challenging to use the by default supported mid-air gestures to interact with visualizations in augmented reality (e.g. due to limited precision). Touch-based interaction (e.g. via mobile devices) can compensate for these drawbacks, but is limited to two-dimensional input. In this work we present STREAM: Spatially-aware Tablets combined with Augmented Reality Head-Mounted Displays for the multimodal interaction with 3D visualizations. We developed a novel eyes-free interaction concept for the seamless transition between the tablet and the augmented reality environment. A user study reveals that participants appreciated the novel interaction concept, indicating the potential for spatially-aware tablets in augmented reality. Based on our findings, we provide design insights to foster the application of spatially-aware touch devices in augmented reality and research implications indicating areas that need further investigation.

In this paper, we present MIRIA, a Mixed Reality Interaction Analysis toolkit designed to support the in-situ visual analysis of user interaction in mixed reality and multi-display environments. So far, there are few options to effectively explore and analyze interaction patterns in such novel computing systems. With MIRIA, we address this gap by supporting the analysis of user movement, spatial interaction, and event data by multiple, co-located users directly in the original environment. Based on our own experiences and an analysis of the typical data, tasks, and visualizations used in existing approaches, we identify requirements for our system. We report on the design and prototypical implementation of MIRIA, which is informed by these requirements and offers various visualizations such as 3D movement trajectories, position heatmaps, and scatterplots. To demonstrate the value of MIRIA for real-world analysis tasks, we conducted expert feedback sessions using several use cases with authentic study data.

Composite data physicalizations allow for the physical reconfiguration of data points, creating new opportunities for interaction and engagement. However, there is a lack of understanding of people’s strategies and behaviors when directly manipulating physical data objects. In this paper, we systematically characterize different reconfiguration strategies using six exemplar physicalizations. We asked 20 participants to reorganize these exemplars with two levels of restriction: changing a single data object versus changing multiple data objects. Our findings show that there were two main reconfiguration strategies used: changes in proximity and changes in atomic orientation. We further characterize these using concrete examples of participant actions in relation to the structure of the physicalizations. We contribute an overview of reconfiguration strategies, which informs the design of future manually reconfigurable and dynamic composite physicalizations.

Virtual pets are an alternative to real pets, providing a substitute for people with allergies or preparing people for adopting a real pet. Recent advancements in mixed reality pave the way for virtual pets to provide a more natural and seamless experience for users. However, one key challenge is embedding environmental awareness into the virtual pet (e.g., identifying the food bowl’s location) so that they can behave naturally in the real world.

We propose a novel approach to synthesize virtual pet behaviors by considering scene semantics, enabling a virtual pet to behave naturally in mixed reality. Given a scene captured from the real world, our approach synthesizes a sequence of pet behaviors (e.g., resting after eating). Then, we assign each behavior in the sequence to a location in the real scene. We conducted user studies to evaluate our approach, which showed the efficacy of our approach in synthesizing natural virtual pet behaviors.

A common task for scatterplots is communicating trends in bivariate data. However, the ability of people to visually estimate these trends is under-explored, especially when the data violate assumptions required for common statistical models, or visual trend estimates are in conflict with statistical ones. In such cases, designers may need to intervene and de-bias these estimations, or otherwise inform viewers about differences between statistical and visual trend estimations. We propose data-driven mark orientation as a solution in such cases, where the directionality of marks in the scatterplot guide participants when visual estimation is otherwise unclear or ambiguous. Through a set of laboratory studies, we investigate trend estimation across a variety of data distributions and mark directionalities, and find that data-driven mark orientation can help resolve ambiguities in visual trend estimates.

We investigated the experiences of 15 parents and their tween children (ages 8-12, n=23) during nature explorations using the NatureCollections app, a mobile application that connects children with nature. Drawing on parent interviews and in-app audio recordings from a 2-week deployment study, we found that tweens’ experiences with the NatureCollections app were influenced by tensions surrounding how parents and tweens negotiate technology use more broadly. Despite these tensions, the app succeeded in engaging tweens in outdoor nature explorations, and parents valued the shared family experiences around nature. Parents desired the app to support family bonding and inform them about how their tween used the app. This work shows how applications intended to support enriching youth experiences are experienced in the context of screen time tensions between parents and tween during a transitional period of child development. We offer recommendations for designing digital experiences to support family needs and reduce screen time tensions.

Parental mediation literature is mostly situated in the contexts of television, Internet use, video games, and mobile devices, while there is less understanding of how parents mediate their children’s engagement with educational-focused media. We examine parental involvement in young children’s use of a creation-oriented educational media, i.e., coding kits, from a mediation perspective through an interview study. We frame parents’ mediation practices along three dimensions: (1) creative mediation, where parents mediate to support children’s creating and learning with media; (2) preparative mediation, where parents explore and prepare media for children’s engagement; and (3) administrative mediation, where parents administer and regulate their children’s media use. Compared to the restrictive, active, and co-using mediation theory, our proposed framework highlights various supportive practices parents take to help their children learn and create with media. We further connect our findings to Joint Media Engagement and reflect on implications for parent involvement in children’s creation-oriented media design.

Novice tangible interaction design students often find it challenging to generate input action ideas for tangible interfaces. To identify opportunities to aid input action idea generation, we built and evaluated a tool consisting of interactive physical artifacts coupled with digital examples of tangible systems and technical implementation guidance. Through video recorded design sessions and interviews with twelve students, we investigated how they used the tool to generate input action ideas, how it supported them, and what challenges they faced. We found that the tool helped in generating input action ideas by enabling to experience input actions, supporting hands-on explorations, and introducing possibilities. However, introducing examples at times caused design fixation. The tool fell short in supporting the planning of technical implementation of the generated ideas. This research is useful for tangible interaction design students, instructors, and researchers to apply in education, design similar tools, or conduct further research.

Distance learning is facing a critical moment finding a balance between high quality education for remote students and engaging them in hands-on learning. This is particularly relevant for project-based classrooms and makerspaces, which typically require extensive trouble-shooting and example demonstrations from instructors. We present RobotAR, a teleconsulting robotics toolkit for creating Augmented Reality (AR) makerspaces. We present the hardware and software for an AR-compatible robot, which behaves as a student’s voice assistant and can be embodied by the instructor for teleconsultation. As a desktop-based teleconsulting agent, the instructor has control of the robot’s joints and position to better focus on areas of interest inside the workspace. Similarly, the instructor has access to the student’s virtual environment and the capability to create AR content to aid the student with problem-solving. We also performed a user study which compares current techniques for distance hands-on learning and an implementation of our toolkit.

Grandparents and grandchildren, who cannot meet face-to-face (e.g., due to dislocation or physical distancing induced by a pandemic), often use audio-visual communication tools in order to maintain their relationship online. In a qualitative online survey (n = 85), we inquired into the various ways that grandparents and grandchildren came up with when being physically distant; many of them are tangible in nature as they include “things” or incorporate “spaces”. In this paper, we illustrate related temporal and spatial trajectories and unpack how online meetings are characterized by constant negotiations of agency. We discuss how online meetings could complement face-to-face meetings, instead of mimicking or replacing them. We finally articulate a collection of design sensitivities with the aim to both inspire and question designing for intergenerational online meetings.

Aging often comes with declining social interaction, a known adversarial factor impacting the life satisfaction of senior population. Such decline appears even in family–a permanent social circle, as their adult children eventually go independent. We present MomentMeld, an AI-powered, cloud-backed mobile application that blends with everyday routine and naturally encourages rich and frequent inter-generational interactions in a family, especially those between the senior generation and their adult children. Firstly, we design a photographic interaction aid called mutually stimulatory memento, which is a cross-generational juxtaposition of semantically related photos to bring natural arousal of context-specific inter-generational empathy and reminiscence. Secondly, we build comprehensive ensemble AI models consisting of various deep neural networks and a runtime system that automates the creation of mutually stimulatory memento on top of the user’s usual photo-taking routines. We deploy MomentMeld in-the-wild with six families for an eight-week period, and discuss the key findings and further implications.

Children are increasingly using wearables with physical activity tracking features. Although research has designed and evaluated novel features for supporting parent-child collaboration with these wearables, less is known about how families naturally adopt and use these technologies in their everyday life. We conducted interviews with 17 families who have naturally adopted child-owned wearables to understand how they use wearables individually and collaboratively. Parents are primarily motivated to use child-owned wearables for children's long-term health and wellbeing, whereas children mostly seek out entertainment and feeling accomplished through reaching goals. Children are often unable to interpret or contextualize the measures that wearables record, while parents do not regularly track these measures and focus on deviations from their children's routines. We discuss opportunities for making naturally-occurring family moments educational to positively contribute to children's conceptual understanding of health, such as developing age-appropriate trackable metrics for shared goal-setting and data reflection.

Physical space and materials we work with, as well as people we interact with affect the design and making processes. These aspects in relation to 3D designing in alignment with 3D printing need considerable exploration within Child-Computer interaction (CCI) research community. We conducted our case study by collecting 9-full-day-observation and interview data and examining 3D modeling and 3D printing activities, as work duties of 15-17-years-old summer trainees organized at the university. We identified, inspired by nexus analysis, different discourses circulating around these activities of novice young people and how the discourses are intermingled with the space, the materials, and the task at hand in complex ways, constructing and shaping the experience of the young participants. In our research and design implications, by signifying the impact of the people, challenges, tasks, spaces and tools, we provide recommendations for maintaining children's engagement in digital fabrication, significantly 3D designing and 3D printing, activities.

Pedagogical agents are theorized to increase humans’ effort to understand computerized instructions. Despite the pedagogical promises of VR, the usefulness of pedagogical agents in VR remains uncertain. Based on this gap, and inspired by global efforts to advance remote learning during the COVID-19 pandemic, we conducted an educational VR study in-the-wild (N = 161). With a 2 × 2 + 1 between subjects design, we manipulated the appearance and behavior of a virtual museum guide in an exhibition about viruses. Factual and conceptual learning outcomes as well as subjective learning experience measures were collected. In general, participants reported high enjoyment and had significant knowledge acquisition. We found that the agent’s appearance and behavior impacted factual knowledge gain. We also report an interaction effect between behavioral and visual realism for conceptual knowledge gain. Our findings nuance classical multimedia learning theories and provide directions for employing agents in immersive learning environments.

With rapid developments in consumer-level head-mounted displays and computer graphics, immersive VR has the potential to take online and remote learning closer to real-world settings. However, the effects of such digital transformations on learners, particularly for VR, have not been evaluated in depth. This work investigates the interaction-related effects of sitting positions of learners, visualization styles of peer-learners and teachers, and hand-raising behaviors of virtual peer-learners on learners in an immersive VR classroom, using eye tracking data. Our results indicate that learners sitting in the back of the virtual classroom may have difficulties extracting information. Additionally, we find indications that learners engage with lectures more efficiently if virtual avatars are visualized with realistic styles. Lastly, we find different eye movement behaviors towards different performance levels of virtual peer-learners, which should be investigated further. Our findings present an important baseline for design decisions for VR classrooms.

Classroom sensing is an important and active area of research with great potential to improve instruction. Complementing professional observers – the current best practice – automated pedagogical professional development systems can attend every class and capture fine-grained details of all occupants. One particularly valuable facet to capture is class gaze behavior. For students, certain gaze patterns have been shown to correlate with interest in the material, while for instructors, student-centered gaze patterns have been shown to increase approachability and immediacy. Unfortunately, prior classroom gaze-sensing systems have limited accuracy and often require specialized external or worn sensors. In this work, we developed a new computer-vision-driven system that powers a 3D “digital twin” of the classroom and enables whole-class, 6DOF head gaze vector estimation without instrumenting any of the occupants. We describe our open source implementation, and results from both controlled studies and real-world classroom deployments.

This project aims to foster shared positive experiences between people living with moderate to advanced dementia and their visitors as they may struggle to find topics to talk about and engaging things to do together. To promote a better visit, we trialed a previously developed app that includes eight games with twenty-one residents and their partners or carers across four care centers for three months each. Through interviews and data logging, we found that residents preferred games that were closer to their interests and skills, and that gameplay and cooperation fostered meaningful and shared interactions between residents and their visitors. The contribution of this work is twofold: (1) insights and opportunities into dyadic interactions when using an app and into promoting positive social experiences through technology design, and (2) reflections on the challenges of evaluating the benefits of technology for people living with dementia.

In recent work, design researchers have sought to ensure that people with disabilities are engaged as competent and valued contributors to co-design. Yet, little is known about how to achieve this with adults with severe intellectual disabilities. Navigating design in the context of complex care practices is challenging, charged with uncertainty, and requires sustained effort of methodological and affective adjustments. To establish a respectful co-design relationship and enrich participation, we turn to Active Support (AS), an evidence-based strategy for engaging adults with severe intellectual disabilities. We present a reflective account of long-term field work that utilized the four aspects of AS, a) every moment has potential; b) graded assistance; c) little and often; d) maximizing choice and control. We discuss how these principles contribute to deepening HCI methods by ensuring interactional turns for adults with severe disabilities, revealing their unique competences, thereby shaping design direction and providing design insight.

Many efforts to increase accessibility in coding for developers with visual impairments (DWVI) focus on supporting interactions with development tools. But, to understand how to appropriately modify and write source code, developers must seek information from a variety of disparate and highly technical sources. DWVI might benefit from technological support in this process. But, it is unclear what accessibility issues arise in technical information sources, whether accessibility impacts strategies for seeking technical information, or how best to support DWVI in information seeking. We conducted observations and interviews with twelve DWVI, to explore their information behaviors. We found that DWVI seek information in many of the same sources as their sighted peers, and accessibility issues in technical information sources were similar to those in nontechnical sources. But, despite these similarities, examining development as an information seeking process highlighted the role of contextual and social factors in determining accessibility for DWVI.

Knitting is a popular craft that can be used to create customized fabric objects such as household items, clothing and toys. Additionally, many knitters find knitting to be a relaxing and calming exercise. Little is known about how disabled knitters use and benefit from knitting, and what accessibility solutions and challenges they create and encounter. We conducted interviews with 16 experienced, disabled knitters and analyzed 20 threads from six forums that discussed accessible knitting to identify how and why disabled knitters knit, and what accessibility concerns remain. We additionally conducted an iterative design case study developing knitting tools for a knitter who found existing solutions insufficient. Our innovations improved the range of stitches she could produce. We conclude by arguing for the importance of improving tools for both pattern generation and modification as well as adaptations or modifications to existing tools such as looms to make it easier to track progress

Command-line interfaces (CLIs) remain a popular tool among developers and system administrators. Since CLIs are text-based interfaces, they are sometimes considered accessible alternatives to predominantly visual developer tools like IDEs. However, there is no systematic evaluation of the accessibility of CLIs in the literature. In this paper, we describe two studies with 12 developers on their experience of using CLIs with screen readers. Our findings show that CLIs have their own set of accessibility issues - the most important being CLIs are unstructured text interfaces. Based on our results, we provide a set of recommendations for improving the accessibility of command-line interfaces.

Technology adoption among older adults has increased significantly in recent years. Yet, as new technologies proliferate and the demographics of aging shift, continued attention to older adults’ adoption priorities and learning preferences is required. Through semi-structured interviews, we examine the factors adults 65+ prioritize in choosing new technologies, the challenges they encounter in learning to use them, and the human and material resources they employ to support these efforts. Using a video prototype as a design probe, we present scenarios to explore older adults’ perceptions of adoption and learning new technologies within the lens of health management support, a relevant and beneficial context for older adults. Our results reveal that participants appreciated self-paced learning, remote support, and flexible learning methods, and were less reliant on instruction manuals than in the past. This work provides insight into older adults’ evolving challenges, learning needs, and design opportunities for next generation learning support.

While HCI researchers have begun designing personalised VR experiences for older adults, there has been limited research examining the use of social VR - where users interact via avatars in a virtual environment. Avatar-mediated communication (AMC) is a crucial component of the social VR experience, but older users’ experience with AMC is poorly understood. We conducted a five-month study with 16 older adults evaluating a co-designed social VR prototype. Results show that AMC in social VR was seen as medium that supported introverted users to express themselves and was viewed as offering advantages when discussing sensitive topics. Our study provides new insights into how older adults view AMC in social VR as a communication medium and we contribute six design reflections, based on our results, that highlight the steps that can be taken to ensure that AMC in social VR can meet the communication needs of older users.

There is an open call for technology to be more playful [5, 79] and for tech design to be more inclusive of people with disabilities [80]. In the era of COVID19, it is often unsafe for the public in general and people with disabilities, in particular, to engage in in-person design exercises using traditional methods. This presents a missed opportunity as these populations are already sharing playful content rich with tacit design knowledge that can be used to inspire the design of playful everyday technology. We present our process of scraping play potentials [4] from TikTok from content creators with disabilities to generate design concepts that may inspire future technology design. We share 7 emerging themes from the scraped content, a catalog of design concepts that may inspire designers, and discuss the relevance of the emerging themes and possible implications for the design concepts.

High-fidelity prototyping tools are used by software designers and developers to iron out interface details without full implementation. However, the lack of visual accessibility in these tools creates a barrier for designers who may use screen readers, such as those who are vision impaired. We assessed conformance of four prototyping tools (Sketch, Adobe XD, Balsamiq, UXPin) with accessibility guidelines, using two screen readers (Narrator and VoiceOver), focusing our analysis on GUI element accessibility and critical workflows used to create prototypes. We found few tools were fully accessible, with 45.9% of GUI elements meeting accessibility criteria (34.2% partially supported accessibility, 19.9% not supporting accessibility). Accessibility issues stymied efforts to create prototypes using screen readers. Though no screen reader-tool pairs were completely accessible, the most accessible pairs were VoiceOver-Sketch, VoiceOver-Balsamiq, and Narrator-Balsamiq. We recommend prioritizing improved accessibility for input and control instruction, alternative text, focus order, canvas element properties, and keyboard operations.

Online shopping has become a valuable modern convenience, but blind or low vision (BLV) users still face significant challenges using it, because of: 1) inadequate image descriptions and 2) the inability to filter large amounts of information using screen readers. To address those challenges, we propose Revamp, a system that leverages customer reviews for interactive information retrieval. Revamp is a browser integration that supports review-based question-answering interactions on a reconstructed product page. From our interview, we identified four main aspects (color, logo, shape, and size) that are vital for BLV users to understand the visual appearance of a product. Based on the findings, we formulated syntactic rules to extract review snippets, which were used to generate image descriptions and responses to users’ queries. Evaluations with eight BLV users showed that Revamp 1) provided useful descriptive information for understanding product appearance and 2) helped the participants locate key information efficiently.

Voice assistants (VAs) – like Amazon Alexa or Siri – offer hands-/eyes-free interactions that are beneficial to a range of users, including individuals who are blind, to fulfill tasks that are otherwise difficult or inaccessible. While these interfaces model conversational interactions to achieve simple tasks, there have been recent calls for VAs that model more transactional interactions for a wider range of complex activities. In this study, we explored the extension of VAs’ capabilities in the context of indoor navigation through mixed-ability focus groups with blind and sighted airport travelers. We found high overlap in the difficulties encountered by blind and sighted travelers, as well as shared interest in a voice-activated travel assistant to improve travel experiences. Leveraging user-elicited recommendations, we present interaction design examples that showcase customization of different and multiple modalities, which collectively demonstrate how VAs can more broadly achieve transactional interactions in complex task scenarios.

There is a growing need to design augmentative and alternative communication (AAC) devices that focus on supporting quality of life goals, such as increased social participation in leisurely activities. Yet, designing AAC applications that support leisurely activities is difficult, as the activity might require novel and specific language in a timely manner. Through observations and contextual interviews with people with aphasia, their social partners, and speech-language therapists, we characterize the important but challenging nature of supporting one specific leisure activity: meal ordering in restaurants. Based on our observational and interview data, we design and explore three prototype AAC systems to support people with aphasia in ordering meals in restaurants. Each prototype integrates a different AI technology, contributing insights into how AI may enhance AAC usage and design. The study opens up questions of designing accessible restaurant experiences for neurodivergent people and the role of AI in AAC devices more broadly.

We present an inexpensive tabletop loom that offers fully computational patterning while maintaining the flexibility of handweaving. Our loom can be assembled for under US$200 with 3D printed parts, and it can be controlled straightforwardly over USB. Our loom is explicitly a hand loom: that is, a weaver is required to operate the weaving process and may mediate row-by-row patterning and material specifics like yarn tension. Our approach combines the flexibility of fully analog handweaving with the computational affordances of digital fabrication: it enables the incorporation of special techniques and materials, as well as allowing for the possibility of computational and creative interventions in the weaving process itself. In taking this approach, we aim to serve a range of end users including artisans and researchers, whether for skill-building, for rapid prototyping, or for creative reflection.

We describe the mechanical and electronic implementation of our loom and show examples of its use for personal fabrication.

Machine knitting is an increasingly accessible fabrication technology for producing custom soft goods. However, recent machine knitting research has focused on knit shaping, or on adapting hand-knitting patterns. We explore a capability unique to machine knitting: producing multilayer spacer fabrics. These fabrics consist of two face layers connected by a monofilament filler yarn which gives the structure stiffness and volume. We show how to vary knit patterning and yarn parameters in spacer fabrics to produce tactile materials with embedded functionality for forming soft actuated mechanisms and sensors with tunable density, stiffness, material bias, and bristle properties. These soft mechanisms can be rapidly produced on a computationally-controlled v-bed knitting machine and integrated directly into soft objects.

In this paper, we present TexYZ, a method for rapid and effortless manufacturing of textile mutual capacitive sensors using a commodity embroidery machine. We use enameled wire as a bobbin thread to yield textile capacitors with high quality and consistency. As a consequence, we are able to leverage the precision and expressiveness of projected mutual capacitance for textile electronics, even when size is limited. Harnessing the assets of machine embroidery, we implement and analyze five distinct electrode patterns, examine the resulting electrical features with respect to geometrical attributes, and demonstrate the feasibility of two promising candidates for small-scale matrix layouts. The resulting sensor patches are further evaluated in terms of capacitance homogeneity, signal-to-noise ratio, sensing range, and washability. Finally, we demonstrate two use case scenarios, primarily focusing on continuous input with up to three degrees-of-freedom.

The craft of improvisational quilting involves working without the use of a predefined pattern. Design decisions are made “in the fabric,” with design experimentation tightly interleaved with the creation of the final artifact. To investigate how this type of design process can be supported, and to address challenges faced by practitioners, this paper presents PatchProv, a system for supporting improvisational quilt design. Based on a review of popular books on improvisational quilting, a set of design principles and key challenges to improvisational quilt design were identified, and PatchProv was developed to support the unique aspects of this process. An evaluation with a small group of quilters showed enthusiasm for the approach and revealed further possibilities for how computational tools can support improvisational quilting and improvisational design practices more broadly.

We present BezelGlide, a novel suite of bezel interaction techniques, designed to minimize screen occlusion and ‘fat finger’ effects, when interacting with common graphs on smartwatches. To explore the design of BezelGlide, we conducted two user studies. First, we quantified the amount of screen occlusion experienced when interacting with the smartwatch bezel. Next, we designed two techniques that involve gliding the finger along the smartwatch bezel for graph interaction. Full BezelGlide (FBG) and Partial BezelGlide (PBG), use the full or a portion of the bezel, respectively, to reduce screen occlusion while scanning a line chart for data. In the common value detection task, we find that PBG outperforms FBG and Shift, a touchscreen occlusion-free technique, both quantitatively and subjectively, also while mobile. We finally illustrate the generzability potential of PBG to interact with common graph types making it a valuable interaction technique for smartwatch users.

We propose a novel type of olfactory device that creates a stereo-smell experience, i.e., directional information about the location of an odor, by rendering the readings of external odor sensors as trigeminal sensations using electrical stimulation of the user's nasal septum. The key is that the sensations from the trigeminal nerve, which arise from nerve-endings in the nose, are perceptually fused with those of the olfactory bulb (the brain region that senses smells). As such, we propose that electrically stimulating the trigeminal nerve is an ideal candidate for stereo-smell augmentation/substitution that, unlike other approaches, does not require implanted electrodes in the olfactory bulb. To realize this, we engineered a self-contained device that users wear across their nasal septum. Our device outputs by stimulating the user's trigeminal nerve using electrical impulses with variable pulse-widths; and it inputs by sensing the user's inhalations using a photoreflector. It measures 10x23 mm and communicates with external gas sensors using Bluetooth. In our user study, we found the key electrical waveform parameters that enable users to feel an odor's intensity (absolute electric charge) and direction (phase order and net charge). In our second study, we demonstrated that participants were able to localize a virtual smell source in the room by using our prototype without any previous training. Using these insights, our device enables expressive trigeminal sensations and could function as an assistive device for people with anosmia, who are unable to smell.

We present HulaMove, a novel interaction technique that leverages the movement of the waist as a new eyes-free and hands-free input method for both the physical world and the virtual world. We first conducted a user study (N=12) to understand users’ ability to control their waist. We found that users could easily discriminate eight shifting directions and two rotating orientations, and quickly confirm actions by returning to the original position (quick return). We developed a design space with eight gestures for waist interaction based on the results and implemented an IMU-based real-time system. Using a hierarchical machine learning model, our system could recognize waist gestures at an accuracy of 97.5%. Finally, we conducted a second user study (N=12) for usability testing in both real-world scenarios and virtual reality settings. Our usability study indicated that HulaMove significantly reduced interaction time by 41.8% compared to a touch screen method, and greatly improved users’ sense of presence in the virtual world. This novel technique provides an additional input method when users’ eyes or hands are busy, accelerates users’ daily operations, and augments their immersive experience in the virtual world.

In traditional body-art, designs are adjusted to the body as they are applied, enabling creative improvisation and exploration. Conventional design and fabrication methods of epidermal interfaces, however, separate these steps. With BodyStylus we present the first computer-assisted approach for on-body design and fabrication of epidermal interfaces. Inspired by traditional techniques, we propose a hand-held tool that augments freehand inking with digital support: projected in-situ guidance assists creating valid on-body circuits and aesthetic ornaments that align with the human bodyscape, while pro-active switching between inking and non-inking creates error preventing constraints. We contribute BodyStylus’s design rationale and interaction concept along with an interactive prototype that uses self-sintering conductive ink. Results of two focus group explorations showed that guidance was more appreciated by artists, while constraints appeared more useful to engineers, and that working on the body inspired critical reflection on the relationship between bodyscape, interaction, and designs.

This paper presents NFCSense, a data-defined rich-ID motion sensing technique for fluent tangible interaction design by using commodity near-field communication (NFC) tags and a single NFC tag reader. An NFC reader can reliably recognize the presence of an NFC tag at a high read rate (∼ 300 reads/s) with low latency, but such high-speed reading has rarely been exploited because the reader may not effectively resolve collisions of multiple tags. Therefore, its human–computer interface applications have been typically limited to a discrete, hands-on interaction style using one tag at a time. In this work, we realized fluent, hands-off, and multi-tag tangible interactions by leveraging gravity and anti-collision physical constraints, which support effortless user input and maximize throughput. Furthermore, our system provides hot-swappable interactivity that enables smooth transitions throughout extended use. Based on the design parameters explored through a series of studies, we present a design space with proof-of-concept implementations in various applications.

This research investigates the design space of combining touchscreens with passive rich-ID building block systems to support the physical construction of contexts in touchscreen interactions. With two proof-of-concept systems, RFIPillars and RFITiles, we explore various schemes for using tangible inputs for context enrichment in touchscreen interactions. Instead of incorporating an electronic touchscreen module that requires per-module maintenance, this work intentionally makes each tangible object passive. We explore rear-projection solutions to integrate touchscreen interactions into these passive building blocks with capacitive touch sensing techniques and deliberate physical forgiving to retain the merits of being both batteryless and wireless. The presented research artifacts embody the interaction designs and elucidate scalability challenges in integrating touchscreen interactions into this emerging tangible user interface.

We present an actuated-interface that is not only a tangible interface but also an autonomous object, designed as an independent entity that takes a similar role to the user’s role in an anagram word game. We highlight two leading interaction paradigms: Turn-taking-actuation and Joint-actuation, and evaluate both in a qualitative interaction study with the autonomous actuated-interface. Our findings reveal that all participants perceived the interaction as a social experience. The different interaction paradigms led to different interpretations: Turn-taking-actuation was interpreted as a competitive experience, while Joint-actuation was interpreted as a collaborative experience. The interaction paradigms also influenced the intensity of emotions and perception of control, with Joint-actuation leading to more intense emotions and higher sensitivity to control in the interaction. To conclude, our findings show that it is possible to design an actuated-interface that users perceive both as a tangible interface and as a social entity with its own intent.

Emotional contagion is a phenomenon in which one’s emotions are transmitted among individuals unconsciously by observing others’ emotional expressions. In this paper, we propose a method for mediating people’s emotions by triggering emotional contagion through artificial bodily changes such as pseudo tears. We focused on shedding tears because of the link to several emotions besides sadness. In addition, it is expected that shedding tears would induce emotional contagion because it is observable by others. We designed an eyeglasses-style wearable device, Teardrop glasses, that release water drops near the wearer’s eyes. The drops flow down the cheeks and emulate real tears. The study revealed that artificial crying with pseudo tears increased sadness among both wearers and those observing them. Moreover, artificial crying attenuated happiness and positive feelings in observers. Our findings show that actual bodily changes are not necessary for inducing emotional contagion as artificial bodily changes are also sufficient.

We present LightTouch, a 3D-printed passive gadget to enhance touch interactions on unmodified capacitive touchscreens. The LightTouch gadgets simulate finger operations such as tapping, swiping, and multi-touch gestures by means of conductive materials and light-dependent resistors (LDR) embedded in the object. The touchscreen emits visible light and the LDR senses the level of this light, which changes its resistance value. By controlling the screen brightness, it intentionally connects or disconnects the path between the GND and the touchscreen, thus allowing the touch inputs to be controlled. In contrast to conventional physical extensions for touchscreens, our technique requires neither continuous finger contact on the conductive part nor the use of batteries. As such, it opens up new possibilities for touchscreen interactions beyond the simple automation of touch inputs, such as establishing a communication channel between devices, enhancing the trackability of tangibles, and inter-application operations.

Emerging research has demonstrated the viability of on-textile actuation mechanisms, however, an easily customizable and versatile on-cloth actuation mechanism is yet to be explored. In this paper, we present ClothTiles along with its rapid fabrication technique that enables actuation of clothes. ClothTiles leverage flexible 3D-printing and Shape-Memory Alloys (SMAs) alongside new parametric actuation designs. We validate the concept of fabric actuation using a base element, and then systematically explore methods of aggregating, scaling, and orienting prospects for extended actuation in garments. A user study demonstrated that our technique enables multiple actuation types applied across a variety of clothes. Users identified both aesthetic and functional applications of ClothTiles. We conclude with a number of insights for the Do-It-Yourself community on how to employ 3D-printing with SMAs to enable actuation on clothes.

Modern vehicles are using AI and increasingly sophisticated sensor suites to improve Advanced Driving Assistance Systems (ADAS) and support automated driving capabilities. Heads-Up-Displays (HUDs) provide an opportunity to visually inform drivers about vehicle perception and interpretation of the driving environment. One approach to HUD design may be to reveal to drivers the vehicle’s full contextual understanding, though it is not clear if the benefits of additional information outweigh the drawbacks of added complexity, or if this balance holds across drivers. We designed and tested an Augmented Reality (AR) HUD in an online study (N = 298), focusing on the influence of HUD visualizations on drivers’ situation awareness and perceptions. Participants viewed two driving scenes with one of three HUD conditions. Results were nuanced: situation awareness declined with increasing driving context complexity, and contrary to expectation, also declined with the presence of a HUD compared to no HUD. Significant differences were found by varying HUD complexity, which led us to explore different characterizations of complexity, including counts of scene items, item categories, and illuminated pixels. Our analysis finds that driving style interacts with driving context and HUD complexity, warranting further study.

Cinematic Virtual Reality (CVR), or 360° video, engages users in immersive viewing experiences. However, as users watch one part of the 360° view, they will necessarily miss out on events happening in other parts of the sphere. Consequently, fear of missing out (FOMO) is unavoidable. However, users can also experience the joy of missing out (JOMO). In a repeated measures, mixed methods design, we examined the fear and joy of missing out (FOMO and JOMO) and sense of presence in two repeat viewings of a 360° film using a head-mounted display. We found that users experienced both FOMO and JOMO. FOMO was caused by the users’ awareness of parallel events in the spherical view, but users also experienced JOMO. FOMO did not compromise viewers’ sense of presence, and FOMO also decreased in the second viewing session, while JOMO remained constant. The findings suggest that FOMO and JOMO can be two integral qualities in an immersive video viewing experience and that FOMO may not be as negative a factor as previously thought.

Precise emotion ground truth labels for 360° virtual reality (VR) video watching are essential for fine-grained predictions under varying viewing behavior. However, current annotation techniques either rely on post-stimulus discrete self-reports, or real-time, continuous emotion annotations (RCEA) but only for desktop/mobile settings. We present RCEA for 360° VR videos (RCEA-360VR), where we evaluate in a controlled study (N=32) the usability of two peripheral visualization techniques: HaloLight and DotSize. We furthermore develop a method that considers head movements when fusing labels. Using physiological, behavioral, and subjective measures, we show that (1) both techniques do not increase users’ workload, sickness, nor break presence (2) our continuous valence and arousal annotations are consistent with discrete within-VR and original stimuli ratings (3) users exhibit high similarity in viewing behavior, where fused ratings perfectly align with intended labels. Our work contributes usable and effective techniques for collecting fine-grained viewport-dependent emotion labels in 360° VR.

Mixed Reality has been shown to enhance remote guidance and is especially well-suited for physical tasks. Conversations during these tasks are heavily anchored around task objects and their spatial relationships in the real world, making referencing - the ability to refer to an object in a way that is understood by others - a crucial process that warrants explicit support in collaborative Mixed Reality systems. This paper presents a 2x2 mixed factorial experiment that explores the effects of providing spatial information and system-generated guidance to task objects. It also investigates the effects of such guidance on the remote collaborator’s need for spatial information. Our results show that guidance increases performance and communication efficiency while reducing the need for spatial information, especially in unfamiliar environments. Our results also demonstrate a reduced need for remote experts to be in immersive environments, making guidance more scalable, and expertise more accessible.

Graphical User Interfaces present commands at particular locations, arranged in menus, toolbars, and ribbons. One hallmark of expertise with a GUI is that experts know the locations of commonly-used commands, such that they can find them quickly and without searching. Although GUIs have been studied for many years, however, there is still little known about how this spatial location memory develops, or how designers can make interfaces more memorable. One of the main ways that people remember locations in the real world is landmarks – so we carried out a study to investigate how users remember commands and navigate in four common applications (Word, Facebook, Reader, and Photoshop). Our study revealed that people strongly rely on landmarks that are readily available in the interface (e.g., layout, corners, and edges) to orient themselves and remember commands. We provide new evidence that landmarks can aid spatial memory and expertise development with an interface, and guidelines for designers to improve the memorability of future GUIs.

Peripheral breathing guides – tools designed to influence breathing while completing another primary task – have been proposed to provide physiological benefits during information work. While research has shown that guides can influence breathing rates under ideal conditions, there is little evidence that they can lead to underlying markers of physiological benefit under interrupted work conditions. Further, even if guides are effective during work tasks, it is unclear how personal and workplace factors affect peoples' willingness to adopt them for everyday use. In this paper, we present the results of a comparative, mixed-methods study of five different peripheral breathing guides. Our findings show that peripheral breathing guides are viable and can provide physiological markers of benefit during interrupted work. Further, we show that guides are effective – even when use is intermittent due to workplace distractions. Finally, we contribute guidelines to support the design of breathing guides for everyday information work.

Virtual Reality (VR) is becoming increasingly popular both in the entertainment and professional domains. Behavioral biometrics have recently been investigated as a means to continuously and implicitly identify users in VR. Applications in VR can specifically benefit from this, for example, to adapt virtual environments and user interfaces as well as to authenticate users. In this work, we conduct a lab study (N = 16) to explore how accurately users can be identified during two task-driven scenarios based on their spatial movement. We show that an identification accuracy of up to 90% is possible across sessions recorded on different days. Moreover, we investigate the role of users’ physiology in behavioral biometrics by virtually altering and normalizing their body proportions. We find that body normalization in general increases the identification rate, in some cases by up to 38%; hence, it improves the performance of identification systems.

Audiovisual recordings of user studies and interviews provide important data in qualitative HCI research. Even when a textual transcription is available, researchers frequently turn to these recordings due to their rich information content. However, the temporal, unstructured nature of audiovisual recordings makes them less efficient to work with than text. Through interviews and a survey, we explored how HCI researchers work with audiovisual recordings. We investigated researchers’ transcription and annotation practice, their overall analysis workflow, and the prevalence of direct analysis of audiovisual recordings. We found that a key task was locating and analyzing inspectables, interesting segments in recordings. Since locating inspectables can be time consuming, participants look for detectables, visual or auditory cues that indicate the presence of an inspectable. Based on our findings, we discuss the potential for automation in locating detectables in qualitative audiovisual analysis.

Avatars in virtual reality (VR) enable embodied experiences and induce the Proteus effect—a shift in behavior and attitude to mimic one’s digital representation. Previous work found that avatars associated with physical strength can decrease users’ perceived exertion when performing physical tasks. However, it is unknown if an avatar’s appearance can also influence the user’s physiological response to exercises. Therefore, we conducted an experiment with 24 participants to investigate the effect of avatars’ athleticism on heart rate and perceived exertion while cycling in VR following a standardized protocol. We found that the avatars’ athleticism has a significant and systematic effect on users’ heart rate and perceived exertion. We discuss potential moderators such as body ownership and users’ level of fitness. Our work contributes to the emerging area of VR exercise systems.

Smartphone users do not deal with notifications strictly in the order they are displayed, but sometimes read them from the middle, suggesting a mismatch between current systems’ display order and users’ needs. We therefore used mixed methods to investigate 34 smartphone users’ desired notification display order and related it with users’ self-reported order of attendance. Classifying using these two orders as dimensions, we obtained seven types of notifications, which helped us not only highlight the distinct attributes but understand the implied roles of these seven types of notifications, as well as the implied meaning of display orders. This is especially manifested in our identification of three main mismatches between the two orders. Qualitative findings reveal several meanings that participants attached to particular positions when arranging notifications. We offer design implications for notification systems, including calling for two-dimensional notification layout to support the multi-purpose roles of smartphone notifications we identified.

Negative body perceptions are a major predictor of physical inactivity, a serious health concern. Sensory feedback can be used to alter such body perceptions; movement sonification, in particular, has been suggested to affect body perception and levels of physical activity (PA) in inactive people. We investigated how metaphorical sounds impact body perception and PA. We report two qualitative studies centered on performing different strengthening/flexibility exercises using SoniBand, a wearable that augments movement through different sounds. The first study involved physically active participants and served to obtain a nuanced understanding of the sonifications’ impact. The second, in the home of physically inactive participants, served to identify which effects could support PA adherence. Our findings show that movement sonification based on metaphors led to changes in body perception (e.g., feeling strong) and PA (e.g., repetitions) in both populations, but effects could differ according to the existing PA-level. We discuss principles for metaphor-based sonification design to foster PA.

In this work we report on two comprehensive user studies investigating the perception of Augmented Reality (AR) visualizations influenced by real-world backgrounds. Since AR is an emerging technology, it is important to also consider productive use cases, which is why we chose an exemplary and challenging industry 4.0 environment. Our basic perceptual research focuses on both the visual complexity of backgrounds as well as the influence of a secondary task. In contrast to our expectation, data of our 34 study participants indicate that the background has far less influence on the perception of AR visualizations. Moreover, we observed a mismatch between measured and subjectively reported performance. We discuss the importance of the background and recommendations for visual real-world augmentations. Overall, our results suggest that AR can be used in many visually challenging environments without losing the ability to productively work with the visualizations shown.

Virtual reality (VR) researchers struggle to conduct remote studies. Previous work has focused on working around limitations imposed by traditional crowdsourcing methods. However, the potential for leveraging social VR platforms for HCI evaluations is largely unexplored. These platforms have large VR-ready user populations, distributed synchronous virtual environments, and support for user-generated content. We demonstrate how social VR platforms can be used to practically and ethically produce valid research results by replicating two studies using one such platform (VRChat): a quantitative study on Fitts’ Law and a qualitative study on tabletop collaboration. Our replication studies exhibited analogous results to the originals, indicating the research validity of this approach. Moreover, we easily recruited experienced VR users with their own hardware for synchronous, remote, and collaborative participation. We further provide lessons learned for future researchers experimenting using social VR platforms. This paper and all supplemental materials are available at osf.io/c2amz.

Current VR systems provide various text input methods that enable users to enter text efficiently with virtual keyboards. However, little attention has been paid to facilitate text revision during the VR text input process. We first summarized existing text revision solutions in current VR text input research and found that backspace is the only tool available for text revision with virtual keyboards with few mentioning designs for caret control. To systematically explore VR text revision designs, we presented a design space for VR text revision based on backspace and caret. With the proposed design space, we further analyzed the feasibility of the combined usage of backspace and caret by proposing and evaluating four VR text revision techniques. Outcomes of this research can provide a fundamental understanding of VR text revision solutions (with backspace and caret) and a comparable basis for evaluating future VR text revision techniques.

Head-Mounted Display based Virtual Reality is proliferating. However, Visually Induced Motion Sickness (VIMS), which prevents many from using VR without discomfort, bars widespread adoption. Prior work has shown that limiting the Field of View (FoV) can reduce VIMS at a cost of also reducing presence. Systems that dynamically adjust a user’s FoV may be able to balance these concerns. To explore this idea, we present a technique for standard 360° video that shrinks FoVs only during VIMS inducing scenes. It uses Visual Simultaneous Localization and Mapping and peripheral optical flow to compute camera movements and reduces FoV during rapid motion or optical flow. A user study (N=23) comparing 360° video with unrestricted-FoVs (90°), reduced fixed-FoVs (40°) and dynamic-FoVs (40°-90°) revealed that dynamic-FoVs mitigate VIMS while maintaining presence. We close by discussing the user experience of dynamic-FoVs and recommendations for how they can help make VR comfortable and immersive for all.

Virtual reality (VR) is increasingly used in complex social and physical settings outside of the lab. However, not much is known about how these settings influence use, nor how to design for them. We analyse 233 YouTube videos of VR Fails to: (1) understand when breakdowns occur, and (2) reveal how the seams between VR use and the social and physical setting emerge. The videos show a variety of fails, including users flailing, colliding with surroundings, and hitting spectators. They also suggest causes of the fails, including fear, sensorimotor mismatches, and spectator participation. We use the videos as inspiration to generate design ideas. For example, we discuss more flexible boundaries between the real and virtual world, ways of involving spectators, and interaction designs to help overcome fear. Based on the findings, we further discuss the ‘moment of breakdown’ as an opportunity for designing engaging and enhanced VR experiences.

Extended Reality (XR) technology - such as virtual and augmented reality - is now widely used in Human Computer Interaction (HCI), social science and psychology experimentation. However, these experiments are predominantly deployed in-lab with a co-present researcher. Remote experiments, without co-present researchers, have not flourished, despite the success of remote approaches for non-XR investigations. This paper summarises findings from a 30-item survey of 46 XR researchers to understand perceived limitations and benefits of remote XR experimentation. Our thematic analysis identifies concerns common with non-XR remote research, such as participant recruitment, as well as XR-specific issues, including safety and hardware variability. We identify potential positive affordances of XR technology, including leveraging data collection functionalities builtin to HMDs (e.g. hand, gaze tracking) and the portability and reproducibility of an experimental setting. We suggest that XR technology could be conceptualised as an interactive technology and a capable data-collection device suited for remote experimentation.

In recent years, pass-through cameras have resurfaced as inclusions for virtual reality (VR) hardware. With modern cameras that now have increased resolution and frame rate, Video See-Through (VST) Head-Mounted Displays (HMD) can be used to provide an Augmented Reality (AR) experience. However, because users see their surroundings through video capture and HMD lenses, there is question surrounding how people perceive their environment with these devices. We conducted a user study with 26 participants to help understand if distance perception is altered when viewing surroundings with a VST HMD. Although previous work shows that distance estimation in VR with an HTC Vive is comparable to that in the real world, our results show that the inclusion of a ZED Mini pass-through camera causes a significant difference between normal, unrestricted viewing and that through a VST HMD.

Arguably one of the most important characteristics of virtual reality (VR) is its ability to induce higher feelings of presence. Still, research has remained inconclusive on how presence is affected by human factors such as emotion and agency. Here we adopt a novel design to investigate their effects by testing virtual environments inducing either happiness or fear, with or without user agency. Results from 121 participants showed that the dominant emotion induced by a virtual environment is positively correlated with presence. In addition, agency had a significant positive effect on presence and, furthermore, moderated the effect of emotion on presence. We show for the first time that the effects of emotion and agency on presence are not straightforward but they can be modelled by separating design factors from subjective measures. We discuss how these findings can explain seemingly conflicting results of related work and their implications for VR design.

Based on a systematic literature review of more than 300 papers published over the last 10 years, we provide indicators that the simulator sickness questionnaire (SSQ) is extensively used and widely accepted as a general discomfort measure in virtual reality (VR) research – although it actually only accounts for one category of symptoms. This results in important other categories (digital eye strain (DES) and ergonomics) being largely neglected. To contribute to a more comprehensive picture of discomfort in VR head-mounted displays, we further conducted an online study (N=352) on the severity and relevance of all three symptom categories. Most importantly, our results reveal that symptoms of simulator sickness are significantly less severe and of lower prevalence than those of DES and ergonomics. In light of these findings, we critically discuss the current use of SSQ as the only discomfort measure and propose a more comprehensive factor model that also includes DES and ergonomics.

Reading is a fundamental activity to obtain information both in the real and the digital world. Virtual reality (VR) allows novel approaches for users to view, read, and interact with a text. However, for efficient reading, it is necessary to understand how a text should be displayed in VR without impairing the VR experience. Therefore, we conducted a study with 18 participants to investigate text presentation type and location in VR. We compared world-fixed, edge-fixed, and head-fixed text locations. Texts were displayed using Rapid Serial Visual Presentation (RSVP) or as a paragraph. We found that RSVP is a promising presentation type for reading short texts displayed in edge-fixed or head-fixed location in VR. The paragraph presentation type using world-fixed or edge-fixed location is promising for reading long text if movement in the virtual environment is not required. Insights from our study inform the design of reading interfaces for VR applications.

A compelling property of virtual reality is that it allows users to interact with objects as they would in the real world. However, such interactions are limited to space within reach. We present Poros, a system that allows users to rearrange space. After marking a portion of space, the distant marked space is mirrored in a nearby proxy. Thereby, users can arrange what is within their reachable space, making it easy to interact with multiple distant spaces as well as nearby objects. Proxies themselves become part of the scene and can be moved, rotated, scaled, or anchored to other objects. Furthermore, they can be used in a set of higher-level interactions such as alignment and action duplication. We show how Poros enables a variety of tasks and applications and also validate its effectiveness through an expert evaluation.

The VR community has introduced many object selection and manipulation techniques during the past two decades. Typically, they are empirically studied to establish their benefits over the state-of-the-art. However, the literature contains few guidelines on how to conduct such studies; standards developed for evaluating 2D interaction often do not apply. This lack of guidelines makes it hard to compare techniques across studies, to report evaluations consistently, and therefore to accumulate or replicate findings. To build such guidelines, we review 20 years of studies on VR object selection and manipulation. Based on the review, we propose recommendations for designing studies and a checklist for reporting them. We also identify research directions for improving evaluation methods and offer ideas for how to make studies more ecologically valid and rigorous.

Evaluating novel authentication systems is often costly and time-consuming. In this work, we assess the suitability of using Virtual Reality (VR) to evaluate the usability and security of real-world authentication systems. To this end, we conducted a replication study and built a virtual replica of CueAuth [52], a recently introduced authentication scheme, and report on results from: (1) a lab-based in-VR usability study (N=20) evaluating user performance; (2) an online security study (N=22) evaluating system’s observation resistance through virtual avatars; and (3) a comparison between our results and those previously reported in the real-world evaluation. Our analysis indicates that VR can serve as a suitable test-bed for human-centred evaluations of real-world authentication schemes, but the used VR technology can have an impact on the evaluation. Our work is a first step towards augmenting the design and evaluation spectrum of authentication systems and offers ground work for more research to follow.

In today’s fast-paced world, stress has become a growing health concern. While more automatic stress tracking technologies have recently become available on wearable or mobile devices, there is still a limited understanding of how they are actually used in everyday life. This paper presents an empirical study of automatic stress-tracking technologies in use in China, based on semi-structured interviews with 17 users. The study highlights three challenges of stress-tracking data engagement that prevent effective technology usage: the lack of immediate awareness, the lack of pre-required knowledge, and the lack of corresponding communal support. Drawing on the stress-tracking practices uncovered in the study, we bring these issues to the fore, and unpack assumptions embedded in related works on self-tracking and how data engagement is approached. We end by calling for a reconsideration of data engagement as part of self-tracking practices with technologies rather than simply looking at the user interface.

Online mental health communities (OMHCs) are prominent resources for improving people’s mental wellbeing. An immediate cue of such improvement is support-seekers’ satisfaction expressed in their replies to the received comments. However, the comments that seekers find satisfying may change with their community knowledge, e.g., measured by tenure and posting experience in that community. In this paper, we first model the amount of satisfaction conveyed in the support-seekers’ replies to the received comments. Then we quantitatively examine how seekers’ expressed satisfaction is affected by their community knowledge, sought and received support in an OMHC. Results show that support-seekers with more posting experience generally display less contentment to the received comments. Compared to newcomers, higher tenured members express less satisfaction when receiving informational support. We also found that support matching positively predicts seekers’ satisfaction regardless of their community knowledge. Our findings have implications for OMHCs to satisfy support-seekers through their community knowledge.

The recent growth of digital interventions for mental well-being prompts a call-to-arms to explore the delivery of personalised recommendations from a user’s perspective. In a randomised placebo study with a two-way factorial design, we analysed the difference between an autonomous user experience as opposed to personalised guidance, with respect to both users’ preference and their actual usage of a mental well-being app. Furthermore, we explored users’ preference in sharing their data for receiving personalised recommendations, by juxtaposing questionnaires and mobile sensor data. Interestingly, self-reported results indicate the preference for personalised guidance, whereas behavioural data suggests that a blend of autonomous choice and recommended activities results in higher engagement. Additionally, although users reported a strong preference of filling out questionnaires instead of sharing their mobile data, the data source did not have any impact on the actual app use. We discuss the implications of our findings and provide takeaways for designers of mental well-being applications.

While Digital Self-Control Tools (DSCTs) mainly target smartphones, more effort should be put into evaluating multi-device ecosystems to enhance digital wellbeing as users typically use multiple devices at a time. In this paper, we first review more than 300 DSCTs by demonstrating that the majority of them implements a single-device conceptualization that poorly adapts to multi-device settings. Then, we report on the results from an interview and a sketching exercise (N=20) exploring how users make sense of their multi-device digital wellbeing. Findings show that digital wellbeing issues extend beyond smartphones, with the most problematic behaviors deriving from the simultaneous usage of different devices to perform uncorrelated tasks. While this suggests the need of DSCTs that can adapt to different and multiple devices, our work also highlights the importance of learning how to properly behave with technology, e.g., through educational courses, which may be more effective than any lock-out mechanism.

The increasing number of mental disorders worldwide calls for novel types of prevention measures. Given the number of commuters who spend a substantial amount of time on the road, the car offers an opportune environment. This paper presents the first in-vehicle intervention study affecting mental health and well-being on public roads. We designed and implemented two in-vehicle interventions based on proven psychotherapy interventions. Whereas the first intervention uses mindfulness exercises while driving, the second intervention induces positive emotions through music. Ten ordinary and healthy commuters completed 313 of these interventions on their daily drives over two months. We collected drivers’ immediate and post-driving feedback for each intervention and conducted interviews with the drivers after the end of the study. The results show that both interventions have improved drivers’ well-being. While the participants rated the music intervention very positively, the reception of the mindfulness intervention was more ambivalent.

Although children’s behavioral and mental problems are generally diagnosed in clinical settings, the prediction and awareness of children’s mental wellness in daily settings are getting increased attention. Toy blocks are both accessible in most children’s daily lives and provide physicality as a unique non-verbal channel to express their inner world. In this paper, we propose a toy block approach for predicting a range of behavior problems in young children (4-6 years old) measured by the Child Behavior Checklist (CBCL). We defined and classified a set of quantitative play actions from IMU-embedded toy blocks. Play data collected from 78 preschoolers revealed that specific play actions and patterns indicate total problems, internalizing problems, and aggressive behavior in children. The results align with our qualitative observations, and suggest the potential of predicting the clinical behavior problems of children based on short free-play sessions with sensor-embedded toy blocks.

Research increasingly shows that technology can improve access to mental health interventions. However, unaccompanied migrant youth (UMY) still struggle in accessing appropriate mental health resources in spite of their high need for mental health support. Through co-design workshops, and using the lens of the social-ecological model of resilience, we explored the social-ecological factors that support or hinder UMY’s use of mental health apps as resources. We identified the strong influence of the macro-system (i.e. resettlement policies) on the bio-and micro-systems, which in turn limits participants’ abilities to use the apps. Our findings highlight the factors specific to each social-ecological system—including personal experiences, technological infrastructure and social environment—that need to be accounted for when designing technological mental health resources for UMY. This contributes: a rich description of the interplay of mental health apps with social-ecological systems in which UMY are embedded; and the corresponding design considerations.

A growing body of work has emphasized the need for customizability and flexibility in mobile health technologies to increase support user autonomy. However, customization may be burdensome for people with motivational and cognitive challenges, such as those with mental illnesses, and the optimal level and type of customizability are unclear. Based on 32 interviews with people who experience symptoms of depression and anxiety, we examine how individuals use and customize mental health apps to manage their symptoms. Our findings suggest that participants’ engagement with the apps is affected by their level of energy and motivation, depending on the severity of symptoms. Customization is deemed desirable when the required user effort does not exceed users’ mental and motivational capacity and when ample resources are available. We discuss how customizable systems can increase autonomy without overburdening users in the context of mental health.

Pregnancy loss is a common yet stigmatized experience. We investigate (non)disclosure of pregnancy loss among LGBTQ people to known ties on identified social media as well as what constitutes ideal socio-technical disclosure environments. LGBTQ persons experiencing loss face intersectional stigma for holding a marginalized sexual and/or gender identity and experiencing pregnancy loss. We interviewed 17 LGBTQ people in the U.S. who used social media and had recently experienced pregnancy loss. We demonstrate how the Disclosure Decision-Making (DDM) framework explains LGBTQ pregnancy loss (non)disclosure decisions, thereby asserting the framework's ability to explain (non)disclosure decisions for those facing intersectional stigma. We illustrate how one's LGBTQ identity shapes (non)disclosure decisions of loss. We argue that social media platforms can better facilitate disclosures about silenced topics by enabling selective disclosure, enabling proxy content moderation, providing education about silenced experiences, and prioritizing such disclosures in news feeds. CAUTION: This paper includes quotes about pregnancy loss.

Digital biomarkers of mental health issues offer many advantages, including timely identification for early intervention, ongoing assessment during treatment, and reducing barriers to assessment stemming from geography, age, fear, or disparities in access to systems of care. Embedding digital biomarkers into games may further increase the reach of digital assessment. In this study, we explore game-based digital biomarkers for social anxiety, based on interaction with a non-player character (NPC). We show that social anxiety affects a player's accuracy and their movement path in a gaming task involving an NPC. Further, we compared first versus third-person camera perspectives and the use of customized versus predefined avatars to explore the influence of common game interface factors on the expression of social anxiety through in-game movements. Our findings provide new insights about how game-based digital biomarkers can be effectively used for social anxiety, affording the benefits of early and ongoing digital assessment.

Individuals in distress adopt varied pathways in pursuit of care that aligns with their individual needs. Prior work has established that the first resource an individual leverages can influence later care and recovery, but less is understood about how the design of a point of care might interact with subsequent pathways to care. We investigate how the design of the Indian mental health helpline system interacts with complex sociocultural factors to marginalize caller needs. We draw on interviews with 18 helpline stakeholders, including individuals who have engaged with helplines in the past, shedding light on how they navigate both technological and structural barriers in pursuit of relief. Finally, we use a design justice framework rooted in Amartya Sen’s conceptualization of realization-focused justice to discuss implications and present recommendations towards the design of technology-mediated points of mental health support.

Prolong Grief Disorder (PGD) is a condition in which mourners are stuck in the grief process for a prolonged period and continue to suffer from an intense, mal-adaptive level of grief. Despite the increased popularity of virtual mourning practices, and subsequently the emergence of HCI research in this area, there is little research looking into how continuing bonds maintained digitally promote or impede bereavement adjustment. Through a one-month diary study and in-depth interviews with 17 participants who recently lost their loved ones, we identified four broad mechanisms of how grievers engage in what we called ”backstage” grieving (as opposed to bereavement through digital public space like social media). We further discuss how this personal and private grieving is important in maintaining emotional well-being hence avoiding developing PGD, as well as possible design opportunities and challenges for future digital tools to support grieving.

Eating disorders (EDs) constitute a mental illness with the highest mortality. Today, mobile health apps provide promising means to ED patients for managing their condition. Apps enable users to monitor their eating habits, thoughts, and feelings, and offer analytic insights for behavior change. However, not only have scholars critiqued the clinical validity of these apps, their underlying design principles are not well understood. Through a review of 34 ED apps, we uncovered 11 different data types ED apps collect, and 9 strategies they employ to support collection and reflection. Drawing upon personal health informatics and visualization frameworks, we found that most apps did not adhere to best practices on what and how data should be collected from and reflected to users, or how data-driven insights should be communicated. Our review offers suggestions for improving the design of ED apps such that they can be useful and meaningful in ED recovery.

A growing body of research in HCI focuses on understanding how social media and other social technologies impact a given user’s mental health, including eating disorders. In this paper, we review the results of an interview study with 10 clinicians spanning various specialties who treat people with eating disorders, in order to understand the clinical contexts of eating disorders and social media use. We found various tensions related to clinician comfort and education into the (mis)use of technologies and balancing the positive and negative aspects of social media use within active disease states as well as in recovery. Understanding these tensions as well as the variation in the current process of diagnosing patients is a critical component in connecting HCI research focused on eating disorders to clinical practice and ultimately assessing how digital self-harm could be addressed clinically in the future.

In 2020, there were widespread Black Lives Matter (BLM) protests in the U.S. Because many attendees were novice protesters, organizations distributed guides for staying safe at a protest, often including security and privacy advice. To understand what advice novice protesters are given, we collected 41 safety guides distributed during BLM protests in spring 2020. We identified 13 classes of digital security and privacy advice in these guides. To understand whether this advice influences protesters, we surveyed 167 BLM protesters. Respondents reported an array of security and privacy concerns, and their concerns were magnified when considering fellow protesters. While most respondents reported being aware of, and following, certain advice (e.g., choosing a strong phone passcode), many were unaware of key advice like using end-to-end encrypted messengers and disabling biometric phone unlocking. Our results can guide future advice and technologies to help novice protesters protect their security and privacy.

This paper provides empirical evidence of a link between the Fear of Missing Out (FoMO) and reluctant privacy behaviours, to help explain a gap between users’ privacy attitudes and their behaviours online (also known as the Privacy Paradox). Using Grounded Theory, we interviewed 25 participants and created a high-level empirically-grounded theory of the relationship between FoMO and reluctant privacy behaviours. We identify three main dimensions in which users feel pressured to participate even when they have privacy concerns, to avoid missing out. We discuss the implications of these results on the design of technologies, and how they may indicate systemic dark design.

Algorithms engineered to leverage rich behavioral and biometric data to predict individual attributes and actions continue to permeate public and private life. A fundamental risk may emerge from misconceptions about the sensitivity of such data, as well as the agency of individuals to protect their privacy when fine-grained (and possibly involuntary) behavior is tracked. In this work, we examine how individuals adjust their behavior when incentivized to avoid the algorithmic prediction of their intent. We present results from a virtual reality task in which gaze, movement, and other physiological signals are tracked. Participants are asked to decide which card to select without an algorithmic adversary anticipating their choice. We find that while participants use a variety of strategies, data collected remains highly predictive of choice (80% accuracy). Additionally, a significant portion of participants became more predictable despite efforts to obfuscate, possibly indicating mistaken priors about the dynamics of algorithmic prediction.

“There’s an app for that” is perhaps the definitive rhetoric of our times. To understand how users navigate the trade-offs involved in using apps that support a variety of everyday activities, we conducted scenario-based semi-structured interviews (n = 25). Despite the technical and regulatory mechanisms that are supposedly meant to empower users to manage their privacy, we found that users express an overarching feeling of resignation regarding privacy matters. Because these apps provide convenience and other benefits, as one participant put it, “there is a very fine line” that marks the divide between feeling empowered in the use of technology and coping with the discomfort and creepiness arising from invasive app behavior. Participants consistently expressed being resigned to disclose data even as they accepted personal responsibility for their own privacy. We apply the findings to discuss the limits of empowerment as a design logic for privacy-oriented solutions.

Research has demonstrated that users’ heuristic decision-making processes cause external factors like defaults and framing to influence the outcome of privacy decisions. Proponents of “privacy nudging” have proposed leveraging these effects to guide users’ decisions. Our research shows that defaults and framing not only influence the outcome of privacy decisions, but also the process of evaluating the contextual factors associated with the decision, effectively making the decision-making process more heuristic. In our analysis of an existing dataset of scenario-based smart home privacy decisions, we demonstrate that defaults and framing not only have a direct effect on participants’ decisions; they also moderate the effect of their cognitive appraisals of the presented scenarios on the decision. These results suggest that nudges like defaults and framing exacerbate the well-researched problem that people often employ heuristics rather than making deliberate privacy decisions, and that privacy-setting interfaces should avoid the effects of heuristic decision-making.

Designing technologies that support the mutual cybersecurity and autonomy of older adults facing cognitive challenges requires close collaboration of partners. As part of research to design a Safety Setting application for older adults with memory loss or mild cognitive impairment (MCI), we use a scenario-based participatory design. Our study builds on previous findings that couples’ approach to memory loss was characterized by a desire for flexibility and choice, and an embrace of role uncertainty. We find that couples don't want a system that fundamentally alters their relationship and are looking to maximize self-surveillance competence and minimize loss of autonomy for their partners. All desire Safety Settings to maintain their mutual safety rather than designating one partner as the target of oversight. Couples are open to more rigorous surveillance if they have control over what types of activities trigger various levels of oversight.

Smart home products aren’t living up to their promise. They claim to transform the way we live, providing convenience, energy efficiency, and safety. However, the reality is significantly less profound and often frustrating. This is particularly apparent in security and privacy experiences: powerlessness, confusion, and annoyance have all been reported.

In order to reduce frustration and help fulfill the promise of smart homes, we need to explore the experience of security and privacy in situ. We analyze an ethnographic study observing six UK households over six months to present a longitudinal view of security and privacy user experiences in smart products. We find inconsistencies in managing security and privacy, e.g., contrasting the ease of granting and difficulty of withholding consent. We identify security and privacy issues in repurposing smart home devices – using devices outside of their initial intended purposes. We conclude with recommendations for design in smart home devices.

We investigate how people’s ‘humor style’ relates to their online photo-sharing behaviors and reactions to ‘privacy primes’. In an online experiment, we queried 437 participants about their humor style, likelihood to share photo-memes, and history of sharing others’ photos. In two treatment conditions, participants were either primed to imagine themselves as the photo-subjects or to consider the photo-subjects’ privacy before sharing memes. We found that participants who frequently use aggressive and self-deprecating humor were more likely to violate others’ privacy by sharing photos. We also replicated the interventions’ paradoxical effects – increasing sharing likelihood – as reported in earlier work and identified the subgroups that demonstrated this behavior through interaction analyses. When primed to consider the subjects’ privacy, only humor deniers (participants who use humor infrequently) demonstrated increased sharing. In contrast, when imagining themselves as the photo-subjects, humor deniers, unlike other participants, did not increase the sharing of photos.

Individuals are known to lie and/or provide untruthful data when providing information online as a way to protect their privacy. Prior studies have attempted to explain when and why individuals lie online. However, no work has examined into how people lie or provide untruthful data online, i.e. the specific strategies they follow to provide untruthful data, or attempted to predict whether people would be truthful or not depending on the specific question/data. To close this gap, we present a large-scale study with over 800 participants. Based on it, we show that it is possible to predict whether users are truthful or not using machine learning with very high accuracy (89.7%). We also identify four main strategies people employ to provide untruthful data and show the factors that influence the choices of their strategies. We discuss the implications of findings and argue that understanding privacy lies at this level can help both users and data collectors.

Smart Home Personal Assistants (SPA) have a complex ecosystem that enables them to carry out various tasks on behalf of the user with just voice commands. SPA capabilities are continually growing, with over a hundred thousand third-party skills in Amazon Alexa, covering several categories, from tasks within the home (e.g. managing smart devices) to tasks beyond the boundaries of the home (e.g. purchasing online, booking a ride). In the SPA ecosystem, information flows through several entities including SPA providers, third-party skills providers, providers of Smart Devices, other users and external parties. Prior studies have not explored privacy norms in the SPA ecosystem, i.e., the acceptability of these information flows. In this paper, we study privacy norms in SPAs based on Contextual Integrity through a large-scale study with 1,738 participants. We also study the influence that the Contextual Integrity parameters and personal factors have on the privacy norms. Further, we identify the similarities in terms of the Contextual Integrity parameters of the privacy norms studied to distill more general privacy norms, which could be useful, for instance, to establish suitable privacy defaults in SPA. We finally provide recommendations for SPA and third-party skill providers based on the privacy norms studied.

Phone numbers are intimately connected to our digital lives. People are increasingly required to disclose their phone number in digital spaces, both commercial and personal. While convenient for companies, the pervasive use of phone numbers as user identifiers also poses privacy, security, and access risks for individuals. In order to understand these risks, we present findings from a qualitative online elicitation study with 195 participants about their negative experiences with phone numbers, the consequences they faced, and how those consequences impacted their behavior. Our participants frequently reported experiencing phone number recycling, unwanted exposure, and temporary loss of access to a phone number. Resulting consequences they faced included harassment, account access problems, and privacy invasions. Based on our findings, we discuss service providers’ faulty assumptions in the use of phone numbers as user identifiers, problems arising from phone number recycling, and provide design and public policy recommendations for mitigating these issues with phone numbers.

Although WhatsApp-based communication is playing an increasingly large role in the professional lives of teachers in low-income schools, the nature of the interactions that occur and how these interactions enable cooperative work are not well understood. We contribute a qualitative analysis of 26 existing WhatsApp groups (35,567 messages) that examines (1) the strategies used to encourage interaction within teacher-focused WhatsApp groups, and (2) how these interactions are sustained by teachers, management, and organizations over a period of time. We use teacher networks and activity awareness model to make sense of WhatsApp-based interactions and show how WhatsApp narrows the gap between management and teachers, leading to additional work and stress for teachers. WhatsApp was also used to circulate polarizing and malicious information, leading to a variety of interesting content moderation strategies. Our findings expand the scope of research on teacher networks to low-income contexts and will inform future interventions that enable cooperative work among teachers.

Configuring community technology to ensure its sustainability has proved challenging. We present a 3-year longitudinal study and evaluation of two independent situated community display networks in rural contexts. We describe how the design of the display systems evolved to reflect the needs and desires of the community. We report on the way stakeholders’ perceptions of the displays changed over time, and examine the community dynamics involved in the administration, maintenance and moderation of the systems. Drawing from our findings, we further explore the role of the community champion and their impact on sustainability and scalability. We provide recommendations for the design of community network display technology that supports democratic inter-community politics and governance, and is sensitive to the hidden emotional labor and social resources that are required from communities to fully adopt and sustain display technology.

Online employment resources are now as important as offline personal and professional networks, which have been pivotal in finding employment. However, it is unclear, which specific online resources are key to employment and how job seekers take advantage of them. Therefore, in an online survey of 768 job seekers, we investigated which online platforms, specific job search phases, behaviors, and job search strategies job seekers used in their job search, and which of these were associated with positive outcomes. We examined whether these results correlated with demographic factors and found differences in online platform use among income, gender, years of education, and race. Our results suggest that higher-income job seekers were more likely to use different strategies and more likely to get callbacks than lower-income job seekers. We raise new questions around demographics and technology and discuss the need for practitioners to design for a wider variety of job seekers.

How do people in a precarious profession leverage technology to grow their business and improve their quality of life? Sex workers sit at the intersection of multiple marginalized identities and make up a sizeable workforce: the United Nations estimates that at least 42 million sex workers are conducting business across the globe. Yet, little research has examined how well technology fulfills sex workers’ business needs in the face of unique social, political, legal, and safety constraints.

We present interviews with 29 sex workers in Germany and Switzerland where such work is legal, offering a first HCI perspective on this population’s use of technology. While our participants demonstrate savvy navigation of online spaces, sex workers encounter frustrating barriers due to an American-dominated internet that enforces puritan values globally. Our findings raise concerns about digital discrimination against sex workers and suggest concrete directions for the design of more inclusive technology.

Crowdsourced design feedback systems are emerging resources for getting large amounts of feedback in a short period of time. Traditionally, the feedback comes in the form of a declarative statement, which often contains positive or negative sentiment. Prior research has shown that overly negative or positive sentiment can strongly influence the perceived usefulness and acceptance of feedback and, subsequently, lead to ineffective design revisions. To enhance the effectiveness of crowdsourced design feedback, we investigate a new approach for mitigating the effects of negative or positive feedback by combining open-ended and thought-provoking questions with declarative feedback statements. We conducted two user studies to assess the effects of question-based feedback on the sentiment and quality of design revisions in the context of graphic design. We found that crowdsourced question-based feedback contains more neutral sentiment than statement-based feedback. Moreover, we provide evidence that presenting feedback as questions followed by statements leads to better design revisions than question- or statement-based feedback alone.

People have scientific questions and folk theories; yet most lack the expertise to investigate them. How might people transform their questions into experiments that inform both science and their lives? This paper demonstrates how online volunteers can collaboratively design and run experiments using a novel social computing system. The Galileo system provides procedural support using three techniques: 1) experimental design workflow that provides just-in-time training; 2) review workflow with scaffolded questions; and 3) automated routines for data collection. We present two empirical investigations: a study and a field deployment with online volunteers across 16 and 8 countries respectively. People generated structurally-sound experiments on personally meaningful topics; three communities ran a week-long experiment each. We identify two key challenges for citizen-led experimentation—supporting different expertise levels and providing recruitment guidance—and provide specific suggestions from the social computing literature. Our results highlight the promise and challenges of citizen-led knowledge work like experimentation.

As cities grow, their people become increasingly distanced from nature except within private and public green spaces. Sensing technologies provide a means to harness curiosity about the animals living in these spaces, and possibly also connect interest to care. Yet little is known as to how people may use these technologies, or the implications for human-nature relations. To learn more, we gave commercial camera traps to ten adult participants to understand how they explored their gardens, what they wanted to learn, and what they did with this knowledge. We discovered trade-offs between control and care; the usefulness of different media and mystery; the temporalities of engaging in natural sensing practice; and a prevalence of sharing media within households. We discuss design for convivial cohabitation with the creatures in our garden. This research contributes to better human-nature relations through citizen sensing, as well as HCI for urban biodiversity conservation.

Wildlife calls are the best witnesses to the health of ecosystems, if only we know how to listen to them. Efforts to understand and inform restoration of healthy ecosystems with environmental audio recordings languish from insufficient tools to learn and identify sounds in recordings. To address this problem, we designed and playtested the Bristle Whistle Challenge prototype with ten players. We explored how to design delightful interactions with audio for gaining awareness of nature sounds and supporting wildlife conservation through citizen science. We found that rather than presenting audio alone, it was necessary to connect sounds to other senses and experiences in creative ways to impart meaning and enhance engagement. We offer recommendations to design creative and contextual interactions with media to build awareness of nature's wonders. We call for greater efforts in interaction design to engage people with nature, which is the key to turning around our environmental crisis.

How to handle gender with machine learning is a controversial topic. A growing critical body of research brought attention to the numerous issues transgender communities face with the adoption of current automatic gender recognition (AGR) systems. In contrast, we explore how such technologies could potentially be appropriated to support transgender practices and needs, especially in non-Western contexts like Japan. We designed a virtual makeup probe to assist transgender individuals with passing, that is to be perceived as the gender they identify as. To understand how such an application might support expressing transgender individuals gender identity or not, we interviewed 15 individuals in Tokyo and found that in the right context and under strict conditions, AGR based systems could assist transgender passing.

Crowdsourcing enables the completion of large-scale and hard-to-automate tasks, while allowing people to earn money. However, 3.6 billion people – a workforce comprising 46.4% of the world population – who could benefit most from this source of income lack the access and literacy to use computers, smartphones, and the internet. In this paper we present Karamad, a voice-based crowdsourcing platform that allows workers in low-resource regions to complete crowd work using low-end phones and receive payments as mobile airtime balance. We explore the usefulness, scalability, and sustainability of Karamad in Pakistan through a 6-month deployment. Without any advertising, training, or airtime subsidies, Karamad organically engaged 725 workers who completed 3,939 tasks (involving 43,006 components) including translations, dataset generation, and surveys on demographics, accessibility, disability, health, employment, and literacy. Collectively, the workers produced a valuable service market for potential customers and included female, unemployed, non-literate, and blind users.

Feedback is an important aspect of design education, and crowdsourcing has emerged as a convenient way to obtain feedback at scale. In this paper, we investigate how crowdsourced design feedback compares to peer design feedback within a design-oriented HCI class and across two metrics: perceived quality and perceived fairness. We also examine the perceived monetary value of crowdsourced feedback, which provides an interesting contrast to the typical requester-centric view of the value of labor on crowdsourcing platforms. Our results reveal that the students (N = 106) perceived the crowdsourced design feedback as inferior to peer design feedback in multiple ways. However, they also identified various positive aspects of the online crowds that peers cannot provide. We discuss the meaning of the findings and provide suggestions for teachers in HCI and other researchers interested in crowd feedback systems on using crowds as a potential complement to peers.

This paper describes results from a study examining how musicians have been affected by the restrictive environments imposed by the COVID-19 pandemic, and the corresponding technological implications. Through a survey of 29 musicians and additional interviews with 7 professional improvisation musicians, we observe the challenges musicians face in 1) finding the technological infrastructure and shared spaces necessary for dynamic, remote creative collaborations, 2) producing the mindset, free time, and sources of inspiration necessary for initiating creativity during times of stress, and 3) maintaining (and potentially expanding) the social connections and creative community vital to creative practice. We also report on how some musicians have creatively leveraged the radical changes in their lives to drive new artistic practices and styles of music. Collectively, these data illustrate the resilience of certain existing remote collaboration and creative tools, while suggesting ways other digital tools could more flexibly accommodate new use cases. The data also suggest the value of tools and services that help people creatively cope with radical changes to their lives and livelihoods.

Cake customization services allow clients to collaboratively personalize cakes with pastry chefs. However, remote (e.g., email) and in-person co-design sessions are prone to miscommunication, due to natural restrictions in visualizing cake size, decoration, and celebration context. This paper presents the design, implementation, and expert evaluation of a social VR application (CakeVR) that allows a client to remotely co-design cakes with a pastry chef, through real-time realistic 3D visualizations. Drawing on expert semi-structured interviews (4 clients, 5 pastry chefs), we distill and incorporate 8 design requirements into our CakeVR prototype. We evaluate CakeVR with 10 experts (6 clients, 4 pastry chefs) using cognitive walkthroughs, and find that it supports ideation and decision making through intuitive size manipulation, color/flavor selection, decoration design, and custom celebration theme fitting. Our findings provide recommendations for enabling co-design in social VR and highlight CakeVR’s potential to transform product design communication through remote interactive and immersive co-design.

Adapting an interface requires taking into account both the positive and negative effects that changes may have on the user. A carelessly picked adaptation may impose high costs to the user – for example, due to surprise or relearning effort – or “trap” the process to a suboptimal design immaturely. However, effects on users are hard to predict as they depend on factors that are latent and evolve over the course of interaction. We propose a novel approach for adaptive user interfaces that yields a conservative adaptation policy: It finds beneficial changes when there are such and avoids changes when there are none. Our model-based reinforcement learning method plans sequences of adaptations and consults predictive HCI models to estimate their effects. We present empirical and simulation results from the case of adaptive menus, showing that the method outperforms both a non-adaptive and a frequency-based policy.

Visual blends are an advanced graphic design technique to seamlessly integrate two objects into one. Existing tools help novices create prototypes of blends, but it is unclear how they would improve them to be higher fidelity. To help novices, we aim to add structure to the iterative improvement process. We introduce a method for improving prototypes that uses secondary design dimensions to explore a structured design space. This method is grounded in the cognitive principles of human visual object recognition. We present VisiFit – a computational design system that uses this method to enable novice graphic designers to improve blends with computationally generated options they can select, adjust, and chain together. Our evaluation shows novices can substantially improve 76% of blends in under 4 minutes. We discuss how the method can be generalized to other blending problems, and how computational tools can support novices by enabling them to explore a structured design space quickly and efficiently.

Visual design provides the backdrop to most of our interactions over the Internet, but has not received as much analytical attention as textual content. Combining computational with qualitative approaches, we investigate the growing concern that visual design of the World Wide Web has homogenized over the past decade. By applying computer vision techniques to a large dataset of representative websites images from 2003–2019, we show that designs have become significantly more similar since 2007, especially for page layouts where the average distance between sites decreased by over 30%. Synthesizing interviews from 11 experienced web design professionals with our computational analyses, we discuss causes of this homogenization including overlap in source code and libraries, color scheme standardization, and support for mobile devices. Our results seek to motivate future discussion of the factors that influence designers and their implications on the future trajectory of web design.

Recent research on creativity support tools (CST) adopts artificial intelligence (AI) that leverages big data and computational capabilities to facilitate creative work. Our work aims to articulate the role of AI in supporting creativity with a case study of an AI-based CST tool in fashion design based on theoretical groundings. We developed AI models by externalizing three cognitive operations (extending, constraining, and blending) that are associated with divergent and convergent thinking. We present FashionQ, an AI-based CST that has three interactive visualization tools (StyleQ, TrendQ, and MergeQ). Through interviews and a user study with 20 fashion design professionals (10 participants for the interviews and 10 for the user study), we demonstrate the effectiveness of FashionQ on facilitating divergent and convergent thinking and identify opportunities and challenges of incorporating AI in the ideation process. Our findings highlight the role and use of AI in each cognitive operation based on professionals’ expertise and suggest future implications of AI-based CST development.

Advertising posters are a commonly used form of information presentation to promote a product. Producing advertising posters often takes much time and effort of designers when confronted with abundant choices of design elements and layouts. This paper presents Vinci, an intelligent system that supports the automatic generation of advertising posters. Given the user-specified product image and taglines, Vinci uses a deep generative model to match the product image with a set of design elements and layouts for generating an aesthetic poster. The system also integrates online editing-feedback that supports users in editing the posters and updating the generated results with their design preference. Through a series of user studies and a Turing test, we found that Vinci can generate posters as good as human designers and that the online editing-feedback improves the efficiency in poster modification.

Representing the semantics of GUI screens and components is crucial to data-driven computational methods for modeling user-GUI interactions and mining GUI designs. Existing GUI semantic representations are limited to encoding either the textual content, the visual design and layout patterns, or the app contexts. Many representation techniques also require significant manual data annotation efforts. This paper presents Screen2Vec, a new self-supervised technique for generating representations in embedding vectors of GUI screens and components that encode all of the above GUI features without requiring manual annotation using the context of user interaction traces. Screen2Vec is inspired by the word embedding method Word2Vec, but uses a new two-layer pipeline informed by the structure of GUIs and interaction traces and incorporates screen- and app-specific metadata. Through several sample downstream tasks, we demonstrate Screen2Vec’s key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks.

What breathes life into an embodied agent or avatar? While body motions such as facial expressions, speech and gestures have been well studied, relatively little attention has been applied to subtle changes due to underlying physiology. We argue that subtle pulse signals are important for creating more lifelike and less disconcerting avatars. We propose a method for animating blood flow patterns, based on a data-driven physiological model that can be used to directly augment the appearance of synthetic avatars and photo-realistic faces. While the changes are difficult for participants to “see”, they significantly more frequently select faces with blood flow as more anthropomorphic and animated than faces without blood flow. Furthermore, by manipulating the frequency of the heart rate in the underlying signal we can change the perceived arousal of the character.

We propose a method that generates a virtual camera layout of a 3D animation scene by following the cinematic intention of a reference video. From a reference video, cinematic features such as the start frame, end frame, framing, camera movement, and the visual features of the subjects are extracted automatically. The extracted information is used to generate the virtual camera layout, which resembles the camera layout of the reference video. Our method handles stylized as well as human characters with body proportions different from those of humans. We demonstrate the effectiveness of our approach with various reference videos and 3D animation scenes. The user evaluation results show that the generated layouts are comparable to layouts created by the artist, allowing us to assert that our method can provide effective assistance to both novice and professional users when positioning a virtual camera.

Showing ads delivers revenue for online content distributors, but ad exposure can compromise user experience and cause user fatigue and frustration. Correctly balancing ads with other content is imperative. Currently, ad allocation relies primarily on demographics and inferred user interests, which are treated as static features and can be privacy-intrusive. This paper uses person-centric and momentary context features to understand optimal ad-timing. In a quasi-experimental study on a three-month longitudinal dataset of 100K Snapchat users, we find ad timing influences ad effectiveness. We draw insights on the relationship between ad effectiveness and momentary behaviors such as duration, interactivity, and interaction diversity. We simulate ad reallocation, finding that our study-driven insights lead to greater value for the platform. This work advances our understanding of ad consumption and bears implications for designing responsible ad allocation systems, improving both user and platform outcomes. We discuss privacy-preserving components and ethical implications of our work.

Recent advances in deep generative neural networks have made it possible for artificial intelligence to actively collaborate with human beings in co-creating novel content (e.g. music, art). While substantial research focuses on (individual) human-AI collaborations, comparatively less research examines how AI can play a role in human-human collaborations during co-creation. In a qualitative lab study, we observed 30 participants (15 pairs) compose a musical phrase in pairs, both with and without AI. Our findings reveal that AI may play important roles in influencing human social dynamics during creativity, including: 1) implicitly seeding a common ground at the start of collaboration, 2) acting as a psychological safety net in creative risk-taking, 3) providing a force for group progress, 4) mitigating interpersonal stalling and friction, and 5) altering users’ collaborative and creative roles. This work contributes to the future of generative AI in social creativity by providing implications for how AI could enrich, impede, or alter creative social dynamics in the years to come.

People increasingly rely on Artificial Intelligence (AI) based systems to aid decision-making in various domains and often face a choice between alternative systems. We explored the effects of users' perception of AI systems' warmth (perceived intent) and competence (perceived ability) on their choices. In a series of studies, we manipulated AI systems' warmth and competence levels. We show that, similar to the judgments of other people, there is often primacy for warmth over competence. Specifically, when faced with a choice between a high-competence system and a high-warmth system, more participants preferred the high-warmth system. Moreover, the precedence of warmth persisted even when the high-warmth system was overtly deficient in its competence compared to an alternative high competence-low warmth system. The current research proposes that it may be vital for AI systems designers to consider and communicate the system's warmth characteristics to its potential users.

Designing intelligent interactive text entry systems often relies on factors that are difficult to estimate or assess using traditional HCI design and evaluation methods. We introduce a complementary approach by adapting function structure models from engineering design. We extend their use by extracting controllable and uncontrollable parameters from function structure models and visualizing their impact using envelope analysis. Function structure models allow designers to understand a system in terms of its functions and flows between functions and decouple functions from function carriers. Envelope analysis allows the designer to further study how parameters affect variables of interest, for example, accuracy, keystroke savings and other dependent variables. We provide examples of function structure models and illustrate a complete envelope analysis by investigating a parameterized function structure model of predictive text entry. We discuss the implications of this design approach for both text entry system design and for critique of system contributions.

Digital text has become one of the primary ways of exchanging knowledge, but text needs to be rendered to a screen to be read. We present AdaptiFont, a human-in-the-loop system that is aimed at interactively increasing readability of text displayed on a monitor. To this end, we first learn a generative font space with non-negative matrix factorization from a set of classic fonts. In this space we generate new true-type-fonts through active learning, render texts with the new font, and measure individual users’ reading speed. Bayesian optimization sequentially generates new fonts on the fly to progressively increase individuals’ reading speed. The results of a user study show that this adaptive font generation system finds regions in the font space corresponding to high reading speeds, that these fonts significantly increase participants’ reading speed, and that the found fonts are significantly different across individual readers.

To create aesthetically pleasing aerial footage, the correct framing of camera targets is crucial. However, current quadrotor camera tools do not consider the 3D extent of actual camera targets in their optimization schemes and simply interpolate between keyframes when generating a trajectory. This can yield videos with aesthetically unpleasing target framing. In this paper, we propose a target framing algorithm that optimizes the quadrotor camera pose such that targets are positioned at desirable screen locations according to videographic compositional rules and entirely visible throughout a shot. Camera targets are identified using a semi-automatic pipeline which leverages a deep-learning-based visual saliency model. A large-scale perceptual study (N ≈ 500) shows that our method enables users to produce shots with a target framing that is closer to what they intended to create and more or as aesthetically pleasing than with the previous state of the art.

According to psychology research, emotional induction has positive implications in many domains such as therapy and education. Our aim in this paper was to manipulate the Regulatory Focus Theory to assess its impact on the induction of regulatory focus related emotions in children in a pretend play scenario with a social robot. The Regulatory Focus Theory suggests that people follow one of two paradigms while attempting to achieve a goal; by seeking gains (promotion focus - associated with feelings of happiness) or by avoiding losses (prevention focus - associated with feelings of fear). We conducted a study with 69 school children in two different conditions (promotion vs. prevention). We succeeded in inducing happiness emotions in the promotion condition and found a resulting positive effect of the induction on children’s social engagement with the robot. We also discuss the important implications of these results in both educational and child robot interaction fields.

This paper uses document theory to analyze Discover, a partially free-to-use application developed by Facebook’s philanthropic initiative Connectivity and released in the Philippines in May 2020. Discover’s design is predicated on the conviction that access to valorized forms of technology—in this case, popular websites viewed via the Internet—promises benefits to marginalized users who are presumed to lack resources needed to fully participate in contemporary informational capitalism. Document theory in HCI provides a framework that allows us to analyze the most popular websites as rendered by Discover. We argue that Discover’s logic of redaction and form moderation reproduces the very structural inequality that access to the Internet frequently claims to ameliorate. We conclude by pointing to potential applications of our approach in research at the intersection of HCI, ICT4D, and political economy.

Across the United States, a growing number of school districts are turning to matching algorithms to assign students to public schools. The designers of these algorithms aimed to promote values such as transparency, equity, and community in the process. However, school districts have encountered practical challenges in their deployment. In fact, San Francisco Unified School District voted to stop using and completely redesign their student assignment algorithm because it was frustrating for families and it was not promoting educational equity in practice. We analyze this system using a Value Sensitive Design approach and find that one reason values are not met in practice is that the system relies on modeling assumptions about families’ priorities, constraints, and goals that clash with the real world. These assumptions overlook the complex barriers to ideal participation that many families face, particularly because of socioeconomic inequalities. We argue that direct, ongoing engagement with stakeholders is central to aligning algorithmic values with real world conditions. In doing so we must broaden how we evaluate algorithms while recognizing the limitations of purely algorithmic solutions in addressing complex socio-political problems.

From robots to autonomous vehicles, the design of artificial moral agents (AMAs) is an area of growing interest to HCI, and rising concern for the implications of deploying moral machines raises questions about which ethical frameworks are being used in AMAs. We performed a literature review to identify and synthesize key themes in how ethical theories have been applied to AMAs. We reviewed 53 papers and performed a thematic analysis to describe and synthesize the current conversation across HCI. We found many describe the value of ethical theories and implement them as technical contributions, but very few conduct empirical studies in real settings. Furthermore, we found AMA development is dominated by two ethical theories: deontology and consequentialism. We argue that the focus on deontology and consequentialism risks creating AMAs based on a narrow set of Western ethical values and concepts at the expense of other forms of moral reasoning.

There is increasing interest within the HCI community about working with living organisms in the design of interactive systems. Bioartists and community lab participants have worked with living organisms for decades. Their motivations for doing so include artistic expression, design innovation, and activism. We interviewed 12 artists, community lab organizers, and researchers who work with or facilitate work with living organisms. Participants expressed perspectives on working with living organisms and described bioart as an effective practice for engaging and informing the public, fostering transdisciplinary collaborations, and facilitating inspiration and learning from organic processes. They also discussed questions of agency and consent, among other ethical issues in this context. Based on our findings, we present future directions to investigate the potential of hybrid living media interfaces to engage and educate human users, open up possibilities for transdisciplinary collaborations, and participate in ethical dialogues on emerging technologies in a new way.

Hybrid interactive systems that combine living and digital components can engage, educate, and inform users, and are of growing interest in the HCI community. Advances in synthetic biology are transforming what is possible to do with these living media interfaces (LMIs). Bioart is a practice in which artists, often using synthetic biology methods, work with living organisms to creatively explore the human relationship with nonhuman organisms. We present results from an interview study with expert bioartists as well as our hands-on experience in a bioart project where we created poetry-infused wine by encoding and inserting a Persian Sufi poem into the DNA sequence of living yeast cells. We find that engaging in bioart practice generates transdisciplinary fluency with implications for access and activism and our understanding of the qualities of living media. We further explore the qualitative aspects of interacting directly with DNA and implications for sustainable futures.

Makerspaces are being introduced in a wide variety of settings, including community settings such as schools and libraries. Older adults are one group for whom making agendas are being pursued, with envisioned outcomes such as supporting agency and well-being. However, research on making and DIY with older adults typically study individuals who are already engaged in making practices or bring individuals in to a technology environment that has already been created. In this paper, we study the older adult-driven formation of a makerspace in an independent living community. Through an ethnographically-informed approach, we studied the ways that individuals considered appropriate allocation of resources towards a makerspace, scoped activities, evaluated goals, and made trade-offs. Our analysis is centered around describing the way that this makerspace formed as well as three ways that individuals made sense of the makerspace as the planning unfolded: the openness of a space that promises to cater to interests of the population; the promise of a makerspace to involve more residents in technology, but the need to obscure the technology to make it appealing; and a valuation of the return on investment for limited financial and space resources. Our discussion contributes to supporting and studying early adoption of technology by older adults, complicates visions of “making for all,” and presents considerations regarding the often under-specified community of a makerspace.

As digital fabrication machines become widespread, online communities have provided space for diverse practitioners to share their work, troubleshoot, and socialize. These communities pioneer increasingly novel fabrication workflows, and it is critical that we understand and conceptualize these workflows beyond traditional manufacturing models. To this end, we conduct a qualitative study of #PlotterTwitter, an online community developing custom hardware and software tools to create artwork with computer-controlled drawing machines known as plotters. We documented and analyzed emergent themes where the traditional interpretation of digital fabrication workflows fails to capture important nuances and nascent directions. We find that #PlotterTwitter makers champion creative exploration of interwoven digital and physical materials over a predictable series of steps. We discuss how this challenges long-running views of digital fabrication and propose design implications for future frameworks and toolkits to account for this breadth of practice.

This paper turns to one of HCI's central value systems, i.e. its commitments to usefulness and the ideal that technology enables social progress, productivity, and excellence. Specifically, we examine how the seemingly “positive” ideal to make technology “useful” – i.e. to build systems and devices that advance social and technological progress – masks various forms of violence and injustice such as colonial othering, racist exclusions, and exploitation. Drawing from ethnographic research, we show how design and computing methods from design thinking to agile theory and entrepreneurial approaches in tech production and higher education are the latest techniques in the cultivation of useful bodies on behalf of the state, the corporation, the university, and the economy. Aligning with feminist, critical race and critical computing commitments, this paper offers a genealogical approach to show how injustice and violence endure, despite and because of a narrative of progress and positive change.

The centrality of the biometric point of sale (POS) machine in the administration of food security in Indian's public distribution system (PDS) invites scrutiny for its primacy as a non-negotiable artifact in the monthly PDS process. In this paper, I critically examine how the POS machine emerges as a site for varying imaginaries of a technologically-mediated welfare system for the three primary stakeholders of the PDS, consisting of the beneficiaries, dealers, and state administrators. Drawing on ethnographic fieldwork, the paper traces the histories of interaction and portraitures that the three stakeholders bring to their description and interpretation of the POS machine. It shows that an active POS machine provokes the stakeholders in the PDS to view it as an artifact that invites engagement on practical, moral, and knowledge dimensions. The varying ‘biographies’ that stakeholders narrate of the POS machine, collectively reveal the design, disposition, and functioning of a social justice infrastructure that rests on the compulsions of biometric technologies to improve inclusion and deter corruption in welfare delivery.

In this paper we report insights from the design and delivery of a process that invited distinct groups of citizens to co-develop and apply social impact assessment criteria for the purpose of reviewing research proposals on HCI, social justice and digital technologies. We describe our process, designed to create dialogic spaces that foster critical engagements with technologies and social issues, cooperation and peer-support. In our findings we explore how people defined and contextualised social impact in lived experiences, negotiated and legitimised their role as reviewers, and articulated the value of HCI research for social justice. We reflect on the significance of involving citizens in the commissioning of research that addresses inequalities and social justice in technology design and draw implications for HCI researchers concerned with ethical dimensions of technology. The work contributes to HCI and civic engagement's traditions to develop effective participatory methods and collaborative processes to produce digital technologies that support social justice.

There has been growing interest in the application of AI for Social Good, motivated by scarce and unequal resources globally. We focus on the case of AI in frontline health, a Social Good domain that is increasingly a topic of significant attention. We offer a thematic discourse analysis of scientific and grey literature to identify prominent applications of AI in frontline health, motivations driving this work, stakeholders involved, and levels of engagement with the local context. We then uncover design considerations for these systems, drawing from data from three years of ethnographic fieldwork with women frontline health workers and women from marginalized communities in Delhi (India). Finally, we outline an agenda for AI systems that target Social Good, drawing from literature on HCI4D, post-development critique, and transnational feminist theory. Our paper thus offers a critical and ethnographic perspective to inform the design of AI systems that target social impact.

Virtual Reality (VR) holds the promise of immersing people in virtual worlds. However, initial work on the relationship between VR and disability suggests that VR is a body-centric technology that poses barriers for disabled users. We supplement this work with a theoretical analysis of immersive VR through the lens of Surrogate Body theory, a concept from media theory for the structured examination of interactive media in use. Leveraging Critical Disability Studies, particularly the theory of the Minority Body, we explore the assumptions about bodies inherent in VR, and we reflect on implications of these assumptions when disabled people engage with the technology. Our findings show that VR is an inherently ableist technology that assumes a ‘corporeal standard’ (i.e., an ‘ideal’, non-disabled human body), and fails to adequately accommodate disabled people. We conclude with implications for HCI research on VR, and discuss design approaches that foster inclusive technology development.

Topic models are widely used analysis techniques for clustering documents and surfacing thematic elements of text corpora. These models remain challenging to optimize and often require a “human-in-the-loop” approach where domain experts use their knowledge to steer and adjust. However, the fragility, incompleteness, and opacity of these models means even minor changes could induce large and potentially undesirable changes in resulting model. In this paper we conduct a simulation-based analysis of human-centered interactions with topic models, with the objective of measuring the sensitivity of topic models to common classes of user actions. We find that user interactions have impacts that differ in magnitude but often negatively affect the quality of the resulting modelling in a way that can be difficult for the user to evaluate. We suggest the incorporation of sensitivity and “multiverse” analyses to topic model interfaces to surface and overcome these deficiencies. Code and Data Availability: https://osf.io/zgqaw

AutoML systems can speed up routine data science work and make machine learning available to those without expertise in statistics and computer science. These systems have gained traction in enterprise settings where pools of skilled data workers are limited. In this study, we conduct interviews with 29 individuals from organizations of different sizes to characterize how they currently use, or intend to use, AutoML systems in their data science work. Our investigation also captures how data visualization is used in conjunction with AutoML systems. Our findings identify three usage scenarios for AutoML that resulted in a framework summarizing the level of automation desired by data workers with different levels of expertise. We surfaced the tension between speed and human oversight and found that data visualization can do a poor job balancing the two. Our findings have implications for the design and implementation of human-in-the-loop visual analytics approaches.

More visualization systems are simplifying the data analysis process by automatically suggesting relevant visualizations. However, little work has been done to understand if users trust these automated recommendations. In this paper, we present the results of a crowd-sourced study exploring preferences and perceived quality of recommendations that have been positioned as either human-curated or algorithmically generated. We observe that while participants initially prefer human recommenders, their actions suggest an indifference for recommendation source when evaluating visualization recommendations. The relevance of presented information (e.g., the presence of certain data fields) was the most critical factor, followed by a belief in the recommender’s ability to create accurate visualizations. Our findings suggest a general indifference towards the provenance of recommendations, and point to idiosyncratic definitions of visualization quality and trustworthiness that may not be captured by simple measures. We suggest that recommendation systems should be tailored to the information-foraging strategies of specific users.

Root cause analysis is a common data analysis task. While question-answering systems enable people to easily articulate a why question (e.g., why students in Massachusetts have high ACT Math scores on average) and obtain an answer, these systems often produce questionable causal claims. To investigate how such claims might mislead users, we conducted two crowdsourced experiments to study the impact of showing different information on user perceptions of a question-answering system. We found that in a system that occasionally provided unreasonable responses, showing a scatterplot increased the plausibility of unreasonable causal claims. Also, simply warning participants that correlation is not causation seemed to lead participants to accept reasonable causal claims more cautiously. We observed a strong tendency among participants to associate correlation with causation. Yet, the warning appeared to reduce the tendency. Grounded in the findings, we propose ways to reduce the illusion of causality when using question-answering systems.

Creating expressive narrative visualization often requires choosing a well-planned narrative order that invites the audience in. The narrative can either follow the linear order of story events (chronology), or deviate from linearity (anachronies). While evidence exists that anachronies in novels and films can enhance story expressiveness, little is known about how they can be incorporated into narrative visualization. To bridge this gap, this work introduces the idea of narrative linearity to visualization and investigates how different narrative orders affect the expressiveness of time-oriented stories. First, we conducted preliminary interviews with seven experts to clarify the motivations and challenges of manipulating narrative linearity in time-oriented stories. Then, we analyzed a corpus of 80 time-oriented stories and identified six most salient patterns of narrative orders. Next, we conducted a crowdsourcing study with 221 participants. Results indicated that anachronies have the potential to make time-oriented stories more expressive without hindering comprehensibility.

Data videos are a genre of narrative visualization that communicates stories by combining data visualization and motion graphics. While data videos are increasingly gaining popularity, few systematic reviews or structured analyses exist for their design. In this work, we introduce a design space for animated visual narratives in data videos. The design space combines a dimension for animation techniques that are frequently used to facilitate data communication with one for visual narrative strategies served by such animation techniques to support story presentation. We derived our design space from the analysis of 82 high-quality data videos collected from online sources. We conducted a workshop with 20 participants to evaluate the effectiveness of our design space. Qualitative and quantitative feedback suggested that our design space is inspirational and useful for designing and creating data videos.

Using visualization requires people to read abstract visual imagery, estimate statistics, and retain information. However, people with Intellectual and Developmental Disabilities (IDD) often process information differently, which may complicate connecting abstract visual information to real-world quantities. This population has traditionally been excluded from visualization design, and often has limited access to data related to their well being. We explore how visualizations may better serve this population. We identify three visualization design elements that may improve data accessibility: chart type, chart embellishment, and data continuity. We evaluate these elements with populations both with and without IDD, measuring accuracy and efficiency in a web-based online experiment with time series and proportion data. Our study identifies performance patterns and subjective preferences for people with IDD when reading common visualizations. These findings suggest possible solutions that may break the cognitive barriers caused by conventional design guidelines.

Controversial understandings of the coronavirus pandemic have turned data visualizations into a battleground. Defying public health officials, coronavirus skeptics on US social media spent much of 2020 creating data visualizations showing that the government’s pandemic response was excessive and that the crisis was over. This paper investigates how pandemic visualizations circulated on social media, and shows that people who mistrust the scientific establishment often deploy the same rhetorics of data-driven decision-making used by experts, but to advocate for radical policy changes. Using a quantitative analysis of how visualizations spread on Twitter and an ethnographic approach to analyzing conversations about COVID data on Facebook, we document an epistemological gap that leads pro- and anti-mask groups to draw drastically different inferences from similar data. Ultimately, we argue that the deployment of COVID data visualizations reflect a deeper sociopolitical rift regarding the place of science in public life.

In response to COVID-19, a vast number of visualizations have been created to communicate information to the public. Information exposure in a public health crisis can impact people’s attitudes towards and responses to the crisis and risks, and ultimately the trajectory of a pandemic. As such, there is a need for work that documents, organizes, and investigates what COVID-19 visualizations have been presented to the public. We address this gap through an analysis of 668 COVID-19 visualizations. We present our findings through a conceptual framework derived from our analysis, that examines who, (uses) what data, (to communicate) what messages, in what form, under what circumstances in the context of COVID-19 crisis visualizations. We provide a set of factors to be considered within each component of the framework. We conclude with directions for future crisis visualization research.

Interaction enables users to navigate large amounts of data effectively, supports cognitive processing, and increases data representation methods. However, there have been few attempts to empirically demonstrate whether adding interaction to a static visualization improves its function beyond popular beliefs. In this paper, we address this gap. We use a classic Bayesian reasoning task as a testbed for evaluating whether allowing users to interact with a static visualization can improve their reasoning. Through two crowdsourced studies, we show that adding interaction to a static Bayesian reasoning visualization does not improve participants’ accuracy on a Bayesian reasoning task. In some cases, it can significantly detract from it. Moreover, we demonstrate that underlying visualization design modulates performance and that people with high versus low spatial ability respond differently to different interaction techniques and underlying base visualizations. Our work suggests that interaction is not as unambiguously good as we often believe; a well designed static visualization can be as, if not more, effective than an interactive one.

Charts often contain visually prominent features that draw attention to aspects of the data and include text captions that emphasize aspects of the data. Through a crowdsourced study, we explore how readers gather takeaways when considering charts and captions together. We first ask participants to mark visually prominent regions in a set of line charts. We then generate text captions based on the prominent features and ask participants to report their takeaways after observing chart-caption pairs. We find that when both the chart and caption describe a high-prominence feature, readers treat the doubly emphasized high-prominence feature as the takeaway; when the caption describes a low-prominence chart feature, readers rely on the chart and report a higher-prominence feature as the takeaway. We also find that external information that provides context, helps further convey the caption’s message to the reader. We use these findings to provide guidelines for authoring effective chart-caption pairs.

Visualizations designed to make readers compassionate with the persons whose data is represented have been called anthropographics and are commonly employed by practitioners. Empirical studies have recently examined whether anthropographics indeed promote empathy, compassion, or the likelihood of prosocial behavior, but findings have been inconclusive so far. This work contributes a detailed overview of past experiments, and two new experiments that use large samples and a combination of design strategies to maximize the possibility of finding an effect. We tested an information-rich anthropographic against a simple bar chart, asking participants to allocate hypothetical money in a crowdsourcing study. We found that the anthropographic had, at best, a small effect on money allocation. Such a small effect may be relevant for large-scale donation campaigns, but the large sample sizes required to observe an effect and the noise involved in measuring it make it very difficult to study in more depth. Data and code are available at https://osf.io/xqae2/.

This research investigates how people engage with data visualizations when commenting on the social platform Reddit. There has been considerable research on collaborative sensemaking with visualizations and the personal relation of people with data. Yet, little is known about how public audiences without specific expertise and shared incentives openly express their thoughts, feelings, and insights in response to data visualizations. Motivated by the extensive social exchange around visualizations in online communities, this research examines characteristics and motivations of people’s reactions to posts featuring visualizations. Following a Grounded Theory approach, we study 475 reactions from the /r/dataisbeautiful community, identify ten distinguishable reaction types, and consider their contribution to the discourse. A follow-up survey with 168 Reddit users clarified their intentions to react. Our results help understand the role of personal perspectives on data and inform future interfaces that integrate audience reactions into visualizations to foster a public discourse about data.

Infographics range from minimalism that aims to convey the raw data to elaborately decorated, or embellished, graphics that aim to engage readers by telling a story. Several studies have shown evidence to negative, but also positive, effects on embellishments. We conducted a set of experiments to gauge more precisely how embellishments affect how people relate to infographics and make sense of the conveyed story. By analyzing questionnaires, interviews, and eye-tracking data simplified by bundling, we show that, within bounds, embellishments have a positive effect on how users get engaged in understanding an infographic, with very limited downside. To our knowledge, our work is the first that fuses the aforementioned three information sources to understand infographics. Our findings can help to design more fine-grained studies to quantify the effects of embellishments and also to design infographics that effectively use the embellishments’ positive aspects identified. I think the contribution does not appear well in the abstract. It’s not just that the visual embellishments are positive. We show a methodology that allows us to see what these effects are (in addition to engagement, memorization and recall) at several levels (scales, interviews, eye tracking) which are therefore physiological and emotional. We could include this idea in the abstract?

Synthesis, or the integration of hitherto separated elements, is a prominent concept in theories of design processes. Synthesis often happens when there is a need to make a decision, though it is often the result of a combination of different alternatives, instead of deciding in favor of one and eliminating another. In many design studies, synthesis has been investigated in the contexts of everyday design—bicycle frames, sewing machines, commercial architecture. We were interested in how it might apply in contexts of reflective design, whose pragmatics often depend on different interrelationships between users and technological products. In this paper, we argue that designing everyday use objects for reflection requires a synthesis of two apparently opposite forms: conventionally practical forms, since they are everyday use objects, and evocative forms, since they make users think. We provide two examples of everyday objects for reflection that we believe synthesize both conventionally practical and evocative forms, analyzing the design processes that led to these forms, and discussing how these reflective designs embody different forms of synthesis.

In this paper, we use fiction as a method to complicate the commonplace narratives of data as intangible and objective, in the particular context of Internet of Things (IoT) in the home. We, a team of two design researchers, partnered with a fiction writer and a single IoT enthusiast, Susan, to create The Data Epics: four short stories based on Susan's monthly home IoT data logs. The Data Epics revealed new imaginaries for data, showing new world-views and lively data, but also surfaced data's entanglement in meshes and hierarchies, and concerns about control and power. Our work also examines the labor of tending to and interpreting data and a particular interest in anomalies. We conclude with discussions of how data imaginaries from fiction might be imperfect, but are uniquely generative, offering a path to get closer to IoT data by trying things on and zooming in and slowing down.

This paper presents Timelines, a design activity to assist values advocates: people who help others recognize values and ethical concerns as relevant to technical practice. Rather than integrate seamlessly into existing design processes, Timelines aims to create a space for critical reflection and contestation among expert participants (such as technology researchers, practitioners, or students) and a values advocate facilitator to surface the importance and relevance of values and ethical concerns. The activity’s design is motivated by theoretical perspectives from design fiction, scenario planning, and value sensitive design. The activity helps participants surface discussion of broad societal-level changes related to a technology by creating stories from news headlines, and recognize a diversity of experiences situated in the everyday by creating social media posts from different viewpoints. We reflect on how decisions on the activity’s design and facilitation enables it to assist in values advocacy practices.

Speculative design, critical design, and other alternative designs have emerged as popular approaches and burgeoning traditions within HCI and design research. While examples of this work abound, comparatively little theory exists for grasping alternative designs, and for explicating their relation to other types of design and to design in general. In response this paper develops the key concepts of progressional design, frictional design, and design as prefiguration. The progressional conceptualization of design holds that designs have a primary purpose, and that purpose is to ultimately converge toward and ideally arrive at production. The frictional conceptualization of design radically relaxes teleological assumptions and productional expectations. Frictional designs prefigure possibilities that are compellingly resistant to further progression and final production. Prefiguration grounds both progression and friction in the idea that designs are partial, provisional, and potentially preliminary actualizations of possible futures. To illustrate frictional design, this paper outlines a framework of 5 frictional tendencies: diverging, opposing, accelerating, counterfactualizing, and analogizing. These tendencies represent ways in which frictional designs are directionally in tension with the arrow-like vector of progressional design. Several additional concepts are discussed in conclusion to further explicate more nuanced relational potentials between friction and progression: transproductional uses, teleological ambiguity, and relational multiplicity.

We describe the research led by the performance Radical Choreographic Object (RCO). RCO is a participatory performance where the audience members respond to instructions and interactions sent to their mobile phones as well as invitations to dance from the performers.

The performance provides an experimental ground where we question how humans participate to the dance and how they relate to their mobile technologies.

Through observations and conversations with participants during several showings, we show that the participants progressed from obeying and feeling hostage of the interactions to re-interpreting and re-appropriating them. We then discuss these findings in the light of the norms and constraints that are imposed by social behaviours and by the abundance of mobile technologies. We also reflect on the alternatives that emerge where people break free from those norms, embrace eccentricity and dare to dance.

In recent years, Sustainable HCI researchers have begun to investigate “noticing” as a design research method useful in efforts to decenter the human in design. Through an autoethnographic bird watching practice combining field observation, journaling, and making practices, we examine how noticing affects us and our way of relating to birds. We found that bird watching surfaces a feeling of abjection, or a simultaneous repulsion and fascination with a part of oneself one rejects in pursuit of personal growth. Along the way, we honed a practice of attunement through deep listening and field recording, which enabled immersive "ecological" experiences. We offer (1) an account of our method and process, (2) the framework of abjection as an approach to designing amongst the complexity of human/non-human interaction, and (3) reflections on how to design for ecological thinking in the push towards a posthuman design.

From providing nutrition to facilitating social exchanges, food plays an essential role in our daily lives and cultures. In HCI, we are interested in using food as an interaction medium and a context of personal fabrication. Yet, the design space of available food printing methods is limited to shapes with minimal overhangs and materials that have a paste-like consistency. In this work, we seek to expand this design space by adapting support bath-assisted printing to the food context. The bath scaffolds the embedded materials and preserves shapes during the printing processes, enabling us to create freeform food with fluid-like materials. We provide users guidelines for choosing the appropriate support bath type and processing methods depending on the printing material's properties. A design tool suite and application examples, including confectionery arts, 4D printed food, and edible displays are also offered to demonstrate the enabled interaction design space.

The HCI community has a rich history of finding new ways to engage people with data beyond the screen. With our work, we aim to expand the scope of how interaction design can engage people, arguing that “eating data” has the potential to allow people to experience “data as delight”. With reference to prior work and our design research findings, we discuss the advantages and the challenges of this approach to integrating data and food. We then identify four themes to guide the design of engagements with data through food: food form, food commensality, food ephemerality, and emotional response to food. Within these design themes, we articulate twelve insights for interaction designers to use when working on serving data as delight.

We are surrounded by sensing devices. We are accustomed to them, appreciate their benefits, and even create affective bonds and might neglect the implications they might have for our daily life. By presenting Eyecam, an anthropomorphic webcam mimicking a human eye, we challenge conventional relationships with ubiquitous sensing devices and call to re-think how sensing devices might appear and behave. Inspired by critical design, Eyecam is an exaggeration of a familiar sensing device which allows for critical reflections on its perceived functionalities and its impact on human-human and human-device relations. We identify 5 different roles Eyecam can take: Mediator, Observer, Mirror, Presence, and Agent. Contributing design fictions and thinking prompts, we allow for articulation on privacy awareness and intrusion, affect in mediated communication, agency and self-perception along with speculation on potential futures. We envision this work to contribute to a bold and responsible design of ubiquitous sensing devices.

Humans frequently discontinue to use certain technologies, services, or platforms: they deactivate accounts, block content, or find workarounds for un-using functionalities or devices. This is noticeable in contemporary trends, such as digital detox, where an entire market emerged to support users in discontinuing their use. In this essayistic paper, we reflect on a collection of examples where humans disengage with technology by applying Vardouli’s concept of ‘making use’ to the context of discontinued use. We propose making un-use as an epistemological perspective to ‘human-artefact engagements’; one that emphasises un-users as enactors of open-ended, temporarily evolving, and creative activities. We depict making un-use as transformational acts, and discuss epistemological and designerly consequences of making un-use to shed light on a not yet explored site of inquiry and a design space that is about to evolve: a design space for making and makers of un-use.

Since entering the HCI lexicon in the 1990s, Probes have been interpreted and used in divergent ways as a designerly approach to research. While originally positioned as a critique of dominant user-research methods, literature on Probes rarely reflects on such critical dimensions nor explicitly articulates the intents of using Probes as research artifacts. We conducted interviews with 12 design researchers who have worked with Probes within diverse Research through Design projects, exploring direct accounts of how and why Probes are used in practice. Our interviews brought to the fore the critical concerns behind Probe practices in relation to the language of Probing, relationships with participants, and motivations to challenge normative practices. While the pluralistic interpretations of Probes offered by our participants brings challenges, we discuss how making visible the critical motivations of our research opens up new ways of practicing and disseminating Probes.

The use of design fiction to speculate an imaginative and critical future has increasingly been recognized in the design research community. Instead of focusing on speculation with a critical position, this paper reports an experiential approach to entangling everyday experiences in the process of speculating. As science fiction has successfully provided fictional-world-building as entanglement material for speculation, we held a workshop to conduct an entanglement experiment of personal photographs with cyberpunk literature, Neuromancer. We built a card deck consisting of 206 quotes from the novel and invited 15 participants to shuffle, draw, and re-compose sentences that best matched their personal photographs. Purposefully selected everyday anchors and sci-fi features in the quotes allowed us to investigate the moment when an everyday photograph encountered a fictional world. We describe the phenomena of imagination and entanglement, explain experiential entanglement, propose a conceptual model for entangled status, and present interpretations and implications in HCI.

This paper illustrates design work carried out to develop an interactive theater performance. HCI has started to address the challenges of designing interactive performances, as both audience and performers’ experiences are considered and a variety of professional expertise involved. Nevertheless, research has overlooked how such design unfolds in practice, and what role artists play in exploring both the creative opportunities and the challenges associated with interweaving digital technologies. A two-day workshop was conducted to tailor the use of the ChameleonMask, a telepresence technology, within a performance. The analysis highlights the artists’ work to make the mask work while framing, exploring and conceptualizing its use. The discussion outlines the artists’ skills and design expertise, and how they redefine the role of HCI in performance-led research.

Public libraries in the U.S. and around the world are rapidly changing due to expanding technological and social needs of their communities. Furthermore, in recent years, public spaces, including libraries, have faced budgetary and other pressures, putting a strain on the resources available to librarians. As such, public libraries are a compelling cite for uncovering socio-technical entanglements between government services, technology use, public space, and civic participation. In this paper, we report on a qualitative study of librarians in a U.S. urban public library system undergoing substantial renovations and transformations. We frame the work of librarians as a range of infrastructuring practices, which help uncover insights about the socio-material and political conditions of public service in libraries. Using the metaphor of cracking public space open, we contribute a set of design provocations to support an increasingly complex work of public librarians.

While floral tributes are commonly used for the public commemoration of victims of disasters, war, and other accidents, flowers in vases color everyday life. In this research, these features of flowers are intertwined with the recent phenomenon of online memorials to develop a virtual floral tribute concept that includes physical rituals. We designed SenseVase, a smart vase to detect flowers placed in it, and a 3DCG Virtual Memorial that illustrates floral tributes given by people using SenseVases at home. This paper describes how we developed our design concept by reviewing previous literature and social aspects, and presents a video illustrating the concept. To validate the current concept, we interviewed several experts knowledgeable in public commemorations, virtual and online communities, and the floral business. Through a discussion of our findings from the design process and interviews, we propose a new direction for how HCI technology can contribute to public commemoration in addition to personal memorialization.

Funerals are irreplaceable events, especially for bereaved family members and relatives. However, the COVID-19 pandemic has prevented many people worldwide from attending their loved ones’ funerals. The authors had the opportunity to assist one family faced with this predicament by webcasting and recording funeral rites held near Tokyo in June, 2020. Using our original 360-degree Telepresence system and smartphones running Zoom, we enabled the deceased’s elder siblings to remotely attend the funeral and did our utmost to make them feel present in the funeral hall. Despite the webcasting via Zoom contributing more to their remote attendances than our system, we discovered thoughtful findings which could be useful for designing remote funeral attendances. From the findings, we also discuss how HCI designers can contribute to this highly sensitive issue, weaving together knowledge from various domains including techno-spiritual practices, thanato-sensitive designs; and other religious and cultural aspects related to death rituals.