Role of context in usability evaluations: A review
---
paper_title: Measuring usability as quality of use

paper_content:

The conventional assumption that quality is an attribute of a product is misleading, as the attributes required for quality will depend on how the product is used. Quality of use is therefore defined as the extent to which a product satisfies stated and implied needs when used under stated conditions. Quality of use can be used to measure usability as the extent to which specific goals can be achieved with effectiveness, efficiency and satisfaction by specified users carrying out specified tasks in specified environments. Practical and reliable methods of measuring quality of use have been developed by the MUSiC project. These provide criteria for usability which can be incorporated into a quality system. A description is given of the MUSiC methods for specifying the context of use and measuring effectiveness, efficiency, and satisfaction.
---
paper_title: Usability engineering methods for software developers

paper_content:

The human-computer interaction community aims to increase the awareness and acceptance of established methods among software practitioners. Indeed, awareness of the basic usability methods will drive an Information Society for all.
---
paper_title: Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability

paper_content:

We describe an empirical, between-subjects study on the use of think-aloud protocols in usability testing of a federal data-dissemination Web site. This double-blind study used three different types of think-aloud protocols: a traditional protocol, a speech-communication protocol, and a coaching protocol. A silent condition served as the control. Eighty participants were recruited and randomly pre-assigned to one of four conditions. Accuracy and efficiency measures were collected, and participants rated their subjective satisfaction with the site. Results show that accuracy is significantly higher in the coaching condition than in the other conditions. The traditional protocol and the speech-communication protocol are not statistically different from each other with regard to accuracy. Participants in the coaching condition are more satisfied with the Web site than participants in the traditional or speech-communication condition. In addition, there are no significant differences with respect to efficiency (time-on-task). This paper concludes with recommendations for usability practitioners.
---
paper_title: Usability studies of WWW sites: heuristic evaluation vs. laboratory testing

paper_content:

This paper describes the strengths and weaknesses of two usability assessment methods frequently applied to web sites. It uses case histories of WWW usability studies conducted by the authors to illustrate issues of special interest to designers of web sites. The discussion not only compares the two methods, but also discusses how an effective usability process can combine them, applying the methods at different times during site development. PREREQUISITES FOR ASSESSING WEB SITE USABILITY The two methods discussed in this paper for assessing the usability of web sites both require the usability specialist to have three vital pieces of background information: the purpose of the web site; profiles of its intended users; and typical scenarios for users accessing the site. These elements are equally important in evaluating the usability of any product or service, but here is how they apply especially to Web site evaluation. • When discussing the purpose of a web site, it’s helpful to consider three categories. Web sites that supply descriptions of companies (or other organizations) and their products, services, informational offerings, or events can be described as informational sites. Web sites that provide explicit links to extensive databases are called search sites. Web sites that behave like products, where users perform other tasks in addition to reading or retrieving information, are referred to as transactional sites. Multi-purpose sites blur these boundaries. • Unfortunately, the definition of user in our increasingly Web-centric environment is becoming more vague, because “anyone can access the site.” However, we must keep in mind which site visitors are the most likely—or the most welcome—and focus usability efforts on those subgroups. • Finally, the scenario for accessing a web site might be a straightforward URL to a home page, or a more roundabout path through a link in search-engine results to a page deep in the bowels of a site. Evaluators should keep in mind that any web page might be the user’s door to that web site. Although users may perform more complex tasks in transactional sites, the free-form nature of navigation in any type of web site makes ensuring (and measuring) success more complex in the Web environment. Home
---
paper_title: Usability engineering methods for software developers

paper_content:

The human-computer interaction community aims to increase the awareness and acceptance of established methods among software practitioners. Indeed, awareness of the basic usability methods will drive an Information Society for all.
---
paper_title: Behavior Settings: A Revision and Extension of Roger G. Barker’s “Ecological Psychology”

paper_content:

Forty years of collaboration in research and writing with Roger G. Barker have uniquely qualified the author to revise Barker's classic Ecological Psychology: Concepts and Methods for Studying the Environment of Human Behavior (1968). The author's primary goal has been to provide an up-to-date and comprehensive description of behavior setting theory and method with sufficient detail and illustration to guide new research applications. Barker's presentation of theory and method has been preserved except where changes were required to reflect the advances reported in Barker and Schoggen's Qualities of Community Life (1973). The lengthy report in Ecological Psychology of empirical findings from the study of behavior settings the town of Midwest has been replaced by extensive summaries of the currently available reports of research applications of behavior setting theory. Four new chapters have been added: a chapter be economist Karl A. Fox on the use of behavior settings in social system accounting, an article by Barker on behavior settings that have figured prominently in his career, a chapter that discusses behavior settings in relation to a number of other concepts in social science and the field of environment and behavior, and a final chapter on the need for an eco-behavioral science taken from two papers by Barker.
---
paper_title: A POWERFUL THEORY AND A PARADOX Ecological Psychologists After Barker

paper_content:

M. M. SCOTT is the Herman B Wells endowed professor of educational psychology at Indiana University, Bloomington. She holds a Ph.D. in psychology from Peabody College (now of Vanderbilt University) in Nashville, Tennessee. Her research interests include natural habitat research methodologies, effects of behavior settings on behavior, theories of human development, and the history of psychology. She teaches ecological psychology, human development, and the history of psychology.
---
paper_title: Context-aware applications: from the laboratory to the marketplace

paper_content:

Current hardware developments are making mobile computing increasingly attractive. An important class of mobile applications are context-aware applications: applications that change their behaviour according to the user's present context-their location, who they are with, what the time of day is, and so on. This article is about software design for context-aware applications. Currently most such applications have been crafted by experts in research laboratories. Our aim is to factor out a simple class of context-aware applications and make the creation of these as easy as, say, creating Web pages.
---
paper_title: Context of Use within Usability Activities

paper_content:

Designing for usability involves establishing user requirements for a new system or product, developing design solutions, prototyping the system and the user interface, and testing it with representative users. However, before any usability design or evaluation activity can begin, it is necessary to understand the Context of Use for the product, i.e. the goals of the user community, and the main user, task and environmental characteristics of the situation in which it will be operated. This paper describes the background to, and importance of, understanding Context of Use, and presents a process for performing a context analysis. The method described is particularly aimed at non-experts in the area of user-centred design and evaluation.
---
paper_title: Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability

paper_content:

We describe an empirical, between-subjects study on the use of think-aloud protocols in usability testing of a federal data-dissemination Web site. This double-blind study used three different types of think-aloud protocols: a traditional protocol, a speech-communication protocol, and a coaching protocol. A silent condition served as the control. Eighty participants were recruited and randomly pre-assigned to one of four conditions. Accuracy and efficiency measures were collected, and participants rated their subjective satisfaction with the site. Results show that accuracy is significantly higher in the coaching condition than in the other conditions. The traditional protocol and the speech-communication protocol are not statistically different from each other with regard to accuracy. Participants in the coaching condition are more satisfied with the Web site than participants in the traditional or speech-communication condition. In addition, there are no significant differences with respect to efficiency (time-on-task). This paper concludes with recommendations for usability practitioners.
---
paper_title: Towards situated computing

paper_content:

Situated computing concerns the ability of computing devices to detect, interpret and respond to aspects of the user's local environment. In this paper, we use our recent prototyping experience to identify a number of challenging issues that must be resolved in building wearable computers that support situated applications. The paper is organized into three areas: Sensing the local environment, interpreting sensor data, and realizing the value of situated applications. We conclude that while it is feasible to develop interesting prototypes, there remain many difficulties to overcome before robust systems may be widely deployed.
---
paper_title: Disseminating active map information to mobile hosts

paper_content:

The article describes an active map service (AMS) that supports context-aware computing by providing clients with information about located-objects and how those objects change over time. The authors focus on the communication issues of disseminating information from an active map server to its clients, and in particular, address how to deal with various overload situations that can occur. Simple unicast callbacks to interested clients work well enough if only a few located-objects are moving at any given time and only a few clients wish to know about any given move. However, if many people are moving about in the same region and many clients are interested in their motion, then the AMS may experience overload due to the quadratic nature of the communications involved. This overload affects both the server as well as any slow communications links being used. Mobile distributed computing enables users to interact with many different mobile and stationary computers over the course of the day. Navigating a mobile environment can be aided by active maps that describe the location and characteristics of objects within some region as they change over time. >
---
paper_title: It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field

paper_content:

The distinction between field and laboratory is classical in research methodology. In human-computer interaction, and in usability evaluation in particular, it has been a controversial topic for several years. The advent of mobile devices has revived this topic. Empirical studies that compare evaluations in the two settings are beginning to appear, but they provide very different results. This paper presents results from an experimental comparison of a field-based and a lab-based usability evaluation of a mobile system. The two evaluations were conducted in exactly the same way. The conclusion is that it is definitely worth the hassle to conduct usability evaluations in the field. In the field-based evaluation we identified significantly more usability problems and this setting revealed problems with interaction style and cognitive load that were not identified in the laboratory.
---
paper_title: A Review of Mobile HCI Research Methods

paper_content:

This paper examines and reviews research methods applied within the field of mobile human-computer interaction. The purpose is to provide a snapshot of current practice for studying mobile HCI to identify shortcomings in the way research is conducted and to propose opportunities for future approaches. 102 publications on mobile human-computer interaction research were categorized in a matrix relating their research methods and purpose. The matrix revealed a number of significant trends with a clear bias towards building systems and evaluating them only in laboratory settings, if at all. Also, gaps in the distribution of research approaches and purposes were identified; action research, case studies, field studies and basic research being applied very infrequently. Consequently, we argue that the bias towards building systems and a lack of research for understanding design and use limits the development of cumulative knowledge on mobile human computer interaction. This in turn inhibits future development of the research field as a whole.
---
paper_title: Usability testing with children: Laboratory vs field studies

paper_content:

Usability testing has traditionally been conducted in dedicated laboratory. Alternatively, field study can be an option to conduct the usability testing. As HCI researchers who work with children, we have experiences in conducting usability tests in both natural and laboratory environments. We tested two different children applications in two different environments: testing drawing applications at their preschool and testing an educational game in our usability laboratory. Thus, this paper highlights our experiences in conducting both usability tests. Although the literature suggests that laboratory and field studies should be done for two different purposes, our experiences have told us that we can actually integrate the steps of field studies into the guidelines for lab tests. These steps are particularly useful for the novice designers and researchers who have never done usability evaluations with children before.
---
paper_title: A structured methodology for comparative evaluation of user interface designs using usability criteria and measures

paper_content:

Abstract Previous research in software ergonomics has indicated the importance of evaluating the usability of computer user interfaces. This paper presents a quantitative basis for selecting from among multiple alternative interfaces relative to multiple criteria of usability concern. The proposed model consists of two main phases: the prescreening phase and the evaluation phase. The prescreening phase involves expert judgment-based assessment with qualitative criteria. The first phase uses absolute measurement analytic hierarchy process (AHP) to filter possible alternative interfaces to a reasonable subset. The evaluation phase involves user-based assessment such as user testing, with quantitative criteria. The objective of the second phase is to evaluate a subset of alternatives using objective measures and to select the best alternative. A set of criteria and measures for evaluating the usability of computer software designs is presented. The proposed methodology is demonstrated in the interface design of a database system used to analyze trip cases information of nuclear power plant. Relevance to industry There is currently a focus on the usability of interactive computer software. Software developers, interface designers or human factors engineers often confront the task of comparative evaluation among systems, versions or interface designs. The proposed methodology provides practitioners with a structured approach to select the best interface based on usability criteria and measures.
---
paper_title: Usability evaluation of handheld devices: A case study for a museum application

paper_content:

In this paper, we describe a usability evaluation study of a system involving PDAs, designed to be used in a traditional historical/ cultural mu- seum. The system permits collaboration of small groups of museum visitors through mobile handheld devices. The key characteristics of the system are de- scribed first, which include a server and a client component and a tool that permits authoring of new activities. The usability evaluation study that in- volved typical users revealed some of the limitations of the design. The re- ported findings can be of use to practitioners interested in following similar ap- proaches relating to evaluation of mobile technology.
---
paper_title: Social Context in Usability Evaluations: Concepts, Processes and Products

paper_content:

This thesis addresses social context of usability evaluations. Context plays an important role in usability evaluations. A major part of the context of a usability evaluation is the people involved. This is also often referred to as the social context of the usability evaluation, and although social context is considered important, only little research has been done to identify how it influences usability evaluations. In this thesis I explore how social context affects the process and product of a usability evaluation and explain the findings in terms of the theory of behaviour settings originating from environmental psychology. This thesis consists of five published paper contributions and a summary. In the summary I motivate three research questions addressing three aspects of social context. These research questions are answered through a literature review, four laboratory experiments and a field experiment. Findings from these activities are presented in five published paper contributions. I furthermore introduce the theory of behaviour settings as a tool to help characterise the key concepts of social context which, together with an understanding of usability evaluations, provide the framework spanning my research. I then present and discuss the research methods applied in my research, followed by a conclusion on my three research questions including limitations. The primary results of my research are: 1. Applying the concept of operatives (single leader, multiple leader and joint leader) and non‐operatives (members, spectators, neutrals and potentials) from the theory of behaviour settings to usability evaluations generates an understanding and create an awareness of the level of power possessed by each of the participants in the social context. 2. On the operative level, the verbalisation and collaboration of multiple leaders in usability evaluations are affected by acquaintance, and a break down in collaboration or a decrease in verbalisation may cause the test leader to dynamically switch role during the usability evaluation to compensate. However, the influence of non‐operatives is subject to some uncertainty. 3. A careful composition of social context can successfully support problem identification. However, problem identification differs between user groups as well as between usability evaluation setups.
---
paper_title: Synchronous remote usability testing: a new approach facilitated by virtual worlds

paper_content:

This study proposes a new methodology for conducting synchronous remote usability studies using a three-dimensional virtual usability testing laboratory built using the Open Wonderland toolkit. This virtual laboratory method is then compared with two other commonly used synchronous usability test methods: the traditional lab approach and WebEx, a web-based conferencing and screen sharing approach. A study was conducted with 48 participants in total, 36 test participants and 12 test facilitators. The test participants completed 5 tasks on a simulated e-commerce website. The three methodologies were compared with respect to the following dependent variables: the time taken to complete the tasks; the usability defects identified; the severity of these usability defects; and the subjective ratings from NASA-TLX, presence and post-test subjective questionnaires. The three methodologies agreed closely in terms of the total number defects identified, number of high severity defects identified and the time taken to complete the tasks. However, there was a significant difference in the workload experienced by the test participants and facilitators, with the traditional lab condition imposing the least and the virtual lab and the WebEx conditions imposing similar levels. It was also found that the test participants experienced greater involvement and a more immersive experience in the virtual world condition than the WebEx condition. These ratings were not significantly different from those in the traditional lab condition. The results of this study suggest that participants were productive and enjoyed the virtual lab condition, indicating the potential of a virtual world based approach as an alternative to the conventional approaches for synchronous usability testing.
---
paper_title: Environmental context sensing for usability evaluation in mobile HCI by means of small wireless sensor networks

paper_content:

In usability evaluations, experiments are often conducted in closed laboratory situations to avoid disturbing influences. Due to non-realistic usage assumptions, this approach has important shortcomings when mobile Human Computer Interactions (m-HCI) have to be evaluated. Field studies allow to perform experiments close to real-world conditions, but potentially introduce influences caused by the environment. In this paper, we aim at distinguishing application shortcomings from environmental disturbances which both potentially cause decreased user performance. Our approach is based on monitoring environmental conditions during the usability experiment, such as light, acceleration, sound, temperature, and humidity, and relating them to user actions. Therefore, a mobile context-framework has been developed based on a small Wireless Sensor Network (WSN). First results are presented that point at increased delays and error rates of user tasks under induced environmental disturbances. Additionally, we demonstrate the potential of environmental monitoring for understanding user performance.
---
paper_title: Evaluating the Effects of the Environmental Context-of-Use on Mobile Website Usability

paper_content:

Environmental conditions arguably influence mobile website usability, especially as users on the move may be exposed to fluctuating contextual conditions. Since the aspects of context cannot be readily simulated in a mobile usability study, the development of theories and models explaining how the context of use may affect usability has yet to be sufficiently addressed. In this paper we propose a model hypothesizing on the effects of environmental distractions on the usability of mobile sites. Environmental distractions are categorized into auditory, visual and social and we suggest both direct and moderated effects of these distractions on efficiency, effectiveness, and perceived satisfaction. The paper contributes to the mobile HCI literature by developing a concrete set of empirically testable hypotheses and a method for their subsequent assessment. A preliminary test (N=30) was conducted to investigate our main hypotheses. Results confirmed that environmental distractions have a direct effect on mobile website usability.
---
paper_title: Exploring multimodality in the laboratory and the field

paper_content:

There are new challenges to us, as researchers, on how to design and evaluate new mobile applications because they give users access to powerful computing devices through small interfaces, which typically have limited input facilities. One way of overcoming these shortcomings is to utilize the possibilities of multimodality. We report in this paper how we designed, developed, and evaluated a multimodal mobile application through a combination of laboratory and field studies. This is the first time, as far as we know, that a multimodal application has been developed in such a way. We did this so that we would understand more about where and when users envisioned using different modes of interaction and what problems they may encounter when using an application in context.
---
paper_title: Usability testing with children: Laboratory vs field studies

paper_content:

Usability testing has traditionally been conducted in dedicated laboratory. Alternatively, field study can be an option to conduct the usability testing. As HCI researchers who work with children, we have experiences in conducting usability tests in both natural and laboratory environments. We tested two different children applications in two different environments: testing drawing applications at their preschool and testing an educational game in our usability laboratory. Thus, this paper highlights our experiences in conducting both usability tests. Although the literature suggests that laboratory and field studies should be done for two different purposes, our experiences have told us that we can actually integrate the steps of field studies into the guidelines for lab tests. These steps are particularly useful for the novice designers and researchers who have never done usability evaluations with children before.
---
paper_title: The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

paper_content:

Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is cer...
---
paper_title: Exploring verbalization and collaboration of constructive interaction with children

paper_content:

Constructive interaction provides natural thinking-aloud as test subjects collaborate in pairs to solve tasks. Since children may face difficulties in following instructions for a standard think-aloud test, constructive interaction has been suggested as evaluation method when usability testing with children. However, the relationship between think-aloud and constructive interaction is still poorly understood. We present an experiment that compares think-aloud and constructive interaction. The experiment involves 60 children with three setups where children apply think-aloud or constructive interaction in acquainted and non-acquainted pairs. Our results show that the pairing of children had impact on how the children collaborated in pairs and how they would afterward assess the testing sessions. In some cases, we found that acquainted dyads would perform well as they would more naturally interact and collaborate while in other cases they would have problems in controlling the evaluations.
---
paper_title: Analyzing the interaction between facilitator and participants in two variants of the think-aloud method

paper_content:

This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods.
---
paper_title: Synchronous remote usability testing: a new approach facilitated by virtual worlds

paper_content:

This study proposes a new methodology for conducting synchronous remote usability studies using a three-dimensional virtual usability testing laboratory built using the Open Wonderland toolkit. This virtual laboratory method is then compared with two other commonly used synchronous usability test methods: the traditional lab approach and WebEx, a web-based conferencing and screen sharing approach. A study was conducted with 48 participants in total, 36 test participants and 12 test facilitators. The test participants completed 5 tasks on a simulated e-commerce website. The three methodologies were compared with respect to the following dependent variables: the time taken to complete the tasks; the usability defects identified; the severity of these usability defects; and the subjective ratings from NASA-TLX, presence and post-test subjective questionnaires. The three methodologies agreed closely in terms of the total number defects identified, number of high severity defects identified and the time taken to complete the tasks. However, there was a significant difference in the workload experienced by the test participants and facilitators, with the traditional lab condition imposing the least and the virtual lab and the WebEx conditions imposing similar levels. It was also found that the test participants experienced greater involvement and a more immersive experience in the virtual world condition than the WebEx condition. These ratings were not significantly different from those in the traditional lab condition. The results of this study suggest that participants were productive and enjoyed the virtual lab condition, indicating the potential of a virtual world based approach as an alternative to the conventional approaches for synchronous usability testing.
---
paper_title: Environmental context sensing for usability evaluation in mobile HCI by means of small wireless sensor networks

paper_content:

In usability evaluations, experiments are often conducted in closed laboratory situations to avoid disturbing influences. Due to non-realistic usage assumptions, this approach has important shortcomings when mobile Human Computer Interactions (m-HCI) have to be evaluated. Field studies allow to perform experiments close to real-world conditions, but potentially introduce influences caused by the environment. In this paper, we aim at distinguishing application shortcomings from environmental disturbances which both potentially cause decreased user performance. Our approach is based on monitoring environmental conditions during the usability experiment, such as light, acceleration, sound, temperature, and humidity, and relating them to user actions. Therefore, a mobile context-framework has been developed based on a small Wireless Sensor Network (WSN). First results are presented that point at increased delays and error rates of user tasks under induced environmental disturbances. Additionally, we demonstrate the potential of environmental monitoring for understanding user performance.
---
paper_title: Exploring verbalization and collaboration of constructive interaction with children

paper_content:

Constructive interaction provides natural thinking-aloud as test subjects collaborate in pairs to solve tasks. Since children may face difficulties in following instructions for a standard think-aloud test, constructive interaction has been suggested as evaluation method when usability testing with children. However, the relationship between think-aloud and constructive interaction is still poorly understood. We present an experiment that compares think-aloud and constructive interaction. The experiment involves 60 children with three setups where children apply think-aloud or constructive interaction in acquainted and non-acquainted pairs. Our results show that the pairing of children had impact on how the children collaborated in pairs and how they would afterward assess the testing sessions. In some cases, we found that acquainted dyads would perform well as they would more naturally interact and collaborate while in other cases they would have problems in controlling the evaluations.
---
paper_title: Evaluating the Effects of the Environmental Context-of-Use on Mobile Website Usability

paper_content:

Environmental conditions arguably influence mobile website usability, especially as users on the move may be exposed to fluctuating contextual conditions. Since the aspects of context cannot be readily simulated in a mobile usability study, the development of theories and models explaining how the context of use may affect usability has yet to be sufficiently addressed. In this paper we propose a model hypothesizing on the effects of environmental distractions on the usability of mobile sites. Environmental distractions are categorized into auditory, visual and social and we suggest both direct and moderated effects of these distractions on efficiency, effectiveness, and perceived satisfaction. The paper contributes to the mobile HCI literature by developing a concrete set of empirically testable hypotheses and a method for their subsequent assessment. A preliminary test (N=30) was conducted to investigate our main hypotheses. Results confirmed that environmental distractions have a direct effect on mobile website usability.
---
paper_title: Analyzing the interaction between facilitator and participants in two variants of the think-aloud method

paper_content:

This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods.
---
paper_title: Teachers' involvement in usability testing with children

paper_content:

This paper describes the involvement of teachers as facilitators in user-based evaluation of educational software. Findings from a case study that compare the behaviour of teachers and designers during the evaluation indicate that there is no strong evidence to support the current practice of teachers' tangential participation in the evaluation of educational software. Further research is needed to better understand the role teachers can play in this type of evaluation so that their knowledge and experience can positively contribute to the evaluation process and outcomes, whilst retaining the children's valuable contribution.
---
paper_title: Exploring multimodality in the laboratory and the field

paper_content:

There are new challenges to us, as researchers, on how to design and evaluate new mobile applications because they give users access to powerful computing devices through small interfaces, which typically have limited input facilities. One way of overcoming these shortcomings is to utilize the possibilities of multimodality. We report in this paper how we designed, developed, and evaluated a multimodal mobile application through a combination of laboratory and field studies. This is the first time, as far as we know, that a multimodal application has been developed in such a way. We did this so that we would understand more about where and when users envisioned using different modes of interaction and what problems they may encounter when using an application in context.
---
paper_title: Usability testing with children: Laboratory vs field studies

paper_content:

Usability testing has traditionally been conducted in dedicated laboratory. Alternatively, field study can be an option to conduct the usability testing. As HCI researchers who work with children, we have experiences in conducting usability tests in both natural and laboratory environments. We tested two different children applications in two different environments: testing drawing applications at their preschool and testing an educational game in our usability laboratory. Thus, this paper highlights our experiences in conducting both usability tests. Although the literature suggests that laboratory and field studies should be done for two different purposes, our experiences have told us that we can actually integrate the steps of field studies into the guidelines for lab tests. These steps are particularly useful for the novice designers and researchers who have never done usability evaluations with children before.
---
paper_title: Mobile Interaction Design

paper_content:

Preface. Acknowledgements. PART I: INTRODUCTION. Chapter 1: Possibilities. Key points. 1.1 Introduction. 1.2 What are mobile devices?. 1.2.1 Communication or information device? 1.2.2 Appliance or swiss-army knife? 1.2.3 Cherished device or commodity tool? 1.3 Impoverished or extraordinary interfaces? 1.3.1 The Fastap keypad. 1.3.2 Peephole displays. 1.3.3 Accommodating human capabilities and limitations. 1.4 Impoverishing interactions? 1.4.1 Reasons for poor design. 1.4.2 Impacts of poor design. 1.5 Outline of the rest of this book. Summary. Workshop questions. Designer tips. Chapter 2: Products for people. Key points. 2.1 Introduction. 2.2 Useful. 2.2.1 Function before form. 2.2.2 Evolving uses. 2.3 Usable. 2.3.1 Usable in itself. 2.3.2 Usable in life. 2.4 User experience. 2.4.1 Strong identity. 2.4.2 Interaction as package. 2.5 Technology acceptance. Summary. Workshop questions. Designer tips. Chapter 3: Innovating. Key points. 3.1 Introduction. 3.2 Technology-centered approaches. 3.3 Transferring from the desktop. 3.3.1 Applications. 3.3.2 Interface styles. 3.4 Building on past mobile success. 3.5 Drama. 3.6 Frameworks for human-centered thinking. Summary. Workshop questions. Designer tips. PART II: DEVELOPING EFFECTIVE MOBILE APPLICATIONS. Chapter 4: Interaction design. Key points. 4.1 Introduction. 4.2 Designing what? Designing how? 4.3 Understanding users. 4.3.1 From biology to psychology. 4.3.2 Field studies. 4.3.3 Direct questioning. 4.3.4 Distilling the findings. 4.4 Developing prototype designs. 4.4.1 Shaping and managing the design space. 4.4.2 Prototyping. 4.5 Evaluation. 4.5.1 Testing with users. 4.5.2 Testing in the absence of users. 4.6 Iterative development. 4.7 Multiple viewpoints. 4.7.1 Many techniques and tools. 4.7.2 Many disciplines. 4.7.3 Participation and collaboration. 4.8 From interaction design to deployment. Summary. Workshop questions. Designer tips. Chapter 5: Watching, asking, probing. Key points. 5.1 Introduction. 5.2 Focusing the study. 5.2.1 How focused should the study be? 5.2.2 Finding people to study. 5.3 Top techniques for understanding people. 5.3.1 Observing. 5.3.2 Enquiring. 5.3.3 Diary studies. 5.3.4 Discount methods. 5.3.5 Focus groups. 5.3.6 Creatively engaging methods. 5.4 Making sense of observations. 5.4.1 Activities. 5.4.2 Analysis perspectives. 5.5 Personas and scenarios. 5.5.1 Personas. 5.5.2 Scenarios. Summary. Workshop questions. Designer tips. Chapter 6: Prototypes. Key points. 6.1 Introduction. 6.2 What is a prototype? 6.3 Different prototypes for different purposes. 6.4 Low-fidelity. 6.4.1 Self-checking. 6.4.2 Communicating with others. 6.4.3 Interaction prototyping. 6.4.4 Empowering users. 6.5 Higher-fidelity. 6.5.1 Deciding what to prototype. 6.5.2 Hardware and software integration. 6.6 Finishing the process. 6.6.1 Evolutionary. 6.6.2 Revolutionary. 6.6.3 Process. 6.7 Issues in prototyping. 6.7.1 Some considerations. 6.8 A final note on development. Summary. Workshop questions. Designer tips. Chapter 7: Evaluation. Key points. 7.1 Introduction. 7.2 Classifying evaluation. 7.3 'Quick And Dirty'. 7.4 Conceptual model extraction. 7.5 Direct observation. 7.5.1 Finding out what users are thinking. 7.5.2 How to record observations. 7.5.3 How to not bias the experiment. 7.5.4 Happy users. 7.6 Interviews. 7.7 Questionnaires. 7.8 Non-user methods. 7.8.1 Heuristic evaluation. 7.8.2 No people whatsoever. 7.9 Experimental evaluation. 7.9.1 Hypothesis. 7.9.2 The users. 7.9.3 Tasks. 7.9.4 Experiment design. 7.9.5 Conducting experiments. 7.9.6 Experimental results. 7.10 Considering context - evaluating mobile systems. 7.10.1 Physical context. 7.10.2 Technological context. 7.10.3 Social context. 7.10.4 Other contexts. 7.11 Complementary evaluation. 7.12 Conclusion. Summary. Workshop questions. Designer tips. PART III: DESIGN GALLERY - DIRECTIONS AND GUIDELINES. Chapter 8: Controlling Complex Functions. Key points. 8.1 Introduction. 8.2 Menus and memory. 8.3 Hierarchical menus. 8.3.1 Learning structure. 8.3.2 Improving classification. 8.3.3 Context information. 8.4 Icons. 8.5 Manuals. 8.5.1 Online manuals. 8.5.2 Website manuals. 8.6 No menus?. 8.6.1 Data structures. 8.6.2 Alternatives. 8.6.3 Design ideas - data structures. 8.6.4 Evaluation via experiment. 8.7 More complex menus. 8.8 Some concluding thoughts. Summary. Workshop questions. Designer tips. Chapter 9: Information access. Key points. 9.1 Introduction. 9.2 Small-screen impacts. 9.2.1 Lessons from the past. 9.2.2 Impact on browsing complex content. 9.2.3 Impact on searching. 9.3 Designs for browsing. 9.3.1 Guidelines. 9.3.2 Schemes. 9.3.3 Packaging content. 9.4 Improving search. 9.4.1 Assessing sets of results. 9.4.2 Judging the value of individual results. 9.5 Mobile information ecologies. 9.5.1 Fitting in. 9.5.2 Case study: the laid-back search tool. 9.5.3 Peer-to-peer schemes. Summary. Workshop questions. Designer tips. Chapter 10: Beyond text - using images on mobile devices. Key points. 10.1 Introduction. 10.2 Ethnography. 10.2.1 Where have all the photographs gone? 10.2.2 Digital ethnography. 10.3 Finding photos. 10.3.1 Meta-data. 10.3.2 Meta-data management. 10.4 Browsing photos. 10.4.1 Timeline. 10.4.2 Treemaps. 10.4.3 RSVP. 10.4.4 Speed Dependent Automatic Zooming. 10.5 Downscaling case study. 10.5.1 Arranging photos. 10.5.2 Screen size. 10.5.3 Write once, run anywhere. 10.5.4 Meanwhile, back with the users. 10.5.5 User testing. 10.5.6 Platform. 10.6 Advanced technology. 10.7 What are photos for? 10.7.1 What are we sharing? 10.7.2 Using audio with photographs. 10.7.3 Video. 10.8 Looking to the future. Summary. Workshop questions. Designer tips. Chapter 11: Impacting the community impacting the world. Key points. 11.1 Introduction. 11.2 The digital divide. 11.3 Mobiles work. 11.3.1 The rise and rise of mobile technology. 11.4 Planning a project. 11.5 That culture thing. 11.6 Case studies. 11.6.1 Empowering people - Cybertracker. 11.6.2 Education. 11.6.3 Communitization. 11.7 Call to arms. Summary. Workshop questions. Designer tips. Resources. References. Index.
---

Title: Role of context in usability evaluations: A review

Section 1: INTRODUCTION
Description 1: Provide an overview of the significance of usability in modern gadgets and introduce the concept of usability engineering, explaining its importance in various design contexts.

Section 2: Usability
Description 2: Define usability and discuss its components, including measurements of ease of use, efficiency, learnability, memorability, error rates, and satisfaction levels according to various standards and experts.

Section 3: Usability Evaluation
Description 3: Discuss the purpose of usability evaluation, its benefits, key metrics, and differentiate between in-vitro (laboratory) and in-situ (field) evaluations. Mention the division of evaluation techniques into inspection and test methods.

Section 4: MOTIVATION
Description 4: Introduce the theory of behaviour settings by Roger Garlock Barker and explain its relevance to understanding the effect of context on human behaviour and usability evaluations.

Section 5: CONTEXT IN USABILITY EVALUATION
Description 5: Define context according to different researchers and detail why considering context is crucial in usability evaluations. Highlight how physical and social contexts influence user behaviour and usability outcomes.

Section 6: Physical Context
Description 6: Explore the physical context of usability evaluations, comparing laboratory (controlled) settings with field (natural) settings, along with their respective advantages and disadvantages. Provide examples from previous studies.

Section 7: Social Context
Description 7: Discuss the impact of social context on usability evaluations, including the roles of people involved and how their presence can affect the outcomes. Summarize relevant findings from previous research.

Section 8: RELATED WORK
Description 8: Review previous studies that have addressed the contextual issues in usability evaluations, particularly focusing on the physical and social contexts. Classify the work based on the context they target and summarize the findings.

Section 9: DISCUSSION
Description 9: Provide an analysis of how physical and social contexts affect usability evaluations, referencing Barker's theory of behaviour settings. Discuss the implications of these findings and identify gaps in current research.

Section 10: CONCLUSION
Description 10: Summarize the key points discussed in the paper regarding the role of context in usability evaluations and suggest directions for future research to develop a comprehensive framework considering both physical and social aspects of context.