A review of quality evaluation of digital libraries based on users ' perceptions
---
paper_title: User tags versus expert-assigned subject terms: A comparison of LibraryThing tags and Library of Congress Subject Headings

paper_content:

Social tagging, as a recent approach for creating metadata, has caught the attention of library and information science researchers. Many researchers recommend incorporating social tagging into the library environment and combining folksonomies with formal classification. However, some researchers are concerned with the quality issues of social annotation because of its uncontrolled nature. In this study, we compare social tags created by users from the LibraryThing website with the subject terms assigned by experts according to the Library of Congress Subject Headings (LCSH). The purpose of this study is to examine the difference and connections between social tags and expert-assigned subject terms and further explore the feasibility and obstacles of implementing social tagging in library systems. The results of our study show that it is possible to use social tags to improve the accessibility of library collections. However, the existence of non-subject-related tags may impede the application of social tagging in traditional library cataloguing systems.
---
paper_title: Digital Library Evaluation: Toward an Evolution of Concepts.

paper_content:

While there were many efforts in the research and practices of digital libraries, evaluation was not a conspicuous activity. It is well recognized that digital library evaluation is a complex and difficult undertaking. Challenges facing digital library evaluation are enumerated. A conceptual framework for evaluation is suggested. A review of evaluation efforts in research and practice concentrates on derivation of criteria used in evaluation. Essential requirements for evaluation are stated. Discussed are constructs, context, and criteria of digital libraries: What should we evaluate? For what purpose do we evaluate? Who should evaluate? At what level do we evaluate? Upon what criteria do we evaluate? In addition, included are suggestions for adaptation of criteria from related activities. The article is considered as a part of the evolution of concepts for digital library evaluation.
---
paper_title: Analyzing the past to prepare for the future: Writing a literature review

paper_content:

A review of prior, relevant literature is an essential feature of any academic project. An effective review creates a firm foundation for advancing knowledge. It facilitates theory development, closes areas where a plethora of research exists, and uncovers areas where research is needed.
---
paper_title: Personalisation and recommender systems in digital libraries

paper_content:

Widespread use of the Internet has resulted in digital libraries that are increasingly used by diverse communities of users for diverse purposes and in which sharing and collaboration have become important social elements. As such libraries become commonplace, as their contents and services become more varied, and as their patrons become more experienced with computer technology, users will expect more sophisticated services from these libraries. A simple search function, normally an integral part of any digital library, increasingly leads to user frustration as user needs become more complex and as the volume of managed information increases. Proactive digital libraries, where the library evolves from being passive and untailored, are seen as offering great potential for addressing and overcoming these issues and include techniques such as personalisation and recommender systems. In this paper, following on from the DELOS/NSF Working Group on Personalisation and Recommender Systems for Digital Libraries, which met and reported during 2003, we present some background material on the scope of personalisation and recommender systems in digital libraries. We then outline the working group’s vision for the evolution of digital libraries and the role that personalisation and recommender systems will play, and we present a series of research challenges and specific recommendations and research priorities for the field.
---
paper_title: Insights into translation students' information literacy using the IL-HUMASS survey

paper_content:

In the context of information literacy studies, the research put forward in this paper is focused on the individual and subjective levels of motivation and self-efficacy, together with the sources of learning used by a population of translation students from three Spanish universities. The data were collected by means of the self-evaluating and specific IL-HUMASS survey, and are analysed using a mixed (quantitative and qualitative) case study methodology. From the 26 variables of the survey, the most and least consistent and important ones with regard to motivation and self-efficacy are detected. In general, central tendency scores regarding motivation and self-efficacy increase as respondents advance academically. The most striking opportunities for improvement emerged in relation to the following issues: using bibliographic reference managers, knowing the laws on the use of information, using database managers and communicating in other languages. The surveyed population prefer autonomous learning, with the library being a much underused source of learning.
---
paper_title: Evaluation of digital libraries

paper_content:

Digital libraries (DLs) are new and innovative information systems, under constant development and change, and therefore evaluation is of critical importance to ensure not only their correct evolution but also their acceptance by the user and application communities. The Evaluation activity of the DELOS Network of Excellence has performed a large-scale survey of current DL evaluation activities. This study has resulted in a description of the state of the art in the field, which is presented in this paper. The paper also proposes a new framework for the evaluation of DLs, as well as for recording, describing and analyzing the related research field. The framework includes a methodology for the classification of current evaluation procedures. The objective is to provide a set of flexible and adaptable guidelines for DL evaluation.
---
paper_title: Digital Libraries: A Generic Classification and Evaluation Scheme

paper_content:

Evaluation of digital libraries (DLs) is essential for further development in this area. Whereas previous approaches were restricted to certain facets of the problem, we argue that evaluation of DLs should be based on a broad view of the subject area. For this purpose, we develop a new description scheme using four major dimensions: data/collection, system/technology, users, and usage. For each of these dimensions, we describe the major attributes. Using this scheme, existing DL test beds can be characterised. For this purpose, we have performed a survey by means of a questionnaire, which is now continued by setting up a DL meta-library.
---
paper_title: Usability Assessment of Academic Digital Libraries: Effectiveness, Efficiency, Satisfaction, and Learnability

paper_content:

This study is to develop and evaluate methods and instruments for assessing the usability of digital libraries. It discusses the dimensions of usability, what methods have been applied in evaluating usability of digital libraries, their applicability, and criteria. It is found in the study that there exists an interlocking relationship among effective ness, effi ciency, and satisfaction. It provides operational cri teria for effectiveness, effi ciency, satisfaction, and learn ability. It discovers users’ criteria on ”ease of use,” ”or gani za tion of in formation,” ”terminology and labeling,” ”visual attractiveness,” and ”mistake recovery.” Common causes of ”user lostness” were found. ”Click cost” was examined.
---
paper_title: Evaluation of digital libraries: An overview

paper_content:

The purpose of this report is to provide an overview of works on digital library evaluation that included data. The approach was first to analyze and then isolate (or even deconstruct) some 80 evaluation studies along the lines of: 1. construct that were evaluated; 2. context in which evaluation were conducted; 3. criteria that were chosen as a basis for evaluation, and 4. methods that were used. For each of these lines of analysis approaches taken in evaluation studies are identified, enumerated, and approximately ranked: a list of constructs evaluated as to entities or processes is given; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified, listed and discussed. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital libray research and practice.
---
paper_title: Developing multi-method, iterative, and user-centered evaluation strategies for digital libraries: Functionality, usability, and accessibility

paper_content:

Usability, functionality, and accessibility testing of digital library information services and products are essential for providing high quality services to users. This paper details a long-term, evolving effort to develop meaningful evaluations for assessing digital libraries. The authors of this paper have been engaged in a multi-year study to determine appropriate evaluation techniques, tools, and methodologies for the Florida Electronic Library (FEL) and other digital libraries. The evaluation protocols and approaches have been designed over time and iteratively through assessment efforts of the research team of other digital library initiatives and with multiple versions of the FEL. As such, this paper examines the process of developing, applying, and refining appropriate evaluation methodologies for the networked environment of libraries, as well as the implications of these methodologies. The approach taken in the research described herein relies on a combination of evaluation strategies applied iteratively ::: to assess libraries from the perspective of patron needs. A number of specific methods, as shown in this paper, can be ::: readily developed to provide such evaluations. The following goals guide the presentation in this paper of the ::: development through time of the methods and instruments created, tested, refined, and operationalized in functionality, usability, and accessibility testing by the researchers. The goals of this paper are to: 1) demonstrate the potential roles of multiple, iterative evaluation strategies in the development and refinement of digital libraries; 2) detail the methodologies that focus on how the services meet the needs of users; and 3) encourage further discussion of the uses of these multiple evaluation approaches in assessing these libraries.
---
paper_title: Evaluation of digital libraries: Criteria and problems from users' perspectives

paper_content:

Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research.
---
paper_title: What is usability in the context of the digital library and how can it be measured

paper_content:

This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes.
---
paper_title: Exploring usefulness and usability in the evaluation of open access digital libraries

paper_content:

Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction.
---
paper_title: Usability Assessment of Academic Digital Libraries: Effectiveness, Efficiency, Satisfaction, and Learnability

paper_content:

This study is to develop and evaluate methods and instruments for assessing the usability of digital libraries. It discusses the dimensions of usability, what methods have been applied in evaluating usability of digital libraries, their applicability, and criteria. It is found in the study that there exists an interlocking relationship among effective ness, effi ciency, and satisfaction. It provides operational cri teria for effectiveness, effi ciency, satisfaction, and learn ability. It discovers users’ criteria on ”ease of use,” ”or gani za tion of in formation,” ”terminology and labeling,” ”visual attractiveness,” and ”mistake recovery.” Common causes of ”user lostness” were found. ”Click cost” was examined.
---
paper_title: What is usability in the context of the digital library and how can it be measured

paper_content:

This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes.
---
paper_title: Evaluation of digital libraries: An overview

paper_content:

The purpose of this report is to provide an overview of works on digital library evaluation that included data. The approach was first to analyze and then isolate (or even deconstruct) some 80 evaluation studies along the lines of: 1. construct that were evaluated; 2. context in which evaluation were conducted; 3. criteria that were chosen as a basis for evaluation, and 4. methods that were used. For each of these lines of analysis approaches taken in evaluation studies are identified, enumerated, and approximately ranked: a list of constructs evaluated as to entities or processes is given; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified, listed and discussed. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital libray research and practice.
---
paper_title: Developing multi-method, iterative, and user-centered evaluation strategies for digital libraries: Functionality, usability, and accessibility

paper_content:

Usability, functionality, and accessibility testing of digital library information services and products are essential for providing high quality services to users. This paper details a long-term, evolving effort to develop meaningful evaluations for assessing digital libraries. The authors of this paper have been engaged in a multi-year study to determine appropriate evaluation techniques, tools, and methodologies for the Florida Electronic Library (FEL) and other digital libraries. The evaluation protocols and approaches have been designed over time and iteratively through assessment efforts of the research team of other digital library initiatives and with multiple versions of the FEL. As such, this paper examines the process of developing, applying, and refining appropriate evaluation methodologies for the networked environment of libraries, as well as the implications of these methodologies. The approach taken in the research described herein relies on a combination of evaluation strategies applied iteratively ::: to assess libraries from the perspective of patron needs. A number of specific methods, as shown in this paper, can be ::: readily developed to provide such evaluations. The following goals guide the presentation in this paper of the ::: development through time of the methods and instruments created, tested, refined, and operationalized in functionality, usability, and accessibility testing by the researchers. The goals of this paper are to: 1) demonstrate the potential roles of multiple, iterative evaluation strategies in the development and refinement of digital libraries; 2) detail the methodologies that focus on how the services meet the needs of users; and 3) encourage further discussion of the uses of these multiple evaluation approaches in assessing these libraries.
---
paper_title: Evaluation of digital libraries: Criteria and problems from users' perspectives

paper_content:

Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research.
---
paper_title: Exploring usefulness and usability in the evaluation of open access digital libraries

paper_content:

Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction.
---
paper_title: Evaluation of digital libraries: An overview

paper_content:

The purpose of this report is to provide an overview of works on digital library evaluation that included data. The approach was first to analyze and then isolate (or even deconstruct) some 80 evaluation studies along the lines of: 1. construct that were evaluated; 2. context in which evaluation were conducted; 3. criteria that were chosen as a basis for evaluation, and 4. methods that were used. For each of these lines of analysis approaches taken in evaluation studies are identified, enumerated, and approximately ranked: a list of constructs evaluated as to entities or processes is given; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified, listed and discussed. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital libray research and practice.
---
paper_title: Evaluation of digital libraries: Criteria and problems from users' perspectives

paper_content:

Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research.
---
paper_title: Exploring usefulness and usability in the evaluation of open access digital libraries

paper_content:

Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction.
---
paper_title: Evaluation of digital libraries: Criteria and problems from users' perspectives

paper_content:

Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research.
---
paper_title: Evaluating a monolingual multinational digital library by using usability: An exploratory approach from a developing country

paper_content:

In this paper the authors present a preliminary discussion on some of the results from a survey aimed to explore, describe and explain some of the usability characteristics in digital libraries evaluation in the Mexican context. The study is framed in the evaluation of a multinational and monolingual digital library: the Miguel de Cervantes Virtual Library, from the University of Alicante in Spain. The evaluators were Mexican “expert” users (i.e. Spanish-speaking professional university librarians specialized in electronic reference services) who were asked to carry on an evaluation instrument based on usability criteria as taken from some models in developed countries. Some questions that might be answered with future research are outlined in this paper.
---
paper_title: Analytical usability evaluation for digital libraries: a case study

paper_content:

There are two main kinds of approach to considering usability of any system: empirical and analytical. Empirical techniques involve testing systems with users, whereas analytical techniques involve usability personnel assessing systems using established theories and methods. We report here on a set of studies in which four different techniques were applied to various digital libraries, focusing on the strengths, limitations and scope of each approach. Two of the techniques, Heuristic Evaluation and Cognitive Walkthrough, were applied in text-book fashion, because there was no obvious way to contextualize them to the Digital Libraries (DL) domain. For the third, Claims Analysis, it was possible to develop a set of re-usable scenarios and personas that relate the approach specifically to DL development. The fourth technique, CASSM, relates explicitly to the DL domain by combining empirical data with an analytical approach. We have found that Heuristic Evaluation and Cognitive Walkthrough only address superficial aspects of interface design (but are good for that), whereas Claims Analysis and CASSM can help identify deeper conceptual difficulties (but demand greater skill of the analyst). However, none fit seamlessly with existing digital library development practices, highlighting an important area for further work to support improved usability.
---
paper_title: Evaluation of a Digital Library by Means of Quality Function Deployment (QFD) and the Kano Model.

paper_content:

ABSTRACT This paper proposes utilizing a combination of the Quality Function Deployment (QFD)–Kano model as a useful tool to evaluate service quality. The digital library of the University of Guadalajara (Mexico) is presented as a case study. Data to feed the QFD–Kano model was gathered by an online questionnaire that was made available to users on the library's website. The study makes clear that by listening to the voice of the customer (VOC), relevant information may be obtained about issues that should be improved in order to increase customer satisfaction.
---
paper_title: Evaluating Digital Libraries: A Longitudinal and Multifaceted View

paper_content:

The Perseus Digital Library, (PDL) is one of the primary digital resources for the humanities. Under continuous development since 1987, the project has included an ongoing evaluation component that aims to understand the effects of access to digitized source materials in the humanities. A summary of the PDL genesis and current status is given and the multifaceted and longitudinal evaluation effort is described. A brief synthesis of results is provided and reflections on the evaluation along with recommendations for DL evaluation are given. Introduction Digital libraries marry the missions, techniques, and cultures of physical libraries with the capabilities and cultures of computing and telecommunications. Evaluating digital libraries is a bit like judging how successful is a marriage. A lot depends on how successful the partners are as individuals as well as the emergent conditions made possible by the union. All three entities —the two individuals and the gestalt union--are of course influenced by their context as well. The difficulties arise from the complexity of mutually self-adapting systems interacting in a rich environment. Metrics for success for component parts of a complex system may be distinct from the metrics for success of the marriage (e.g., success for an individual partner is typically necessary but not sufficient to insure success for the marriage). Digital libraries (DLs) are extensions and augmentations of physical libraries (Marchionini & Fox, 1999). As extensions, we might evaluate the individual partners using existing techniques and metrics. Assessing the impacts of libraries on patron's lives and the larger social milieu are the ultimate goals of evaluation but the practical difficulties of assessing such complex and varied impacts causes us to measure the effectiveness and efficiencies of library operations and services as surrogates for these impacts. Metrics such as circulation, collection size and growth rate, patron visits, reference questions answered, patron satisfaction, and financial stability may be used to assess physical library performance in this regard. Clearly, these are metrics that may be points of departure for evaluating digital libraries, but they are not sufficient to characterize the new, rapidly emerging entity. Evaluation criteria for digital technologies can also be useful. For example, metrics such as response time, storage capacity, transfer rate, user satisfaction, and cost per operation may be useful in assessing technological components but may not be sufficient to characterize DL performance, let alone impact. As extensions of physical libraries and digital technologies, these metrics are good starting points but we must look further to consider the effects of DLs as augmentations that provide new services, products, and capabilities.
---
paper_title: What is usability in the context of the digital library and how can it be measured

paper_content:

This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes.
---
paper_title: Exploring usefulness and usability in the evaluation of open access digital libraries

paper_content:

Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction.
---
paper_title: A usability study of the Belgian‐American Research Collection: measuring the functionality of a digital library

paper_content:

This study presents findings with a particular digital library system, the Belgian‐American Research Collection at the University of Wisconsin Digital Collections Center (UWDCC). It focuses on the ways in which a usability study can inform the future design of the collection and other UWDCC projects. The study uses two formal testing instruments: a focus group and a task‐oriented field test. Both of these techniques yielded solid data about user expectations, problems and actions with the collection. While the findings can aid UWDCC designers as they begin to envision another iteration of the Belgian‐American Research Collection, the findings also point to a need for a shift in the development of more complete digital library systems. This study proposes the development of digital library systems that have a strong orientation toward users and usability as well as advanced search interfaces and full metadata. In the end, it is not just about cataloging and organizing information; it is also about presenting information logically and intuitively for your users.
---
paper_title: THE EVALUATION OF AIRLINE SERVICE QUALITY BY FUZZY MCDM

paper_content:

Abstract This study applies the fuzzy set theory to evaluate the service quality of airline. Service quality is a composite of various attributes, among them many intangible attributes are difficult to measure. This characteristic introduces the obstacles for respondent in replying to the survey. In order to overcome the issue, we invite fuzzy set theory into the measurement of performance. By applying AHP in obtaining criteria weight and TOPSIS in ranking, we found the most concerned aspects of service quality are tangible and the least is empathy. The most concerned attribute is courtesy, safety and comfort.
---
paper_title: Evaluating Digital Libraries: A Longitudinal and Multifaceted View

paper_content:

The Perseus Digital Library, (PDL) is one of the primary digital resources for the humanities. Under continuous development since 1987, the project has included an ongoing evaluation component that aims to understand the effects of access to digitized source materials in the humanities. A summary of the PDL genesis and current status is given and the multifaceted and longitudinal evaluation effort is described. A brief synthesis of results is provided and reflections on the evaluation along with recommendations for DL evaluation are given. Introduction Digital libraries marry the missions, techniques, and cultures of physical libraries with the capabilities and cultures of computing and telecommunications. Evaluating digital libraries is a bit like judging how successful is a marriage. A lot depends on how successful the partners are as individuals as well as the emergent conditions made possible by the union. All three entities —the two individuals and the gestalt union--are of course influenced by their context as well. The difficulties arise from the complexity of mutually self-adapting systems interacting in a rich environment. Metrics for success for component parts of a complex system may be distinct from the metrics for success of the marriage (e.g., success for an individual partner is typically necessary but not sufficient to insure success for the marriage). Digital libraries (DLs) are extensions and augmentations of physical libraries (Marchionini & Fox, 1999). As extensions, we might evaluate the individual partners using existing techniques and metrics. Assessing the impacts of libraries on patron's lives and the larger social milieu are the ultimate goals of evaluation but the practical difficulties of assessing such complex and varied impacts causes us to measure the effectiveness and efficiencies of library operations and services as surrogates for these impacts. Metrics such as circulation, collection size and growth rate, patron visits, reference questions answered, patron satisfaction, and financial stability may be used to assess physical library performance in this regard. Clearly, these are metrics that may be points of departure for evaluating digital libraries, but they are not sufficient to characterize the new, rapidly emerging entity. Evaluation criteria for digital technologies can also be useful. For example, metrics such as response time, storage capacity, transfer rate, user satisfaction, and cost per operation may be useful in assessing technological components but may not be sufficient to characterize DL performance, let alone impact. As extensions of physical libraries and digital technologies, these metrics are good starting points but we must look further to consider the effects of DLs as augmentations that provide new services, products, and capabilities.
---
paper_title: A technique for the measurement of attitudes

paper_content:

THIS paper describes a technique which has been developed for the measurement of race prejudice. This technique differs from most prejudice inventories in that it avoids the following assumptions: (a) that the individual can say, to his own or the investigator's satisfaction, "This is how prejudiced I am," and (b) that, to the extent that the individual can accurately assess his degree of antipathy, he will report honestly the findings of such introspection. Most sociologists would perhaps agree that race attitudes rarely reside on a completely articulate level. Even where the individual holds to intellectual or ideological convictions which would seem to leave no room for out-group antipathies, such do persevere. Thus, we may expect the number of Americans who honestly think themselves "unprejudiced" to be considerably larger than effective research would reveal. Moreover, the number who present themselves as unprejudiced probably exceeds considerably the number who honestly, though often inaccurately, see themselves in this light. Most indirect techniques for the measurement of attitudes have their rationale in observations such as these. The instrument to be described here is not, however, indirect in the usual sense of the word; it does not seek responses to items apparently unrelated to the attitudes investigated. We do, however, seek to measure prejudice in a manner less direct than is true of the usual prejudice scale. In our instrument we seek to measure anti-Negro prejudice. Persons are called upon to respond on social distance scales to whites and Negroes who occupy a variety of occupational positions. The measure of prejudice is derived through the summation of the differences in distance responses to Negroes as opposed to whites in the same occupations. Thus, for lack of a better label,
---
paper_title: Site navigation and its impact on the content viewed by the virtual scholar : a deep log analysis

paper_content:

This paper presents early findings of a unique analysis that related questionnaire data to site usage as recorded in the transaction log reports of ScienceDirect, for the same people. Its focus is the differences in the online behaviour of three types of navigational users: those accessing the site via a gateway (either via a reference hyperlink or subject search facility), those using the on site search facility and those employing menus. Towards this end 16,865 sessions were analysed and grouped by navigational entry and compared over three types of online behaviour: the viewing of articles in press (AIP), the number of different journals viewed in a session and the viewing of old material. A strong association was found between form of navigation and behavioural trait. Those using menus were more likely to view AIPs, while those using the search facility were more likely to view a greater number of different journals and were more likely to view older material. This supports a hypothesis proposed by Nicholas et al. (2006) that use of the online searching facility increases the visibility of material irrespective of journal and age and results in a greater use of older material and a more diverse journal use compared to other online and off-line information retrieval methods. Although research has been undertaken on the different strategies that users employ to navigate and find their way around a collection of content (e.g. a digital library), this we believe is the first time the effect of different navigational strategies on outcomes (for example, what is viewed) has been investigated.
---
paper_title: Understanding users’ experiences: evaluation of digital libraries

paper_content:

Evaluation of digital libraries from a user perspective poses immense challenges. Any digital library serves different types of user, with different needs and levels of understanding, and has to be acceptable - not just in terms of basic usability, but also in terms of supporting work as a social, contextualized activity. Different approaches to evaluation are needed for answering different evaluation questions; their broader relationship to design practice also needs to be considered. These issues are discussed in outline in this extended abstract.
---
paper_title: The state of the art in automating usability evaluation of user interfaces

paper_content:

Usability evaluation is an increasingly important part of the user interface design process. However, usability evaluation can be expensive in terms of time and human resources, and automation is therefore a promising way to augment existing approaches. This article presents an extensive survey of usability evaluation methods, organized according to a new taxonomy that emphasizes the role of automation. The survey analyzes existing techniques, identifies which aspects of usability evaluation automation are likely to be of use in future research, and suggests new ways to expand existing approaches to better support usability evaluation.
---
paper_title: Streams, structures, spaces, scenarios, societies (5s): A formal model for digital libraries

paper_content:

Digital libraries (DLs) are complex information systems and therefore demand formal foundations lest development efforts diverge and interoperability suffers. In this article, we propose the fundamental abstractions of Streams, Structures, Spaces, Scenarios, and Societies (5S), which allow us to define digital libraries rigorously and usefully. Streams are sequences of arbitrary items used to describe both static and dynamic (e.g., video) content. Structures can be viewed as labeled directed graphs, which impose organization. Spaces are sets with operations on those sets that obey certain constraints. Scenarios consist of sequences of events or actions that modify states of a computation in order to accomplish a functional requirement. Societies are sets of entities and activities and the relationships among them. Together these abstractions provide a formal foundation to define, relate, and unify concepts---among others, of digital objects, metadata, collections, and services---required to formalize and elucidate "digital libraries". The applicability, versatility, and unifying power of the 5S model are demonstrated through its use in three distinct applications: building and interpretation of a DL taxonomy, informal and formal analysis of case studies of digital libraries (NDLTD and OAI), and utilization as a formal basis for a DL description language.
---
paper_title: User behavior tendencies on data collections in a digital library

paper_content:

We compare the usage of a Digital Library with many different categories of collections, by examining its log files for a period of twenty months, and we conclude that the access points that the users mostly refer to, depend heavily on the type of content of the collection, the detail of the existing meta-data and the target user group. We also found that most users tend to use simple query structures (e.g. only one search term) and very few and primitive operations to accomplish their request. Furthermore, as they get more experienced, they reduce the number of operations in their sessions.
---
paper_title: User-Centered Evaluation Model for Medical Digital Libraries

paper_content:

It remains unclear whether the recent explosion of medical Internet Digital Libraries (DLs), enabled by substantial investments into eHealth by national governments and international agencies, has brought the desired improvements. As ultimately life-critical applications, medical DLs play a crucial role in delivering evidence to professionals and empowering patients. However, little attention has been given to impact evaluation with domain experts in real settings to assess whether they actually make a difference to clinical practice. ::: ::: In this paper we describe a novel evaluation framework --- Impact-ED --- developed at CeRC to fill the gap in impact evaluation research taking into account the community, content, services and technology dimensions of DLs. We present an account of Impact-ED's application in assessing the impact of the National Resource for Infection Control in the UK (NRIC www.nric.org.uk) --- a real-world medical DL used by over 40 000 professionals monthly.
---
paper_title: 5SL: a language for declarative specification and generation of digital libraries

paper_content:

Digital libraries (DLs) are among the most complex kinds of information systems, due in part to their intrinsic multi disciplinary nature. Nowadays DLs are built within monolithic, tightly integrated, and generally inflexible systems -- or by assembling disparate components together in an ad-hoc way, with resulting problems in interoperability and adaptability. More importantly, conceptual modeling, requirements analysis, and software engineering approaches are rarely supported, making it extremely difficult to tailor DL content and behavior to the interests, needs, and preferences of particular communities. In this paper, we address these problems. In particular, we present 5SL, a declarative language for specifying and generating domain-specific digital libraries. 5SL is based on the 5S formal theory for digital libraries and enables high-level specification of DLs in five complementary dimensions, including: the kinds of multimedia information the DL supports (Stream Model); how that information is structured and organized (Structural Model); different logical and presentational properties and operations of DL components (Spatial Model); the behavior of the DL (Scenario Model); and the different societies of actors and managers of services that act together to carry out the DL behavior (Societal Model). The practical feasibility of the approach is demonstrated by the presentation of a 5SL digital library generator for the MARIAN digital library system.
---
paper_title: A transaction log analysis of a digital library

paper_content:

Abstract.As experimental digital library testbeds gain wider acceptance and develop significant user bases, it becomes important to investigate the ways in which users interact with the systems in practice. Transaction logs are one source of usage information, and the information on user behavior can be culled from them both automatically (through calculation of summary statistics) and manually (by examining query strings for semantic clues on search motivations and searching strategy). We have conducted a transaction log analysis on user activity in the Computer Science Technical Reports Collection of the New Zealand Digital Library, and report insights gained and identify resulting search interface design issues. Specifically, we present the user demographics available with our library, discuss the use of operators and search options in queries, and examine patterns in query construction and refinement. We also describe common mistakes in searching, and examine the distribution of query terms appearing in the logs.
---
paper_title: Exploring behavior of E-journal users in science and technology: Transaction log analysis of Elsevier's ScienceDirect OnSite in Taiwan

paper_content:

In the era of digital libraries, Web-based electronic databases have become important resources for education and research, providing functionality and ease of use superior to print products. Analysis of usage of such online systems can provide valuable information on user behavior, and on usage of electronic information in general. Furthermore, the findings can be used to improve effectiveness of these electronic systems and identify areas for improvement, ranging from user interface and functionality to documentation and product training. This article analyzes usage of the Taiwan-based ScienceDirect OnSite E-journal system, one of the largest and most heavily used full-text Science, Technology, and Medicine (STM) databases worldwide.
---
paper_title: Aggregation operators for linguistic weighted information

paper_content:

The aim of this paper is to model the processes of the aggregation of weighted information in a linguistic framework. Three aggregation operators of weighted linguistic information are presented: linguistic weighted disjunction operator, linguistic weighted conjunction operator, and linguistic weighted averaging operator. A study of their axiomatics is presented to demonstrate their rational aggregation.
---
paper_title: Evaluating the effectiveness of visual user interfaces for information retrieval

paper_content:

An integrated visual thesaurus and results browser to support information retrieval was designed using a task model of information searching. The system provided a hierarchical thesaurus with a results cluster display representing similarity between retrieved documents and relevance ranking using a bullseye metaphor. Latent semantic indexing (LSI) was used as the retrieval engine and to calculate the similarity between documents. The design was tested with two information retrieval tasks. User behaviour, performance and attitude were recorded as well as usability problems. The system had few usability problems and users liked the visualizations, but recall performance was poor. The reasons for poor/good performance were investigated by examining user behaviour and search strategies. Better searchers used the visualizations more effectively and spent longer on the task, whereas poorer performances were attributable to poor motivation, difficulty in assessing article relevance and poor use of system visualizations. The bullseye browser display appeared to encourage limited evaluation of article relevance on titles, leading to poor performance. The bullseye display metaphor for article relevance was understood by users; however, they were confused by the concept of similarity searching expressed as visual clusters. The conclusions from the study are that while visual user interfaces for information searching might seem to be usable, they may not actually improve performance. Training and advisor facilities for effective search strategies need to be incorporated to enhance the effectiveness of visual user interfaces for information retrieval.
---
paper_title: Evaluating Digital Libraries: A Longitudinal and Multifaceted View

paper_content:

The Perseus Digital Library, (PDL) is one of the primary digital resources for the humanities. Under continuous development since 1987, the project has included an ongoing evaluation component that aims to understand the effects of access to digitized source materials in the humanities. A summary of the PDL genesis and current status is given and the multifaceted and longitudinal evaluation effort is described. A brief synthesis of results is provided and reflections on the evaluation along with recommendations for DL evaluation are given. Introduction Digital libraries marry the missions, techniques, and cultures of physical libraries with the capabilities and cultures of computing and telecommunications. Evaluating digital libraries is a bit like judging how successful is a marriage. A lot depends on how successful the partners are as individuals as well as the emergent conditions made possible by the union. All three entities —the two individuals and the gestalt union--are of course influenced by their context as well. The difficulties arise from the complexity of mutually self-adapting systems interacting in a rich environment. Metrics for success for component parts of a complex system may be distinct from the metrics for success of the marriage (e.g., success for an individual partner is typically necessary but not sufficient to insure success for the marriage). Digital libraries (DLs) are extensions and augmentations of physical libraries (Marchionini & Fox, 1999). As extensions, we might evaluate the individual partners using existing techniques and metrics. Assessing the impacts of libraries on patron's lives and the larger social milieu are the ultimate goals of evaluation but the practical difficulties of assessing such complex and varied impacts causes us to measure the effectiveness and efficiencies of library operations and services as surrogates for these impacts. Metrics such as circulation, collection size and growth rate, patron visits, reference questions answered, patron satisfaction, and financial stability may be used to assess physical library performance in this regard. Clearly, these are metrics that may be points of departure for evaluating digital libraries, but they are not sufficient to characterize the new, rapidly emerging entity. Evaluation criteria for digital technologies can also be useful. For example, metrics such as response time, storage capacity, transfer rate, user satisfaction, and cost per operation may be useful in assessing technological components but may not be sufficient to characterize DL performance, let alone impact. As extensions of physical libraries and digital technologies, these metrics are good starting points but we must look further to consider the effects of DLs as augmentations that provide new services, products, and capabilities.
---
paper_title: Evaluation of digital libraries: Criteria and problems from users' perspectives

paper_content:

Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research.
---
paper_title: A technique for the measurement of attitudes

paper_content:

THIS paper describes a technique which has been developed for the measurement of race prejudice. This technique differs from most prejudice inventories in that it avoids the following assumptions: (a) that the individual can say, to his own or the investigator's satisfaction, "This is how prejudiced I am," and (b) that, to the extent that the individual can accurately assess his degree of antipathy, he will report honestly the findings of such introspection. Most sociologists would perhaps agree that race attitudes rarely reside on a completely articulate level. Even where the individual holds to intellectual or ideological convictions which would seem to leave no room for out-group antipathies, such do persevere. Thus, we may expect the number of Americans who honestly think themselves "unprejudiced" to be considerably larger than effective research would reveal. Moreover, the number who present themselves as unprejudiced probably exceeds considerably the number who honestly, though often inaccurately, see themselves in this light. Most indirect techniques for the measurement of attitudes have their rationale in observations such as these. The instrument to be described here is not, however, indirect in the usual sense of the word; it does not seek responses to items apparently unrelated to the attitudes investigated. We do, however, seek to measure prejudice in a manner less direct than is true of the usual prejudice scale. In our instrument we seek to measure anti-Negro prejudice. Persons are called upon to respond on social distance scales to whites and Negroes who occupy a variety of occupational positions. The measure of prejudice is derived through the summation of the differences in distance responses to Negroes as opposed to whites in the same occupations. Thus, for lack of a better label,
---
paper_title: User tags versus expert-assigned subject terms: A comparison of LibraryThing tags and Library of Congress Subject Headings

paper_content:

Social tagging, as a recent approach for creating metadata, has caught the attention of library and information science researchers. Many researchers recommend incorporating social tagging into the library environment and combining folksonomies with formal classification. However, some researchers are concerned with the quality issues of social annotation because of its uncontrolled nature. In this study, we compare social tags created by users from the LibraryThing website with the subject terms assigned by experts according to the Library of Congress Subject Headings (LCSH). The purpose of this study is to examine the difference and connections between social tags and expert-assigned subject terms and further explore the feasibility and obstacles of implementing social tagging in library systems. The results of our study show that it is possible to use social tags to improve the accessibility of library collections. However, the existence of non-subject-related tags may impede the application of social tagging in traditional library cataloguing systems.
---
paper_title: What is usability in the context of the digital library and how can it be measured

paper_content:

This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes.
---
paper_title: Likert scales: how to (ab)use them

paper_content:

Dipping my toe into the water of educational research, I have recently used Likert-type rating scales to measure student views on various educational interventions. Likert scales are commonly used to measure attitude, providing a range of responses to a given question or statement . Typically, there are 5 categories of response, from (for example) 1 1⁄4 strongly disagree to 5 1⁄4 strongly agree, although there are arguments in favour of scales with 7 or with an even number of response categories.
---
paper_title: Analyzing Quantitative Data: From Description to Explanation

paper_content:

Introduction About the Book Social Research and Data Analysis Demystifying Basic Concepts Data Analysis in Context Working with Two Data Sets Descriptive Analysis - Univariate Looking for Characteristics Descriptive Analysis - Bivariate Looking for Patterns Explanatory Analysis Looking for Influences Inferential Analysis From Sample to Population Data Reduction Preparing to Answer Research Questions Real Data Analysis Answering Research Questions
---
paper_title: Exploring usefulness and usability in the evaluation of open access digital libraries

paper_content:

Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction.
---
paper_title: Simple Statistics: A Course Book for the Social Sciences

paper_content:

Acknowledgements 1. Why do we need statistics? 2. Measures of central tendency 3. Measures of dispersion 4. The normal distribution 5. Probability 6. What are statistical tests all about? 7. Hypotheses 8. Significance 9. Simple statistical tests 10. What's in a number 11. Two parametric tests 12. Tests of goodness of fit 13. The design of experiments 14. Sampling 15. Correlation 16. In the last analysis ... 17. Operation schedules Answers to exercises Appendix Index.
---
paper_title: Evaluation of a Digital Library by Means of Quality Function Deployment (QFD) and the Kano Model.

paper_content:

ABSTRACT This paper proposes utilizing a combination of the Quality Function Deployment (QFD)–Kano model as a useful tool to evaluate service quality. The digital library of the University of Guadalajara (Mexico) is presented as a case study. Data to feed the QFD–Kano model was gathered by an online questionnaire that was made available to users on the library's website. The study makes clear that by listening to the voice of the customer (VOC), relevant information may be obtained about issues that should be improved in order to increase customer satisfaction.
---
paper_title: A new application of a fuzzy linguistic quality evaluation system in digital libraries

paper_content:

In this contribution, we present a new application based on fuzzy linguistic information to evaluate the quality of digital libraries. The quality evaluation of digital libraries is defined using users' perceptions on the quality of digital services provided through their Web sites. We assume a fuzzy linguistic modeling to represent the users' perception and apply automatic tools of fuzzy computing with words based on some weighted aggregation operators to compute global quality evaluations of digital libraries.
---
paper_title: A Selection Process Based on Additive Consistency to Deal with Incomplete Fuzzy Linguistic Information

paper_content:

In group decision making situations, there may be cases in which experts do not have an in-depth knowledge of the problem to be solved and, as a result, they may present incomplete information. In this paper, we present a new selection process to deal with incomplete fuzzy linguistic information. As part of it, we use an iterative procedure to estimate the missing information. This procedure is guided by the additive consistency property and only uses the preference values provided by the experts. In addition, the additive consistency property is also used to measure the level of consistency of the information provided by the experts. The main novelties of this selection process are both the possibility to manage decision situations under incomplete fuzzy linguistic information and the importance of the experts' preferences in the aggregation processes is modeled by means of the experts' consistency.
---
paper_title: A consensus model for group decision making problems with unbalanced fuzzy linguistic information

paper_content:

Most group decision making (GDM) problems based on linguistic approaches use symmetrically and uniformly distributed linguistic term sets to express experts' opinions. However, there exist problems whose assessments need to be represented by means of unbalanced linguistic term sets, i.e. using term sets that are not uniformly and symmetrically distributed. The aim of this paper is to present a consensus model for GDM problems with unbalanced fuzzy linguistic information. This consensus model is based on both a fuzzy linguistic methodology to deal with unbalanced linguistic term sets and two consensus criteria, consensus degrees, and proximity measures. To do so, we use a new fuzzy linguistic methodology improving another approach to manage unbalanced fuzzy linguistic information,1 (Int. J. Intell. Syst. 22(11) (2007) 1197–1214), which uses the linguistic 2-tuple model as representation base of unbalanced fuzzy linguistic information. In addition, the consensus model presents a feedback mechanism to help experts for reaching the highest degree of consensus possible. There are two main advantages provided by this consensus model. First, its ability to cope with GDM problems with unbalanced fuzzy linguistic information overcoming the problem of finding different discrimination levels in linguistic term sets. Second, it supports the consensus process automatically, avoiding the possible subjectivity that the moderator can introduce in this phase.
---
paper_title: A fuzzy linguistic quality evaluation model for digital libraries

paper_content:

The aim of this paper is to present a model based on a fuzzy linguistic approach to evaluate the quality of digital libraries. The quality evaluation of digital libraries is defined using users' perceptions on the quality of digital services provided through their Web sites. We assume a fuzzy linguistic modeling to represent the users' perception and apply automatic tools of fuzzy computing with words based on the LOWA and LWA operators to compute global quality evaluations of digital libraries.
---
paper_title: Aggregation operators for linguistic weighted information

paper_content:

The aim of this paper is to model the processes of the aggregation of weighted information in a linguistic framework. Three aggregation operators of weighted linguistic information are presented: linguistic weighted disjunction operator, linguistic weighted conjunction operator, and linguistic weighted averaging operator. A study of their axiomatics is presented to demonstrate their rational aggregation.
---
paper_title: Evaluation of digital libraries: An overview

paper_content:

The purpose of this report is to provide an overview of works on digital library evaluation that included data. The approach was first to analyze and then isolate (or even deconstruct) some 80 evaluation studies along the lines of: 1. construct that were evaluated; 2. context in which evaluation were conducted; 3. criteria that were chosen as a basis for evaluation, and 4. methods that were used. For each of these lines of analysis approaches taken in evaluation studies are identified, enumerated, and approximately ranked: a list of constructs evaluated as to entities or processes is given; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified, listed and discussed. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital libray research and practice.
---
paper_title: What is usability in the context of the digital library and how can it be measured

paper_content:

This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes.
---
paper_title: Analytical usability evaluation for digital libraries: a case study

paper_content:

There are two main kinds of approach to considering usability of any system: empirical and analytical. Empirical techniques involve testing systems with users, whereas analytical techniques involve usability personnel assessing systems using established theories and methods. We report here on a set of studies in which four different techniques were applied to various digital libraries, focusing on the strengths, limitations and scope of each approach. Two of the techniques, Heuristic Evaluation and Cognitive Walkthrough, were applied in text-book fashion, because there was no obvious way to contextualize them to the Digital Libraries (DL) domain. For the third, Claims Analysis, it was possible to develop a set of re-usable scenarios and personas that relate the approach specifically to DL development. The fourth technique, CASSM, relates explicitly to the DL domain by combining empirical data with an analytical approach. We have found that Heuristic Evaluation and Cognitive Walkthrough only address superficial aspects of interface design (but are good for that), whereas Claims Analysis and CASSM can help identify deeper conceptual difficulties (but demand greater skill of the analyst). However, none fit seamlessly with existing digital library development practices, highlighting an important area for further work to support improved usability.
---
paper_title: Evaluation of a Digital Library by Means of Quality Function Deployment (QFD) and the Kano Model.

paper_content:

ABSTRACT This paper proposes utilizing a combination of the Quality Function Deployment (QFD)–Kano model as a useful tool to evaluate service quality. The digital library of the University of Guadalajara (Mexico) is presented as a case study. Data to feed the QFD–Kano model was gathered by an online questionnaire that was made available to users on the library's website. The study makes clear that by listening to the voice of the customer (VOC), relevant information may be obtained about issues that should be improved in order to increase customer satisfaction.
---

Title: A Review of Quality Evaluation of Digital Libraries Based on Users' Perceptions

Section 1: Introduction
Description 1: Introduce the concept of digital libraries, their growth, and the focus of the paper on evaluating DL quality based on users' perceptions.

Section 2: Criteria for Quality Evaluation of DLs
Description 2: Outline the criteria proposed to evaluate the quality of DLs from the user's perspective, including a conceptual model for DL evaluation.

Section 3: Usability and Usefulness
Description 3: Discuss the definitions and criteria of usability and usefulness as proposed by various authors within the context of digital libraries.

Section 4: Criteria Prioritization
Description 4: Summarize efforts to identify the most important evaluation criteria from the user's perspective, covering experiments and findings on criteria importance.

Section 5: Criteria Correlation
Description 5: Analyze the possible correlation among evaluation criteria to minimize the number of criteria needed for evaluating DLs.

Section 6: Criteria Measurement and Analysis
Description 6: Explain the approaches to measure and analyze the quality criteria, including qualitative and quantitative methods, Likert scales, and fuzzy linguistic modelling.

Section 7: Discussions and Challenges
Description 7: Discuss the current state and challenges of evaluating DL quality from a user-centric perspective, identifying key areas for future research.

Section 8: Conclusions
Description 8: Summarize the findings of the review, including the criteria used to evaluate DL quality, their importance, and the identified challenges for future research.