A Survey of Human Computation Systems
---
paper_title: KA-CAPTCHA: An Opportunity for Knowledge Acquisition on the Web

paper_content:

Any Web user is a potential knowledge contributor, but it remains a challenge to make them devote their time contributing to some purpose. In order to align individual with social interests, we selected the CAPTCHA Web resource protection application to embed knowledge elicitation within the users' main task of accessing a Web resource. Consequently, unlike previous knowledge acquisition approaches, no extra effort is expected from users since they are already willing to use a CAPTCHA to perform some particular task. We present an application where we extract pictorial knowledge from Web users, and experiments suggest that our approach enables knowledge acquisition while still satisfying CAPTCHA's security requirements.
---
paper_title: CAPTCHA: Using Hard AI Problems for Security

paper_content:

We introduce CAPTCHA, an automated test that humans can pass, but current computer programs can't pass: any program that has high success over a CAPTCHA can be used to solve an unsolved Artificial Intelligence (AI) problem. We provide several novel constructions of CAPTCHAS. Since CAPTCHAs have many applications in practical security, our approach introduces a new class of hard problems that can be exploited for security purposes. Much like research in cryptography has had a positive impact on algorithms for factoring and discrete log, we hope that the use of hard AI problems for security purposes allows us to advance the field of Artificial Intelligence. We introduce two families of Al problems that can be used to construct CAPTCHAS and we show that solutions to such problems can be used for steganographic communication. CAPTCHAS based on these AI problem families, then, imply a win-win situation: either the problems remain unsolved and there is a way to differentiate humans from computers, or the problems are solved and there is a way to communicate covertly on some channels.
---
paper_title: A Web-Based Game for Collecting Music Metadata

paper_content:

Abstract We have designed a web-based game, MajorMiner, that makes collecting descriptions of musical excerpts fun, easy, useful, and objective. Participants describe 10 second clips of songs and score points when their descriptions match those of other participants. The rules were designed to encourage players to be thorough and the clip length was chosen to make judgments objective and specific. To analyse the data, we measured the degree to which binary classifiers could be trained to spot popular tags. We also compared the performance of clip classifiers trained with MajorMiner's tag data to those trained with social tag data from a popular website. On the top 25 tags from each source, MajorMiner's tags were classified correctly 67.2% of the time, while the social tags were classified correctly 62.6% of the time.
---
paper_title: A Game-Based Approach for Collecting Semantic Annotations of Music

paper_content:

Games based on human computation are a valuable tool for collecting semantic information about images. We show how to transfer this idea into the music domain in order to collect high-quality semantic information about songs. We present Listen Game, a online, multiplayer game that measures the semantic relationship between music and words. In the normal mode, a player sees a list of semantically related words (e.g., instruments, emotions, usages, genres) and is asked to pick the best and worst word to describe a song. In the freestyle mode, a user is asked to suggest a new word that describes the music. Each player receives realtime feedback about the agreement amongst all players. We show that we can use the data collected during a two-week pilot study of Listen Game to learn a supervised multiclass labeling (SML) model. We show that this SML model can annotate a novel song with meaningful words and retrieve relevant songs from a database of audio content.
---
paper_title: LabelMe: A Database and Web-Based Tool for Image Annotation

paper_content:

We seek to build a large collection of images with ground truth labels to be used for object detection and recognition research. Such data is useful for supervised learning and quantitative evaluation. To achieve this, we developed a web-based tool that allows easy image annotation and instant sharing of such annotations. Using this annotation tool, we have collected a large dataset that spans many object categories, often containing multiple instances over a wide variety of images. We quantify the contents of the dataset and compare against existing state of the art datasets used for object recognition and detection. Also, we show how to extend the dataset to automatically enhance object labels with WordNet, discover object parts, recover a depth ordering of objects in a scene, and increase the number of labels using minimal user supervision and images from the web.
---
paper_title: CYC: a large-scale investment in knowledge infrastructure

paper_content:

Since 1984, a person-century of effort has gone into building CYC, a universal schema of roughly 10 5 general concepts spanning human reality. Most of the time has been spent codifying knowledge about these concepts; approximately 10 6 commonsense axioms have been handcrafted for and entered into CYC's knowledge base, and millions more have been inferred and cached by CYC. This article examines the fundamental assumptions of doing such a large-scale project, reviews the technical lessons learned by the developers, and surveys the range of applications that are or soon will be enabled by the technology.
---
paper_title: Wildfire Wally: a volunteer computing game

paper_content:

Online casual games can be used to significantly enhance the productivity of volunteer computing. We call games which perform volunteer computing volunteer computing games. We introduce Wildfire Wally, a volunteer computing game capable of solving the maximum clique problem.
---
paper_title: Games with a purpose

paper_content:

Through online games, people can collectively solve large-scale computational problems. Such games constitute a general mechanism for using brain power to solve open problems. In fact, designing such a game is much like designing an algorithm - it must be proven correct, its efficiency can be analyzed, a more efficient version can supersede a less efficient one, and so on. "Games with a purpose" have a vast range of applications in areas as diverse as security, computer vision, Internet accessibility, adult content filtering, and Internet search. Any game designed to address these and other problems must ensure that game play results in a correct solution and, at the same time, is enjoyable. People will play such games to be entertained, not to solve a problem - no matter how laudable the objective
---
paper_title: Designing games with a purpose

paper_content:

Data generated as a side effect of game play also solves computational problems and trains AI algorithms.
---
paper_title: The gopher game: a social, mobile, locative game with user generated content and peer review

paper_content:

This paper introduces 'Gophers', a social game for mobile devices that utilises task oriented gameplay to create a novel entertainment experience. The study combines a number of key research themes: mobile social gaming, acquiring useful data through gameplay and content sharing in mobile settings. The experience of trialling the game in the real world is discussed and the findings from the study are presented.
---
paper_title: Human computation

paper_content:

This talk is about harnessing human brainpower to solve problems that computers cannot. Although computers have advanced dramatically over the last 50 years, they still do not possess basic conceptual intelligence or perceptual capabilities that most humans take for granted. By leveraging human abilities in a novel way, I solve large-scale computational problems and collect data to teach computers basic human talents. To this end, I treat human brains as processors in a distributed system, each performing a small part of a massive computation.
---
paper_title: Designing location-based mobile games with a purpose: collecting geospatial data with CityExplorer

paper_content:

The games with a purpose paradigm proposed by Luis von Ahn [9] is a new approach for game design where useful but boring tasks, like labeling a random image found in the web, are packed within a game to make them entertaining. But there are not only large numbers of internet users that can be used as voluntary data producers but legions of mobile device owners, too. In this paper we describe the design of a location-based mobile game with a purpose: CityExplorer. The purpose of this game is to produce geospatial data that is useful for non-gaming applications like a location-based service. From the analysis of four use case studies of CityExplorer we report that such a purposeful game is entertaining and can produce rich geospatial data collections.
---
paper_title: Agent architecture in social games -- the implementation of subsumption architecture in diplomacy

paper_content:

Social games are challenging for AI research because they involve not only the mechanism of game (taking actions, planning, etc), but also the social aspects including communicating, cooperating and reaching agreements. Some social games also require cunning, duplicity, or bad faith. In this paper we present the implementation of a prototype that utilizes the properties of the subsumption architecture to provide an interesting computer opponent in a social board game -- Diplomacy. The evaluation result indicates that the subsumption architecture is appropriate for social games.
---
paper_title: Playful Geospatial Data Acquisition by Location-based Gaming Communities

paper_content:

The success of Web 2.0 communities demonstrates that the users of an information service are willing to participate in the content creation process on a voluntary basis. In this paper we adapt this idea to the task of collecting geographic data by player communities of location-based games. We identify three types of geographic data that players can collect as part of the game experience: (1) data about the localization/communication network, (2) data about the geographic environment, and (3) related non-geographic information. Furthermore, we present game design patterns that permit to gather all three types of data. The approach is illustrated with the Geogame “CityExplorer”. First empirical results are reported from the evaluation of geographic data acquired over a one-year period with two Geogames, “GeoTicTacToe” and “CityPoker”.
---
paper_title: Verbosity: a game for collecting common-sense facts

paper_content:

We address the problem of collecting a database of ""common-sense facts"" using a computer game. Informally, a common-sense fact is a true statement about the world that is known to most humans: ""milk is white,"" ""touching hot metal hurts,"" etc. Several efforts have been devoted to collecting common-sense knowledge for the purpose of making computer programs more intelligent. Such efforts, however, have not succeeded in amassing enough data because the manual process of entering these facts is tedious. We therefore introduce Verbosity, a novel interactive system in the form of an enjoyable game. People play Verbosity because it is fun, and as a side effect of them playing, we collect accurate common-sense knowledge. Verbosity is an example of a game that not only brings people together for leisure, but also collects useful data for computer science.
---
paper_title: Supporting selective information sharing with people-tagging

paper_content:

We explore the idea of tagging people for selective information sharing in the context of a social-personal information management (SPIM) system. Each new tag applied to a person has a distinctly specifiable visibility, and people tagged with the same key word form a relationship group that can be used as an access control option for each piece of information. We introduce the idea and motivation behind people-tagging, describe a research implementation we have created to experiment with the concept, and provide a preliminary discussion of potential usage and directions for future work.
---
paper_title: Improving Image Search with PHETCH

paper_content:

Keyword-based image search engines are hindered by the lack of proper labeling for images in their indices. In many cases the labels do not agree with the contents of the image itself, since images are generally indexed by their filename and the surrounding text in the Web page. Another popular approach to image search, content based image retrieval, suffers from a gap between the available low level data and the semantic needs of user searches. To overcome these problems we suggest human annotation of images with natural language descriptions. To this end we present Phetch, an engaging multiplayer game that allows people to attach accurate explanatory text captions to arbitrary images on the Web. People play the game because it is fun, and as a side effect we collect valuable information that can be applied towards improving image retrieval. Furthermore, the game can also be used for other novel applications.
---
paper_title: Online Gaming to Apply for Jobs - The Impact of Self- and E-Assessment on Staff Recruitment

paper_content:

The process of recruiting employees has changed since the internet entered enterprises. From simply posting job ads and information on the internet to online application forms and holistic e-recruiting system architectures the way of recruiting has changed a lot. With the introduction of virtual worlds and the increasing number of online games this paper is discussing the next step of e-recruiting: online games to apply for jobs. With a single explorative case study the implementation of self- and e-assessment as online games could be explained. Furthermore with an SEM based on the empirical data of a survey with the Fortune 1,000 companies in Germany (response rate 19.1%) evidence for the motivation of companies to use online games can be provided. Perceived quality improvements, perceived cost savings and perceived time savings as well as perceived usefulness and perceived ease of use are the most important drivers for companies intending to use online games in staff recruitment.
---
paper_title: Learning consensus opinion: mining data from a labeling game

paper_content:

We consider the problem of identifying the consensus ranking for the results of a query, given preferences among those results from a set of individual users. Once consensus rankings are identified for a set of queries, these rankings can serve for both evaluation and training of retrieval and learning systems. We present a novel approach to collecting the individual user preferences over image-search results: we use a collaborative game in which players are rewarded for agreeing on which image result is best for a query. Our approach is distinct from other labeling games because we are able to elicit directly the preferences of interest with respect to image queries extracted from query logs. As a source of relevance judgments, this data provides a useful complement to click data. Furthermore, the data is free of positional biases and is collected by the game without the risk of frustrating users with non-relevant results; this risk is prevalent in standard mechanisms for debiasing clicks. We describe data collected over 34 days from a deployed version of this game that amounts to about 18 million expressed preferences between pairs. Finally, we present several approaches to modeling this data in order to extract the consensus rankings from the preferences and better sort the search results for targeted queries.
---
paper_title: Labeling images with a computer game

paper_content:

We introduce a new interactive system: a game that is fun and can be used to create valuable output. When people play the game they help determine the contents of images by providing meaningful labels for them. If the game is played as much as popular online games, we estimate that most images on the Web can be labeled in a few months. Having proper labels associated with each image on the Web would allow for more accurate image search, improve the accessibility of sites (by providing descriptions of images to visually impaired individuals), and help users block inappropriate images. Our system makes a significant contribution because of its valuable output and because of the way it addresses the image-labeling problem. Rather than using computer vision techniques, which don't work well enough, we encourage people to do the work by taking advantage of their desire to be entertained.
---
paper_title: EyeSpy: supporting navigation through play

paper_content:

This paper demonstrates how useful content can be generated as a by-product of an enjoyable mobile multiplayer game. In EyeSpy, players tag geographic locations with photos or text. By locating the places in which other players' tags were created and 'confirming' them, players earn points for themselves and verify the tags' locations. As a side effect of game-play, EyeSpy produces a collection of recognisable and findable geographic details, in the form of photographs and text tags, that can be repurposed to support navigation tasks. Two user trials of the game successfully produced an archive of geo-located photographs and tags, and in a follow-up experiment we compared performance in a navigation task using photographs from the game, with geo-referenced photos collected from the Flickr website. Our experiences with EyeSpy support reflection upon the design challenges presented by 'human computation' and the production of usable by-products through mobile game-play.
---
paper_title: Dialogue management for social game characters using statecharts

paper_content:

Character development in games tend to refer to those features that augment the character's skills and experiences in order for it to become better equipped to meet the challenges in the game. We mean that the next step should be to provide the characters with social skills, including the ability to interact using natural language. To date an extensive amount of significant work has been done in the field of believable characters, but not specifically aimed at being placed in game worlds. This means that the methods used not necessarily meet the necessary requirements. In this paper we present a method we believe is capable of handling both game flow as well as dialogue flow, namely statecharts. By example we also show how to specify statecharts for describing socially intelligent dialogue behavior.
---
paper_title: Secure distributed human computation

paper_content:

This paper is a preliminary exploration of secure distributed human computation. We consider the general paradigm of using large-scale distributed computation to solve difficult problems, but where humans can act as agents and provide candidate solutions. We are especially motivated by problem classes that appear to be difficult for computers to solve effectively, but are easier for humans; e.g., image analysis, speech recognition, and natural language processing. This paradigm already seems to be employed in several real-world scenarios, but we are unaware of any formal and unified attempt to study it. Nonetheless, this concept spawns interesting research questions in algorithm design, human computer interfaces, and programming language / API design, distributed systems, and cryptography, among other fields. There are also interesting implications for Internet commerce and the B24b model. We describe this research area and suggest a basic framework for the design of such systems. We analyze security and reliability against malicious parties using standard probability theory tools. We then derive design principles using standard decision-theory concepts. Finally, we list extensions and open problems.
---
paper_title: Shared virtual environment (SVE): a framework for developing social games

paper_content:

The communication channels are changing. More and more, face to face communication is being replaced by computer mediated communication. Using mediated communication, the non-verbal communication cues that are available in face to face communication are not. PASION is a European project where mediated communication is augmented. In this paper we present the Shared Virtual Environment (SVE), the framework to be used for the development of two types of applications (social gaming and collaborative work) where augmentation will be studied. The first version of SVE has been used for the development of the first version of a pervasive game for treasure hunting, Bluenet.
---
paper_title: A context-aware recognition survey for data collection using ubiquitous sensors in the home

paper_content:

Identifying what people do in the home can both inform ubiquitous computing application design decisions and provide training data to the machine learning algorithms used in their implementation. This paper describes an unsupervised technique in which contextual information gathered by ubiquitous sensors is used to help users label a multitude of anonymous activity episodes. This context-aware recognition survey is a game-like computer program in which users attempt to correctly guess which activity is happening after seeing a series of symbolic images that represent sensor values generated during the activity. We report a user study of the system, focusing on how well subjects were able to recognize their own activities, the activities of others, and counterfeits that did not correspond to any activity.
---
paper_title: Human-aided computing: utilizing implicit human processing to classify images

paper_content:

In this paper, we present Human-Aided Computing, an approach that uses an electroencephalograph (EEG) device to measure the presence and outcomes of implicit cognitive processing, processing that users perform automatically and may not even be aware of. We describe a classification system and present results from two experiments as proof-of-concept. Results from the first experiment showed that our system could classify whether a user was looking at an image of a face or not, even when the user was not explicitly trying to make this determination. Results from the second experiment extended this to animals and inanimate object categories as well, suggesting generality beyond face recognition. We further show that we can improve classification accuracies if we show images multiple times, potentially to multiple people, attaining well above 90% classification accuracies with even just ten presentations.
---
paper_title: The Structure of Collaborative Tagging Systems

paper_content:

Collaborative tagging describes the process by which many users add metadata in the form of keywords to shared content. Recently, collaborative tagging has grown in popularity on the web, on sites that allow users to tag bookmarks, photographs and other content. In this paper we analyze the structure of collaborative tagging systems as well as their dynamical aspects. Specifically, we discovered regularities in user activity, tag frequencies, kinds of tags used, bursts of popularity in bookmarking and a remarkable stability in the relative proportions of tags within a given url. We also present a dynamical model of collaborative tagging that predicts these stable patterns and relates them to imitation and shared knowledge.
---
paper_title: Social heroes: games as APIs for social interaction

paper_content:

Social Heroes is a pervasive social game in which players trade points by tagging each other using Twitter. Through the accumulation of tags and points in defined combinations, they earn achievement awards and gain the ability to create their own tags and achievements. Taking the metaphor of APIs from software development and drawing parallels between the way we model game state, Social Heroes attempts to provide a ludic interface for our personal relationships. The result is a system for social play surrounding identity, relationships, and communication.
---
paper_title: Dogear: Social bookmarking in the enterprise

paper_content:

We describe a social bookmarking service de-signed for a large enterprise. We discuss design principles addressing online identity, privacy, information discovery (including search and pivot browsing), and service extensi-bility based on a web-friendly architectural style. In addi-tion we describe the key design features of our implementa-tion. We provide the results of an eight week field trial of this enterprise social bookmarking service, including a de-scription of user activities, based on log file analysis. We share the results of a user survey focused on the benefits of the service. The feedback from the user trial, comprising survey results, log file analysis and informal communica-tions, is quite positive and suggests several promising en-hancements to the service. Finally, we discuss potential extension and integration of social bookmarking services with other corporate collaborative applications.
---
paper_title: Games with a Purpose for the Semantic Web

paper_content:

Weaving the semantic Web requires that humans contribute their labor and judgment for creating, extending, and updating formal knowledge structures. Hiding such tasks behind online multiplayer games presents the tasks as fun and intellectually challenging entertainment. The games we've presented are the first prototypes of the OntoGame series. We're extending and improving the scenarios in several directions.
---
paper_title: The dogear game: a social bookmark recommender system

paper_content:

We describe the Dogear Game, which works with an enterprise social bookmarking system. The game is designed to accomplish individual, collaborative, and organization goals. Individual players receive entertainment and learn about their colleagues' bookmarks. The player's colleagues receive recommendations of websites and documents of potential interest to them. And the organization benefits from a richer knowledge-base of bookmarks as recommendations are accepted. The Dogear Game builds on von Ahn's "serious games," useful in motivating and distributing game-like entertaining "work" to a large group of game players. This note presents the design and implementation of a working prototype and some initial user feedback.
---
paper_title: MobiMissions: the game of missions for mobile phones

paper_content:

Mobimissions is a location-based pervasive social game in which players use camera phones with location-based capabilities to create, share and reply to real-world missions. Missions and their responses are defined by sequences of digital photographs and text annotations. Players create missions, search locations for available missions, create responses to and submit missions for others to find. They can visit a website to review and rate all missions and responses. Feedback from a trial involving 11 players over five weeks reveals patterns of play, preferred locations for play, the limitations of location-based play, and the need for greater social awareness and exchange. Future directions for such location-based social games are suggested.
---
paper_title: Peekaboom: a game for locating objects in images

paper_content:

We introduce Peekaboom, an entertaining web-based game that can help computers locate objects in images. People play the game because of its entertainment value, and as a side effect of them playing, we collect valuable image metadata, such as which pixels belong to which object in the image. The collected data could be applied towards constructing more accurate computer vision algorithms, which require massive amounts of training and testing data not currently available. Peekaboom has been played by thousands of people, some of whom have spent over 12 hours a day playing, and thus far has generated millions of data points. In addition to its purely utilitarian aspect, Peekaboom is an example of a new, emerging class of games, which not only bring people together for leisure purposes, but also exist to improve artificial intelligence. Such games appeal to a general audience, while providing answers to problems that computers cannot yet solve.
---
paper_title: The gopher game: a social, mobile, locative game with user generated content and peer review

paper_content:

This paper introduces 'Gophers', a social game for mobile devices that utilises task oriented gameplay to create a novel entertainment experience. The study combines a number of key research themes: mobile social gaming, acquiring useful data through gameplay and content sharing in mobile settings. The experience of trialling the game in the real world is discussed and the findings from the study are presented.
---
paper_title: Improving accessibility of the web with a computer game

paper_content:

Images on the Web present a major accessibility issue for the visually impaired, mainly because the majority of them do not have proper captions. This paper addresses the problem of attaching proper explanatory text descriptions to arbitrary images on the Web. To this end, we introduce Phetch, an enjoyable computer game that collects explanatory descriptions of images. People play the game because it is fun, and as a side effect of game play we collect valuable information. Given any image from the World Wide Web, Phetch can output a correct annotation for it. The collected data can be applied towards significantly improving Web accessibility. In addition to improving accessibility, Phetch is an example of a new class of games that provide entertainment in exchange for human processing power. In essence, we solve a typical computer vision problem with HCI tools alone.
---
paper_title: A comprehensive human computation framework: with application to image labeling

paper_content:

Image and video labeling is important for computers to understand images and videos and for image and video search. Manual labeling is tedious and costly. Automatically image and video labeling is yet a dream. In this paper, we adopt a Web 2.0 approach to labeling images and videos efficiently: Internet users around the world are mobilized to apply their "common sense" to solve problems that are hard for today's computers, such as labeling images and videos. We first propose a general human computation framework that binds problem providers, Web sites, and Internet users together to solve large-scale common sense problems efficiently and economically. The framework addresses the technical challenges such as preventing a malicious party from attacking others, removing answers from bots, and distilling human answers to produce high-quality solutions to the problems. The framework is then applied to labeling images. Three incremental refinement stages are applied. The first stage collects candidate labels of objects in an image. The second stage refines the candidate labels using multiple choices. Synonymic labels are also correlated in this stage. To prevent bots and lazy humans from selecting all the choices, trap labels are generated automatically and intermixed with the candidate labels. Semantic distance is used to ensure that the selected trap labels would be different enough from the candidate labels so that no human users would mistakenly select the trap labels. The last stage is to ask users to locate an object given a label from a segmented image. The experimental results are also reported in this paper. They indicate that our proposed schemes can successfully remove spurious answers from bots and distill human answers to produce high-quality image labels.
---
paper_title: WebInSight:: making web images accessible

paper_content:

Images without alternative text are a barrier to equal web access for blind users. To illustrate the problem, we conducted a series of studies that conclusively show that a large fraction of significant images have no alternative text. To ameliorate this problem, we introduce WebInSight, a system that automatically creates and inserts alternative text into web pages on-the-fly. To formulate alternative text for images, we present three labeling modules based on web context analysis, enhanced optical character recognition (OCR) and human labeling. The system caches alternative text in a local database and can add new labels seamlessly after a web page is downloaded, resulting in minimal impact to the browsing experience.
---
paper_title: The Restaurant Game: Learning Social Behavior and Language from Thousands of Players Online

paper_content:

We envision a future in which conversational virtual agents collaborate with humans in games and training simulations. A representation of common ground for everyday scenarios is essential for these agents if they are to be effective collaborators and communicators. Effective collaborators can infer a partner’s goals and predict future actions. Effective communicators can infer the meaning of utterances based on semantic context. This article introduces a computational model of common ground called a Plan Network, a statistical model that encodes context-sensitive expected patterns of behavior and language, with dependencies on social roles and object affordances. We describe a methodology for unsupervised learning of a Plan Network using a multiplayer video game, visualization of this network, and evaluation of the learned model with respect to human judgment of typical behavior. Specifically, we describe learning the Restaurant Plan Network from data collected from over 5,000 gameplay sessions of a minimal investment multiplayer online (MIMO) role-playing game called The Restaurant Game. Our results demonstrate a kind of social common sense for virtual agents, and have implications for automatic authoring of content in the future.
---
paper_title: Designing games with a purpose

paper_content:

Data generated as a side effect of game play also solves computational problems and trains AI algorithms.
---
paper_title: A game theoretic analysis of games with a purpose

paper_content:

We present a simple game-theoretic model for the ESP game, an interactive game devised to label images on the web, and characterize the equilibrium behavior of the model. We show that a simple change in the incentive structure can lead to different equilibrium structure and suggest the possibility of formal incentive design in achieving desirable system-wide outcomes, complementing existing considerations of robustness against cheating and human factors.
---
paper_title: Learning consensus opinion: mining data from a labeling game

paper_content:

We consider the problem of identifying the consensus ranking for the results of a query, given preferences among those results from a set of individual users. Once consensus rankings are identified for a set of queries, these rankings can serve for both evaluation and training of retrieval and learning systems. We present a novel approach to collecting the individual user preferences over image-search results: we use a collaborative game in which players are rewarded for agreeing on which image result is best for a query. Our approach is distinct from other labeling games because we are able to elicit directly the preferences of interest with respect to image queries extracted from query logs. As a source of relevance judgments, this data provides a useful complement to click data. Furthermore, the data is free of positional biases and is collected by the game without the risk of frustrating users with non-relevant results; this risk is prevalent in standard mechanisms for debiasing clicks. We describe data collected over 34 days from a deployed version of this game that amounts to about 18 million expressed preferences between pairs. Finally, we present several approaches to modeling this data in order to extract the consensus rankings from the preferences and better sort the search results for targeted queries.
---
paper_title: Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

paper_content:

Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.
---
paper_title: Secure distributed human computation

paper_content:

This paper is a preliminary exploration of secure distributed human computation. We consider the general paradigm of using large-scale distributed computation to solve difficult problems, but where humans can act as agents and provide candidate solutions. We are especially motivated by problem classes that appear to be difficult for computers to solve effectively, but are easier for humans; e.g., image analysis, speech recognition, and natural language processing. This paradigm already seems to be employed in several real-world scenarios, but we are unaware of any formal and unified attempt to study it. Nonetheless, this concept spawns interesting research questions in algorithm design, human computer interfaces, and programming language / API design, distributed systems, and cryptography, among other fields. There are also interesting implications for Internet commerce and the B24b model. We describe this research area and suggest a basic framework for the design of such systems. We analyze security and reliability against malicious parties using standard probability theory tools. We then derive design principles using standard decision-theory concepts. Finally, we list extensions and open problems.
---
paper_title: Internet-scale collection of human-reviewed data

paper_content:

Enterprise and web data processing and content aggregation systems often require extensive use of human-reviewed data (e.g. for training and monitoring machine learning-based applications). Today these needs are often met by in-house efforts or out-sourced offshore contracting. Emerging applications attempt to provide automated collection of human-reviewed data at Internet-scale. We conduct extensive experiments to study the effectiveness of one such application. We also study the feasibility of using Yahoo! Answers, a general question-answering forum, for human-reviewed data collection.
---
paper_title: An Analytical Approach to Optimizing the Utility of ESP Games

paper_content:

In this paper, we propose an analytical model for computing the utility of ESP games, i.e., the throughput rate of appropriate labels for given puzzles. The model targets generalized games, where the number of players, the consensus threshold, and the stopping condition are variable. Via extensive simulations, we show that our model can accurately predict the stopping condition that will yield the optimal utility of an ESP game under a specific setting. A service provider can therefore utilize the model to ensure that the hosted ESP games produce high-quality labels efficiently, given that the number of players willing to invest time and effort in the game is limited.
---
paper_title: An Analytical Study of Puzzle Selection Strategies for the ESP Game

paper_content:

"Human computation" represents a new paradigm of applications that take advantage of people's desire to be entertained and produce useful metadata as a by-product. By creating games with a purpose, human computation has shown promise in solving a variety of problems that computer computation cannot currently resolve completely. Using the ESP game as an example, we propose a metric, called system gain, for evaluating the performance of human computation systems, and also use analysis to study the properties of the ESP game. We argue that human computation systems should be played with a strategy. To this end, we implement an optimal puzzle selection strategy (OPSA) based on our analysis to improve human computation. Using a comprehensive set of simulations, we demonstrate that the proposed OPSA approach can effectively improve the system gain of the ESP game, as long as the number of puzzles in the system is sufficiently large.
---
paper_title: PhotoSlap: a multi-player online game for semantic annotation

paper_content:

Multimedia content presents special challenges for the search engines, and could benefit from semantic annotation of images. Unfortunately, manual labeling is too tedious and time-consuming for humans, whereas automatic image annotation is too difficult for the computers. In this paper, we explore the power of human computation by designing a multi-player online game, PhotoSlap, to achieve the task of annotating metadata for a collection of digital photos. PhotoSlap engages users in an interactive game that capitalizes on human ability in deciphering quickly whether the same person shows up in two consecutive images presented by the computer. The game mechanism supports the objection and trap actions to encourage truthful input from the players. This research extends human computation research in two aspects: game-theoretic design principles and quantitative evaluation metrics. In particular, PhotoSlap can be shown to reach subgame perfect equilibrium with the target strategy when players are rational and without collusion. Experiments involving four focus groups have been conducted, and the preliminary results demonstrated the game to be fun and effective in annotating people metadata for photo collections.
---
paper_title: Rethinking the ESP game

paper_content:

The ESP Game was designed to harvest human intelligence to assign labels to images - a task which is still difficult for even the most advanced systems in image processing. However, the ESP Game as it is currently implemented encourages players to assign "obvious" labels, which can be easily predicted given previously assigned labels. We present a language model which can assign probabilities to the next label to be added. This model is then used in a program, which plays the ESP game without looking at the image. Even without any use of the actual image, the program manages to agree with the randomly assigned human partner on a label for 69% of all images, and for 81% of images which have at least one "off-limits" term assigned to them. We discuss how the scoring system and the design of the ESP game can be improved to encourage users to add less predictable labels, thereby improving the quality of the collected information.
---
paper_title: Get another label? improving data quality and data mining using multiple, noisy labelers

paper_content:

This paper addresses the repeated acquisition of labels for data items when the labeling is imperfect. We examine the improvement (or lack thereof) in data quality via repeated labeling, and focus especially on the improvement of training labels for supervised induction. With the outsourcing of small tasks becoming easier, for example via Rent-A-Coder or Amazon's Mechanical Turk, it often is possible to obtain less-than-expert labeling at low cost. With low-cost labeling, preparing the unlabeled part of the data can become considerably more expensive than labeling. We present repeated-labeling strategies of increasing complexity, and show several main results. (i) Repeated-labeling can improve label quality and model quality, but not always. (ii) When labels are noisy, repeated labeling can be preferable to single labeling even in the traditional setting where labels are not particularly cheap. (iii) As soon as the cost of processing the unlabeled data is not free, even the simple strategy of labeling everything multiple times can give considerable advantage. (iv) Repeatedly labeling a carefully chosen set of points is generally preferable, and we present a robust technique that combines different notions of uncertainty to select data points for which quality should be improved. The bottom line: the results show clearly that when labeling is not perfect, selective acquisition of multiple labels is a strategy that data miners should have in their repertoire; for certain label-quality/cost regimes, the benefit is substantial.
---

Title: A Survey of Human Computation Systems

Section 1: INTRODUCTION
Description 1: This section introduces human computation, defining it and explaining its significance and the categories of existing human computation systems.

Section 2: INITIATORY HUMAN COMPUTATION
Description 2: This section covers the early human computation systems designed to leverage human common sense for tasks difficult for computers, such as CAPTCHA and reCAPTCHA.

Section 3: DISTRIBUTED HUMAN COMPUTATION
Description 3: This section surveys systems that utilize the collective efforts of large groups of internet users to solve AI problems, with examples like Vipul's Razor, Proofreader, and Wikipedia.

Section 4: SOCIAL GAME-BASED HUMAN COMPUTATION WITH VOLUNTEERS OR PAID ENGINEERS
Description 4: This section examines systems that engage volunteers or paid engineers in human computation tasks, discussing projects like Cyc, Open Mind, and Mindpixel.

Section 5: SOCIAL GAME-BASED HUMAN COMPUTATION WITH ONLINE PLAYERS
Description 5: This section presents human computation systems that use games played by online users to gather data, detailing different types of game structures and their characteristics.

Section 6: THE PERFORMANCE ASPECT OF HUMAN COMPUTATION
Description 6: This section discusses the evaluation of human computation systems, including security, reliability, quality of data, and metrics for system performance.

Section 7: CONCLUSION
Description 7: This section provides a summary of the survey's findings and reflects on the categorization and performance aspects of human computation systems, suggesting future research directions.