To study how mental object representations are related to behavior, we estimated sparse, non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories. These representations predicted a latent similarity structure between objects, which captured most of the explainable variance in human behavioral judgments. Individual dimensions in the low-dimensional embedding were found to be highly reproducible and interpretable as conveying degrees of taxonomic membership, functionality, and perceptual attributes. We further demonstrated the predictive power of the embeddings for explaining other forms of human behavior, including categorization, typicality judgments, and feature ratings, suggesting that the dimensions reflect human conceptual representations of objects beyond the specific task. A central goal in understanding the human mind is to determine how object concepts are represented and how they relate to human behavior. Given the near-infinite number of tasks or contexts of usage, this might appear to be an impossible prospect. Take the example of picking a tomato in a grocery store. A person may recognize it by its color, shape, size, or texture; they may also know many of the more conceptual dimensions of a tomato, such as it being a fruit, or functional ones such as being a salad item. In most cases, however, only a few of these aspects might matter, depending on task context (e.g. as a grocery item or as a projectile in an audience that is not pleased with a presentation). Thus, to understand how we interact meaningfully with the objects around us, we need to tackle three problems simultaneously: (1) determine the object concept, the cognitive representation of behaviorally-relevant dimensions that distinguish an object from other objects. (2) determine which object concept representations are integrated into our perceptual decisions. (3) characterize the influence of the context -determined by the surrounding objects -on those decisions.There have been many attempts to represent object concepts in terms of semantic features, a vector of variables indicating the presence of different aspects of their meaning. These representations have been used to model phenomena such as judgments of typicality or similarity between concepts, or reaction times in various semantic tasks, with the goal of drawing conclusions about mental representations of those concepts (see BID13 for an extensive review of this literature). These features have usually been binary properties, postulated by researchers. A landmark study BID9 departed from this approach by instead asking hundreds of subjects to name binary properties of 541 objects, yielding 2,526 such semantic features. These corresponded to different types of information, e.g. for the concept "hammer" subjects might list taxonomic ("is a tool"), functional ("is used to hammer nails"), perceptual ("heavy"), among others. The results revealed that concepts for objects in the same basic level category shared many features; at the same time, there were also features that distinguished even very similar concepts. This effort was later replicated and extended by BID4 , generating 5,929 semantic features for 638 objects.The main issues with features produced in either study are the lack of degree (they can only be present or absent, albeit with varying naming frequencies), the extreme specificity of some features, and the omission of many features shared by the majority of concepts. A separate concern is the fact that, absent a specific context of use for each object, subjects will not likely think of many valid properties (e.g. that a tomato can be thrown). A different approach is to postulate the existence of certain semantic features and ask subjects to judge the degree to which features are salient for each concept, instead of assuming binary features. BID3 did this for 65 features corresponding to types of information for which there is evidence of brain representation (in their terminology: sensory, motor, spatial, temporal, affective, social, cognitive, etc) . This requires experts to specify the features in advance, and not all features can be judged easily by salience. In all three approaches outlined above, there is also no clear way of determining which features are critical to semantic behavior.Here we introduce an approach that uses only information from behavioral judgments about the grouping of object images in the context of other objects to estimate representations of object concepts. We demonstrate that this approach can predict human behavior in face of new combinations of objects and that it also allows prediction of results in other behavioral tasks. We show that individual dimensions in the low-dimensional embedding represent complex combinations of the information in the binary features in BID4 and, furthermore, are interpretable as conveying taxonomic, functional, or perceptual information. Finally, we will discuss the way in which this representation suggests a simple, effective model of judgments of semantic similarity in context. In this paper, we show that human behavioral judgments are well-explained by a strikingly lowdimensional semantic representation of concrete concepts. This representation, which embeds each object in a 49-dimensional vector, allows prediction of subject behavior in face of new combinations of concepts not encountered before, as well as prediction of other behavioral or human-annotated data, such as typicality ratings or similarity judgments. Moreover, the representation is readily interpretable, as positive, sparse dimensions make it easy to identify which concepts load on each dimension. Further, we demonstrate that the value of each dimension in this space can be explained in terms of elementary features elicited directly from human subjects in publicly available norms. Given this converging evidence, we conclude that dimensions represent distinct types of information, from taxonomic (indicators of category membership) to functional or perceptual.As the representations were estimated solely from behavioral data, this suggests a simple model of decision making in the triplet task. This can be viewed in terms of the distinction discussed in Navarro & Lee (2004) for judging concept similarity from semantic feature vectors. They distinguish a dimensional approach for representing stimuli (each feature is a continuous value, each concept is a point in a high-dimensional space, and similarity corresponds to proximity in the space), and a featural approach (each feature is binary, or discrete, and similarity is a function of the number of features that are common to both concepts, or that distinguish them). More refined schemes use modified distance metrics (dimensional) or combine commonality and distinctiveness (featural).The use of sparsity and positivity in the SPoSE representation, and the vector dot product for computing concept similarity, blends the featural and dimensional approaches when making decisions about a triplet of concepts. First , if any two concepts share a semantic category, and the other one does not, the two concepts will likely be grouped together. Because of sparsity, the dot product between concepts will be driven primarily by the number of features are shared between the two concepts in the same category, versus the different one. Second, if any three concepts share a semantic category, they also share most, if not all of their non-zero features. The decision becomes a function of the values of the features shared between them, and hence dimensional rather than featural. Third, if all concepts belong to different categories, there may be very few features in common between any two of them. The results will likely be determined by which of those few features takes a higher value. Results might be idiosyncratic, e.g. two objects grouped because their pictures are both very red, while the alternative grouping would be because they are both string-like, and the former feature is more salient. This is another reason why our features are unbounded: their scale can reflect their importance in decision making. This is akin to learning a distance metric in dimensional approaches.Our object representations capture the information that is necessary to explain subject behavior in the triplet task. Obviously, subjects have a lot more information about each concept that is not necessary or relevant for task performance. A promising direction for further work is to sample additional triplets so as to obtain more fine-grained, within-category distinctions. Beyond this, we have also considered the possibility of there being information influencing behavior that might be too infrequent to be estimated from this type of data, or elicited from human subjects. Yet another possible extension is to consider different types of similarity judgments BID19 , e.g. resulting from asking subjects to group objects based on a specific attribute (size, color, etc.). One avenue for trying to identify this type of information is to predict synset vectors from SPoSE vectors and/or semantic features elicited from subjects, and represent the residuals in terms of a dictionary of new sparse, positive concept features. These could then be used as a complement to SPoSE dimensions.
Human behavioral judgments are used to obtain sparse and interpretable representations of objects that generalize to other tasks This paper describes a large-scale experiment on human object/sematic representations and a model of such representations. This paper develops a new representation system for object representations from training on data collected from odd-one-out human judgements of images. A new approach to learn a sparse, positive, interpretable semantic space that maximizes human similarity judgements by training to specifically maximize the prediction of human similarity judgements.