Article Figures & Data
Figures
- Figure 1.
A, Two example images for each of the 24 basic-level objects, sampled from the test set (each row corresponds to a group of 8 objects). To enforce true object recognition behavior (rather than image matching) and tackle the invariance problem, we generated thousands of naturalistic images, each with one foreground object, by rendering a 3D model of each object with randomly chosen viewing parameters (2D position, 3D rotation, and viewing distance) and placing that foreground object view onto a randomly chosen, natural image background. B, Behavioral paradigm (for monkey M). Each trial was initiated when the monkey held gaze fixation on a central fixation point for 200 ms, after which a square test image (spanning 6° of visual angle) appeared at the center of gaze for 100 ms. Immediately after extinction of the test image, two choice images, each displaying the canonical view of a single object with no background, were shown to the left and right (see Materials and Methods). Test and choice images are shown to scale. The monkey was allowed to freely view the response images for up to 1500 ms and responded by holding fixation over the selected image for 700 ms. Monkey Z performed the exact same tasks but used touch to initiate trials and indicate its choice (see Materials and Methods). Successful trials were rewarded with juice, and incorrect choices resulted in timeouts of 1.5–2.5 s.
- Figure 2.
A, B, Learning of novel objects. For each monkey, performance relative to the human pool, quantified as a Turing ratio d′/d′ human pool (one-versus-all d′), for each of these 16 objects (colored dots) and the mean over objects (black line) are shown. For these 16 objects, animals were exposed to full-variation images in groups of eight objects and reached high performance in 1000–2000 image presentations per object. C, Performance relative to the human pool, quantified as a Turing ratio, of two monkeys and 15 unique individual humans subjects with sufficient longitudinal data on the same tasks (mean ± SE over subjects). Monkeys learned each new object rapidly, whereas humans performed at a high initial performance and exhibited no change in performance as a function of (unsupervised) experience with the objects. The Test marker indicates monkeys' relative performance on held-out test images, after all behavioral training. D, Generalization to novel images. Top, Pooling data from both monkeys, the first-trial performance of 2400 test images (mean ± SE) is plotted as a function of the corresponding distance to the nearest training image; black line denotes linear regression. Distance is computed on gray background images, as a Euclidean pixel distance of each test image to the nearest training image of the same object (see Materials and Methods, Visual images). The distance values have been discretized into 20 bins of varying lengths with equal number of images. The overall performance, including all subsequent exposures to test images, is shown on the left (at 0 distance). Bottom, Overlap between training and test image sets. The distribution of distances of test images to the nearest trained image is shown relative to actual training images (black line) and to “ideal” generalization surrogate training images (green line). The last histogram bin includes all distances >2500.
- Figure 3.
A, Pattern of behavioral performances for the pooled human and pooled monkey. Each 24 × 24 matrix summarizes confusions of all two-way tasks: the color of bin (i,j) indicates the unbiased performance (d′) of the binary recognition task with objects i and j. Objects have been reordered based on a hierarchical clustering of human confusion patterns to highlight structure in the matrix. We observe qualitative similarities in the confusion patterns. For example, (camel, dog) and (tank, truck) are two often confused object pairs in both monkeys and humans. B, Comparison of d′ estimates of all 276 tasks (mean ± SE as estimated by bootstrap, 100 resamples) of the pooled human with that of the pooled monkey (top) and a low-level pixel representation (bottom). C, Quantification of consistency as noise-adjusted correlation of d′ vectors. The pooled monkey shows patterns of confusions that are highly correlated with pooled human subject confusion patterns (consistency of pooled monkey, 0.78). Importantly, low-level visual representations do not share these confusion patterns (pixels, 0.37; V1+, 0.52). Furthermore, a state-of-the-art deep convolutional neural network representation was highly predictive of human confusion patterns (CNN2013, 0.86), in contrast to an alternative model of the ventral stream (HMAX, 0.55). The dashed lines indicate thresholds at p = 0.1, 0.05 confidence for consistency to the gold-standard pooled human, estimated from pairs of individual human subjects. D, Comparison of d′ estimates of all 276 tasks (mean ± SE as estimated by bootstrap, 100 resamples) between the two monkeys.
- Figure 4.
Accounting for intersubject variability. For each of three groups of eight objects, the absolute performance and consistency of individual human subjects, individual monkeys, and machine features are shown. Error bars for consistency relative to pooled human (mean ± SD) are shown for individual monkeys and machine features for each group (error bars for monkeys are not visible in object group 2 because of small variability). The shaded gray areas indicate the distribution of performance/consistency over individual human subjects (mean ± SD). There is significant intersubject variability: individual human subjects are on average not perfectly correlated with the pooled human (average consistency, 0.74, 0.77, and 0.68 for the 3 groups). As a result, monkeys are statistically indistinguishable from individual human subjects in their correlation to the human pool (monkey M, p = 0.4, 0.21, and 0.22; monkey Z, p = 0.16, 0.22, and 0.27). In contrast, low-level visual representations were falsified on both performance and consistency grounds for two of three groups of objects (V1, p = 0.30, 0.03, and 0.03; pixels, p = 0.17, 0.02, and 0.03).
