Previous studies have suggested that visual short-term memory (VSTM) has a storage limit of approximately four items. However, the type of high-threshold (HT) model used to derive this estimate is based on a number of assumptions that have been criticized in other experimental paradigms (e.g., visual search). Here we report findings from nine experiments in which VSTM for color, spatial frequency, and orientation was modeled using a signal detection theory (SDT) approach. In Experiments 1-6, two arrays composed of multiple stimulus elements were presented for 100 ms with a 1500 ms ISI. Observers were asked to report in a yes/no fashion whether there was any difference between the first and second arrays, and to rate their confidence in their response on a 1-4 scale. In Experiments 1-3, only one stimulus element difference could occur (T = 1) while set size was varied. In Experiments 4-6, set size was fixed while the number of stimuli that might change was varied (T = 1, 2, 3, and 4). Three general models were tested against the receiver operating characteristics generated by the six experiments. In addition to the HT model, two SDT models were tried: one assuming summation of signals prior to a decision, the other using a max rule. In Experiments 7-9, observers were asked to directly report the relevant feature attribute of a stimulus presented 1500 ms previously, from an array of varying set size. Overall, the results suggest that observers encode stimuli independently and in parallel, and that performance is limited by internal noise, which is a function of set size.

A “follow-the-dot ” method was used to investigate the visual memory systems supporting accumulation of object information in natural scenes. Participants fixated a series of objects in each scene, following a dot cue from object to object. Memory for the visual form of a target object was then tested. Object memory was consistently superior for the two most recently fixated objects, a recency advantage indicating a visual short-term memory component to scene representation. In addition, objects examined earlier were remembered at rates well above chance, with no evidence of further forgetting when 10 objects intervened between target examination and test and only modest forgetting with 402 intervening objects. This robust prerecency performance indicates a visual long-term memory component to scene representation.

Traditional memory research has focused on identifying separate memory systems and exploring different stages of memory processing. This approach has been valuable for establishing a taxonomy of memory systems and characterizing their function but has been less informative about the nature of stored memory representations. Recent research on visual memory has shifted toward a representation-based emphasis, focusing on the contents of memory and attempting to determine the format and structure of remembered information. The main thesis of this review will be that one cannot fully understand memory systems or memory processes without also determining the nature of memory representations. Nowhere is this connection more obvious than in research that attempts to measure the capacity of visual memory. We will review research on the capacity of visual working memory and visual long-term memory, highlighting recent work that emphasizes the contents of memory. This focus impacts not only how we estimate the capacity of the systemVgoing beyond quantifying how many items can be remembered and moving toward structured representationsVbut how we model memory systems and memory processes. Keywords: memory, working memory, long-term memory, visual cognition, memory capacity, memory fidelity Citation: Brady, T. F., Konkle, T., & Alvarez, G. A. (2011). A review of visual memory capacity: Beyond individual items and toward structured representations. Journal of Vision, 11(5):4, 1-34, http://www.journalofvision.org/content/11/5/4, doi:10.1167/11.5.4. Introduction Tulving Early on, William James (1890) proposed the distinction between primary memoryVthe information held in the "conscious present"Vand secondary memory, which consists of information that is acquired, stored outside of conscious awareness, and then later remembered. This distinction maps directly onto the modern distinction between short-term memory (henceforth working memory) and long-term memory The emphasis on memory systems and memory processes has been quite valuable in shaping cognitive and neural models of memory. In general, this approach aims to characterize memory systems in a way that generalizes over representational content Research on visual perception takes the opposite approach, attempting to determine what is being represented and to generalize across processes. For example, early stages of visual representation consist of orientation and spatial frequency features. Vision research has measured the properties of these features, such as their tuning curves and sensitivity (e.g., Here, we review recent research in the domains of visual working memory and visual long-term memory, focusing on how models of these memory systems are altered and refined by taking the contents of memory into account. Visual working memory The working memory system is used to hold information actively in mind and to manipulate that information to perform a cognitive task The study of visual working memory has largely focused on the capacity of the system, both because limited capacity is one of the main hallmarks of working memory and because individual differences in measures of working memory capacity are correlated with differences in fluid intelligence, reading comprehension, and academic achievement In the broader working memory literature, a significant amount of research has focused on characterizing memory limits based on how quickly information can be refreshed (e.g., Here, we review research that focuses on working memory representations, including their fidelity, structure, and effects of stored knowledge. While not an exhaustive review of the literature, these examples highlight the fact that working memory representations have a great deal of structure beyond the level of individual items. This structure can be characterized as a hierarchy of properties, from individual features to individual objects to acrossobject ensemble features (spatial context and featural context). Together, the work reviewed here illustrates how a representation-based approach has led to important advances, not just in understanding the nature of stored representations themselves but also in characterizing working memory capacity and shaping models of visual working memory. The fidelity of visual working memory Recent progress in modeling visual working memory has resulted from an emphasis on estimating the fidelity of visual working memory representations. In general, the capacity of any memory system should be characterized both in terms of the number of items that can be stored and in terms of the fidelity with which each individual item can be stored. Consider the case of a USB drive that can store exactly 1000 images: the number of images alone is not a complete estimate of this USB drive's storage capacity. It is also important to consider the resolution with which those images can be stored: if each image can be stored with a very low resolution, say 16  16 pixels, then the drive has a lower capacity than if it can store the same number of images with a high resolution, say 1024  768 pixels. In general, the true capacity of a memory system can be estimated by multiplying the maximum number of items that can be stored by the fidelity with which each individual item can be stored (capacity = quantity  fidelity). For a memory system such as your USB drive, there is only an information limit on memory storage, so the number of files that can be stored is limited only by the size of those files. Whether visual working memory is best characterized as an information- or whether it has a predetermined and fixed item limit Importantly, this standard change detection paradigm provides little information about how well each individual object was remembered. The change detection paradigm indicates only that items were remembered with sufficient fidelity to distinguish an object's color from a categorically different color. How much information do observers actually remember about each object? Several new methods have been used to address this question (see Journal of Vision Fidelity of storage for complex objects While early experiments using large changes in a change detection paradigm found evidence for a slot model, in which memory is limited to storing a fixed number of items, subsequent experiments with newer paradigms that focused on the precision of memory representations have suggested an information-limited model. Specifically, This result was not due to an inability to discriminate the more complex shapes, such as 3D cubes: observers could easily detect a change between cubes when only a single cube was remembered, but they could not detect the same change when they tried to remember 4 cubes. This result suggests that encoding additional items reduced the resolution with which each individual item could be remembered, consistent with the idea that there is an information limit on memory. Using the same paradigm but varying the difficulty of the memory test, Awh, Fidelity of simple feature dimensions While the work of Alvarez and Cavanagh Wilken and Ma's However, Zhang and Luck Conclusion To summarize, by focusing on the contents of visual working memory, and on the fidelity of representations in particular, there has been significant progress in models of visual working memory and its capacity. At present, there is widespread agreement in the visual working memory literature that visual working memory has an extremely limited capacity and that it can represent 1 item with greater fidelity than 3-4 items. This finding requires the conclusion that working memory is limited by a resource that is shared among the representations of different items (i.e., information-limited). Some models claim that resource allocation is discrete and quantized into slots Research on the fidelity of working memory places important constraints on both continuous and discrete models. If working memory is slot-limited, then those slots must be recast as a flexible resource, all of which can be allocated to a single item to gain precision in its representation or which can be divided separately among multiple items yielding relatively low-resolution representations of each item. If memory capacity is informationlimited, then it is necessary to explain why under some conditions it appears that there is an upper bound on memory storage of 3-4 objects (e.g., The representation of features vs. objects in visual working memory Any estimate of memory capacity must be expressed with some unit, and what counts as the appropriate unit depends upon how information is represented. Since George Miller's (1956) seminal paper claiming a limit of 7 T 2 chunks as the capacity of working memory, a significant amount of work has attempted to determine the units of storage in working memory. In the domain of verbal memory, for example, debate has flourished about the extent to which working memory capacity is limited by storing a fixed number of chunks vs. time-based decay Objects are not always encoded in their entirety A significant body of work has demonstrated that observers do not always encode objects in their entirety. When multiple features of an object appear on distinct object parts, observers are significantly impaired at representing the entire object Costs for encoding multiple features within an object Furthermore, another body of work has demonstrated that encoding more than one feature of the same object does not always come without cost. In addition to limits on the number of values that may be stored within a particular feature dimension, data on the fidelity of representations suggest that even separate visual features from the same object are not stored completely independently. In an elegant design combining elements of the original work of Benefits of object-based storage beyond separate buffers While observers cannot completely represent 3-4 objects independently of their information load, there is a benefit to encoding multiple features from the same object compared to the same number of features on different objects Journal of Vision (2011) 11(5):4, 1-34 Brady, Konkle, & Alvarez 6 Jiang showed that it is easier to remember the color and orientation of 2 objects (4 features in total) than the color of 2 objects and the orientation of 2 separate objects (still 4 features in total). In addition, while Conclusion So what is the basic unit of representation in visual working memory? While there are significant benefits to encoding multiple features of the same object compared to multiple features across different objects (e.g., One possibility is that the initial encoding process is object-based (or location-based), but that the "unit" of visual working memory is a hierarchically structured feature bundle This proposal for the structure of memory representations is consistent with the full pattern of evidence described above, including the benefit for remembering multiple features from the same objects relative to different objects and the cost for remembering multiple features from the same object. Moreover, this hierarchical working memory theory is consistent with evidence showing a specific impairment in object-based working memory when attention is withdrawn from items (e.g. Furthermore, there is some direct evidence for separate capacities for feature-based and object-based working memory representations, with studies showing separable priming effects and memory capacities It is important to note that our proposed hierarchical feature bundle model is not compatible with a straightforward item-based or chunk-based model of working memory capacity. A key part of such proposals (e.g., Thus far, we have considered only the structure of individual items in working memory. Next, we review research demonstrating that working memory representations include another level of organization that represents properties that are computed across sets of items. Interactions between items in visual working memory In the previous two sections, we discussed the representation of individual items in visual working memory. However, research focusing on contextual effects in memory demonstrates that items are not stored in memory completely independent of one another. In particular, several studies have shown that items are encoded along with spatial context information (the spatial layout of items in the display) and with featural context information (the ensemble statistics of items in the display). These results suggest that visual working memory representations have a great deal of structure beyond the individual Journal of Vision Influences of spatial context Visual working memory paradigms often require observers to remember not only the featural properties of items (size, color, shape, identity) but also where those items appeared in the display. In these cases, memory for the features of individual items may be dependent on spatial working memory as well (for a review of spatial working memory, see Influence of feature context or "ensemble statistics" In addition to spatial context effects on item memory, it is likely that there are feature context effects as well. For instance, even in a display of squares with random colors, some displays will tend to have more "warm colors" on average, whereas others will have more "cool colors" on average, and others still will have no clear across-item structure. This featural context, or "ensemble statistics" (Alvarez, 2011), could influence memory for individual items (e.g., Given that ensemble information would be useful for remembering individual items, it is important to consider the possibility that these ensemble statistics will influence Journal of Vision Perceptual grouping and dependence between items Other research has shown that items tend to be influenced by the other items in visual working memory, although such work has not explicitly attempted to distinguish influences due to the storage of individual items and influences from ensemble statistics. For example, Lin and Luck (2008; using colored squares) and Viswanathan, Perl, Bisscher, Kahana, and Sekuler (2010; using Gabor stimuli) showed improved memory performance when items appear more similar to one another (see also Cases of explicit perceptual grouping make the nonindependence between objects even more clear. For example, Woodman, Vecera, and Luck Perceptual grouping vs. chunking vs. hierarchically structured memory What is the relationship between perceptual grouping, chunking, and the hierarchically structured memory model we have described? Perceptual grouping and chunking are both processes by which multiple elements are combined into a single higher order description. For example, a series of 10 evenly spaced dots could be grouped into a single line, and the letters F, B, and I can be chunked into the familiar acronym FBI (e.g., :4, 1-34 Brady, Konkle, & Alvarez 9 assume that the only limits on memory capacity come from the number of chunks or groups that can be encoded Conclusion Taken together, these results provide significant evidence that individual items are not represented independent of other items on the same display and that visual working memory stores information beyond the level of individual items. Put another way, every display has multiple levels of structure, from the level of feature representations to individual items to the level of groups or ensembles, and these levels of structure interact. It is important to note that these levels of structure exist and vary across trials, even if the display consists of randomly positioned objects that have randomly selected feature values. The visual system efficiently extracts and encodes structure from the spatial and featural information across the visual scene, even when, in the long run over displays, there may not be any consistent regularities. This suggests that any theory of visual working memory that specifies only the representation of individual items or groups cannot be a complete model of visual working memory. The effects of stored knowledge on visual working memory Most visual working memory research requires observers to remember meaningless, unrelated items, such as randomly selected colors or shapes. This is done to minimize the role of stored knowledge and to isolate working memory limitations from long-term memory. However, in the real world, working memory does not operate over meaningless, unrelated items. Observers have stored knowledge about most items in the real world, and this stored knowledge constrains what features and objects we expect to see and where we expect to see them. The role of such stored knowledge in modulating visual working memory representations has been controversial. In the broader working memory literature, there is clear evidence of the use of stored knowledge to increase the number of items remembered in working memory Biases from stored knowledge One uncontroversial effect of long-term memory on working memory is that there are biases in working memory resulting from prototypes or previous experience. For example, Stored knowledge effects on memory capacity While these biases in visual working memory representations are systematic and important, they do not address the question of whether long-term knowledge can be used to store more items in visual working memory. This Journal of Vision In contrast to this earlier work, Brady, Konkle, and Alvarez (2009) have recently shown clear effects of learned knowledge on working memory. In their paradigm, observers were shown standard working memory stimuli in which they had to remember the color of multiple objects It is possible that Brady, Konkle, and Alvarez (2009) found evidence for the use of stored knowledge in working memory coding because their paradigm teaches associations between items rather than attempting to make the items themselves more familiar. For instance, seeing the same set of colors for hundreds of trials might not improve the encoding of colors or shapes, because the One group of observers saw certain color pairs more often than others (e.g., yellow and green might occur next to each other 80% of the time), whereas the other group saw completely random color pairs. For the group that saw repeated color pairs, the number of color remembered increased across blocks, nearly doubling the number remembered by the random group by the end of the session. Journal of Vision Conclusion Observers have stored knowledge about most items in the real world, and this stored knowledge constrains what features and objects we expect to see and where we expect to see them. There is significant evidence that the representation of items in working memory is dependent on this stored knowledge. Thus, items for which we have expertise, like faces, are represented with more fidelity Visual working memory conclusion A great deal of research on visual working memory has focused on how to characterize the capacity of the system. We have argued that in order to characterize working memory capacity, it is important to take into account both the number of individual items remembered and the fidelity with which each individual item is remembered. Moreover, it is necessary to specify what the units of working memory storage are, how multiple units in memory interact, and how stored knowledge affects the representation of information in memory. In general, we believe that theories and models of working memory must be expanded to include memory representations that go beyond the representation of individual items and include hierarchically structured representations, both at the individual item level (hierarchical feature bundles) and across individual items. There is considerable evidence that working memory representations are not based on independent items, that working memory also stores ensembles that summarize the spatial and featural information across the display, and further, that there are interactions between working memory and stored knowledge even in simple displays. Moving beyond individual items toward structured representations certainly complicates any attempt to estimate working memory capacity. The answer to how many items can you hold in visual working memory depends on what kind of items you are trying to remember, how precisely they must be remembered, how they are presented on the display, and your history with those items. Even representations of simple items have structure at multiple levels. Thus, models that wish to accurately account for the full breadth of data and memory phenomena must make use of structured representations, especially as we move beyond colored dot objects probed by their locations toward items with more featural dimensions or toward real-world objects in scenes. Visual long-term memory Before discussing the capacity of long-term memory, it is important to make the distinction between visual longterm memory and stored knowledge. By "visual long-term memory," we refer to the ability to explicitly remember an image that was seen previously but that has not been continuously held actively in mind. Thus, visual long-term memory is the passive storage and subsequent retrieval of visual episodic information. By "stored knowledge," we refer to the preexisting visual representations that underlie our ability to perceive and recognize visual input. For example, when we first see an image, say of a red apple, stored knowledge about the visual form and features of apples in general enables us to recognize the object as such. If we are shown another picture of an apple hours later, visual long-term memory enables us to decide whether this is the exact same apple we saw previously. While working memory is characterized by its severely limited capacity, long-term memory is characterized by its very large capacity: people can remember thousands of episodes from their lives, dating back to their childhood. However, in the same way that working memory capacity cannot be characterized simply in terms of the number of items stored, the capacity of long-term memory cannot be fully characterized by estimating the number of individual episodes that can be stored. Long-term memory representations are highly structured, consisting of multiple levels of representation from individual items to higher level conceptual representations. Just as we proposed for working memory, these structured representations should be taken into account, both when quantifying and characterizing the capacity of the system and when modeling memory processes such as retrieval. Generally, work in the broader field of long-term memory has not emphasized the nature of stored representations and has focused instead on identifying different memory systems (e.g., declarative vs. nondeclarative, episodic vs. semantic) and understanding the processing stages of those systems, particularly the encoding and retrieval of information (e.g., Critically, in order to account for the range of performance across these manipulations, such models have postulated a role for some form of "psychological similarity" between items, like how many features they share (e.g., Clearly, the more complete our model of the structure and content of long-term memory representations, the more accurately we will be able to model retrieval processes. Thus, the rich, structured nature of long-term memory representations and the role of distinctiveness in long-term memory retrieval pose challenges to quantifying and characterizing the capacity of visual long-term memory. Here, we review recent work that has examined these representation-based issues within the domain of visual long-term memory: What exactly is the content of the representations stored in visual long-term memory? What features of the incoming visual information are critical for facilitating successful memory for those items? By assessing both the quantity and the fidelity of the visual long-term memory representations, we can more accurately quantify the capacity of this visual episodic memory system. By measuring the content of visual long-term memory representations, and what forms of psychological similarity cause this information to be forgotten, we can use memory as a probe into the structure of stored knowledge about objects and scenes.

The ability to keep track of locations in a dynamic, multimodal environment is crucial for successful interactions with other people and objects. The authors investigated the existence and flexibility of spatial indexing in adults and 6-month-old infants by adapting an eye-tracking paradigm from D. C. Richardson and M. J. Spivey (2000). Multimodal events were presented in specific locations, and eye movements were measured when the auditory portion of the stimulus was presented without its visual counterpart. Experiment 1 showed that adults spatially index auditory information even when the original associated locations move. Experiments 2 and 3 showed that infants are capable of both binding multimodal events to locations and tracking those locations when they move. Certain aspects of perception—encoding of object location, binding object properties across modalities, and tracking objects through time and space, for example—have been experimentally isolated and dissected in the literature. By probing the limitations of these systems, researchers have learned much about their pos-sible architectures. A question remains, however, concerning how these abilities operate in concert in the course of everyday life. The

Meaningful visual experience requires computations that identify objects as the same persisting individuals over time, motion, occlusion, and featural change. This article explores these computations in the tunnel effect: When an object moves behind an occluder, and then an object later emerges following a consistent trajectory, observers irresistibly perceive a persisting object, even when the pre- and postocclusion views contrast featurally. This article introduces a new change detection method for quantifying percepts of the tunnel effect. Observers had to detect color changes in displays where several objects oscillated behind occluders and occasionally changed color. Across comparisons with several types of spatiotemporal gaps, as well as manipulations of occlusion versus implosion, performance was better when objects ’ kinematics gave the impression of a persisting individual. The results reveal a temporal same-object advantage: better change detection across temporal scene fragments bound into the same persisting object representations. This suggests that persisting objects are the underlying units of visual memory.

This study examined the role of attention in maintaining feature bindings in visual short-term memory. In a change-detection paradigm, participants attempted to detect changes in the colors and orientations of multiple objects; the changes consisted of new feature values in a feature-memory condition and changes in how existing feature values were combined in a binding-memory condition. In the critical experiment, a demanding visual search task requiring sequential shifts of spatial attention was interposed during the delay interval of the change-detection task. If attention is more important for the maintenance of feature bindings than for the maintenance of unbound feature values, the attention-requiring search task should specifically disrupt performance in the binding-memory task. Contrary to this proposal, it was found that memory for bindings and memory for features were equally impaired by the search task.

It has recently been shown, using functional magnetic resonance imaging with a change detection paradigm, that activity in posterior parietal cortex (PPC) correlates with the limited number of objects held in visual short-term memory (VSTM). We replicate this finding and extend it to tasks that use similar stimuli, but without explicit memory requirements. As well as a perceptual task used previously (detecting an item at fixation), 2 additional tasks were designed to increase attentional demands across space (searching for a red item anywhere in the array) and across both space and time (detecting a staggered offset after prolonged viewing of the array). During the VSTM task, a capacity-limited set-size effect was seen in PPC as well as occipital and frontal regions. However, the PPC showed similar activity during 2 of the tasks not requiring VSTM. These findings cannot easily be explained by behavioral perfor-mance measures or memory demands alone, suggesting a role of the PPC in processing a limited number of discrete object representations, whether in the current perceptual scene or working memory. The differential influence of item load across perceptual tasks is consistent with task requirements affecting the form of these representations.

The information that individuals can hold in working memory is quite limited, but researchers have typically studied this capacity using simple objects or letter strings with no associations between them. However, in the real world there are strong associations and regularities in the input. In an information theoretic sense, regularities introduce redundancies that make the input more compressible. The current study shows that observers can take advantage of these redundancies, enabling them to remember more items in working memory. In 2 experiments, covariance was introduced between colors in a display so that over trials some color pairs were more likely to appear than other color pairs. Observers remembered more items from these displays than from displays where the colors were paired randomly. The improved memory performance cannot be explained by simply guessing the high-probability color pair, suggesting that observers formed more efficient representations to remember more items. Further, as observers learned the regularities, their working memory performance improved in a way that is quantitatively predicted by a Bayesian learning model and optimal encoding scheme. These results suggest that the underlying capacity of the individuals ’ working memory is unchanged, but the information they have to remember can be encoded in a more compressed fashion.

■ Does the capacity of visual short-term memory (VSTM) de-pend on the complexity of the objects represented in memory? Although some previous findings indicated lower capacity for more complex stimuli, other results suggest that complexity ef-fects arise during retrieval (due to errors in the comparison pro-cess with what is in memory) that is not related to storage limitations of VSTM, per se. We used ERPs to track neuronal activity specifically related to retention in VSTM by measuring the sustained posterior contralateral negativity during a change detection task (which required detecting if an item was changed between a memory and a test array). The sustained posterior contralateral negativity, during the retention interval, was larger for complex objects than for simple objects, suggesting that neu-rons mediating VSTM needed to work harder to maintain more complex objects. This, in turn, is consistent with the view that VSTM capacity depends on complexity. ■

Two dot arrays, each containing a different set of 6 randomly selected locations from a 5x5 matrix, were presented briefly and separated by an inter-stimulusinterval (ISI) of 0, 200, 500, or 1500ms. Subjects were asked to remember these locations and to report whether a probe dot matched the memory locations. To find out whether subjects formed an integrated representation of the two arrays, the probe dot was accompanied by matrix elements from the first array, from the second array, or from both arrays. Memory for array 1 was significantly impaired when the retrieval context was drawn from array 2, and vice versa, suggesting that the two arrays were represented separately. This effect was observed only at an ISI of 500ms or longer. We propose that as array 1 is better consolidated, it representation becomes more separated from that of array 2. 2 An important challenge confronting the human visual system during natural viewing is to extract complex visual information and to retain it momentarily. A lot of vision research has focused on how the visual system perceives objects and scenes (Palmer, 1999), and how such information is retained in visual short-term memory (Intraub, 1997; Jiang, Olson, & Chun, 2000; Luck & Vogel, 1997; Phillips, 1974; Rensink, O'Regan, & Clark, 1997; Wheeler & Treisman, 2002). Recent studies suggest that approximately four visual objects or six spatial locations can be stored in VSTM simultaneously (Luck & Vogel, 1997; Pashler, 1988). More features can be stored in VSTM if they conjoin to form integrated objects than if they are separate (Lee & Chun, 2001; Luck & Vogel, 1997; Olson & Jiang, 2002; Xu, 2002; Wheeler & Treisman, 2001). These studies have focused on the representation of a single visual display in VSTM. However, visual events evolv...