Memento logo Memento

Modeling Effects of Semantics and Decay on Video Memorability

Anelise Newman*1, Camilo Fosco*1, Vincent Casser1, Allen Lee1, Barry McNamara1, and Aude Oliva1
1Massachusetts Institute of Technology, *Equal Contribution

A key capability of an intelligent system is deciding when events from past experience must be remembered and when they can be forgotten. Towards this goal, we develop a predictive model of human visual event memory and how those memories decay over time. We introduce Memento10k, a new, dynamic video memorability dataset containing human annotations at different viewing delays. Based on our findings we propose a new mathematical formulation of memorability decay, resulting in a model that is able to produce the first quantitative estimation of how a video decays in memory over time. In contrast with previous work, our model can predict the probability that a video will be remembered at an arbitrary delay. Importantly, our approach is multimodal, combining visual and semantic information to fully represent the meaning of events. Our experiments on two video memorability benchmarks, including Memento10k, show that our model significantly improves upon the best prior approach (by 12% on average).

Short Video Presentation (1 minute)
Full Video Presentation (10 minutes)

Dataset icon Memento10k Dataset

Memento10k Image

We present a new multi-temporal, multimodal memory dataset of 10,000 video clips, Memento10k. With 900,000 human memory annotations at different delay intervals and 50,000 captions describing the events, it is the largest repository of dynamic visual memory data.

code icon Code

Memento code Image

Download the code for the Memento project here.

Paper icon Paper

dataset logo
  • Newman, A.*, Fosco, C.*, Casser, V., Lee, A., Mcnamara, B., Oliva, A. Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability. European Conference on Computer Vision (ECCV), 2020

Demo icon Demo

Examine the memorability of scenes within a video using our online demo. Simply select a video from our list of examples or upload your own.