Encode-Store-Retrieve: Enhancing Memory Augmentation through
  Language-Encoded Egocentric Perception
Junxiao Shen John Dudley Per Ola Kristensson
We depend on our own memory to encode, store, and retrieve our experiences.
However, memory lapses can occur. One promising avenue for achieving memory
augmentation is through the use of augmented reality head-mounted displays to
capture and preserve egocentric videos, a practice commonly referred to as life
logging. However, a significant challenge arises from the sheer volume of video
data generated through life logging, as the current technology lacks the
capability to encode and store such large amounts of data efficiently. Further,
retrieving specific information from extensive video archives requires
substantial computational power, further complicating the task of quickly
accessing desired content. To address these challenges, we propose a memory
augmentation system that involves leveraging natural language encoding for
video data and storing them in a vector database. This approach harnesses the
power of large vision language models to perform the language encoding process.
Additionally, we propose using large language models to facilitate natural
language querying. Our system underwent extensive evaluation using the QA-Ego4D
dataset and achieved state-of-the-art results with a BLEU score of 8.3,
outperforming conventional machine learning models that scored between 3.4 and
5.8. Additionally, in a user study, our system received a higher mean response
score of 4.13/5 compared to the human participants' score of 2.46/5 on
real-life episodic memory tasks.