You are here

SEALINCMedia (Socially-enriched access to linked cultural media)

SocialZap, a multimedia search engine that finds the most interesting fragments, zap points, in a television broadcast based on microblog posts and socially tagged photos


Although computers are far better at computational tasks than humans, humans still have an advantage in other, cognitively complex tasks, like object recognition. When we combine human and computer capacities together, we can benefit from the best of both worlds, which opens new possibilities in deploying ICT in solving socio-economic problems.

The ICT science challenge of SEALINCMedia is exactly this: modeling and evaluating human input for multimedia content access for the purpose of integrating this input with automatic data analysis. The project emphasizes the need for integrating large crowds in the multimedia annotation processes, maximization of the quality of human input and smart use of CPU-power for improving the efficiency and effectiveness of multimedia content access in large data collections. This leads to the notion of human computation power, defined as crowd size quality of input CPU power, which is the main paradigm of the project.

The inspiration for SEALINCMedia comes from the emerging trend of digital heritage going social. Digital cultural heritage is immersing itself in dedicated user communities on the web to increase the accessibility and value of the collection, both as economic asset and merit good. To turn these potential benefits into reality, the challenge is to let the public participate via the full social media paradigm in the process of enriching the heritage collections. The project aims to achieve a virtuous circle, in which social network users interact with the heritage collection to enrich it. This enrichment improves the collection value, but also makes it more interesting for people to interact more with it, which again leads to further interactions and more enrichment.

Biggest results so far

Social Zap, finding the most interesting fragments in a tv-broadcast

For Slideshare Social Zap click on image
​We have developed SocialZap, a multimedia search engine that finds the most interest­ing fragments (‘zap points’) in a television broadcast, based on microblog posts like tweets and socially tagged photos. The main novelty of SocialZap is the fully-au­tomatic transfer of the learned viewer’s interest from textual posts to the visual channel. More.

ICT science question: what happens where in digital video? The fundamental problem in video retrieval is that computers can − at present − only extract low-level features from a video signal, whereas humans interpret the data in a high-level conceptual way. It’s a grand scientific challenge to bridge this so called ‘semantic gap’. In particular we consider here the synchronization of audio and text signals when they refer to one and the same event.

Involved COMMIT/partners: Auxilium, Videodock, Beeld & Geluid, TUDelft, UvA.

MediaMill Semantic Video Search Engine

The demo shows the MediaMill semantic video search engine, a system facilitating access to the content of large collections of video streams. The system is based on a large lexicon of visual concept detectors, complex event detectors and an interactive vid­eo browser learned from example video material. MediaMill provides access to the content of videos without the need to label each and every video. More.

ICT science question: The world is full of digital videos and images. In this deluge of visual information, the grand challenge for computer vision is to unlock its content directly by understanding what is in the image.

Involved COMMIT/partners: UvA, Beeld & Geluid, Videodock.

​Nichesourced annotation of cultural heritage collections

For Slideshare Nichsourced click on image
Cultural heritage institutions more and more provide their collections online. To make them accessible to a interested users, the collections need short descriptions (annotations). Creating such annotations requires knowledge and expertise that is not always possessed by the collection curators. The usage of crowdsourcing and nichesourcing techniques provides cultural heritage institutions with new tools to create high-quality annotations in an inexpensive and fast way. More.

ICT science question: how to design efficient and effective crowdsourced content annotation is still an open research question. With our work we seek answers to the following research questions: How can large crowds and expert niches be activated to share their knowledge? How can expertise be identified on the Web? How can experts be discovered and engaged? How can linked data support and drive the content annotation process? How can semantically enriched knowledge repositories improve search and recommendation applications?

Involved COMMIT/partners: TUDelft, VUAmsterdam, CWI, Rijksmuseum, Beeld & Geluid, Koninklijke Bibliotheek, Gridline, CIT, Nieuwe Prinsenhof