Extract-Transform-Load for Video Streams
Ferdinand Kossmann Ziniu Wu Eugenie Lai Nesime Tatbul Lei Cao Tim Kraska Samuel Madden
Social media, self-driving cars, and traffic cameras produce video streams at
large scales and cheap cost. However, storing and querying video at such scales
is prohibitively expensive. We propose to treat large-scale video analytics as
a data warehousing problem: Video is a format that is easy to produce but needs
to be transformed into an application-specific format that is easy to query.
Analogously, we define the problem of Video Extract-Transform-Load (V-ETL).
V-ETL systems need to reduce the cost of running a user-defined V-ETL job while
also giving throughput guarantees to keep up with the rate at which data is
produced. We find that no current system sufficiently fulfills both needs and
therefore propose Skyscraper, a system tailored to V-ETL. Skyscraper can
execute arbitrary video ingestion pipelines and adaptively tunes them to reduce
cost at minimal or no quality degradation, e.g., by adjusting sampling rates
and resolutions to the ingested content. Skyscraper can hereby be provisioned
with cheap on-premises compute and uses a combination of buffering and cloud
bursting to deal with peaks in workload caused by expensive processing
configurations. In our experiments, we find that Skyscraper significantly
reduces the cost of V-ETL ingestion compared to adaptions of current SOTA
systems, while at the same time giving robustness guarantees that these systems
are lacking.