Industry collaboration – Spotify

Description

At the SCS department, in the ICT School at KTH, as part of the SeRC “Big Data for Social Science” project, a collaboration is going on with Spotify on their peer-to-peer and Big Data problems. Spotify's client produces an enormous amount of data that is stored in large-scale storage and as of early 2014, they manage many petabytes of data. This “Big Data” is processed to produce both reports for the copyright owners of the music that has been listened to, as well as to create value for Spotify AB.

 

Two PhD-projects have been involved in this collaboration so far and the first project finished in December 2013. That project aimed at reducing the power consumed by smartphones running the Spotify client, while still enabling the smartphone clients to use Spotify's peer-to-peer (P2P) network. Spotify's P2P network handles over 90% of their network traffic, and thus generates significant savings in bandwidth costs for Spotify. The main contribution of the work was to reduce the “trickling” traffic, while maintaining energy consumption at a level comparable to the (more expensive for Spotify) client-server configuration.

 

The other PhD project is still ongoing and deals with improvement with one of the back end tasks at Spotify that involves cleaning up meta-data. This is currently done by a sequence of operations forming a pipeline. This pipeline is executed daily and takes several hours to complete, where most time is spent on format conversations and disk I/O. By improving the pipeline operations there should be a performance increase due to reduced disk I/O and also a decrease in maintenance and development times for that pipeline.

 

Status

Ongoing (2011-)

 

Contact person
Anders Arpteg, arpteg@spotify.com