3 min

When the mobile phone secretly listens

Project of St. Pölten University of Applied Sciences develops defence against unwanted audiotracking by acoustic cookies

When the mobile phone secretly listens
Copyright: St. Pölten UAS

Mobile phones and tablets through so called audiotracking, can be used by means of ultrasound to unnoticeably track the behaviour of their users: for example, viewing certain television advertising or staying in specific rooms and places. In the project SoniControl, St. Pölten  University of Applied Sciences is developing a method for how this undetected (and usually unwanted) spying can be exposed and blocked.

Acoustic cookies

The permanent networking of mobile devices can endanger the privacy of users and lead to new forms of monitoring. For example, a television can emit inaudible acoustic signals ("acoustic cookies") during a commercial. Mobile devices in the environment receive these signals, decode them and send back information to the transmitters of the signal over the Internet.

For instance, it can be ascertained that the television and the mobile device belong to the same person and which advertisement is currently being viewed. This allows user behaviour to be captured similar to the long time use of cookies in the web but across multiple devices and without the knowledge of users.

Detecting and blocking

The SoniControl project of St. Pölten UAS is developing a mobile application that detects acoustic cookies, alerts the users and if desired, blocks tracking.

"Acoustic tracking is particularly dangerous for users and attractive for content providers, because users notice nothing and can be tracked across multiple devices. The aim of the project is to create awareness of acoustic tracking and to purposefully protect privacy by recognizing and filtering acoustic cookies", said Matthias Zeppelzauer, Head of the project and Senior Researcher at the Research Group Media Computing in the Institute for Creative\Media/Technologies at  St. Pölten UAS.

Determining interests and location

Such acoustic signals are used for so-called "cross-device tracking". This makes it possible to track the user's behaviour across multiple devices, and relevant user profiles can be merged with one other. In this way, more accurate user profiles can be created for targeted advertising and filtering of Internet content.

Unlike their electronic counterparts when visiting web pages, up to now it has not been possible to block acoustic cookies. In addition, in order to allow mobile phones to communicate with one another, acoustic cookies can be sent via mobile phone speakers actively and for people inaudibly. In this way, the location and current social environment of users can be identified, for example, which people are nearby.   

"In order to accept voice commands, the mobile phone microphone is permanently active.  Every mobile application that has access to the microphone as well as the operating system itself can at any time without notice: activate the microphone of a mobile terminal, listen to it, detect acoustic cookies and synchronise it over the Internet", said Zeppelzauer. Users are often not informed of this information transmission during ongoing operation. Only a permanent deactivation of the microphone would help, whereby the device as a telephone would become unusable.

Marking from interfering noise

In the project SoniControl, Zeppelzauer and his colleagues are developing a procedure to expose the cookies and inform device users. "We use signal processing methods to detect acoustic cookies. For masking, interference signals are transmitted via the speaker of the mobile device. Thus, acoustic cookies can be neutralized before operating systems or mobile applications can access them", explained Zeppelzauer. Users can selectively block cookies without affecting the functionality of the smartphone.

The masking of the cookies occurs by means of ultrasound, which is inaudible to humans. "There is currently no technology on the market that can detect and block acoustic cookies. The application being developed in this project represents the first approach to give people control over this type of tracking. We want to create a stronger awareness of this technology among users", said Zeppelzauer.

Research open source

All project results and the application will be made publicly available after completion of the funded project; intermediate steps are already documented in advance in a blog. "In this way we would like to promote the shared further development of our technology and the new application, as well as the development of new applications based on our technology", explained Zeppelzauer.

The system is therefore directly usable and expandable for everyone. The application should also be available in the Google Play Store. All project results will be published under Creative Commons License.

With a pitch at this year's Open Source Community Camp of the netidee Internet Foundation Austria, the jury awarded the project first place.


Project SoniControl

The project is funded by the netidee initiative. The support campaign is organized and financed by the non-profit Internet Foundation Austria (IPA).

Duration: January to December 2017.