“You could have, you could have,” says Hitesh Chawla.
What are those brands, if you could name them?
In his dimly lit glass cabin on the third floor of a mall in Gurugram, Chawla, co-founder of SilverPush, a company which sells technology solutions to target consumers with ads across television and digital media, hesitates slightly. “I am not sure if I should share names.”
“I can tell you one,” he then continues. “They were a large e-commerce company called Askme.com. They aren’t in business, so I can tell you. They were a big client.”
In 2014, the e-commerce company was running ads starring the Bollywood actors Ranbir Kapoor and, later, Kangana Ranaut, Askme’s still-available profile on YouTube shows. Askme, connected to the Malaysian billionaire Ananda Krishnan, shut shop in 2016. Chawla says the company still owes him half a million dollars.
Either of those adverts on TV may have emitted an ultrasonic audio beacon, an identifier embedded in the television signal, using sound close to or beyond the human hearing capability. If you were watching it and your mobile had an app using SilverPush’s software development kit (SDK), it would have promptly pinged the company’s server.
Unbeknownst to the mobile user, this allowed SilverPush and its clients to know which of its adverts were being watched in the vicinity of the device. What’s more, unlike electromagnetic waves used, for example, in Bluetooth, ultrasonic sound waves cannot pass through walls. This can let you precisely determine a device’s location. The aim, at the time, was to have more granularity and real-time data on television viewers by connecting an offline and an online device.
Chawla, 36, comes across as charming in an earthy way. Dressed in a blue sweater, jeans and light shoes with streaks of grey in his curly hair, he later breaks our conversation to take a call in animated, laughing Punjabi. A 2004 batch IIT Delhi chemical engineer who was born near the India border, and spent years in research in Australia, he doesn’t appear to be the stereotypical villain.
Not long ago though, for US privacy groups and tech media, he was.
Things fall apart
It was between October 2015 and March 2016 that SilverPush’s audio beacon model came apart. To follow the trail of events is an object lesson in how the interconnected world operates.
Referencing newspaper reports from the US media and India, the Washington-based Center for Democracy and Technology had written to the US Federal Trade Commission (FTC) in October 2015, in anticipation of a workshop on cross-device tracking.
“Compared to probabilistic tracking through browser fingerprinting, the use of audio beacons is a more accurate way to track users across devices,” the letter read in part. “The industry leader of cross-device tracking using audio beacons is SilverPush.” Citing an Indian newspaper report, it further added that SilverPush’s policy is not to “divulge the names of the apps the technology is embedded” and that the company monitors “18 million smartphones”. (Today, Chawla disputes the 18 million figure.)
Picked up possibly right after the workshop by Forbes magazine for an article, Chawla recalls it as being just after Diwali holidays in November 2015. He began hearing from clients and even a colleague in Australia called him. The Forbes article, headlined “Meet The ‘Ultrasonic’ Tracking Company Privacy Activists Are Terrified Of”, also quoted a SilverPush company executive who states that the tracking was confined to India.
But the damage gets done, says Chawla.
Then, the US FTC wrote a warning letter accompanied by a press release in March 2016, which asked twelve app developers to refrain from using the SilverPush SDK, in turn citing the Forbes article.
A warning letter is a formal legal notice to a private sector entity that the FTC, the US consumer protection and antitrust agency, sends when its staff is considering moving to a civil enforcement action involving the FTC’s commissioners. The FTC has a range of such actions in its regulatory arsenal, from forcing change in behaviour to mandating changes to how products are labelled or described, or even more stringent measures sanctioned by courts.
The FTC can also formally issue rules to govern conduct in a particular industry segment or practice. In information privacy, the FTC has generally taken up the more serious enforcement actions by forcing companies to enter into “consent decrees” by which firms commit to undertaking certain actions in exchange for the FTC not formally moving for civil enforcement measures or other coercive measures against them.
The letter in this case did not find any unlawful action but served as a warning that if actual monitoring was done, it would be a violation of the Federal Trade Commission Act. It was addressed to app developers, not SilverPush, and was signed by a staff member.
According to Chawla, the reason the company came up on the radar was that it was incorporated in California. (It had been part of an American accelerator programme called 500 startups in 2013.) Today, its parent entity is in Singapore and SilverEdge Technologies Pvt. Ltd functions as an Indian subsidiary.
Though Chawla doesn’t seem to want to dwell much on those days, he recalls a dark time. People were spreading rumours that we were out of business, he says. “Company band hi ho jayegi (the company’s going to shut down for sure),” was the refrain he heard.
It affected rounds of investment and a few existing investors wanted to sell. “Two Japanese investors had faith in us,” says Chawla, referring to marketing technology company Freakout and venture capital firm M&S Partners. “I didn’t get the same support from here.”
Staff morale was low. His co-founder Mudit Seth would leave later that year. In the meantime, it was decided that the argument that SilverPush wasn’t active in the US wasn’t enough.
“We didn’t want to keep explaining,” says Chawla, conceding too that the argument that it works for India or outside the US was relatively weak. In the end, the decision was “let’s not build our business on this”. “We decided we were in the right space but we have to find a different way,” he says.
The company had to pivot. The idea was to flip its model around. As Chawla puts it, to allay privacy issues, “we’ll not track the user, we’ll synchronise between TV and digital”.
Back from the brink
In the backroom of SilverPush’s offices run graphics processor units monitoring TV feeds in real time from set-top boxes. For each TV channel, a feed goes into the processor, which breaks it into audio and video. The adverts are then matched and their required online response triggered.
“Let’s say on IPL, as soon as there is an ad break, everyone is on their mobile,” says Chawla, “and you’ll see an ad on Facebook.”
The patents SilverPush has filed give a sense of why it may consider the system a technical achievement. In patent filings published in December 2018, it is stated that to automatically detect an advertisement with older methods took about seven seconds. The claim is that SilverPush’s technology can do this in between 0.6 to one second.
Chawla has found a cosy niche between microtargeting and broadcast advertising, a Venn diagram of convenience, which correlates user behaviour in real-time from TV. And the company claims over 100 brands as clients, including the likes of Unilever, Nestle, Coca-Cola, Cisco, Spotify, Ola and Nissan.
In the aftermath of the FTC letter, after the audio beacon product was done away with, revenues halved. More than halved. “From Rs 20 crore ($2.86 million) they came to around Rs 8 crore ($1.14 million),” says Chawla. But things are looking up after the pivot. “Now we are around Rs 40 crore ($5.71 million).”
Two weeks ago, SilverPush announced it had raised $5 million in a Series B funding round led by existing investor Freakout. Revenues, the company said, grew over 100% in 2018 after it expanded to Southeast Asia, and it’s now targeting revenues of $25 million in the next couple of years.
Chawla shows me SilverPush’s marketing case studies on YouTube, among them a Coca-Cola ad in Thailand during the 2018 football World Cup, for which Facebook data was used to synchronise adverts to a specific demographic.
Coca-Cola was already a sponsor of the World Cup, but wanted to further target football fans aged 15-30 in Bangkok. It had creatives done on the nine teams that were most popular in Thailand. Key in-game moments—goals, penalties, half-time, free kicks and the like—were then synced with advertisements on social media. The ads then pointed to a popular food app, called Lineman.
Chawla says that all the technology does is monitor the TV feed, match the ads it catches with its database and trigger an advertisement on social media to a certain demographic. It would do this even to a user at an office not watching the match. It remains ingeniously well-targeted, and allows for a connect between actions and triggers in ads, though wading through what is correlation and what is causation here can make one’s head hurt.
Other advertising campaigns, available on YouTube, show a type of competitive strategy the technology makes possible. By knowing when rivals’ ads are on TV, brands can target the same audience on what marketers refer to as the second screen—i.e. mobile. Nissan in India and Unilever’s Breeze in the Philippines were two brands that used this product, which SilverPush calls competitive sync.
According to a report last year by market research firm Nielsen, almost half the respondents in a survey of TV viewers in the US said that they used digital devices “very often” or “always” while watching television. In the battle for users’ attention, the smartphone or second screen is increasingly a source of interest for advertisers, around the world. (And in India too.)
Another client, Nestle’s iced tea brand Nestea, synced to the weather, targeting 12-32 year-olds during rains and at temperatures higher than 30 degrees Celsius. Such campaigns do give a small disquiet about being gently prodded by our social media minders.
Beyond TV, SilverPush’s move is to video more broadly, on the net and otherwise. The trigger does not have to be an advertisement, it could be an object, face, logo or emotion, detected using the company’s proprietary machine learning software. Chawla argues that contextual ads based on text have been around but not based on what’s playing on the video itself. “That problem had to be solved,” he says.
After training the machine vision system, around 200,000 videos are processed by the company every day to identify whatever it is that the clients want and trigger a related ad. For instance, “even if in video a character was wearing a Spiderman T-shirt,” it will identify that and perhaps have an ad for the movie pop up alongside the video, says Chawla, whose pitch deck mentions Sony Pictures as a client.
A changing privacy world
In hindsight, it was just in time that Chawla pivoted. Stricter legislation on data security and privacy, like the European General Data Protection Regulation (GDPR), would soon kick in. Privacy researchers would be able to track down apps that were using SilverPush’s SDK and ask Google to remove these from its Play Store. In India too, while a data protection bill is yet to clear Parliament, the Supreme Court in 2017 ruled that privacy is a fundamental right.
What SilverPush was doing was “definitely against the spirit” if not the word of the landmark K.S. Puttaswamy versus Union of India judgement, says Raman Jit Singh Chima, Asia Policy director at Access Now, an international digital rights organisation.
Chima sees the 2016 FTC action against SilverPush as an instance of “outsourcing our legal e-policing”. Rather than regulatory agencies in India taking action (and technically, a few could have), the impetus will come from the US or other countries to enforce privacy standards in emerging technology.
He agrees that there are all kinds of intrusive technologies entering living rooms. From TVs that may know what’s in the room to deeper data recorded by smart devices like Amazon’s Echo. If we are heading into this dystopian future, says Chima, at least we need not rush headlong into it.
The audio beacon fiasco continued to haunt SilverPush for some time. In 2017, technology news website Ars Technica published an article that over 200 apps were still using the company’s audio beacons, a full year after SilverPush had dropped the tech. The article was pegged to a paper by German privacy researchers that were able to track down several apps, whose provenance was mainly from the Philippines, that used SilverPush’s SDK.
Daniel Arp, one of the researchers, clarifies in an email that while most of the apps in their dataset containing the SDK were created before the FTC letter in 2016, there were some created afterwards too. “However, it is possible that the SDK has been embedded into these apps without the knowledge of the company,” he wrote.
According to Chawla, the Ars Technica article is misleading, given that at the time, over 90% of the apps were inactive. But either way, the consensus is that SilverPush is out of audio beacons.
“In response to our article, Google removed all apps carrying the functionality from Google Play,” Arp wrote. The functionality is to “continuously listen for ultrasonic audio beacons, which can be embedded into various media”.
“Moreover, we haven’t found any new apps containing the functionality,” Arp wrote.
Most devices, he added, today run Android 6 or higher, it creates a higher bar for unauthorised access as the user will be informed during run-time if an application wants to access the device’s microphone.
However, close to 18% of Android phones in India still run older versions of Android. Moreover, Chima points out that smartphones, especially those from Chinese manufacturers, use “forked versions” of Android which have their own app stores not governed by Google and may alter Android controls on apps collecting private information.
What are Chawla’s own views on privacy? “Don’t use Android,” he says bluntly. Indeed, he has an iPhone as his device of choice. When I point out that my phone runs Android, and indeed that is the case for a majority in India, he doesn’t say anything.
His broader view is that while some level of data collection is alright, as in the end that’s what “subsidises” free services like Gmail and Facebook, “deep level data” should not be accessible.
What does he mean by “deep level”? Well, he points out that there are several apps that take SMS access, which means if you have banking alerts, that they are privy to your bank balance. He draws the line there.
Telecom companies sell your data, not identifying you but at scale, he says. Then there are data collection agencies. Any marketing manager or ad agency should know a data collection agency—which, for example, analyse SMSes to build consumer profiles—he says.
Does he know, or work with such companies. Yes, he says but doesn’t want to name names. He doesn’t want a Forbes incident visited on them.
Does he believe in the GDPR’s assertion that data belongs to the user, and the user has the right to delete it, if he or she wants? Yes, he does.
Opportunity beckons
It has taken several iterations for SilverPush to get where it is. In 2012 when it was incorporated, it was interested in smart push notifications (hence the name). Then came the audio beacon model. The initial focus on cross-device tracking came because of the realisation that there wasn’t a company in India like Drawbridge, which built profiles of users from data across devices.
The specific implementation, though, was inspired by Shopkick, a US-based company that tracked users at retail outlets, and was brought by SK Telecom of South Korea for $200 million in 2014. “That’s where we got the idea, to do it for TV,” he says.
Audio beacons don’t need Wi-Fi or Bluetooth. “This was the most convenient way that didn’t send [personal data] to the servers,” Chawla says.
Today, companies have found ways to track TV adverts without audio beacons. Facebook filed a patent for such a system in 2018 with audio watermarks.
Referring to new television tracking companies, Arp writes, that a company called Alphonso, covered by The New York Times in 2017, uses a similar technique and even exhibits several parallels to SilverPush. “In contrast to SilverPush, this technique does not require to embed additional beacons into the audio of the media,” he wrote. “Instead, it creates a fingerprint of the audio signal and sends it to an external server.”
“This technique has the same privacy implications as SilverPush,” he added, but it always depends on the exact implementation, whether it is a threat to user privacy. For instance, if an app only uses a random, resettable ID to identify a particular device and if the user is informed that an app uses this technology, the effect on privacy is limited, he wrote.
In the current pivot, SilverPush comes closest to the American machine vision company called Gum-Gum, which also helps advertisers display contextual ads alongside online videos, says Chawla. SilverPush, he adds, is now planning on increasing its presence in Southeast Asia, as well as “see if we can crack the developed markets” like the US.
Going forward, Chawla also wants to experiment with bringing contextual ads from just online video to TV—banner ads that display based on the content being broadcast in real time. And the use cases keep increasing for the existing technology. From instances where companies actively avoid video content they don’t want to be associated with to actual tallying of a film star or a politician’s facetime on TV.
Inevitably, political parties seem to want to be part of the mix. I see a whiteboard where words like state, constituency, leader and SilverPush’s two main products, Parallels (synching social media ads with TV) and Mirrors (contextual ad triggers for streaming video), are mentioned.
Given how much political parties spend on advertising, it doesn’t quite come as a surprise. But I can imagine wanting to be in a place that’s unreachable when the marketing hits.