Smarter smartphones for smarter people

This is a piece I wrote for the BBC World Service..

So, the iPhone 5 is here, and while it will sell well, probably better than any phone before it, there’s a sense of anticlimax: this, we are told, is evolution, not revolution. None of the mind-bending sense of newness and change that the iPhone and iPad used to engender. This is a sign, we’re told, that the market is mature, that there’s not much more that can be done.

I’d like to suggest another way of looking at this. For sure, not every new product that comes out of Apple HQ can blow our minds. But that doesn’t mean the mobile device is now doomed for a stodgy and reliable plateau of incremental improvements, like cars, washing machines or TVs.

In fact, quite the opposite. The world of the mobile device has already made extraordinary changes to our world, and we’re only at the start of a much larger set of changes. Our problem is that we’re just not very good judging where we sit amidst all this upheaval.

Consider these little factlets from a survey conducted last year by Oracle. At first glance they seem contradictory, but I’ll explain why they’re not.

More than half of those surveyed thought their mobile phone would replace their iPod/MP3 player by 2015. A year later when they asked them again, a third said it already had. Oracle found more or less the same was true of people’s global positioning systems, or GPS.

Then there’s this. More than two thirds of the people surveyed said they use a smartphone, and of those people, 43% have more than one.

In other words, more and more functions that used to be a separate device are now part of our mobile phone. And yet at the same time a significant chunk of users have more than one mobile phone.

What this means, I think, is that we are integrating mobile phones into our lives in a way that even those who spend time researching this kind of thing don’t really get. In fact we’ve integrated them so much we need two.

That’s because, of course, they’re not really phones: they’re devices that connect us to all sorts of things that we hold dear, whether it’s social, work or personal.

But there’s still a long way to go. The device of the future will make everything more seamless. A company in Thailand, for example, allows you to use your smartphone to open your hotel door, tweak the room lights and air con, order food and switch TV channels.

In other words interact with your surroundings. Some via connected devices, from air conditioning units to washing machines, from street signs to earthquake sensors. Other services will sift piles and piles of big data in the cloud, and push important information to us when we need it. Google already has something called Google Now which tries to anticipate your problems and needs before you do: a traffic jam up ahead, a sudden turn in the weather, a delayed flight.

Devices will also interact with the disconnected world, measuring it for us — whether it’s our blood sugar levels or the air quality. Sense movement, odors, colors, frequencies, speed. It may even, one day, see through walls for us.

So our smart phones are just starting to get smart. We’re already smart enough to see how useful they can be. The bits that are missing are the technologies that blend this all together. This could still take some time, but don’t for a moment think the mobile world is about to get boring.

Social Media and Politics: Truthiness and Astroturfing

By Jeremy Wagstaff

(this is a column I wrote back in November. I’m repeating it here because of connections to astroturing in the HBGary/Anonymous case.)

Just how social is social media? By which I mean: Can we trust it as a measure of what people think, what they may buy, how they may vote? Or is it as easy a place to manipulate as the real world.

The answers to these questions aren’t of academic interest only. They go right to the heart of what may be our future. More and more of our world is online. And more and more of our online world is social media: A quarter of web pages viewed in the U.S. are on Facebook. So it’s not been lost on those who care about such things that a) what we say online may add up to be a useful predictor of what we may do at the shops, the movies, at the polling booth. And b) that social media is a worthwhile place to try to manipulate what we think, and what we do at the shops, the movies—and at the ballot box.

There is plenty of evidence supporting the former. Counting the number of followers a candidate has on Facebook, for example, is apparently a pretty good indicator of whether they’ll do well at the ballot box. The Daily Beast set up something called the Oracle which scanned 40,000 websites—including Twitter—to measure whether comments on candidates in the recent U.S. elections were positive, negative, neutral or mixed. It predicted 36 out of 37 Senate races and 29 out of 30 Governors’ races and nearly 98% of the House races. That’s pretty good.

Dan Zarrella, a self-styled social media scientist, counted the followers of the twitter feeds of 30 senate, house and governor races and found that in 71% of the races, the candidate with the most Twitter followers was ahead in the polls. And Facebook found that candidates with more Facebook fans than their opponents won 74% of House races, and 81% of Senate races. More than 12 million people used the “I Voted” button this year, more than double that in 2008.

Why is this interesting? Well, social media, it turns out, is quite a different beast to even recent phenomena such as blogs. Social media, it turns out, really is social, in that more than previous Internet methods of communication, it reflects the views of the people using it. It is, one might say, democratic.

A study by researchers from the Technical University of Munich of the 2009 federal parliamentary elections in Germany, for example, revealed that, in contrast to the bulletin boards and blogs of the past, Twitter was reflective of the way Germans voted. Unlike bulletin boards and blogs, they wrote, “heavy users were unable to impose their political sentiment on the discussion.” The large number of participants, they found, “make the information stream as a whole more representative of the electorate.”

In other words, social media is as much a battleground for hearts and minds as the rest of the world. Even more so, perhaps, because it’s easier to reach people. Forget knocking on doors or holding rallies: Just build a Facebook page or tweet.

And, maybe, hire some political operators to build a fake movement, aka astroturfing?

Astroturfing, for those not familiar with the term, is the opposite of grassroots. If you lack the support of ordinary people, or don’t have time to get it, you can still fake it. Just make it look like you’ve got grassroots support. Since the term was coined in the mid 1980s it’s become popular activity by marketers, political operators and governments (think Chinese 50-cent blogging army). Astroturfing, in short, allows a politician to seem a lot more popular than he really is by paying folk to say how great he is.

Whether social media is ripe for astroturfing isn’t clear. On one hand, we know that the Internet is full of fakery and flummery: Just because your inbox is no longer full of spam doesn’t mean the Internet isn’t full of it—87%, according to the latest figures from MessageLabs. You don’t see it because the filters are getting better at keeping it away from you. Twitter, by contrast, is much less spammy: the latest figures from Twitter suggest that after some tweaks earlier this year the percentage of unwanted messages on the service is about 1%.

So Twitter isn’t spammy, and it broadly reflects the electorate. But can it be gamed?

We already know that Twitter can spread an idea, or meme, rapidly—only four hops are needed before more or less everyone on Twitter sees it. In late 2009 Google unveiled a new product: Real time search. This meant that, atop the usual results to a search, Google would throw in the latest matches from the real time web—in other words, Twitter and its ilk. So getting your tweets up there would be valuable if, say, you were a political operator and you wanted people to hear good things about your candidate, or bad things about your rival. But were people doing this? Two researchers from Wellesley College in Massachusetts wondered.

Panagiotis Takis Metaxas and Eni Mustafaraj studied the local senate race and found that they were. They looked at 185,000 Twitter messages which mentioned the two competing candidates and found that there was plenty of astroturfing going on—where political supporters were creating fake accounts and repeating each other’s messages, and sending them to likely sympathizers, in the hope of their messages hitting the mainstream.

The researchers found one group, apparently linked to an Iowa Republican group, was sending out one tweet a second linking to websites “exposing” their rival’s missteps and misstatements. Overall, the message they sent reached more than 60,000 users. The researchers concluded that “the fact that a few minutes of work, using automated scripts and exploiting the open architecture of social networks such as twitter, makes possible reaching a large audience for free…raises concerns about the deliberate exploitation of the medium.”

The point here is not merely that you’re propagating a point of view. That’s just spam. But by setting up fake Twitter accounts and tweeting  and then repeating these messages, you’re creating the illusion that these views are widespread. We may ignore the first Twitter message we see exposing these views and linking to a website, but will we ignore the second or the third?

This discovery of Twitter astroturfing in one race has prompted researchers at Indiana University to set up a tool they call Truthy—after comedian Stephen Colbert’s term to describe something that someone knows intuitively from the gut—irrespective of evidence, logic or the facts. Their tool has exposed other similar attacks which, while not explosive in terms of growth, are, they wrote in an accompanying paper,  “nevertheless clear examples of coordinated attempts to deceive Twitter users.” And, they point out, the danger with these Twitter messages is that unless they’re caught early, “once one of these attempts is successful at gaining the attention of the community, it will quickly become indistinguishable from an organic meme.”

This is all interesting, for several reasons. First off, it’s only in the past few months that we’ve woken up to what political operators seem to be doing on Twitter. Secondly, while none of these cases achieves viral levels, the relative ease with which these campaigns can be launched suggests that a lot more people will try them out. Thirdly, what does this tell us about the future of political manipulation in social media?

I don’t know, but it’s naïve to think that this is just an American thing. Or a ‘what do you expect in a thriving democracy?’ thing. Less democratically minded organizations and governments are becoming increasingly sophisticated about the way they use the Internet to control and influence public opinion. Evgeny Morozov points to the Lebanon’s Hezbollah, “whose suave manipulation of cyberspace was on display during the 2006 war with Israel”; my journalist friends in Afghanistan say the Taliban are more sophisticated about using the Internet than the Karzai government or NATO.

The good news is that researchers are pushing Twitter to improve their spam catching tools to stop this kind of thing from getting out of hand. But I guess the bigger lesson is this: While social media is an unprecedented window on, and reflection of, the populace, it is also an unprecedented opportunity for shysters, snake oil salesmen and political operators to manipulate what we think we know.

It may be a great channel for the truth, but truthiness may also be one step behind.

Social Media and Politics: Truthiness and Astroturfing

(This is a longer version of my syndicated newspaper column)

By Jeremy Wagstaff

Just how social is social media? By which I mean: Can we trust it as a measure of what people think, what they may buy, how they may vote? Or is it as easy a place to manipulate as the real world?

The answers to these questions aren’t of academic interest only. They go right to the heart of what may be our future. More and more of our world is online. And more and more of our online world is social media: A quarter of web pages viewed in the U.S. are on Facebook. So it’s not been lost on those who care about such things that a) what we say online may add up to be a useful predictor of what we may do at the shops, the movies, at the polling booth. And b) that social media is a worthwhile place to try to manipulate what we think, and what we do at the shops, the movies—and at the ballot box.

There is plenty of evidence supporting the former. Counting the number of followers a candidate has on Facebook, for example, is apparently a pretty good indicator of whether they’ll do well at the ballot box. The Daily Beast set up something called the Oracle which scanned 40,000 websites—including Twitter—to measure whether comments on candidates in the recent U.S. elections were positive, negative, neutral or mixed. It predicted 36 out of 37 Senate races and 29 out of 30 Governors’ races and nearly 98% of the House races. That’s pretty good.

Dan Zarrella, a self-styled social media scientist, counted the followers of the twitter feeds of 30 senate, house and governor races and found that in 71% of the races, the candidate with the most Twitter followers was ahead in the polls. And Facebook found that candidates with more Facebook fans than their opponents won 74% of House races, and 81% of Senate races. More than 12 million people used the “I Voted” button this year, more than double that in 2008.

Why is this interesting? Well, social media, it turns out, is quite a different beast to even recent phenomena such as blogs. Social media, it turns out, really is social, in that more than previous Internet methods of communication, it reflects the views of the people using it. It is, one might say, democratic.

A study by researchers from the Technical University of Munich of the 2009 federal parliamentary elections in Germany, for example, revealed that, in contrast to the bulletin boards and blogs of the past, Twitter was reflective of the way Germans voted. Unlike bulletin boards and blogs, they wrote, “heavy users were unable to impose their political sentiment on the discussion.” The large number of participants, they found, “make the information stream as a whole more representative of the electorate.”

In other words, social media is as much a battleground for hearts and minds as the rest of the world. Even more so, perhaps, because it’s easier to reach people. Forget knocking on doors or holding rallies: Just build a Facebook page or tweet.

And, maybe, hire some political operators to build a fake movement, aka astroturfing?

Astroturfing, for those not familiar with the term, is the opposite of grassroots. If you lack the support of ordinary people, or don’t have time to get it, you can still fake it. Just make it look like you’ve got grassroots support. Since the term was coined in the mid 1980s it’s become popular activity by marketers, political operators and governments (think Chinese 50-cent blogging army). Astroturfing, in short, allows a politician to seem a lot more popular than he really is by paying folk to say how great he is.

Whether social media is ripe for astroturfing isn’t clear. On one hand, we know that the Internet is full of fakery and flummery: Just because your inbox is no longer full of spam doesn’t mean the Internet isn’t full of it—87%, according to the latest figures from MessageLabs. You don’t see it because the filters are getting better at keeping it away from you. Twitter, by contrast, is much less spammy: the latest figures from Twitter suggest that after some tweaks earlier this year the percentage of unwanted messages on the service is about 1%.

So Twitter isn’t spammy, and it broadly reflects the electorate. But can it be gamed?

We already know that Twitter can spread an idea, or meme, rapidly—only four hops are needed before more or less everyone on Twitter sees it. In late 2009 Google unveiled a new product: Real time search. This meant that, atop the usual results to a search, Google would throw in the latest matches from the real time web—in other words, Twitter and its ilk. So getting your tweets up there would be valuable if, say, you were a political operator and you wanted people to hear good things about your candidate, or bad things about your rival. But were people doing this? Two researchers from Wellesley College in Massachusetts wondered.

Panagiotis Takis Metaxas and Eni Mustafaraj studied the local senate race and found that they were. They looked at 185,000 Twitter messages which mentioned the two competing candidates and found that there was plenty of astroturfing going on—where political supporters were creating fake accounts and repeating each other’s messages, and sending them to likely sympathizers, in the hope of their messages hitting the mainstream.

The researchers found one group, apparently linked to an Iowa Republican group, was sending out one tweet a second linking to websites “exposing” their rival’s missteps and misstatements. Overall, the message they sent reached more than 60,000 users. The researchers concluded that “the fact that a few minutes of work, using automated scripts and exploiting the open architecture of social networks such as twitter, makes possible reaching a large audience for free…raises concerns about the deliberate exploitation of the medium.”

The point here is not merely that you’re propagating a point of view. That’s just spam. But by setting up fake Twitter accounts and tweeting  and then repeating these messages, you’re creating the illusion that these views are widespread. We may ignore the first Twitter message we see exposing these views and linking to a website, but will we ignore the second or the third?

This discovery of Twitter astroturfing in one race has prompted researchers at Indiana University to set up a tool they call Truthy—after comedian Stephen Colbert’s term to describe something that someone knows intuitively from the gut—irrespective of evidence, logic or the facts. Their tool has exposed other similar attacks which, while not explosive in terms of growth, are, they wrote in an accompanying paper,  “nevertheless clear examples of coordinated attempts to deceive Twitter users.” And, they point out, the danger with these Twitter messages is that unless they’re caught early, “once one of these attempts is successful at gaining the attention of the community, it will quickly become indistinguishable from an organic meme.”

This is all interesting, for several reasons. First off, it’s only in the past few months that we’ve woken up to what political operators seem to be doing on Twitter. Secondly, while none of these cases achieves viral levels, the relative ease with which these campaigns can be launched suggests that a lot more people will try them out. Thirdly, what does this tell us about the future of political manipulation in social media?

I don’t know, but it’s naïve to think that this is just an American thing. Or a ‘what do you expect in a thriving democracy?’ thing. Less democratically minded organizations and governments are becoming increasingly sophisticated about the way they use the Internet to control and influence public opinion. Evgeny Morozov points to the Lebanon’s Hezbollah, “whose suave manipulation of cyberspace was on display during the 2006 war with Israel”; my journalist friends in Afghanistan say the Taliban are more sophisticated about using the Internet than the Karzai government or NATO.

The good news is that researchers are pushing Twitter to improve their spam catching tools to stop this kind of thing from getting out of hand. But I guess the bigger lesson is this: While social media is an unprecedented window on, and reflection of, the populace, it is also an unprecedented opportunity for shysters, snake oil salesmen and political operators to manipulate what we think we know.

It may be a great channel for the truth, but truthiness may also be one step behind.

Q&A: X1 and The Future of Finding Stuff

  Full text of email interview with Mark Goodstein of X1 (see my column in WSJE and FEER this week)
 
– Who are you aiming at with this product?
 
Not to be too simplistic, we’re aiming at two groups: consumers and professionals, specifically those who have a lot of email and files and who spend more time than they want searching for information on the Internet or intranet. The free version offers a substantial set of features that we hope will entice legions of users to use the product at
home and work, for all their information finding needs. The pro version has features that power users will demand, like indexing network drives and viewing files in their native formats, regardless of whether they have the native application installed. Both versions will continue to get richer over the coming weeks and months, as we add more consumer features, like media-specific tabs (pictures, music, etc.) and more powerful web searching and eCommerce-related features. The pro version will get support for indexing attachments, contacts, events, PDFs, and archives. We think these two prongs will encourage great numbers of people to use the product and will eventually allow us to crack the enterprise market, which is straining for simple interfaces to complex data: X1’s specialty.
 
 
– I’ve always thought this kind of product was really basic, and when Enfish came out in 1999, I assumed it would be massive. But it wasn’t, and nothing since has really caught on. Why is this? Does it have to do with new paradigms, or just the product wasn’t right, or people aren’t ready for it, or what?
 
Our approach isn’t that much different than others, but we’re staying focused on simplicity and speed. X1’s interface is visceral and innovative: allowing the user to winnow the searches down from all to just a few, instantly, as opposed to the normal none to many (sometimes with a coffee break) of today’s search engines and desktop search utilities. This interface gives the user the feeling of control over chaos, which is hard to underestimate. Many people have built up complicated directory structures for storing their files and email, all in an effort to just keep track. X1 allows the user to stop caring about the organization and more about the work!
 
This is a difficult question to answer because it seems like Enfish and others have done many of the things we’ve done, but several years in advance. I’m not sure why they failed to catch on like you assumed, but I don’t think the fundamentals have changed. The amount of data we’re responsible for is large and always growing; it’s in disparate formats and locations; the tools that help users wade into this sea of information are, maybe justifiably, difficult to understand and use; and there’s no incentive for market leaders, like Microsoft, to innovate. It doesn’t help that the dotcom bubble excited expectations and the companies responsible never followed through.
 
That said, we really do think we’ve created a beautiful interface to complicated data sets. We think of it as something between a spreadsheet and a database. So, like you said, Enfish should have caught on big, and didn’t. Just like databases were supposed to catch on big at the end-user level, and didn’t. Spreadsheets have tried to fill the gap,
becoming more database-y over time. But that’s a little ridiculous, as many people have come to realize.
 
– What’s under the hood? Presumably these programs have different technologies underpinning them? Could you explain a little of the challenges to minimize the downside of such programs — index size, performance loss, ease of use, success ratio of finding what you’re looking for, etc?
 
I assume most indexing technologies are actually pretty close cousins, separated by clever coding and intelligent choices. We all deal with the same limitations of compression, physical memory, disk space, etc., and all have to make trade-offs to deliver a product to market. X1 has an inverted index with all sorts of clever tricks to manage memory and
processor use to keep the indexing as invisible and painless as possible. Our goal, from the beginning, was to make a product that was as simple to use as possible, as fast as a machine would allow, and as invisible as possible. We’ve had success on all fronts and we’ll continue to improve and innovate as time goes by. We think the bottom line here is speed and simplicity. Speed allows us to skip all those complicated, frankly under-used, search features, while allowing the user to iteratively search (quickly) through their data. They may search twice before success, but certainly it’ll be faster and more satisfying. This is compounded by our innovative multi-field search interface. That’s it.
 
– Where do you see this going? Is searching a hard drive going to get more sophisticated a la data mining? Or is this a rough and ready product that will always fit the brute force approach?
 
Not to harp on this too much, but we honestly believe that our mission will be fulfilled and we’ll achieve big success if we stick to our dual goals of speed and simplicity. We can let Oracle do the OLAP while we do away with the DBA…