Renee DiResta is the head of policy for Data for Democracy, an Ideas contributor for Wired magazine, and a Mozilla Fellow in Media, Misinformation and Trust.
This is the transcript of an interview with FRONTLINE’s James Jacoby conducted on May 17, 2018. It has been edited in parts for clarity and length.
Let’s start with the birth of your son at the beginning and how that led to your journey in the world of disinformation.
Yeah. My son was born in December 2013, and in November 2014, I started going through the process of getting him on the list for preschool applications, which is an unfortunate thing you’ve got to do in San Francisco. I started looking at the vaccination rates in the local schools, and I was really disappointed to see some were at 38-40 percent, incredibly low. I called my congressman-I’ve never done that before, but I called, and I said: “You know, this seems really unsafe. Why is this the way it is?” And he said, “You know, anytime anybody tries to introduce legislation to change it, there’s a huge, huge public outcry by the anti-vaccine movement, and it’s just really hard to get anything done.”
So, [a] couple months later the measles outbreak happened, the Disneyland measles outbreak, and I called back, and I said, “Hey, this seems like an ever more pressing problem. Two hundred people get sick with the measles. Is there really nothing that we can do about this? It’s a lot of public outcry in reverse right now.” And they said: “Yeah, we’re going to try to do a bill. Would you like to help?” And I said, “Yes, and specifically I would like to help understand the dynamics and the conversations that are happening online, because I am online, and I’m seeing the anti-vaxxers on Twitter, and I’m wondering why this is, where this activity is coming from.” And as the bill made its way through the California Legislature – it was a bill to eliminate vaccine opt-outs. As the bill made its way through the legislature, a lot of the congressmen would poll their districts to see how their districts felt that they should vote, and it would be around 85 percent positive, and then we would go to look at the conversation on Twitter, and we would see that it was overwhelmingly negative, 99 percent negative. Everything [in] the hashtag was negative-a lot of memes, a lot of threats, a lot of harassment. We were saying, “Where is this coming from?”
So I partnered up with a data scientist named Gilad Lotan, and we started mapping the conversation and looking at how it evolved over time, and what we started to notice was the ways that these communities were doing targeted outreach into very specific hashtags and really using it as an organizing tool. But there were also these groups that were coordinating this activity on Facebook, and there were instructional videos being put out on YouTube every night, the night before, telling people what to tweet, who to tweet it at. There were people helping regular people who had just created Twitter accounts create automated accounts, saying, “Here are all the tools that you can use, so you can cue up everything, and we can just always have as many messages as possible being pushed into the hashtag because we need to make sure that when people search for this, we’re the first thing that they see.”
We started to realize that this wasn’t bot activity per se, but it was leveraging automation to ensure that they were just the absolute dominant message in this hashtag at all times and that this coordination was happening across multiple platforms at the same time. And the effect it created-there were a lot of people who created accounts specifically for this. They created extra accounts, so there was fake people, and a lot of the conversation was being run to look like they were a much bigger group than they really were.
I took some of these findings, and I went back to senators, and we started talking about to what extent these were really their constituents weighing in on the bill versus to what extent how many of these people were real at all. We would see things like 0.6 percent of the participants sending out 25 percent of the tweets in the hashtag, so some incredibly automated accounts that were just always on, always on. This was the first indication that groups were doing this mass coordination to shape public opinion about particular policies, particularly smaller groups who were able to leverage the social ecosystem to make themselves look a lot larger and to have an impact. We did wind up getting a law passed in California, in part because I think we were able to show that this wasn’t the majority opinion; this was a small group of people leveraging a social platform to look like the majority opinion. But the way to reconcile what we were seeing online with how we were polling in the district was the fact that there was so much automation and so much outside, non-California voices participating in this hashtag.
That was the first indication that manufactured consensus was becoming an issue in policy campaigns. And shortly after, right around the same time in late 2015, the problem of ISIS on Twitter had gotten to be incredibly large, and I connected with a data scientist named Jonathon Morgan. Jonathon had done an “ISIS Twitter Census,” is what it was called. It was released by the Brookings Institute, and it looked at automation, and it looked at-The same types of tactics that I had seen in the anti-vax community, he was observing with regard to ISIS. And ISIS, again, they weren’t pretending to be something that they weren’t; this wasn’t like Russia, where they were pretending to be Americans. They were just brazenly out there: “This is our message, and we are creating a virtual caliphate, and we are building it on all social platforms simultaneously, and we are coordinating on all platforms simultaneously.” Again, you have this small group of people achieving this kind of disproportionate shared voice and really building a brand, almost, on Twitter, using the social platforms to carry out these very coordinated campaigns.
That was when he and I started comparing notes about how these various groups were leveraging social channels and realizing that we could see it in a number of different groups. It really didn’t matter what the message was. What mattered was that we had built this system, and nobody was really in charge of looking at any kind of systemic coordination. What was basically happening was that we were reliant on each platform to take action individually. So at the time the government was-since ISIS was of course a major global security threat – the government was trying to prevail upon the platforms to take some action here, and there was initially a lot of resistance, a lot of reluctance, in late October 2015 and early 2016, to acknowledge that social network manipulation was a real problem.
When you say reluctance, reluctance by whom?
There was a reluctance by the platforms. There was a lot of concern. You know, if you go back and read articles written in early 2016, you’ll see a lot about, “If we take down ISIS content, what next?” “One man’s terrorist is another man’s freedom fighter.” We don’t want the government to be telling our tech platforms what to silence and what not to silence.” “If the U.S. government does this, authoritarian regimes will do it.” There was a sense that we wanted the platforms to be wholly independent in how they dealt with this problem. At the same time, there was a very strong response from the platform[s] saying that they were very much committed to free speech and didn’t necessarily want to intervene in these things in the early days of the conversation. I think that gradually started to change, I think in part because of public concern, as more and more articles came to light showing that young women had been contacted by ISIS recruiters on Twitter with the intent of recruiting them to come over and be jihadi brides. Young men were radicalizing because of content they were seeing on YouTube and Twitter and then going to launch attacks or going to try to fight overseas.
There was a beginning of a sense that the prevalence of this content, how easy it was to find, the ease with which ISIS activists could reach out to Americans and others on social platforms was leading to the growth of this group. So people started to realize I think, and the public started to get a little bit angry, and that was when we started to see actions beginning to take place.
The Algorithm And Amplification
Right. It’s one thing to say it’s a content moderation issue or a free speech issue. But weren’t you also discovering that the nature of these platforms is that they actually amplify, right? Help me understand systemically why this wasn’t necessarily a free speech issue.
There’s a couple things that happened. First is, if a sufficiently large group of people manage to put out content and get other people to pay attention to it, then the platforms treat that as signal, right? So highly engaging content is seen as something that the platforms want to show to other users. That happens in a couple different ways on search, right? This is a very old phenomenon, people trying to game search algorithms. On the social platforms, it’s a little bit easier to game than on Google because the signals are a little bit less sophisticated.
There is Trending topics. If you can get enough people talking about a particular concept or sharing a particular article or using a hashtag, you’ll see that begin to trend. So on both Facebook and Twitter, it was relatively easy, because there was no editorial oversight, so people would deliberately coordinate just to get something trending, or they would use bots, automated accounts that were not people, to push out this content in an automated way, again with the intent of gaming the Trending algorithm. Once it hit the Trending algorithm, you would see mainstream media come in and pick it up and report on it. This was a way to get mass attention to whatever it was that you wanted people to become more aware of.
Then the third what we call “curatorial surface,” where the platform algorithms are curating what people see, is the recommendation engine. And in the recommendation engine, that’s where the platforms recommend content to you based on one of two things. The first is what we call content-based filtering, where you’ve looked at a particular topic, and the platform knows that you engage with the topic for a longer period of time or you’ve read multiple things on the topic. You’ve communicated that you have an interest in that topic, so the platform shows you more of that topic and then topics related to that.
You like dogs; maybe it will show you some things related to dog shows; maybe it will show you some things related to dog training. There’s a number of different ways the algorithm can go, but it’s all related to that topic. But the other way that the recommendation engine decides what you see is what’s called collaborative filtering. That says that you are similar to me in a couple of different ways. I am interested in dogs; therefore you might be interested in dogs because you’re similar to me. Even though you haven’t actually searched for this topic, the recommendation engine is going to suggest it to you, because it knows that you and I have a fair amount of overlap, and you’re probably interested in some of the topics that I’m interested in.
So that’s where you start to see topic correlations come into play, and sometimes it’s very mundane, [as] in the example of dogs, but if I’m looking at something that’s a little bit more out there-maybe if I’m a chemtrails conspiracy theorist or a flat earth conspiracy theorist, for example. The recommendation engine says a sufficient number of people who are interested in chemtrails and flat earth are also interested in Pizzagate, who are also interested in anti-vaccine content. Even though that person may never have searched for the word “Pizzagate” or searched for an anti-vaccine group, the recommendation engine will proactively serve it up, because it knows that this person is like all these other people, and they like this other type of content. That’s where we start to see the unintended consequences where on YouTube, in particular, on Facebook as well, Facebook groups you see this a lot: People receive suggestions for topics that we would consider a little bit further down the rabbit hole because there is a similarity between them and the people who actually go searching for that stuff.
Bring me back into the timeline, though. Were you trying to study that phenomenon when it came to how anti-vaccination campaigns spread or how disinformation spread?
Yeah, because we were interested in how the growth of the groups was happening. And this is something that we still don’t have a very good handle on as outside researchers, because I can see that an anti-vaccine group is growing, and I can’t see is it organically being searched for. Are people coming to that Facebook group, for example, because of Facebook’s recommendation engine volunteering it to them? We see this a lot with people who appreciate natural parenting, as it’s called, or attachment parenting, or people who are very concerned about organic food. These people are in groups talking about their particular method of raising their children, and if there [are] a lot of parallels to the profiles of the mostly women in the anti-vaccine groups, that’s how that group is targeted with recommendations for anti-vaccine content.
The other way that they can be targeted is very deliberately through the ads interface, and that’s where groups that want to grow their numbers will put out ads. There’s a category on Facebook’s ads interface for parenting, for example, sSo a group can choose to run ads to new parents. They can geographically target. The ability to microtarget is highly sophisticated, both through just the ready-made options that Facebook includes in the interface, and then also through a thing known as Custom Audiences, where if somebody’s engaged with your website or you have an email list, you can target people who have engaged with your content previously and then continue to develop a relationship with those people. This is what we saw Russia do.
Facebook’s Reaction To Warnings
At the time, how concerned were companies like Facebook with the premise or the phenomenon of disinformation being spread on their platform?
Well, they really weren’t. Facebook had a security team, of course, as all the large companies did, but Facebook’s team was much more focused on the bigger threats-threats that it was expecting, spammers or people trying to do identity theft or people selling drugs or guns or other things that were in violation of terms of service. The idea that a state intelligence agency would run an operation targeting Americans to create societal divisions ahead of an election, no, I don’t think that was on their radar. But what I would say is the ISIS problem was very much on the radar, and Facebook did take much more proactive steps to get ahead of it than some of the other platforms did. I think the big difference there was the terrorist content was sort of universally condemned. Even in 2014, we had this “One man’s terrorist is another man’s freedom fighter” narrative, but that was largely gone by 2016.
They were very proactive about dealing with the terrorism problem. But the idea that people were creating polarizing communities or radicalization, not in the terrorism sense of the term but just more general people being pushed into more extreme political groups or political views, that was just seen as hosting content, right? And the algorithms, the recommendation engine, Trending, Search, they were performing exactly as they were designed to perform. The ads interface was targeting people because that’s what it was designed to do. So as far as they were concerned, all of this was functioning exactly as intended, and as long as the specific actions didn’t violate the terms of service, it was kind of like an anything-goes environment for a long time, which is why I think Russia was able to operate largely uninvestigated. This just wasn’t something that people were thinking about.
What were you thinking about? What were you saying at the time about the danger of that entire system?
I was talking about conspiracy theorists, and I was trying to say that we had some really clear indications that that the recommender systems were really pushing people into groups that it wasn’t clear to me we necessarily wanted the platforms volunteering up. I mean, I got served up Pizzagate recommendations before I’d ever heard of Pizzagate, before it was a news story. And then I’m there Googling for “What is Pizzagate?” And once you search for that term, prior to the point where there is a ton of press explainer articles explaining what Pizzagate is and clarifying it as a conspiracy theory, the initial search environment is what we call a news void, right, so the only terms you’re going to see in there are the terms of the people who are creating the content they want you to see. So you’re on Facebook; you’re served up this suggestion that you go look at Pizzagate content. That pushes you to YouTube where a lot of the content is, and then YouTube’s recommendation engines gradually walks you down the path, and Pizzagate is, of course, a big topic on Alex Jones and InfoWars and these other more conspiratorial media properties.
On the flip side, on Google, you’re looking for articles about what this thing is, and you’re just-it’s constantly being reinforced. This is where you see this environment where there’s no pushback. In the early days of a lot of this, of these kind of stories coming into being, the concern that we were having was more about are we serving up, or are our platforms disproportionately amplifying things that are sensational and false, and if that’s happening, are they responsible, or is this just sort of a free-for-all? At the time we would hear things back like, “Well, the antidote to bad speech is more speech,” or “The antidote to people seeing misinformation online is people writing fact-checks,” and things like that. But it seemed like there was no way for those fact-checks to ever get the kind of lift that the original conspiratorial push was getting, in part because it was being amplified by bots on Twitter, Facebook’s recommendation engine, YouTube’s recommendation engine. How do you push back against this wall of misinformation that’s just being pushed at people? And what does that mean for us, if we believe that a democracy requires informed citizens, and our citizens are being fed this nonsense all day long? How do we think about that, what we expect there as a society? I also wasn’t thinking, oh, Russia is going to co-opt this. I was just thinking, ooh, boy, we have some real problems here.”
Did you meet with Facebook?
I didn’t meet with Facebook in any kind of official capacity. I started writing about it, and in response to some of the articles I wrote-I wrote one for Fast Company; I wrote a couple on some blog called Ribbonfarm, looking at what happens to society when we have these perpetual crowds that are moving from one conspiracy to another and trying to recruit the mainstream, trying to bring people in. I wrote a lot about this stuff, and I started getting inbound emails from people who worked at the companies, not in any kind of official capacity but just saying, “Yeah, we’re seeing this, too, and I don’t know what to do about it.” Then, I had some friends there who I could say, “Hey, I just saw this thing; maybe you should look at it.” It was more of an unofficial kind of “How can I help people who are there see what’s going on?”
I did sit down with Twitter. That was more to talk about some of the tactics that went along with automation as it pertained to gaming Trending topics. One of their product managers just wanted to talk about how they could be thinking about this more strategically inside, so there were a couple of things, and Twitter was really much more concerned about harassment than disinformation and manipulation. They hadn’t quite made the connection there that the same tactics that were being used to harass could also be used to just push a ton of nonsense into a feed or a hashtag.
But it seems like what you were basically worried about was a fundamental flaw in the design of the platforms, which is that they’re propagating, they’re pushing bad information toward their users, right?
Yeah it’s a really hard problem, because terrorism violates their Terms of Service provisions about exhortations to violence and these sorts of things, …and now we’ve seen the Terms of Service evolve to deal with some of the harassment issues that the platforms have a little more leeway. There is a strong sense in the valley that-some very loud voices-that moderation is censorship. And it’s a big issue, because it means that the platforms hear more often than not that they shouldn’t be moderating, that moderation is a bad thing, which means that even where they do have some leeway with their Terms of Service to decide what will or will not be allowed, there is a very strong pushback to kind of keep up with this “anything goes” attitude. I think that what we’re starting to see is a shift in public opinion where people are no longer-the negative downstream effects of this “anything goes” policy we’re beginning to see in terms of election manipulation, increasing polarization, increasing radicalization. These are the sorts of things where public sentiment is finally shifting, and that’s what they’re going to listen to more than anything else.
It seems to me what you were raising is different than a content moderation issue, right? What you were raising, it sounds like-and help me clarify this-was not content moderation; it was that bad content or more incendiary or controversial or conspiratorial content was actually given an advantage.
Yes.
Drawbacks Of The Engagement Algorithm
Right. So help me get that.
Yeah. So the platforms’ algorithms, all of them, decide what you’re going to see based on engagement. First of all, it’s like, “Is this something that fits their perception of what you would like?” That’s that content-filtering, collaborating-filtering thing. Then it’s, “Are people engaging with this content?” You might notice, if you put up a baby picture, how quickly you start to-all the likes start to roll in, versus if you share an article. If you’re a person who shares an article and a lot of your friends comment on it, Facebook will push it into people’s feeds because it thinks it knows that you and your audience have that relationship. But if you’re a regular person who shares an article versus a baby picture, the baby picture is what the algorithm is going to push into your friends’ newsfeeds. I think a lot of people don’t realize that what they see is controlled by the algorithm. Once you get away from just pure timeline-based filtering, which is not a good experience once there’s a certain number of users on the platform, the algorithm is weighting what you’re going to see based on what it thinks you and other people like. Baby pictures almost always get a ton of engagement.
Unfortunately, some of the other things that get a ton of engagement are things that make people very angry. This is where you’ll see the really sensational news articles were getting a ton of lift. For a while, click-bait headlines were something that were taking over Facebook, because people would click in to see what was happening, and the algorithm was rewarding that engagement until the platforms ultimately decided that was gaming the system and they needed to downrank that. There is this ongoing arms race with how the algorithm interacts with the type of things people push onto the platform. Because they do control it already, because they’re already controlling the order of things that you see, what we’ve been trying to suggest over the last couple of years now is that they do start to pay attention to the downstream effects of what happens when you’re serving up conspiratorial content nonstop; what happens when you’re so good at predicting who’s going to be receptive to radical content and you serve them that radical content. Well, yes, you’ve kept them on the platform. You’ve gotten that engagement, but is that really what you want to be doing? Is that something that-is that ethical? Is it good for society? These are the much bigger questions where it’s thorny, because there’s a lot of people who really feel that it should be anything goes.
If you want to see that content and the platform serves you that content, that’s great. I feel it should be the other direction. If the platform chooses to host the content and you go searching for it, that’s fantastic, but once the platform proactively gives it to you, finds you and says you’re going to be receptive to this conspiracy theory so I’m going to serve it to you, that’s where I think the platform’s made a decision, and that’s where I think that the platform does bear some responsibility for the polarization and radicalization that we see in our communities.
And you’re calling that out about the anti-vax campaign specifically at the time.
The anti-vax campaign was like a canary in the coal mine, right? It was: how is this movement doing so well online? And it’s because they have really emotionally heart-wrenching tactics. There’s a campaign going on right now where one particular group has decided-it went on to GoFundMe; it raised $10,000-and what it plans to do with that $10,000 is push stories of baby deaths into the feeds of new parents. What it is doing is it is collecting SIDS cases that are correlated with a child getting vaccinations, and it is actively targeting new parents with those stories as ads. And this is a thing that, you know, what Terms of Service does that violate? Is that something that we feel comfortable with, with this behavior happening? Since the ads are targeted to people and they’re not just out there-everyone can’t see it-how do you push back? It’s a really hard thing. Imagine you’re a new parent, and all of a sudden, your Facebook feed is stories of child death and suggesting that if you want to know more, you should go join this anti-vax page and this anti-vax group and this anti-vax mailing list. This group has 140,000 members in it. This is not small.
Aren’t these just marketing campaigns? Isn’t that what it comes down to?
It’s run exactly as a marketing campaign, yeah, and that’s why it’s so challenging, right? There are times that the platforms come in and say things like, “We are not going to support the hosting of pro-anorexia content of pro-suicide content.” Sometimes it’s economics. “We’re not going to support bitcoin-related ICO stories. We’re not going to allow payday lending ads.” So there are times when the platforms come in and say, “This is harmful; it’s harmful to our users, and it’s harmful to society.” That’s usually done in a very ad hoc way, and it leads to a lot of pushback and leads a lot of people to wonder, “Why this topic but not this other topic? How do we think about what’s an OK topic? How do we think about an ethical framework?”
It’s a really big, thorny issue because, yes, the tactics, the way these things are executed, looks exactly like a marketing campaign, and it’s just a marketing campaign for an idea. You don’t really know who’s running the marketing campaigns. You don’t really know who’s paying for the marketing campaigns. You don’t know if, “Do we need the Department of Public Health to run a counter-marketing campaign?,” because you can’t see that the campaign is being run anyway unless you’re actively out there looking for it. It’s a little bit more insidious than selling a product, I think. I think that large parts of public opinion are beginning to think that, too. That’s because we are seeing so much polarization, and we’re trying to figure out, is it the platforms that are causing it, or are they just reflecting it? And that’s an answer that really only the platforms can answer…
Facebook’s Reaction To Warnings
…With Facebook, I know that you had more informal conversations with them about what you were finding, but how would you characterize the response? Was there a deep concern, or was it dismissive? How would you characterize the response?
I think that one of the things that was really difficult in the early conversations with the platforms was gathering enough different stories, like case studies, basically. We could say, “We’re seeing this rise in conspiratorial content,” but we don’t have the hard numbers to back that up. With Twitter we have some hard numbers, right? That’s why the “ISIS Twitter Census” was so powerful, because they did just go in there and meticulously catalog all of these accounts. The anti-vax Twitter mapping we did again. We were like meticulously cataloging the sub-communities and the way that they were growing over time. We couldn’t do that on Facebook. The only thing we had was somebody with a researcher account, which was a very informal thing, by the way-there’s no official researcher account. It was just like you have an account that you use to engage with or look at a particular community and look at the pages, look at the content. It was very anecdotal. There was no way to go to them and say, “Hey, you need to take this more seriously as a systemic issue internally.” That just wasn’t happening.
Even though you asked them to?
We tried to impress upon them that this maybe wasn’t necessarily good for society and, per what you said earlier, it’s a marketing campaign. It’s just a marketing campaign for an idea. What we kept coming back to them with was these are not only ideas; these are manipulative tactics to spread the idea. So this was where the automation piece in particular comes into play, where we would say: “We’re seeing this relationship between what’s happening on Twitter, on Facebook, on YouTube, on Imgur, on Reddit. We’re seeing such incredible coordination across platforms. You’re not looking at what’s happening across platforms; you’re only seeing what’s happening on yours.” We can’t just be looking at the content. We have to be looking at what types of accounts are sharing the content. If these are all sock puppets and bots, that’s a pretty big a red flag, right, that this isn’t just a normal marketing campaign; this is a manipulative campaign. These are people trying to make themselves look bigger, trying to manufacture consensus about a topic when in reality they are a very small fringe group.
We would look at, basically, the dissemination patterns. All of a sudden, does it just appear overnight? All these pages suddenly appear; everybody’s posting the same thing at the same time. We would be looking at the dissemination pattern, the types of accounts doing it, and then the content. The content was usually like third in the list of how we talked about the problem, because we didn’t want it to be about “I don’t like this idea; therefore it shouldn’t exist.” But we wanted the conversation to be: “Here’s evidence of manipulated narratives. Here’s how people are manipulating the narratives. Here is how they’re doing it across the entire social ecosystem. No one is in charge, and we’re just letting this happen.”
And when you said that and laid that out for Facebook, for instance, the response was what?
“People have a right to free speech on the platform.” You know, there were some social pressures that I think it would be impossible to ignore the impact of, right? Which was Facebook’s Trending algorithm had some editorial oversight for a while, and then there were allegations of conservative bias, and they responded by eliminating the human editorial oversight because they felt that adding people increased bias. What this did, though, was it eliminated that last line of defense against purely manipulated campaigns, in the sense that then at that point there was no human looking and saying, “Oh, I can see that this story about Megyn Kelly getting fired from Fox News, that’s just not true.” So what started to happen instead was as soon as they eliminated the human oversight from that Trending topics, in response to allegations that it was biased against conservatives, was that all of a sudden, anything, anyone that could game the algorithm could get to the top of Trending stories. In some ways eliminating the people increased the problem, but because this is a tech company, the sense was like: “Oh, the algorithm is neutral. What the algorithm is going to show us is going to be that there’s no way to accuse the algorithm of bias, so we’ll just eliminate the people, and that solves the problem.” In fact it did the exact opposite.
The idea of the anti-vaccination movement or narrative spreading: Describe for me simply how is that the canary in the coal mine. Why is that the canary in the coal mine?
That was just because it was highly visible. I don’t think that-it’s not related to the content, the topic of vaccines. It’s that the anti-vaccine movement is actually bipartisan or nonpartisan, depending on how you want to frame it. There’s kind of the natural types on the left, and then there is the libertarian “Government doesn’t have a right to tell me to vaccinate my kids” on the right. So it was sort of almost an apolitical example. When you presented the growth and the way that the recommender systems were picking up the content and the fact that these are objectively a very small number of people in society. I mean, when you actually go and look at the numbers, it’s small, but when you look at its prevalence and its representation on social platforms, it’s far larger than the pro-vaccine communities, because most people just vaccinate their kids and go about their day. They’re not on there on the platforms pushing back against this.
You can see this with the rise of the flat earth conspiracy theories. Nobody creates groups saying, “Hey, the earth is round.” That would be-you know, that would be absurd. What you have is this asymmetry of passion, where the communities that are making this content, pushing out this content, are the more conspiratorial truther communities, because they feel that they’re underrepresented in mainstream media, underrepresented in society, generally use these social platforms in this entire unregulated, unmanaged ecosystem to make themselves look much bigger than they are and increase their reach. It’s a huge incentive, and there’s nothing to push back against that.
Were there teams inside the social media platforms like Facebook that were studying this, that were studying the spread of rumors or studying the spread of disinformation?
I don’t know. I didn’t get the sense that that was anybody’s sole function. I think that they were always looking at ways to tweak the algorithms, like these things change. Google search engine changes its weightings periodically, and there’s a lot of tweaks that go into these things. I don’t think they were looking at it with, “Is this topical content we want to be amplifying?,” or, “To what extent are people coming to this group on Facebook because of a manipulative campaign on another platform?” I don’t believe that anybody was responsible for looking at that.
I think it’s really important to understand what it meant, for instance, that Facebook, they would have had the data that you were looking for to understand how these things are spreading, correct?
Yes.
So were you asking them for the data, or were you asking them for some visibility into what they know about how conspiracy theories spread on their platform?
Yes. First, to preface, each platform knows what’s happening on its own platform, so they’re not looking at the system, which is one of the problems, and this is where I think third-party researchers provide a valuable service, because that is what we look at. Facebook had a-you could ask for a research relationship. There were a series of requirements you have to go through. The project had to be really careful scoped. It wasn’t like they would just give you data, and that’s as one would hope. As we’ve seen in the era of Cambridge Analytica one would hope that those protections were in place for privacy reasons. But you could enter into a research relationship with them. There were stipulations that I think made that unattractive to academics, to people at universities. There was some question about whether you can publish if the findings were unfavorable, that sort of thing. So one of the things that we started asking for was ways to improve the research relationship, and that was conversations that went on for a few months. They did just take those steps last month, to create a council to approve projects, to eliminate the right of refusal for publication, these sorts of things, so that that is beginning to take some shape.
But in 2015, 2016, no, the conversations were much more, “Thank you; we’ll look at it internally,” as opposed to “Let’s work in a partnership capacity where we see what you see and then we work together to understand the problem.”
Was there a lack of curiosity on their part in knowing whether disinformation or conspiracies were actually spreading easily on the platform as propagated by outsiders looking to game it?
I really don’t know what those conversations were like. I know that there were a handful of people, like I said, who were very concerned about it. I know that there was a real deep concern about being the content police, and we kept always coming back to the idea, “We don’t want to be the arbiters of truth,” and for a long time, I think that it’s hard to push back against that, right? It’s hard to say, “No, we do want Facebook to be the arbiters of truth,” because no, very few people would say that.
It was impressing upon them that this wasn’t about truth. This was about information integrity. This was about narrative integrity. This was about the deliberate gaming of their algorithms, the deliberate gaming of algorithms across the social ecosystem. That was what was happening, and saying-take out the fact that this is about vaccines; take out the fact that this is about whatever conspiracy theory. It shouldn’t matter what it’s about. What should matter is this inauthentic narrative pushed by real people, is this grassroots activism, or is this gaming a system to get a narrative in front of as many people as possible?
Is it-coming back to this again-that if there is a metric for engagement, and conspiracies are more engaging than truth, then isn’t this a flawed system?
I would say yes.
And what did they say at the time to that allegation?
Well, that people have a right to freedom of how they’re going to think about things. If you believe the moon landing is fake, I can tell you that it’s not, but who am I to tell you [that] you don’t have the right to believe that it’s not so, and that’s true, right? So again, what I would talk a lot about was, it’s possible for this content to exist in the world and for you not to give it voluntary amplification, and for you to realize that your algorithm is incentivizing and rewarding highly sensational, highly conspiratorial, highly emotionally-fraught content.
That’s where the weighting is shifting so that this content that can very well exist on the internet. Nobody is arguing for the wholesale removal of every anti-vaccine, chemtrail, or flat earth group. What we are saying is the way that the algorithms engage with the behaviors of these communities, the content style that they’re leveraging, the sensational highly emotional – when that’s happening over and over and over again on all social platforms, what is your responsibility to be thinking about what to do about that?
And you saw them as shirking their responsibility at the time?
I saw it as “This is not a priority for us.” There are other things that are-you know, this is a tech company, and I’ve worked at [and] run tech companies. It’s hard to-you can’t do everything all the time. This was just not a thing that was really high up on the radar.
The Misinformation Playbook
ISIS has been described as being almost the beginnings of the playbook, right? I’m curious about that. I mean, in terms of how ISIS was using the platforms to game-gaming the platforms, do you see that as sort of the beginning of the playbook?
ISIS and the conspiracy theorists ran largely the same playbook. That’s what was so interesting about it. That was the phenomenal parallels. That was the realization that “Hey, anybody who wants to do this can do this.” ISIS was absolutely transparent about what they were doing, and so are, honestly, some of these conspiracy theorists. This isn’t like, “We’re going to mislead you into this page, and then the reveal it that, ‘Oh, we’ve been flat earthers all along.’” No, this is all right out there and overt. I think that what ISIS showed more than anything else, though, was that even when the stakes were life and death, even when the stakes were “This is a terrorist organization recruiting the vulnerable,” nobody was in charge. It was this surreal series of conversations where, what do you do when the platforms are concerned about being seen as censoring content or censoring groups? Because the recognition was since ISIS was largely running a marketing campaign for a terrorist organization, but it looked like a marketing campaign, like these were the same things that you could do with any other marketing campaign. If you could police this one idea, would that mean that by taking action there, would their users become concerned that they could police any idea, and would you see this kind of slippery-slope argument be used to-actually, let me try to explain that one again.
So in terms of the playbook, and you brought up something that’s even better than the playbook, which is one playbook is the same with ISIS and the conspiracy theorists but that Facebook and other platforms were grappling with what their responsibility would be on policing that. I guess the question is, why is the ISIS development important in retrospect as a point along our timeline in understanding the growing realization that they maybe should be responsible for?
The ISIS thing is important for a couple of reasons. First, it showed that there was no one in charge, and it showed that to the world, because the fact that American tech companies were having fights in the press through proxies about what they were going to do to stop a national security threat was something that I imagine was very, very interesting for Russia and others, right? It was really hard. No one was really seeing eye-to-eye. The government couldn’t really do anything; the platforms didn’t really want to do anything. The platforms are indemnified from the content that they host, right? So there is this indemnification under something called the Communications Decency Act Section 230 that says that they’re not liable for what they host. There was a lot of concern that if they started to take responsibility and do takedowns and de-platform this one particular issue that they would be seen as being lax on other issues. If you started with this one, then people would ask them, “Why aren’t you doing these other five things, these other five communities, these other five lesser terrorist organizations?”, etc.
That was a big concern for them, I think, the idea that they would be seen as responding to government pressure to take down this community’s content, what that would communicate to the rest of their users and also what that would mean for their own indemnification.
So they didn’t want to draw a line as to who is a bad actor or a good actor?
I think that initially that was a big part of it. That was definitely a concern, especially in the early days of ISIS. I think that was something that they were concerned about.
OK. Bring me into what you were doing at the time on the ISIS issue and what you were observing.
Yeah I was not an expert in ISIS. There were people at the State Department who were. What they wanted was a collection of people with tech backgrounds who understood social networks, who understood social network manipulation, to come and help and just advise them on what was happening and what they could do about it, so myself and Jonathon Morgan and a handful of other people started to advise the State Department on: “Here is how these campaigns are conducted. Here is how we see them play out. Here are tactics that we see from various groups A, B, C, D.” We helped them communicate with the platforms. We also helped them think about what the U.S. government’s own strategy could be for pushing back against this, because at the time, the sense was since the content wasn’t really coming down as fast as one would hope, since the ISIS propagandists were so active on Twitter in particular, maybe the U.S. government’s approach was to try to counterpropagandize or counter the narrative on Twitter. And this was not the most successful initiative. I was kind of panned in the press. The U.S. State Department tweeting at terrorists wasn’t really effective in any way.
What they started to do is think about what could we do about this. We did have conversations with Twitter where there were a couple moments we were trying to make headway and find common ground. Someone I spoke with said something about how difficult it was to find these accounts. And I said, “You know, if I take out my phone and follow one right now, your recommendation engine is going to give me three more.” And that’s because once you’ve started to follow one of these accounts, the recommender system was pushing me all the others. It is this fascinating insight into [the fact that] the company doesn’t necessarily feel like it has a very good handle on the problem. But ironically, the recommender system is giving me the accounts. That was sort of an interesting point in the conversation, where we were saying: “It seems almost like nobody knows what to do here. No one knows what’s going on. No one’s in charge. And oh, boy, if we can’t even get there and find common ground on how to deal with overt terrorists, what are we going to do about all the rest of it?”
…I know that this was a specific conversation with Twitter about “If I start following one account, oh, your engine will recommend others.” Would that go for all the platforms as well, or that was something that was specific to Twitter? I’m just trying to-
That was more specific to Twitter, because YouTube had been blasted for hosting the beheading videos, and that of course was a violation of the Terms of Service, but the fact that they were playing whack-a-mole with extremely violent content was something that they had begun to realize that they had to get ahead of that somehow. They were a little more proactive on the terrorism front after that.
What about Facebook, though?
Facebook had taken down quite a lot of ISIS content and tried really, really hard to keep it down. The problem was, I think-again, you had to staff up teams. Like, they didn’t necessarily have counterterrorism experts within the moderation community or within the management chain that was deciding what the policies were going to be. You didn’t have necessarily enough Arabic-language [speakers] and other languages that ISIS was targeting the speakers of looking at the content there. There was a sense that a couple of the platforms did a little bit more. There’s always been variations in the degree of response. You know, YouTube wound up testing out this method developed by Google Jigsaw, called Project Redirect, which said if people were coming to YouTube looking for extremist content, YouTube would try instead to push them on a deradicalization path. So the autoplay would serve up content intended to deradicalize.
It was a really fascinating experiment. It’s something I wish we learned more about. It’s something I wish that we focused on more, because some people were deeply uncomfortable with the idea that Google was trying to sway an opinion one way or another. I think the irony there is that the recommender system just goes the opposite direction normally. So why is it that we’re uncomfortable with the idea that we would try to proactively deradicalize or push someone back, but we’re not uncomfortable with the fact that the recommender system is doing this in the other direction? I think that’s something that I don’t fully understand public opinion on that particular issue yet.
What were you recommending at the time? What were you advising the State Department or the platforms to do about this problem at the time?
We were trying to say that there should be better channels of communication, for starters, trying to say that information sharing about these things needed to be more robust. It was hard. You know, there was a lot of distrust between the government and tech, between people and the government. This was after [Edward] Snowden, right after the Snowden revelations. The tech companies really didn’t want to be seen as cooperating with the government. So we were advocating for a couple of really basic things: One, create a task force or a center in government that’s responsible for monitoring this and ultimately became the Global Engagement Center, which exists. It’s focused on terrorist propaganda but has also recently had its mandate expanded to foreign propaganda in general as a result of what we saw with Russia, the idea that at least if we had a kind of a government center in charge of thinking about what the government’s response was going to be, or detecting threats that would be a good starting point.
The platforms-in mid-2016, I think it was; you might want to check this; it might be in 2017-created what was called the Global Internet Forum to Counter Terrorism and that Global Internet Forum to Counter Terrorism was an effort where they would share information with each other, so when terrorist content appeared on Facebook, Facebook would make that content available to YouTube and Twitter so they could block it also. That was a really positive response. That was the kind of thing that we were advocating for, like there must be information sharing. We’ve got to treat this almost like we treated spam, where the platforms came together to think about ways to deal with the spam issue. It was somewhat remarkable that we do have these instances of cooperation in the past to deprecate the impact of bad actors, but it took so long to get us there on terrorism. That was a real surprise.
…Did you feel very alone, to some degree? What was the feeling of finding some of the other people that were having similar concerns? Tell me a little bit about that at the time.
Well, when I first started looking at this, there’s a lot of soul searching. I was like, am I being Chicken Little? Am I overreacting? I don’t have a complete picture of the problem. Is this “a tree falls in the woods” issue where, yes, I’m getting served up these weird recommendations but maybe nobody’s clicking on them, and they’re just noise? I remember when I met Jonathon Morgan and realized that he was seeing the same things with ISIS that I was seeing with conspiracy communities and feeling like, OK, someone else is starting to realize there’s a problem here; it’s not just me. Then when I started speaking about it, like I gave a talk, an Ignite talk at Google I/O, and I had 10 or 12 engineers come up to me afterward and say, “We know; we see it.” That was when I thought, OK, so somebody-people know this is happening. I would say election 2016 was where it became-that was when everyone came out of the woodwork, right? That was where the engineers themselves, talking very publicly on Twitter that night as the data was coming in, [were] saying, as the election results are coming in, saying: “What did we build? What did we build?” And then-
What does that mean? When they were asking, “What did we build?” what does that mean specifically?
The valley is a liberal place, liberal and libertarian. But the conversations that started to happen were what was the responsibility of the engineers in building the platforms and increased polarization? If you remember, huge swaths of the campaign covered harassment on Twitter. It was the big topic of conversation: reporters being targeted for being Jewish, you know. A number of different, unfortunate things that were happening were really showing that the degree of vitriol on the platforms was a huge topic of conversation throughout the campaign.
The results came in, and yes, some of it was absolutely partisan handwringing over the fact that Trump had won. But the subsequent conversations as people began to point the finger at fake news was: “By not doing anything, are we responsible? Would we have had a different outcome if we had been more proactive at shutting down the fake news and the conspiratorial partisan content? Is anybody even looking at quantifying the degree to which what we think happened in terms of much more of the conspiratorial content targeting one candidate versus the candidate who won or is anybody looking at to what extent that did impact the election?” When the Russia stories began to break, when Mark Zuckerberg said the total amount of fake news on Facebook is a very, very small percentage of the content on Facebook, and the engineers internally had a small revolt saying, “We can’t ignore that this happened; we have to dig in,” that was where I think the power of the people who work at those companies to push this along-previously, like I said, a lot of them, we would have back-channel conversations, and it was just conversational, but I think that it was really November 2016 that you started to see the people who worked at the companies say: “OK, we’ve got to figure out what’s going on here. We can’t just let this go.”
Misinformation In The 2016 Election
What were you observing during the 2016 election?
I was still looking mostly at the conspiratorial communities, and what I was observing was the increasing cross-pollination of extremist conspiracies, or the cross-pollination of the chemtrails groups being served up Pizzagate content, the fact that certain far-right conspiracy theories were rapidly crossing over and crossing back-and-forth in the sense that we were increasing the number of followers of all of these different types of conspiracy theories by facilitating this cross-pollination and realizing that a lot of the conspiratorial communities were becoming increasingly partisan. A lot of the support for President Trump even among chemtrails believers living in Los Angeles was a really interesting thing to see; people who believed that the earth was flat really going hard for President Trump. It was a little bit bizarre how the justifications for this poor candidate were rooted in the conspiratorial worldview. But this cross-pollination that was happening, where people were saying, “Oh, yeah, I’ve always known that the CDC [Centers for Disease Control] was secretly controlled by industry, but finding out that all of these other different groups are also controlled and finding out that there is a vast cabal working to-” … The deep-state conspiracies, a lot of these sorts of things became this morass where all the conspiracies kind of blended together. But there is this-
Well, I’m just curious. Did you see it as sort of a perfect storm? Here you’ve been observing for years, right, how disinformation spreads on these platforms. Was what you were observing kind of a perfect storm in 2016? Is that how it felt? 2016 felt like we were starting to see that the recommender systems pushing anyone who had any vaguely conspiratorial worldview into all of the different conspiracies all at once, and it was a really fascinating thing to see, especially because so many of those groups would also be talking about President Trump and how he was going to destroy their preferred conspiracy of choice. He was going to crush the CDC; he was going to destroy the deep state; he was going [to] drain the swamp. That weird intersection of that candidacy and those social conversations with all of the different conspiratorial groups all at once was a really interesting thing to see happen, and it was deeply disturbing. [We] started seeing the growth of the numbers in these groups climbing.
I think also there was a sense that no matter how many times we tried to point out that this was happening, there was nobody really receptive to it. I felt like the conversations around harassment and around brigading and around some of the more obvious techniques for amplifying fake content of Twitter, that started to get some attention. Facebook’s Trending algorithm got a lot of attention. There were these interesting little sort of sub-facets that the media began to cover, but the fact that the entire system was misused, the entire system was broken, that didn’t really make it into the public consciousness until the Russia investigation was well underway.
Were you essentially asking the platforms to question what is it that you’re building here?
Yes. I was asking them to think about what are the downstream impacts of what you’re building here. How much of this is a joke versus how much of this is really growing-like, are you looking at this? Is anybody looking at this? We see these flat earth groups getting surfaced, and we all treat it as a joke. How much of a joke is it? What do you see these people who join these groups do elsewhere on the platform? What do they go search for next? Is there a sense that-flat earth sounds like a a joke, but is it a gateway conspiracy? It’s definitely-if you’re in there, you’re going to get recommended all sorts of other conspiracy theories, so does it gradually lead to people joining these more and more extreme kind of conspiratorial communities? Is anybody looking at this, or is it just like, well, people have a right to join flat earth groups and that’s the end of it.
So essentially, are you guys building a disinformation machine?
That’s a good way to put it, yeah.
And their feeling was that they weren’t or that they weren’t just going to look into it?
I just think it was not really on the radar. It was, you know, sometimes there are flat earthers, and they share their content, and who are we to tell them not to?
Fast-forward again to the 2016 election. Right after the election, Mark Zuckerberg says, “It’s crazy,” right? I mean, there’s the kind of famous “crazy” moment about [whether] misinformation or fake news played any role in this at all. Do you remember hearing that?
Yeah, I had just had a baby. I was in the hospital, and my daughter was due on Election Day. She came a week late, and I was in the hospital. And, you know, you’re kind of [inaudible ], but you also have your phone, and all of a sudden I felt like the story was taking shape, and I was like, “Wait, this is stuff I’ve been talking about for years, and here I am in a hospital with this baby.” As I’m holding the baby, and I’m texting people and saying like, “Oh, my gosh, these comments.” And I knocked out a Medium post. I don’t really remember the timeline. It was all so hazy. [A] couple of days after I think, saying, again pointing the finger at groups and their recommendation engine and saying: It does not matter if it’s a small percentage of the total content on the platform. What matters is, is it targeted at the people who are going to be receptive to it?”
That was the thing where I felt like this was such an obvious statement, because this is how we think about epidemiology, right? If a small cluster is unprotected, that does leave the broader population vulnerable because you can gain a foothold in that cluster, and so thinking about it in the same way, right, when you think about how we think about ways that small communities get radicalized and then spread that more broadly. Those are pieces that we haven’t even talked about, which is like the organic spread of content.
We’ll get there. We’ll get there. But going back to your hospital story. So you’re sitting there in the hospital, and you’re hearing that they think it’s actually crazy, right, and this is something you’ve worked on for years. What, in basic terms, is going through your head at that point?
What was going through my head at that point was, all of a sudden, every major news organization was paying attention, and this was an opportunity to point them in the right direction and say: “This is what you need to look at. This is what we’ve been saying for years now, and please do the investigative work to put it out there and to put it out there to the public, to make the public understand what’s happening, to make the public understand that it might be a small amount of content given the entire sum total of Facebook’s 2 billion users, but for the communities that were targeted, it’s something that they see every single day. It’s their experience on Facebook.”
Facebook And Russian Disinformation
…When was Russia on the radar when it came to the potential susceptibility of social media platforms to disinformation campaigns?
Russia was part of the conversation in the White House and in the State Department advising I was doing in October 2015. I don’t know when it started for them, but that was when it started for me. Adrian Chen’s piece about the Troll Factory had come out in July, I think, of 2015, and I had read it and registered it. The interesting thing for me was him, Adrian Chen, making the connection to the story of the Louisiana chemical plant explosion which we had treated as a hoax. I thought that that was really interesting because there was this idea that we had things called “hoaxes,” which was a very quaint way of referring to an extremely coordinated attempt to make Americans believe that this truly terrible thing had happened.
When we were doing the work, thinking about how we can manage this easily manipulatable system, Russia was part of the conversation then. So were other state-level actors that the government was concerned about, because we were saying over and over again, we can’t be building a response team just for ISIS because ISIS isn’t the only problem here. ISIS is the problem today, but the real problem, the bigger problem is that the entire system can be manipulated in this way, as we’ve seen happen now, through so many different actors. Why would we build a response specific to ISIS when what we should be thinking about more is the idea that our adversaries seem to have this almost like “Department of the Internet,” and we just don’t have anything even comparable to that?
So in some ways, what was to come was not a big surprise to you.
When the reports began to trickle out confirming it, no, it was not a surprise. It was, “OK, they’ve got some proof now.”
But, for instance, going back to October 2015, was there information sharing between the State Department, the White House and companies like Facebook about thinking about the vulnerabilities of their system to foreign actors looking to spread misinformation or disinformation?
I don’t know. Not in any of the conversations I was in. The conversations I was in were very focused on ISIS. It was actually a point of frustration for a couple of us who were part of the Silicon Valley contingent saying, “You know, we would never build for this one problem; we would want to be thinking about how does it apply at a broader level.” But I don’t know to what extent the government, the high-level government and high-level tech executives were talking about anything beyond terrorism.
But Russia was on your radar.
Russia was brought up by other people in the government who were concerned about the potential for manipulation. Russia was mentioned specifically in that context, not the election, not that anybody had any idea that this had actually happened. It was a “Have you read Adrian Chen’s article? This seems like something that is going to be a problem. This seems like something that has already been a problem. They already manipulated us with this Louisiana chemical explosion hoax.”
OK. So going back to January then, after the election, there is this report that’s issued by [Director of National Intelligence], which said that the Russians had launched a campaign to interfere with our elections and that social media played a role in it. Do you remember your thinking at that time when that report came out? What did you think when that report came out?
I thought, now we’re going to start to pay a lot more attention to this problem, because it just went from being fringe groups and nonstate actors to being a serious, elaborate, geopolitically motivated campaign by a state actor. And that’s bigger. That’s a much bigger problem.
And the bigger problem being the susceptibility of these systems to that?
Bigger problem being that people with real resources and capabilities were doing the same thing that terrorists and small fringe groups were doing. Terrorists and small fringe groups were doing it because it was easy and cheap. The thought of a state-level intelligence service actively going in there and doing it means that you don’t even have to do it small and cheap; you can put real money and real people behind it. You don’t even need to use bots then; you can just hire people. You can evade a lot of the even primitive detection systems that we’ve been advocating for with regard to bots and automation, really kind of low-level, low-hanging fruit stuff that we wanted to see done, recognizing that there’s still some importance to doing that. But this is a much bigger problem requiring a much different level of thinking if state-level intelligence agencies are running the same playbook.
Did you see the acknowledgment of election interference on social media as having been an intelligence failure?
…The government is constrained from engaging with U.S. citizens on social platforms, and it’s constrained-it can’t propagandize to its own citizens. There’s a series of laws governing how it can interact with Americans. This means that its hands are kind of tied in terms of doing large-scale digging and investigation, especially of groups targeting Americans on social platforms. That’s considered a feature, not a bug, right?
But what it does mean is that when platforms aren’t looking, and the government isn’t and really can’t be looking, then nobody is looking. This is where you have this vacuum. No one is in charge.
…At first, there were reports about Russian hacking and of obviously the DCLeaks and some of the other things, but when did that change from kind of a hacking concern to a disinformation concern on social media?
I don’t think it changed. I think it was just two different vectors of attack. One, you’re infiltrating a system getting as much information as you can and then doing these kind of coordinated leaks, like more of a media-you’re engaging with the media at that point. Yes, it’s on social platforms, but it’s different than running a long-term relationship development campaign with communities. So the WikiLeaks and the DNC leaks and all the other things that came out- that was much more of a traditional hack-you know, infiltrate, hack, social engineering your way in to get somebody’s password, so on and so forth. That side of things was one facet.
The other facet, though, is that the relationships with Americans and the building up of these societally divisive pages and things, that started in 2015. So that wasn’t even really focused on the election. That was simply a covert propaganda campaign to develop relationships with American communities, to activate Americans to do things that Russia wanted done, meaning coordinate protests, have these real-world meetups, hire people to show up to rallies, these sorts of things. That was a very different type of campaign. One was really, “How do we get the media to pick up and cover these stories?” The other was, “How do we reach Americans directly and form very long-term relationships?”
And how was it that you were learning about the social media aspect of that campaign?
The social media aspect came when we started realizing that these Facebook groups had existed and wanting to dig into what had happened. For me the curiosity there was zero surprise that this was done but a desire to understand how, like what were the mechanisms they used, particularly because most of the stuff that I had seen and I think most researchers had seen was overt propaganda, where nobody is making any secret of who they are.
This was the first time we’d seen the complete fabrication of identities and the co-opting of narratives and the radicalization pathway of using every single social platform to push a message and to target it to particular groups of people. It was sort of like the playbook we had seen others run but taken to that next level where it was done covertly.
Facebook’s Security Reports
…Do you remember the April security report from Facebook and what you thought at the time when that report came out?
I thought that it was vague, but I believe that it triggered a series of media investigations that got us a lot more in the way of specifics. That was where we got the names of some of the pages. Once we had the names of some of the pages, that was when we were able to start doing real investigation into the content on the pages and the patterns that they used to share the content. This was when Jonathan Albright found the CrowdTangle data showing the content for the last 500 posts on six pages that we knew the names of at that point.
From there, independent researchers, myself and others, were able to take the content and really say, “These are the phrases they’re using; this is the writing style they’re going for.” Some of the language was flawless English, and we would Google it, and it would turn out it was from a news clip from some obscure local paper that they were just cutting and pasting. Sometimes the language was very, very formal business English but advocating for Texas secession, or it was like a formal business, English-style treatise on why Hillary Clinton was such a terrible person, which was sort of incongruous, like these sorts of little breadcrumbs where you would realize that something was off here; something was wrong.
But the other thing that we had was engagement data, and that was where we were really able to say hundreds of millions of people saw this content. We were able to track the content to Instagram and realize that there were another tens or maybe 100 million more there, able to track it to Pinterest and say here’s the repository that they were using to store some of this stuff, track it to Reddit. This is where we started to get a handle on-once we had these little snippets of information about what the memes looked like or what the content was, then we could see where it was elsewhere.
And as you and other independent researchers were looking into the Russian disinformation campaign, were you assuming that Facebook was also doing a deep dive into its data and was going to come out with a thorough investigation?
Yes. It seemed like a reasonable thing to do.
In September, there’s a blog post, right?
Telling us it was $100,000 worth of ads, no big deal. That was the moment for me when I thought-you know there’s that saying about never attribute to malice that which can be adequately explained by incompetence? I really thought, oh, they just didn’t know. No, they just didn’t know what to look for, and when they released a statement saying it was just $100,000 worth of ads, that was when I thought, but you know how your platform works. The whole point is to drive organic engagement. How has nobody mentioned your organic engagement? How is nobody mentioning the numbers in the CrowdTangle data, and why are we minimizing this? Why is minimization the approach that we’re taking here? It should be the opposite. It should be, “Mea culpa, we found this stuff, and we’re going to keep looking.”
That was I think a misstep on their part, that I believe that the tech hearings and the public outcry and, again, the media not dropping it. I think that’s the other thing that’s been remarkable about this story since November 2016, is how the media has not dropped it. The media has gone after every single platform, has continued the investigation, has continued exposing purchases of fake accounts, fake followers, review manipulation. The wholesale investigation into social platform manipulation and misuse has been a press narrative now for over a year, and I think that’s remarkable.
What did the September blog post say?
It said that it was- “We investigated, and we found $100,000 worth of ads paid for in [Russian] rubles.” That was what their investigation had found, so they concluded that there were so few impressions on the ads relative to the sum total of impressions on Facebook that this really supported the idea that this was a very small problem because of how small it was relative to the total amount of content on Facebook. Rather than saying, “We found this thing; we’ve realized that a hostile intelligence service manipulated our platforms; we’re going to continue to dig and dig and dig until we find all of the various iterations of this on the platform,” instead it was, “Don’t worry, don’t worry; we’ve got this,” and, “Don’t worry; this is not something that was ever really a big deal, and you can just go on about your day.”
And you knew otherwise?
Well, the CrowdTangle dataset that Jonathan Albright found certainly seemed to indicate otherwise. And it’s important to note that CrowdTangle is owned by Facebook, so it would be one thing if it was a third-party, random platform that was saying that the engagement data was orders of magnitude larger than Facebook stated. But because it had come from an analytics tool owned by Facebook, there is a sense that one of two things is true.
Either Facebook’s analytics tools are not that great, or what’s being said is not accurate. So one of those, yeah, it’s really the only two options.
Can you explain to my grandmother, who would not understand what CrowdTangle is, what basically Jonathan and you were looking into, what that research was?
…What Jonathan found was the analytics data-so the number of people who engaged with the content, who clicked on it maybe, or the number of likes that the page had, the number of likes that the content got. It had a lot of data about what people did when they saw those Russian posts. And it wasn’t just the ads. It was the organic posts that the page was putting out.
What happens is if you see an ad and the ad tells you to like a page, you click the “like” button. At that point you start to see posts from the page in your News Feed. You don’t see all of them, but you see some amount of them. The organic posts are not paid for. They’re free, so they would never show up as an ad buy. But people will reshare. Let’s say I put up a meme that you think is really funny or interesting. You’re going to reshare it so that your friends see it, too.
Even if your friends haven’t liked that page, you’ve done the work of spreading the content from that page out to your broader social network. Even if a lot of these pages had about a 100,000 to 120,000 people who had liked them, even if there’s just that 120,000 people, let’s assume some 50 percent of them see a given post, there’s still an opportunity for hundreds of thousands more people to see it if each of those people who sees the post goes on and reshares it and then it goes to their network, and then those people reshare, it goes to their network, so this is how you get the propagation of content across the internet. When it happens fast, we say something’s gone viral. But the organic push of content out to very, very broad networks, that’s what social platforms are really used for. That’s why marketers spend tons of money on them also, because you buy an ad, you get people to like your page, and then they’re going to see a lot of the content that you’re putting out, and they’re going to share it for you.
…Facebook comes out in September and says what?
They said that they had investigated the reports of Russian interference and Russian propaganda pages and that it had only been $100,000 worth of ads paid for in rubles, and that was what they found. Those of us who had been paying attention to this, particularly who had seen campaigns be executed by other actors, found this incredibly disingenuous, because the whole point of Facebook is that you have organic reach.
We listened to this line about ads and how few people had seen the ads, and we were saying: “But the whole point of the ads is to drive people to pages to give that organic reach. Facebook knows how its platform works, so why aren’t they talking about organic reach?”
Tell me what you mean by organic reach.
Sure. When you see an ad, there’s a call to action on the ad, and oftentimes that call to action is to “like” a page, and sometimes it’s to join a group. So if I am running an ad campaign to get people to like my page, the reason I want them on the page is so that I can continue to reach them for free. I no longer have to pay money to run ads. I can just-they’ve liked my page, and I can tell them to follow the page for all of the updates.
I had the opportunity to form a long-term relationship with them. If I’m producing really good content, maybe they’re even coming back voluntarily without even waiting for my content to be pushed into their News Feed. But it’s entirely free.
…So if I run content that’s really sensational or engaging, there’s a high likelihood that they’re going to then share that content out to their network. So even though those people don’t like my page, they’re still going to see my content. And if those people find it scandalous or engaging or enraging, they’re going to share it again, and then their friends are going to see my content. Now we’re kind of two levels out, and you can see how this propagates across an entire network, and that is what’s called organic reach.
My initial engagement with people might have been paid, and the paid piece is what Facebook is serving up to people. I’m paying Facebook to push this content to people. But then, as people go and organically continue to share it, that’s organic reach. I’m not paying for any of that, and it’s reaching a much wider group of people.
So you suspected at the time that Facebook was not being forthright or honest about how many Americans were exposed to this Russian content?
Yes, because they said that it was only $100,000 worth of ads, and they mentioned, I think, a couple hundred thousand people who had been targeted. Recognizing that if a couple hundred thousand people see your ads and then go on and engage with your page and then go on and share your content, there is the potential that millions and millions of people have seen your content.
Which, presumably, Facebook would have the tools to know.
Yes.
But didn’t reveal.
Correct.
And you thought what?
I thought this went from seeming like they just didn’t know-this went from incompetence to malice. This went to feeling like they were deliberately trying to minimize and pretend that this was really not a big deal at all rather than do a full accounting for what had happened.
So the next phase in your work is teaming up to some degree with Jonathan Albright, is that correct? Let’s get this right.
A number of us who had been raising issues about platform dynamics and algorithms and recommender systems, we had all been aware of each other. But all of a sudden, we were all actively talking to each other, because there was a sense that, particularly after the September announcement was made, that this felt so incredibly disingenuous that we felt that we really needed to call attention and really lay out for people to help them understand why this was a much bigger deal than Facebook was acknowledging at the time.
Jonathan Albright, a researcher who was a member of the group out of New York, was digging around and found this remarkable dataset … through a tool called CrowdTangle, which is an analytics tool owned by Facebook. What he found was even though Facebook had deleted all the content of the pages that they had identified as Russian, the information about engagements with the content had not been deleted from this analytics tool.
So all of a sudden, for the six or seven pages that we knew about, what Jonathan found was this remarkable dataset detailing all the engagements on the last 500 posts for each of these six pages. So we could see things like how many people had liked the page, and realized that hundreds of thousands of people had liked these pages. We could see on a lot of the engaging content that they’d been shared hundreds and hundreds of times. We could see the likes and the impressions. In aggregate, this was millions of engagements with this content.
So what we had suspected-and I think Facebook had said that there were a couple dozen pages, and we only knew the names of six, but for these six pages, there were hundreds of millions of engagements. That meant that this was way bigger than just a couple of hundred thousand people being targeted with some ads.
And armed with this information and your knowledge, you’re at the same time talking to the U.S. government?
At the time, as we became aware that network manipulation had been an issue in the election, I was in contact with members of Congress who were trying to understand the dynamics by which this happened. The conversation was not specific to Russia. The conversation was a more broad “Here is how social network manipulation takes place.”
As specific information began to make its way into the discourse through research finds like Jonathan, through investigative reporting, through a number of different exceptional publications, as we began to get more and more of these breadcrumbs, we were able to tie it back together. And one of the things that I was doing was helping communicate that back to members of Congress who were responsible for investigating what happened.
Congress And Facebook
How would you describe the level of aptitude or knowledge in Congress at that time about these problems, the susceptibility of social platforms like Facebook to manipulation by foreign actors or bad actors?
There’s a broad spectrum. A number of the lead voices that you heard step into the role of explaining to the public and pushing for the hearings and holding the companies accountable-Sen. [Mark] Warner (D-Va.), Rep. [Adam] Schiff (R-Calif.), Sen. [Richard] Burr (R-N.C.), Sen. [Lindsey] Graham (R-S.C.)-were incredibly well-briefed. They really understood what was going on, and to their credit it was an absolutely bipartisan consensus that this had been an attack on American democracy and that as such, it merited a full investigation in hearings.
It was never about relitigating the outcome of the election. It was much more about what does this say about our ability to ensure election integrity in the future; what does this say about our ability to protect our citizens from direct contact with hostile foreign intelligence services. They were very well-briefed, very sophisticated. There were other members who didn’t really know as much, but that’s not necessarily-this is not a thing that members of Congress study before they get elected. I think our role was to help them get there. I think this is where techies who have a deep understanding of what a click-through rate is, how the platforms are tracking engagement, how the platforms are tracking what comes-how information moves through an ecosystem, these are the sorts of things where we just wanted to help them get up to speed on how the system worked.
And was part of your aim to try and figure out ways to hold platforms like Facebook accountable?
I had been a real big believer in self-regulatory models for a long time. It’s just my personal alignment. I think that when you push for regulation, you get a very specific thing that is defined and regulated, and it’s not as easily malleable as an industry deciding that it’s going to take something on itself and adapt for the good of its customers usually.
So we started to get the sense, though, as Facebook put out statements saying this is only $100,000 worth of ads, that maybe self-regulation wasn’t going to do enough here, and that maybe we did have to be thinking about ultimately who is responsible for the oversight of the system. That’s sort of an ongoing conversation here. As the platforms have begun to accept more responsibility and acknowledge more of what they’ve done, they have started to take proactive steps to fix a lot of the problems.
Facebook, in particular, has done a pretty big about-face in terms of the number of people it’s hired to staff up, the way it’s created these integrity teams. They’ve done some remarkable – well, “remarkable” is probably not the best word. They’ve done some things to indicate that they finally accept the seriousness of the problem and their role in protecting Americans and protecting democracy and elections worldwide.
But before we get too ahead of ourselves, I mean, the November hearings, right? Tell me a little bit. One, it’s not the chief executives of these companies; it’s their general counsels appearing. Give me the story of leading up to those hearings and what work you were doing to prep the congresspeople and also what you thought of the fact they were going to put up their lawyers.
We were asking for the hearings-a number of people were calling congressmen and asking for these hearings actually, because we felt that the American people had a right to know, and that was partly because the issue had become so politicized that there were-It had become all about the election and whether or not you liked President Trump, and what we wanted to say was, “This maybe went your way this time, but it’s not-this is a big problem independently of who wins an election in this particular instance.” We wanted to break through the partisanship and hoped that a bipartisan hearing would facilitate that.
The fact that they sent their lawyers was remarkable to me, because I was on Wall Street in 2008, and I remembered the series of hearings, the bankers, the automakers, the insurers, all having to come and all sending their CEOs. It was remarkable to me, having been part of an industry that was going through a major reckoning that seemed far more apologetic, versus being in an industry that was now going through its first big reckoning, sending the lawyers. I thought it was a very surprising and disappointing choice.
And tell me about those hearings. Situate me in the highlights or your feelings as [to] whether there were sufficient answers.
There had been a number of hearings. First, there was the Judiciary hearings. They were sort of a late add, and I didn’t think that they understood the issues as intimately as the two subsequent hearings in the next days, the Senate and the House Intelligence Committees. Remarkably good questions, really understood what was going on, really appreciated the significance of it as an attack on America and an attack on democracy.
What we heard from the platforms over and over again was the refrain of like, “We have to do better; we need to investigate more; we’ll get back to you,” the indications [being] that they were only then beginning to really do deep investigations because they had to account for what third-party researchers had released to the press at this point. They were now being called to account for not releasing that themselves, so it became much more of a hearing-but in my opinion-I was trying to think of the best way to say this. Are you asking what did I think of their questions, or-?
Or what are their answers? For instance, how did Facebook justify not having released the accurate numbers of how many Americans were exposed to Russian content?
Right before the hearings happened, Facebook pre-emptively announced that it was going to voluntarily disclose more information about ads and people who were paying for them. It stepped back from its traditional opposition to having to put “paid for by” on election-related ads.
There were a number of capitulations that happened right in advance of the hearings, that were intended to indicate some degree of support for the Honest Ads Act. There was also an acknowledgment that they had been incorrect in their initial assessment of how many people were impacted. I think that there was still really no mention of Instagram.
Google talked a lot during the hearings about not being a social network. That’s objectively true, but then they didn’t really tackle their YouTube problem head-on. It was an interesting show of apology, while also acknowledging that the real investigation wasn’t even done yet, and that was in late 2017. There had been a full year in there, so it was somewhat remarkable that they were still promising to investigate more.
Even though a year had passed since the election basically.
Yeah.
One thing is this, Renee: The term “tip of the iceberg” has been used in terms of what’s really to be found out for real about what happened. Putting yourself back in that moment in November, after those hearings, did you feel like, even with Jonathan’s findings and what you were finding and discovering, did you feel like this was just the tip of the iceberg of what we knew about what had happened on Facebook?
I think what we heard in the hearings was the tip of the iceberg. They acknowledged they’ve had more investigations left to do. There still wasn’t a ton of conversation happening at that point with third-party researchers. Instead, their response was to almost diminish, discredit, deny, you know, much more vague, how you would try to wrangle your way through a bad PR cycle as opposed to a recognition that this was all going to continue to come out. So there was a sense of frustration for me that we were still dependent on the platforms to do their own investigations. Either because they were incapable or because they were really slow or didn’t have resources, I don’t know, but these things just weren’t being prioritized. So we started wondering: “When are we going to get the full accounting of what happened here? We don’t even know the names of all the pages.”
The other thing that we were really advocating for during the hearings was the idea that people have the right to know. The hearings were an opportunity for the senators to ask questions to get at what had happened and to help people understand that this was real and that they had been targeted. But then, helping the specific people who were targeted to know that they personally had seen or engaged with this content, that was kind of the next step, right, to personalize it for people, to say “My community was targeted.”
And for that, we were again reliant on the platforms to feel compelled, to feel morally responsible, because there was no legal responsibility, so to feel morally responsible for telling Americans: “You liked a propaganda page. You engaged with this content. They targeted you, and you were fooled. You shouldn’t feel bad about being fooled, but you need to know that this happened to you, and we’re sorry, and we’re taking steps to prevent it from ever happening again.”
This is, again, the sort of thing that seemed like a relatively common-sense way to manage the situation, but did take Sen. [Richard] Blumenthal (D-Conn.) calling for it on the Senate floor before it became a thing that anyone indicated a willingness to do.
Should we get to [the Special Counsel Robert] Mueller indictment? So the Mueller indictment comes out. What did the Mueller indictment reveal for you?
Oh, the Mueller indictment. The sophistication of the money-laundering operation; the fact that they had sent people here to kind of drive around American communities and figure out what to talk about; the sophistication of the outrage over DM, over direct message; the sophistication of the one-on-one communication with organizers and activists-like, that’s remarkable. It’s one thing to run a media campaign and target people or have some sock puppets communicating on Facebook pages, but when you’re actually reaching out to people and trying to get them to hire protesters to show up to events or pretending you’re a local community organizer trying to co-sponsor a rally and then having two opposing rallies show up at the same place, at the same time, that leaves the truly online world behind and really gets at the impact that this had on communities. The fact that real people were enticed to show up with signs to protest and to counterprotest in real American cities-this went way beyond just some people saying some fake things on the internet.
So the surprise, to me was like, that’s the sort of thing where no third-party researcher is privy to the DMs of random American citizens. So that’s the kind of thing where it makes you realize how difficult this is to track, how difficult this is to get ahead of. And for Facebook at that point, the importance of having these third-party relationships to share information, the importance of having what we call “red teams” to look at all of the different ways the features on these platforms can be abused-just even now, with the House releasing the dataset, the cataloging of unique and innovative tactics that were used to personally touch individual people was remarkable.
So it was a sophisticated campaign?
Yeah, incredibly sophisticated. And this is where we get at the challenge of, this is not some low-rent spam ring or some even terrorist organization. This is a highly sophisticated, coordinated state intelligence service deciding to use [a] completely open social platform to target the citizens of a country directly.
Testifying Before Congress
Do you think of it as sort of ironic that the thing that gets Mark Zuckerberg in the chair is Cambridge Analytica? How important do you think Cambridge Analytica is on this timeline? Tell me that. Why was that significant?
Cambridge Analytica was another example, a different type of example, that you could argue had absolutely nothing to do with the Russian intelligence operation, right? Simply, it makes people realize that their information is vulnerable; their information is out there, God knows where, and that the platforms have not prioritized their personal privacy and personal security. Independently of disinformation and manipulation, this is another example of a platform that got big very quickly not necessarily having protections in place, and it registers on a deeply personal level with people.
I think that was sort of the secondary layer of outcry around both people who thought that this was relevant to the election as well as people who thought that this was simply [an] egregious privacy violation.
What about the impact of it actually getting Mark Zuckerberg to testify?
The platforms-they’re in an interesting position. They have a lot of power from being kind of entrenched, somewhat monopolistic, as Sen. Graham alluded to during the hearing, somewhat monopolistic – there’s no alternative, but they are still vulnerable to the actions of their users. And if your brand becomes synonymous with Russian disinformation, terrorism, trafficking, harassment, and now privacy and data leaks, that’s not a good place to be. And I think at that point they began to recognize that none of these press cycles were going away and that perhaps sending the CEO to speak for himself to put a face to the company was their best bet as far as trying to undo some of the damage that this steady stream of news stories had done.
And what were your observations or thoughts during those two days of hearings?
I thought, here’s an example, you know, a clear example of some lawmakers being remarkably sophisticated and prepared and others just not knowing what was going on. I thought that particularly in the Senate, Mr. Zuckerberg did perform well as apologetic and contrite. I thought that the hearings were challenging because they were so broad, there was so much complexity, so many different things to talk about-
You thought he did a good job?
I did. I thought he did a good job.
…Is there a larger takeaway from those hearings?
I think what we’ve seen [in] the investigations and the public accountings [that] have taken place is that this is a huge problem with so many different facets. We’ve had so many hearings, and each one is focused on a different thing. We had a terrorism-focused hearing; we had a privacy-focused hearing; we had three focused on Russia. We had a subsequent hearing focused on censorship which the platforms didn’t show up to, but the topic was still discussed-the idea that the conversations on the internet have become our new public square; a not-insignificant portion of those conversations appear to be manipulated; the fact that there doesn’t seem to be an easy fix or solution, so they continue to happen. The government is not responsible; the platforms are trying to get a little bit more responsible, but there’s ways to go there. The adversaries certainly haven’t stopped trying.
I think that it leaves a lot of people with the feeling that our social and political systems remain under threat, and I think it’s very hard to get your head around how these different threats fit together or what can be done to mitigate them.
Does it feel like an intractable problem to you?
It feels like a problem that is going to require the creation of some sort of third-party oversight body. It feels like a problem that’s going to require massive hiring, restructuring on the platforms, which to their credit they have begun to do. [It] feels like a problem that’s going to require information sharing and collaboration and cooperation between the platforms and then also between third-party researchers and the platforms.
So there’s a number of-we have to create a framework for monitoring and addressing this problem, and that does require the companies to do some things that they really don’t want to do.
…In some ways, Mark Zuckerberg has indicated that the answers lie within Facebook right now; the fix to Facebook is something Facebook needs to do. He said he’s welcomed-
If we want the 2018 elections to go well, then he’s absolutely right then [that] the fix to Facebook is within Facebook, and we don’t have time for anything else. At that point we have the third-party and platforms working hand-in-hand, academic researchers and platforms working together to solve it. There’s no regulation that’s going to happen between now and 2018 that’s going to do anything to ensure election integrity in 2018. That’s just a reality.
So given that, I do think that facilitating open channels of communication and working with the platforms to the greatest extent possible is the best possible thing we can do in the short term while recognizing that in the long term there does need to be some sort of independent oversight to think about the fact that we now have a privatized public square.
originally posted on pbs.org
