The Algorithm is Still a Lie

Debunking (Yet Again) One of the Biggest Myths of the Streaming Wars

May 31, 2023

Today’s article is free for all to read. If you’d like to cross-post it to your newsletter, feel free!)

Last year, four days after I published my not-seminal-but-it-should-be article, “The Algorithm is a Lie”, HBO’s Barry had this scene:

The fake executive in the episode literally says, “The algorithm felt it wasn’t hitting the right taste clusters.” as if the “algorithm” is a thing that actually exists. (Or that it gets data analysis back that quickly.) Really, that (fictional) executive is bald-faced lying to Sally, since the “algorithm” doesn’t exist, but Sally believes her, taking this absurd claim seriously anyway like so many people in Hollywood today.

(To be clear, in reality, a show like the fictional Joplin, with 98% on Rotten Tomatoes, would absolutely stay up on the front page so the streamer could win an Emmy, even with horrible/non-existent ratings. That’s the actual reality of Hollywood over the last five years. Frankly, it’s Apple TV+’s entire business model.)

But it wasn’t just Barry. Four months later, She-Hulk: Attorney-At-Law’s final episode revealed that Marvel Studios is run by...an algorithm named “KEVIN”!

Obviously, this is comedy. Parody. Satire. That said, as the old cliche goes, there’s a grain of truth in all humor. The grain (and it’s not even a grain, but like the actual point and what actual people actually think) is that humans are no longer making decisions in Hollywood; data does. Or “algorithms”. (Or in the alliterative parlance of countless online cultural critics and writers, the “almighty algorithm”.) And writers feel powerless to push back against “data” and “algorithms”. Well, I have good news:

The algorithm is still a lie.

This isn’t news to longtime readers, but I’m revisiting this topic, because...

I have way, way more subscribers than I did last year.
I still see this silly idea floating around Hollywood all the time.
I have way more thoughts on this topic.
And more evidence that the algorithm doesn’t exist.

I mean, when writers as smart as Bill Hader, Alec Berg, and Jessica Gao believe in the “algorithm” or even think that “data” is driving decision-making more than humans, you know there’s a problem. Plus I’ve collected so many links, almost weekly, of reporters, critics and writers repeating this idea that the Netflix algorithm picks shows, I just have to debunk it again.

Again and again, if you ask yourself, “Did a human make this decision or a computer/algorithm?”, the human explanation makes way more sense.

Even More Examples of “The Algorithm”

Lest you think I’m making a straw man argument, here’s a brief selection of headlines and quotes blaming the “algorithm” for a new Netflix show that isn’t very good.

That’s not even a complete list. I’ve been collecting links of people lamenting the algorithm for six months now, and I have so many links that I published a companion post today, showing all of the examples I’ve found, more than would fit here. (I didn’t email it out, since I try to limit the number of newsletters filling up your inbox each week.)

You can find some examples here. Or here. Oh, also here, here, and here. And here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, and here.

That’s over twenty different publications! Over fifty examples! Sure, a bunch of these are clickbait-y pop culture websites (We Got This Covered, CBR and Screen Rant) but also prominent websites and news outlets like Vanity Fair, The LA Times, The Boston Herald, LA Magazine, the Guardian and the AV Club.

A Reminder from the Last Time We Discussed This: Algorithms Don’t Make Decisions; People Do

If you’d like, just read my article from last year, but to sum up...

“Enthusiastic” entrepreneurs have been selling products that they say can predict what screenplays are going to be hits for a long time now, literally for decades, predating Netflix’s vaunted “algorithm” by a while. They usually work well explaining past data, but fail to predict future hits.
Predicting the future is hard! Predicting what movies and TV shows are going to be hits is insanely hard. It is possible to model future outcomes, but that modeling requires lots and lots of data.
Compared to Google, Facebook, and other tech companies,1 the streamers do not have billions of data points from millions of users; they have performance data on, at best, thousands of shows and films. And this dataset isn’t robust enough to make the conclusions that many people think they do. (What about “recommendation algorithms”? Those do exist, as I discuss later in this article.)
Specifically looking at Netflix, they don’t know what TV shows are going to be hits ahead of a time. A lot of their hits surprised them! Especially Squid Game and Tiger King, but also shows like Manifest.
Algorithms can’t give notes. They just can’t. This is probably the argument/talking point that drives me craziest. Seriously, read the quote from Cary Fukunaga about his (not hit) show Maniac and ask yourself, “What dataset could have given that Netflix exec that insight?”

Algorithms Still Can’t Give Notes

Let’s expand, briefly, on that last point with two new examples. If you take nothing else from this article, know this: Netflix (and other streamers) don’t have the same level of data as other tech giants. Again, Google has billions of users and billions of data points (search terms); while Netflix has 200 million users, their data points (film and TV shows) is in the low thousands. It’s not enough data to generate these “algorithmic” conclusions. At least not very good ones.

What do I mean? Well, take a look at the Hollywood Reporter article from last fall on streaming documentaries:

“But then, a red flag: Gibney started to get notes from the streamers “that tried to scientifically rationalize the process,” he says: “‘Our algorithm states that by minute 10 you should do X, Y or Z.’”

Vulture earlier this year got even more specific on the algorithm and documentaries:

“There might now be a production company, with a streamer — and an algorithm — looking over its shoulder, asking the director to produce a moment “around 75% of the way into the story” when “something must happen to your hero that makes it seem to the audience that all is lost.”

Um, no “algorithm” said that.

Even if algorithms could recommend the types of shows Netflix and other streamers should make—those are actually called film comps and studios have been using them for decades without fancy algorithms—they certainly can’t give notes on story structure. There’s just not enough data. Despite the glut of (mostly true crime and biopic) documentaries from the last half decade, we’re talking (maybe) about hundreds of films. Even if you don’t isolate variables like runtime, language, subject matter and genre—which, if you’re doing this type of analysis, you absolutely have to isolate variables like that—that’s not enough data for an algorithm to find a trend like “have an “All is Lost” moment 75% of the way through a film”.

Basically, figuring out which documentaries had an “All is Lost” moment versus which ones didn’t, and then pinpointing where that “All is Lost” moment happened in the documentary (was if 50% of the way through? 65%? 80%?), then seeing which films performed better is just won’t provide statistically meaningful results, if the data even exists. There’s just not a big enough dataset to find this correlation.

Second, algorithms are no where close to even figuring out what would constitute an “All is Lost" moment! Maybe someday in the future, but not with our current level of technology.

But don’t take my word for it. The “All is Lost” moment is nothing new! I know, because this recommendation actually comes from the human-written 2005 screenwriting book, Save the Cat. (Here's the website which still applies beat sheet analysis to new movies.)2 No computers were needed for this note!

Again, don’t blame machines. Blame humans!

Don’t Take My Word For It. Check Out These News Articles

One of the main reasons that I’m updating this series/article today is because, in the last year, other reporters haveconfirmed that Netflix doesn’t have an algorithm. Again, people have to make decisions, using data. And this Insider headline sort of says it all:

As Elaine Low writes, “Humans ultimately hold the reins over whether a project gets made...each decision — and the risk that comes with it — is ‘owned by the creative exec.’” To which I would say, “Of course!” but, again, so many people in Hollywood believe otherwise. Read this whole article (it’s great) to get a sense of how messy and complicated data and data analysis can be.

That’s maybe my biggest frustration with people complaining about “algorithm”. It makes it seem as if using data to make decisions is easy and uncomplicated. It’s not. At the end of the day, humans have to make decisions using data.

Similarly, according to Variety and later The Wall Street Journal, even Netflix has started using focus groups:

“Netflix has been getting member feedback on original content — ahead of its public release — for about a year, Variety has learned.
Since May 2021, the streamer has been reaching out to small groups of subscribers with a proposition: The company is inviting them to participate in a panel to provide feedback on Netflix’s upcoming movies and TV shows before they’re released publicly.”

There’s nothing wrong with focus groups; indeed, I like and support them strategically. There’s insights to be gleaned that viewership alone can’t provide; but if you have an algorithm to make these sorts of decisions, why would Netflix be using an old school focus group? Isn’t that way more inefficient?

Netflix’s Algorithm And Kids Shows

As I wrote in my streaming ratings report a few weeks ago, Netflix has pivoted away from making creator-friendly kids shows, staying with what works: kids shows like Boss Baby: Back in the Crib. But here’s the thing: Netflix has always had kids shows from DreamWorks Animation on their platform! They’ve had a deal with DreamWorks since 2013!

So back to the non-existent “algorithm”: did the algorithm tell Netflix to spend big on creator-friendly, often niche, borderline-prestige kids content? If so, when did the algorithm change its mind and tell them to pivot back to licensed content? In the context of an algorithm, this makes no sense.

Conversely, in the context of development execs who are hungry to win awards and critical acclaim working for a company that’s willing to lose billions a year to gain marketshare, it makes total sense why they pivoted to “creator-friendly” kids shows! Even that article points out that they won awards. The human explanation actually matches the timeline.

The other, more charitable explanation is that sometimes (often?) business strategy and economics trump any so-called “algorithm”. In this case, owning fully-owned kids franchises makes more sense than licensing third party shows. At least, that was the hope in 2018, but clearly Netflix wasn’t able to pull it off completely.

I could run this exercise for half a dozen or more types of TV shows, films and genres where I don’t think the data justifies the decisions that Netflix makes. Again, humans are making decisions, not algorithms.

Netflix Does Have a Recommendation Engine Powered by “Algorithms”

To be clear, Netflix does have a powerful, influential algorithm: a recommendation engine. As I wrote last time:

"And they have a wonderful recommendation engine. (Which is powered by algorithms, perhaps the best in the business!)...
...Contrariwise, their recommendation algorithm does use the 200 million users’ choices to predict if they’ll like a show. That’s a lot more data! And even that is far from perfect."

But even the power of this recommendation is probably really overstated, especially as the “secret sauce” to Netflix’s success.

First, Netflix “perfected” this algorithm nearly a decade ago. Seriously, look it up. In 2006, Netflix’s recommendation had already plateaued, so they ran a contest (with a million dollar prize!) to improve it another 10%. And, initially, teams competing to improve the recommendation engine immediately improved the results, but improving it over 10% was really, really tough. Future gains were tiny, as happens when you reach upper-limits of technology like this. It shows how hard future gains perfecting the recommendation engine could be.

Which isn’t to say that Netflix’s engineers haven’t improved their recommendation engine any further in the last fifteen or so years, but they almost certainly haven’t made gains like they had in the first few years.

And I would argue that the recommendation engine has probably gone down in quality since. Why? Well, at the time of the contest, streaming video didn’t exist yet. The engine was recommending DVDs, or, essentially, every available movie from all time, over 100,000 movies at one point. Today, Netflix’s US library consists of 6,473 titles. Basically, with a lot less TV shows and movies to recommend, that recommendation engine has to be even better to keep the accuracy of its recommendations as high as it’s been in the past.

Plus a recommendation engine is only as good as what it can recommend and, as I linked to a couple of weeks ago, the quality of Netflix’s titles might be going down, according to IMDb scores (H/T Omdia).

Finally, back to my main question: is it humans or machines? So let me ask it bluntly:

Do you think executives at Netflix can influence what shows get on the front page and for how long?

Obviously they can. My proof? Well, last October, I reviewed every streamers’ Halloween landing pages...and found that Netflix was promoting unliked, barely-watched Original horror films. I can’t think of any reason why the Netflix recommendation engine would actively promote bad films, over good films, unless the recommendation engine prioritizes Netflix Originals.

Again, blame people, not machines/data/algorithms.

Why Do People Believe in the “Algorithm”?

There’s a psychological theory that the reason that many people believe in conspiracy theories is that they’re actually comforting to the conspiracy theorist. Basically, conspiracy theories give them the sense that someone is out there controlling everything, which is strangely reassuring to some people. Someone is in control! Instead of just random chance bringing misfortune to people, there’s a puppet master controlling the strings. And in some ways, that’s better than things being random.

I think the “algorithm” provides the same comfort to many people in Hollywood.

Instead of blaming a person, you can blame a machine. Instead of blaming executives at Netflix, you can blame “the algorithm”. I think this is especially true for online cultural critics. Countless millennials and Gen-Z reporters and writers grew up watching Netflix and it’s a part of their lives. But now, so many people often feel betrayed by their beloved streaming service cancelling so many of their favorite shows, especially YA fantasy shows, and they need someone (or something) to blame. And that thing is the algorithm.

For creatives, the “algorithm” provides an external factor to blame, shifting the locus of control (“I didn’t write a bad show; the algorithm buried it!”) to an external force, which is definitely a comforting notion.3

And it maintains relationships. You can stay friendly with the executive at a streamer who cancelled your show if it wasn’t their fault/decision, but the algorithm’s. It’s a faceless, literally inhuman villain.

Unfortunately, though, the algorithm doesn’t exist. But humans do.

As I wrote about in my original article, Facebook and Google do have the data needed to make really effective algorithms...and those algorithms can be really, really bad for you and the world. And artificial intelligence and machine learning is about to turbo-charge this. Just wait until large language models are creating social media posts, videos, and music, then tailoring future changes to the feedback it’s getting.

I’m just debunking the notion, today, that Netflix uses an algorithm to pick and choose TV shows. In general, skepticism of algorithms and their power is more than warranted.

The Vulture article even mentioned one production company’s “six-page “Story Structure Template,” complete with breakdowns for what it wanted to happen at specific moments”. I’m guessing that’s where this came from, but people love blaming the “algorithm”.

If you are a screenwriter, the algorithms you should be worried about power large language models like Chat-GP3/4. And I think the WGA is absolutely right to hold the line on AI-generated content in Hollywood for this exact reason. Maybe large language models aren’t powerful enough, right now, to write great screenplays, but I wouldn’t bet on it for long, especially based on the gains with programs in Chat-GPT in just the last two years.

I don’t think screenwriting and screenwriters will go extinct, even with AI writing scripts. But I’d bet that 90% of screenwriting jobs would go away, with development execs, working with a handful of screenwriters to polish screenplays written by large language models.

The Entertainment Strategy Guy