Ah, showbiz. The glitz! The glamour! The dedicated, optimized AI tools creating a new economy built on the dreams of writers hoping for their big break!
If that last line is unfamiliar territory, allow me to enlighten you. This might be one of the most controversial side effects of the AI writing debate, and it’s been building to a boil for more than a decade.
The old doorways for new screenwriters to enter the business of penning potential blockbusters used to be entirely human-led, but customized AI tools and services designed to read scripts and provide detailed notes in minutes have flipped the script. I wanted to see just how far these AI script-reading tools have come by testing them against some human feedback.
What I found was a strange duality in which AI trained on human blood, sweat and tears can sometimes create an uncanny note-for-note facsimile of the real McCoy. But while the algorithm can copy-paste the analysis, there are certain things it can’t fake.
How we got here
In the good old days, one way to become a screenwriter was that you’d go to school for screenwriting and then become a screenwriter’s assistant (or know the right people). There’s also the most straightforward, labor-intensive way to secure a career: being a person who reads scripts and writes feedback on those scripts.
One current WGA writer told me that reading and offering feedback on scripts was something an aspiring writer could never turn down; you’d never get called back.
They’re called “readers” in the biz, and AI services that claim to do their jobs faster and at a scale that’s impossible to humanly match have multiplied with the efficiency of tribbles. OnDesk, Greenlight AI, Scriptreader.ai, AIScriptReader, PreScene and Storyflow are just a few, not to mention your basic ChatGPT, Gemini and Claude AI-generated notes.
And then there’s the service that may be responsible for starting it all — Scriptbook, which, according to its marketing documents, “is the original patented AI for screenplay analysis and box office prediction.”
The arguments against using these tools are loud and oftentimes angry for obvious reasons. Unions, professional screenwriters, film fest runners and readers themselves are actively fighting a battle for the freedom to perform their work without competing with tools trained on the same standards, conventions and formats on which they honed their craft. Tools that are perhaps even trained on their own work, without their consent.
The use of AI in writing scripts is explicitly forbidden in most festivals and open screenwriting contest cases, but only a few services and communities apply that same scrutiny to people reading the submissions. That’s where our testing comes in.
The rules of the game
The Black List is one script-hosting and reading platform that explicitly states it uses only human readers who don’t use AI in their processes.
The service came under fire in 2017 for a very brief association with Scriptbook, offering the AI platform’s services to its users for an extra $100 fee. The Black List backtracked on the partnership shortly thereafter, and the service’s founder, Franklin Leonard, remains skeptical about AI’s value in the space.
“The notion that people would even want AI responding to their work seems to be based on an assumption that LLMs have been optimized to provide writers feedback that will improve their work,” Leonard said in an email to CNET. “I don’t believe that’s true.”
Man vs. machine
In an era where whipping up a story can be as simple as writing a prompt, I tested how AI readers compared to the real McCoy on The Black List by penning my own feature-length script over the course of a weekend, no AI involved.
The Black List doesn’t regularly reveal how many submissions it gets over specific periods of time, making it mostly a black box when it comes to overall reader workload and volume.
Once I had the human-written notes in hand — which are broken into categories of Strengths, Weaknesses and Prospects — I compared them with notes from two of the most popular AI script readers, OnDesk and Greenlight.
After a simple upload of the script, which took minutes, the resultant feedback was close to The Black List’s, in both detail and overall impressions.
For example, The Black List’s evaluation pointed out issues with dialogue, stating that “dialogue is stilted and functions primarily as a vehicle for spoken exposition.” (I said it was a script, not a good script.)
The OnDesk feedback stated, “The dialogue often tells us what the scene already shows, with repeated speeches that explain rules and repeat insults.”
The Black List evaluation praised the use of a “warped two-hander approach,” while the AI OnDesk feedback mimicked that praise with almost identical words, calling it “a two-hander about abuse, control and revenge.”
Overall, industry jargon, positive, negative and tonal notes made the same appearances in both the OnDesk feedback and The Black List evaluation, as well as notes on commercial
I ran another test with a different script and evaluation from The Black List, as well as notes from an active WGA writer, through OnDesk and received similar feedback from both the AI and the human reader versions.
For example, they included notes that B-plot elements felt too disconnected from the main plot, and the setting was a strong suit. Both the AI tool and The Black List evaluation also called out character notes.
From The Black List: The script has some fun moments, but the plot underneath it all is too thin for any of the characters or their storylines to develop.
From the AI tool: Character work is solid but limited… Dialogue is often funny.
According to the WGA writer I talked to for this piece, the biggest difference they found in AI script readers and human ones was subjectivity.
“It’s kind of miraculous how they can give you really interesting, kind of like, detailed feedback in a way that, you know, does not seem like something you’d be able to get out of a machine,” said the WGA writer who has used both The Black List’s evaluations and AI script reading tools like Greenlight as well as Claude AI, ChatGPT and Google Gemini.
“At first blush, it’s fairly remarkable,” the writer added. “The thing that gets a little bit more challenging is if you put a bunch of scripts through there, you start to see some patterns.”
After reviewing the similarities in coverage, I made a support ticket and sent it to The Black List, asking for a secondary evaluation, and expressing suspicion of AI use to see what the typical response to a customer might be like.
The Black List’s director of customer experience, Lillian Beaudoin, told me in part, “I can assure you we have systems in place to both detect the use of AI in evaluations and, most importantly, to help prevent its use in the first place.” She also noted readers at The Black List are not allowed to download the work they evaluate.
She added that, “while an AI-generated evaluation might employ the specific format of The Black List’s evaluations and use similar vocabulary, the expertise and insight of a human reader with a background in the entertainment industry will not be present.”
The bottom line on AI script readers
The Black List representatives maintain a posture of close scrutiny around AI script reader usage in their paid evaluations, and have never discovered a reader in their employ using those tools. This may be because it has never happened.
It may also be that when someone says they suspect AI was used in the development of a script evaluation, it is literally impossible to discover if it was, and the feedback is frankly so close to human feedback that it may not even matter. That’s the nature of using models trained on reams of script evaluations.
“As I’m sure you know, AI text detection – particularly over short lengths of text – is far from accurate or consistent at this point,” Beaudoin told me. “It’s also worth noting that, as likely thousands of Black List evaluations have been shared across Reddit and other platforms beyond our website, the Black List style has almost certainly been scraped into more than one LLM’s training datasets.”
Still, AI script-reading services like PreScene and Callaia cite major talent agencies like Paradigm, studios like Warner Bros, Sony and Lionsgate and producers as “leading entertainment companies that trust” their products.
Franklin, for his part, doesn’t see a threat from a technology that, inherently, can’t do what a person can.
“On some level, I hope every other company in the marketplace decides to use AI exclusively to evaluate material, because if they do, the Black List’s human approach will beat them by a country mile,” he said.
Although these AI script reading tools can give feedback that sounds very human because they’re trained on human work, an algorithm can’t pass your screenplay onto an agent, or recommend a writer for a gig that gets them into the right rooms.
AI may be bad news for Hollywood hopefuls looking to bust through the barriersfor an increasingly crowded field of AI script reader tools that position themselves as gatekeepers who hold the golden tickets
In the meantime, this little pocket of Hollywood’s creative pipeline is still open for business, if you can compete with the machine.
