Researchers Build AI Model That Automatically Generates Movie Trailers

Artificial intelligence is a technology that can be applied in any field. Everything happening today is proof of this: projects based on machine and deep learning are being used in a variety of areas, from medicine to entertainment and art. 

Trailers are short video clips that introduce viewers to new films and are often key elements of advertising strategies used by film studios. To be most effective, trailers should briefly outline the film's plot, conveying its artistic style and overall mood in an appealing way.

According to one school of filmmaking, trailers should have a narrative structure consisting of three acts: the first act establishes the characters and plot, the second introduces the main conflict, and the third raises the stakes and offers teasers for the ending. 

Another school relies more on the trailer's mood, determined by the story's highs and lows. According to this approach, trailers should first have medium intensity to engage viewers, then low intensity to convey key information about the story, and then gradually increase intensity to reach a climax at the end of the trailer. 

Until now, movie trailers have mostly been created by humans. However, some scientists working with AI have recently begun exploring whether these promotional videos could also be created automatically by machines.

Researchers from the University of Edinburgh have developed a model based on an artificial neural network that can automatically create movie trailers. The model, presented in a paper on arXiv, was built using an unsupervised graph-based machine learning algorithm.

To best handle the task of automatically creating movie trailers, the researchers divided it into two subtasks: identifying the film's narrative structure and predicting the sentiment (that is, the mood) it conveys. As a result, their technique processes both parts of the film (that is, video) and text excerpts from the movie script.

"We model movies as graphs, where nodes are frames and edges denote semantic relationships between them. We learn these relationships by applying co-contrastive learning that uses privileged textual information (e.g., characters, actions, situations) from screenplays. Then an unsupervised algorithm traverses the graph and generates trailers," said Pinelopi Papalampidi, Frank Keller and Mirella Lapata, who conducted the study. 

In essence, the trailer creation method they developed consists of two neural networks. While one of these networks processes multimodal representations of frames taken from the film's video stream, the other analyzes textual representations of moments based on the movie script.

Together, the two neural networks can identify turning points in the film, which are especially important parts of the movie that should be shown in trailers. Turning points in films typically include an opportunity, a change of plan, a point of no return, a major setback, and a climax.

The scientists evaluated their movie trailer technique in a series of tests. They found that it can identify turning points in films with significantly greater accuracy than other baseline trailer generation methods.

"The advantage of our approach is that it is interpretable and can easily be used as a human-in-the-loop tool. In particular, given the immediate neighborhood at each step, one could select frames based on various automatic criteria or even manually. Our approach dramatically reduces the number of frames that need to be reviewed to create trailer episodes, to 10% of the entire film. Moreover, our criteria allow users to explore different sections of the film and create diverse trailers," the study says. 

In addition, the researchers used their model to create trailers for 40 different films. They then compared the quality of these generated trailers with trailers created using methods based on supervised learning, that is, with human assistance. 

The scientists asked questions to viewers selected on Amazon Mechanical Turk (AMT). Participants were asked which trailers they preferred. Interestingly, most respondents favored trailers created by their technique over those produced by supervised models.

Although the model created by Papalampidi, Keller and Lapata may not yet produce perfect trailers, film studios could eventually use it to ease and speed up trailer production. Meanwhile, the team plans to continue working on its technique to improve the quality of the videos it produces.

The researchers said: "In the future, we would like to focus on methods for predicting subtle emotions (e.g., grief, disgust, terror, joy) in films. In this work, we treat positive/negative sentiment as a proxy for emotions due to the lack of labeled in-domain datasets. Opportunities for future work include new emotion datasets for films, as well as emotion detection models based on text data and audiovisual cues."