PROBLEM AND MOTIVATION
Analyzing large volumes of audiovisual material to identify narrative, aesthetic, and thematic patterns requires more than manual interpretative approaches, which often fail to capture subtle visual relationships. To address these challenges, we investigated how computer vision models can complement traditional methods, enabling contextual and systematic analysis of extensive image datasets.
Initially developed for static collections, the Semantic Imagery Mapping (SIM) methodology was adapted to apply computer vision models and attention algorithms to audiovisual libraries. This led to the creation of Movie Scene Sensing (MSS), a tool designed to analyze scenes in complex visual narratives. MSS detects narrative patterns, editing styles, and thematic axes, revealing how these elements emerge, connect, and evolve throughout a work.
Viewing Films Through Algorithmic Lenses
DEVELOPMENT CONTEXT
Movie Scene Sensing is more than a technical tool; it is a methodology that repositions computer vision algorithms as "critical partners" and "co-creators" in the analysis of visual narratives. Developed within the research project Borrowing Algorithmic Epistemologies, coordinated by Prof. Elias Bitencourt, MSS explores how computational epistemologies can be adapted to study and reinterpret visualities in cinema and audiovisual media.
Alongside other methodologies like Semantic Imagery Mapping and the Digital Practices Matrix, MSS repositions algorithms not merely as classifiers but as mediators in the creation and interpretation of visual narratives.
This approach uncovers narrative and aesthetic patterns, offering alternatives for studying and producing audiovisual works. By combining critical analysis, creative experimentation, and methodological imagination, MSS expands the possibilities of using computer vision models as methodological resources for the humanities, media studies, and applied social sciences.
Reframing Machine Learning as a Method for Audiovisual Analysis
HOW IT WORKS
Keyframes Extraction
Keyframes are selected based on visual and narrative changes identified by computer vision models. These frames encapsulate essential narrative and stylistic moments of the film.
Clustering with Semantic Imagery Mapping (SIM)
Keyframes are grouped by conceptual and compositional similarity, independent of chronological order. SIM identifies not only superficial elements like colors and textures but also compositional patterns, framing, costumes, facial recognition of characters, and narrative themes.
Visualization in Thematic Maps
The groups are projected into a vector space and displayed as topic maps. These maps allow exploration of relationships between narrative and stylistic patterns, highlighting contrasts, connections, and the centrality of elements within the work, regardless of chronological linearity.
Labeling and Visualization
Clusters are labeled and differentiated by colors, facilitating the analysis of stylistic and thematic relationships. It is possible to organize the clusters chronologically in an image wall, reconstructing the film's narrative by highlighting the distribution of patterns over time.
Clip Generation
The tool enables the generation of isolated clips for each cluster, automatic creation of representative scenes, or remixing of groups, allowing for alternative audiovisual analyses and creations
Beyond Visual Similarity and Linearity
POTENTIAL APPLICATIONS
Initially conceived as a methodological interface for media, communication, and audiovisual researchers, Movie Scene Sensing can also be utilized by filmmakers, editors, and directors to explore raw material, identify narrative and stylistic patterns, seek references, and support editing processes.
Beyond analysis, the tool allows for new interpretations of existing works, generating clips or remixes that offer alternative perspectives for audiovisual analysis and production.
Support for Analysis and Creation of Audiovisual Material
AVAILABILITY AND ACCESS
Currently, Movie Scene Sensing is under development and limited to ongoing research within the laboratory. It is not yet available for general use or broad distribution.
Collaborations may be considered for specific projects aligned with the lab's research agenda and team availability. Proposals for collaborations or partnerships will be reviewed based on their potential contribution to ongoing investigations.
If you are interested in exploring collaboration or investment opportunities, please contact the group leader for preliminary discussions: eliasbitencourt@gmail.com.