Updated: Jun 29, 2021
By Matteo Antona, Felix Hanschke, & Hannah Pamplin
Tim Smith has found the perfect middle-ground of research to satisfy his interests. At the intersection of science and the arts, his research involves building theories of film-making processes that guide our engagement – what mechanisms are required to create an environment in which we willingly suspend our disbelief for two or so hours. A significant amount of his understanding comes straight from the horse's mouth: Smith believes that talking with filmmakers, directors, and craftspeople and learning from their creative processes is key to creating his knowledge framework.
He touched on a number of aspects important to engaging filmmaking in his talk; from this we parsed three sections into which he appears to divide his work – attentional theory, synchrony, and the importance of audio in scene creation.
Attentional Theory of Continuity
Continuity editing is the baseline of filmmaking. It is the process of combining shots into a sequence that hooks spectators’ attention. To do so, filmmakers have to maintain a consistent perception of time and space in a sequence. To put it simply, in classic Hollywood style, film editing has to be “invisible” (Dmytryk, 1986; Smith, 2012).
But is this just a visual style? Let's start with the first shot of a scene. Smith explains that establishing shots gives directors awareness of what camera angles will make sense for an audience. Once the opening shot is established, the director has a 180-degree axis around which to shoot. This is a guideline regarding the on-screen spatial relationship between a main character and others. By keeping the camera on one side of the imaginary axis between two characters, the first person is always located in the frame right of the second. Working within this arc means that the audience will not be distracted by jarring perspective changes between shots,(Kachkovski, Vasilyev, Kuk, Kingstone, & Street, 2019), (Levin, 2010), creating a spatial relationship between the audience and the action.
Fig 1. The image represents a violation of the 180-degree rule, (Levin, 2010). In Shot 3 (180 violation) the character A is in the same frame position of the character B in Shot 2. This means that the audience perception can be disturbed by the prospective change between shots 2 and 3, and thus perceive less continuity.
Smith also discusses how continuity editing can help guide our attention between scenes. Much of the research in this area comes from creators themselves - trial and error processes of filmmaking, experimentation, audience box office beta-testing, and eye-tracking studies that have led to an understanding of how best to create seamless transitions that are similar enough to our real-world experiences that we can attend in the same way we would with daily life, (Anderson, 1996). Tim suggests that continuity editing (such as making a scene cut right as movement begins) works, because it replicates how we attend in the real-world, (Smith, 2012), creating a fluid attentional experience.
Fig 2. From Every Frame of Painting, Joel & Ethan Coen - Shot | Reverse Shot. Coen’s brothers are masters of shot/reverse shot continuity editing technique, helping viewers to orientate themselves by informing them where each actor is located. For more information regarding the film editing continuity visit: https://nofilmschool.com/what-is-continuity-editing.
Movies Synchronize our Brains
The second topic introduced by Smith is the phenomenon that movies produce similar patterns of brain activity among spectators. This predominantly occurs within the brain regions associated with vision, audition, language, and recognition for faces and places (Hasson et al, 2004). A study by Hasson and colleagues (2004), revealed that 65% of audience members' brains showed synchronized activity while watching Hitchcock’s “Bang! You’re Dead”, while watching Sergio Leone’s “The Good, the Bad and the Ugly” produced synchrony in 45% of the audience. These results were put in contrast to a naturalist scene from Washington Square Park, which resulted in only 5% brain synchrony among audience members - demonstrating that masters of the art, like Hitchcock and Leone, utilize filmmaking techniques that capture our thoughts, feelings and sensations, to unify us through our aesthetic experience.
Smith explains that shot composition and length determine the extent to which the brains tick together. This is due to the viewer's tendency to always place the point of interest within their foveal region - the most sensitive part of the eye's visual field, approximately two degrees wide. If the point of interest exceeds these two degrees, viewers automatically shift their gaze to fully grasp the scene, leading to a greater variance in viewer brain synchrony, (Bordwell, 2006; Smith, 2012). In empirical work, heat-maps are often used. Here eye movements are recorded by eye trackers and compared with movements of others watching the same scene (Smith, 2015). To this end, choosing the correct shot size will optimize how people perceive the image.
Fig 3. Example of heat-map generated from real-time gaze tracking. The red dots represent the focal points of attention of different viewers looking at the scene. In this case the attention is directed towards the face and legs movements of the character. Movements, as well as faces and expressions, are usually great eye-catchers. https://www.neurotechnology.com/sentigaze.html for more information.
Other than shot size, gaze synchrony also depends on the length of a scene, and how it is cut (Bordwell, 2006). Usually, in a long scene, viewers often visually explore the screen and this leads to greater divergence of attention. However, in a comparison of movies produced between 1935 and 2010, among other differences, psychologists noticed a decreasing shot length over time (Cutting, 2011). Indeed, in contemporary cinema techniques promoting attentional synchrony, such as rapid editing, extremely short or long lens lengths, wide-ranging camera movements and reliance on close-ups, are always more used. But why does this happen, it has something to do with sound synchrony?
Fig 4. According to research by James Cutting, the average shot length in Hollywood films is increasingly shortening! It has moved from more than 10s in the thirties to less than 4s now. A similar trend was noticed by the Film Historian Barry Salt as shown in In the top right graph (Miller, 2014).
Sound in Continuity Perception
How does sound, a key film component, influence our perception of continuity? “The Jazz Singer”(1927) is the first talkie film, (Jolsonville, 2013), and a few years later (1929) Walt Disney directed “The Skeleton Dance”; an animation with non-post-sync sound (Devilmanozzy, 2009) - considered to be one of the first ever music videos!
While US directors started to experiment with unlimited sound possibilities, on the other side of the world, another father of cinema was captivated by the power of music. In a sequence from “Alexander Nevsky” (1938), Russian director Sergei Eisenstein claimed that it was the music score composed by Prokofiev that guided the audience's gaze, (Eisenstein, 1943, p. 137). While Eisenstein did not have means to test his intuition at the time, fortunately, Tim Smith and colleagues (2014) have exactly the right technology today. Using eye movement tracking, Smith asked audiences to watch the film sequence with and without audio. In the two conditions there was no difference in eye movement - suggesting that visuals are prioritized in the audio-visual correspondence (Smith et al., 2014), however, this might also mean that Prokofiev just knew exactly how to complement Eisenstein’s vision.
Fig 5. Audio-visual correspondences in the sequence "Battle on the Ice" (Alexander Nevsky, 1938). The shots depict the picture frames, music phrases, music score, duration, diagram (composition), and diagram of the gaze movement (gesture) (Eisenstein, 1943).
But how does audio affect editing perception? In a recent experiment, Smith asked participants to detect cuts in eighty clips from different films. With audio they missed 2/3 of the cuts, compared with very few misses without sound, (Smith & Martin-Portugues Santacreu, 2017), leading to the idea that the audio serves as scaffolding - where images and motion can build upon and reinforce a seamless attentional span from one frame to the next.
In the course of his lecture, Tim Smith gave an intriguing peek behind the curtain into what makes engaging cinema. More of Tim’s musings on the attentional theory of cinematic continuity can be found in research from CineLab, which he heads, and his blog, Continuity Boy.
Anderson, J. (1996). The reality of illusion: An ecological approach to cognitive film theory. SIU Press. https://www.amazon.co.uk/Reality-Illusion-Ecological-Approach-Cognitive/dp/809321963
Batten, J. P., & Smith, T. J. (2018). Looking at Sound: Sound Design and the Audiovisual Influences on Gaze. In T. Dwyer, C. Perkins, S. Redmond, &J.Sita https://cognitiveresearchjournal.springeropen.com/articles/10.1186/s41235-08-01405
Bordwell, D. (2006). The way Hollywood tells it: Story and style in modern movies. Univ of California Press. https://www.fulcrumorg.gold.idm.oclc.org/concern/monographs/7m01bm30s
Cutting, J. E., Brunick, K. L., DeLong, J. E., Iricinschi, C., & Candan, A. (2011). Quicker, faster, darker: Changes in Hollywood film over 75 years. i-Perception, 2(6), 569-576. Cognitive Media Theory (0 ed., pp. 97–117). Routledge. https://doi.org/10.4324/9780203098226-14
Devilmanozzy, Oct.7 2009, Skeleton Dance, accessed Mar. 15, 2020, https://cartoons.fandom.com/wiki/Skeleton_Dance
Dmytryk, E. (1986). On Filmmaking. London: UK Focal Press. Eisenstein, S., & In Leyda, J. (1947). The film sense: By Sergei M. Eisenstein. New York: Harcourt, Brace and World. https://www.goodreads.com/book/show/291225.The_Film_Sense
Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., & Malach, R. (2004). Intersubject synchronization of cortical activity during natural vision. science, 303(5664), 1634-1640. https://science.sciencemag.org/content/303/5664/1634
Kachkovski, G. V., Vasilyev, D., Kuk, M., Kingstone, A., & Street, C. N. (2019). Exploring the effects of violating the 180-degree rule on film viewing preferences. Communication Research, 46(7), 948-964. https://pure.hud.ac.uk/en/publications/exploring-the-effects-of-violating-the-180-degree-rule-on-film-vi
Levin, D. (2010). Spatial representations of the sets of familiar and unfamiliar television programs. Media Psychology, 13(1), 54-76. https://www.tandfonline.com/doi/abs/10.1080/15213260903563006
Miller G. (2014). Data From a Century of Cinema Reveals How Movies Have Evolved, Wired. https://www.wired.com/2014/09/cinema-is-evolving/
Nannicelli, T., & Taberham, P. (Eds.). (2014). audiovisual correspondences in sergei eisenstein’s alexander nevsky: A case study in viewer attention tim j. smith. https://eprints.bbk.ac.uk/id/eprint/12587/1/Alex_Nev_CogMediaTheory_2014_timjsmith_figs.pdf
Smith, T. J., & Martin-Portugues Santacreu, J. Y. (2017). Match-Action: The Role of Motion and Audio in Creating Global Change Blindness in Film. MediaPsychology, 20(2), 317–348.
Smith, T. J., & Mital, P. K. (2013). Attentional synchrony and the influence of viewing task on gaze behavior in static and dynamic scenes. Journal of vision, 13(8), 16-16. https://doi.org/10.1080/15213269.2016.1160789
Smith, T. J. (2012). The attentional theory of cinematic continuity. Projections, 6(1), 1-27. https://eprints.bbk.ac.uk/id/eprint/6679/
Smith, Tim J. (2015) Read, watch, listen: a commentary on eye tracking and moving images. Refractory: A Journal of Entertainment Media 25 (9), ISSN1447-4905. https://eprints.bbk.ac.uk/id/eprint/12583/1/Refractory_commentary_timj mith_ccepted.pdf
Smith, Tim J. and Henderson, J.M. (2008) Edit blindness: the relationship between attention and global change blindness in dynamic scenes. Journal of Eye Movement Research 2 (2), pp. 1-17. ISSN 1995-8692. https://eprints.bbk.ac.uk/id/eprint/6678/
Jolsonville, Oct. 9, 2013, The first talkie - the "The Jazz Singer", accessed Mar. 15, 2020, https://jolsonville.net/2013/09/10/the-first-talkie/#more-1016
Alexander Nevsky (1938), 35mm, black & white, sound mix mono,112 min. Sergei M. Eisenstein, Mosfilm, available on Amazon Prime. https://www.imdb.com/title/tt0029850/?ref_=fn_al_tt_1
Bang! you’re Dead (1955-1962), black & white, sound mix mono, 30min. Alfred J. Hitchcock, Alfred J. Hitchcock Productions, Shamley Productions. https://www.imdb.com/title/tt0508131/
The Good, the Bad and the Ugly (1967), 35 mm (Anamorphic), Color (Technicolor), sound mix Dolby Digital, Technicolor S.p.a., Roma, Italy (color). Sergio Leone, Produzioni Europee Associate (PEA), Arturo González Producciones Cinematográficas, Constantin Film. https://www.imdb.com/title/tt0060196/?ref_=nv_sr_srsg_0
The Jazz Singer (1927), 35mm, black & white, sound mix mono, 88min. Alan Crosland Warner Bros., available on Amazon Prime. https://www.imdb.com/title/tt0018037/?ref_=tt_mv_close
The Skeleton Dance (1929), 35mm, black & white, sound mix Cinephone, 6 min. Walt Disney, Walt Disney, available on Youtube. https://www.imdb.com/title/tt0020414/?ref_=fn_al_tt_1