Gestures and multilevel discourse in spontaneous speech corpora: the case of reported speech

1. Introduction

This work deals with the analysis of spontaneous speech to study the reported speech units and the gestural units that are produced in parallel to them. For this purpose, gestures are understood as deliberate actions that the interlocutor perceives as aiming to expression – and not to some purely practical goal (Kendon 2004).

Furthermore, in this perspective, besides being synchronously manifested with speech and expression-oriented, gestures are idiosyncratic, since they are produced instantly by the speaker as they express themselves (McNeill & Duncan 2000). In other words, although a few gestures are conventional in certain cultures, gestures produced parallel to speech do not have a pre-established or conventionalized pattern among speakers.

Given their expressive scope, the gestures we perform when engaging in trivial interaction can be conceived as integral parts of an utterance. They unfold consistently with what is expressed in speech, i.e., the spoken component of that same utterance (Kendon, 2004). Thus, speech and gesture, indeed, correlate in such an intrinsic way that they can be treated as parts of a single semiotic sphere (McNeill 1992).

Regarding the relation between gestures and prosody – an aspect of speech that is of particular interest to this research –, there is a tendency for gestural and prosodic units to be synchronized, as already noted by Kendon (1972) in his primordial analyses (see also McNeill 1992, 2005; Kita et al. 1998). Subsequent corpus-based studies grounded in the Language into Act Theory (Cresti 2000; Moneglia & Raso 2014; Cavalcante 2020) have shown, among other things, that boundaries of gestural units end where prosodic boundaries are completed (Cantalini 2018; Cantalini & Moneglia 2020; Barros 2021).  The present research was also based on the Language into Act Theory framework. Therefore, section 2 will briefly discuss its main theoretical guidelines to contextualize our analyses.

2. The language into Act Theory

This study was grounded on the theoretical framework of the Language into Act Theory, henceforth L-AcT (Cresti 2000; Moneglia & Raso 2014; Cavalcante 2020), a pragmatic corpus-driven theory aimed at studying the informational structure of spontaneous speech. Within this framework, the units of reference for linguistic analysis are the minimum units that can be pragmatically and prosodically interpreted in the flow of speech (Izre'el et al, 2020). Such units are identified through prosodic and pragmatic parameters: they contain at least one illocution and are considered terminated units since a terminal prosodic boundary can be perceived after them.

These units of reference can be constituted by one or more prosodic and informational patterns. According to the L-AcT, informational functions are conveyed by prosodic units. A prosodic/informational pattern, in turn, is made up of a nuclear unit that carries the illocutionary force, the comment unit (COM). A pattern can also host other non-illocutionary units, which can be of two types: textual units, which compose the semantic text of the pattern, and dialogic units, which are intended to regulate dialogic interaction. These can be classified based on three criteria: function, position, and prosodic form (f0 curve, duration, intensity, and alignment with the syllabic structure) (t’ Hart; Collier; Cohen 1990; Firenzuoli 2003). The prosodic units are internally separated by non-terminal prosodic boundaries.

Based on the concept of prosodic/informational pattern, the units of reference can be of two types: utterances or stanzas. Utterances can be defined as the terminated units made up of just one prosodic/informational pattern; stanzas are terminated units made up of more than one juxtaposed pattern linked by a boundary with a continuity signal.

In our study, the locutive introducer (INT) is of interest among the non-illocutionary units that make up a prosodic/informational pattern. This unit is important because INT signals that what comes next is on a different level, not directly related to the current conversation (Maia Rocha & Raso 2011).

3. Discoursive level

While analyzing spontaneous speech, three different discursive levels are observed: the level of the utterance, the level of the parentheticals, and the meta-illocutionary level (see example 3.4 and figure 3.1). In the speech flow, when the speaker moves from one level to another, they mark this transition prosodically in order to indicate that adjacent prosodic units belong to different hierarchical levels.

The level of the utterance is where most of the interactional flow between speakers takes place. It is grounded in the here and now of the interaction. The parenthetical level, in turn, conveys any comments made by the speaker about what they say at the level of the utterance in order to make its interpretation clearer. These comments can be metanarrative – when the speaker adds information considered crucial to understanding what has been said (see example 3.1); modal – when the speaker shows their commitment to what has been said (see example 3.2); or metalinguistic in a lexical sense – when the speaker reformulates something they said (see example 3.3) (Barros 2021; Santos 2020; Tucci 2010).

The examples below were all extracted from the C-ORAL-BRASIL I (Raso & Mello 2012) corpus, and the parentheticals are labeled PAR.

Prosodically, the parenthetical level often exhibits a change in f0, usually a fall, compared to adjacent units belonging to the utterance level. It also shows a reduction in intensity and may exhibit a higher articulation rate relative to its surroundings. There is often a pause before and/or after the parenthetical (see Figure 3.1).

From another perspective, the meta-illocutionary level indicates that the here and now of the situation is pragmatically suspended. The most common metaillocutions are emblematic exemplification, instruction, and, above all, reported speech, which will be discussed in section 3.1. Example (3.4) illustrates the instruction.

Example (3.5), extracted from the C-ORAL-BGEST corpus (Mello et al. in preparation), illustrates a situation where all three levels can be observed. This excerpt’s units were tagged using the informational labels adopted by L-AcT – COM, COB and CMM stand for the illocutionary units; TOP stands for topic, i.e., the domain of identification for the interpretation of the illocution; and PAR stands for parenthetical².

To indicate that a certain unit is reported, « _r » (r for reported) is added to the label assigned to that unit. Therefore, if there is, for instance, a reported illocutionary unit, its label will be COM_r, COB_r, or CMM_r.

In the example below, meta-illocutionary level is indicated in bold, and the parenthetical level is evidenced in italic.

                 Figure 3.1 – Intensity and f0 contour for example (3.5)

On the other hand, since speech and gesture can be treated as parts of a single semiotic sphere, we wanted to verify whether the differences between the three discursive levels are marked not only prosodically but also gesturally. In fact, Barros 2021 noted that the shift from the utterance level to the parenthetical level is marked through strategies of gestural pattern contrast. This paper aimed to analyze the shift to the meta-illocutionary level, focusing on the case of the reported speech.

3.1 - The reported speech

Reported speech can be defined as a direct discourse the speaker inserts into the flow of his  own speech. In reported speech, there is a change in deictic parameters; that is, the here and now is suspended to insert something that was said at another time, either by the speaker or by someone else.

However, when reporting a past discourse, the speaker not only repeats (or tries to repeat) what has been said but may also incorporate other strategies that allow the hearer to notice the suspension of the here and now of the current situation. Indeed, Good (2015) argues that reported speech is actually just one facet of a whole reported action.

According to Good 2015, some features that may follow the reported speech are shifts in body posture, gaze, prosodic variation, and enacted gestures. Regarding prosodic variation, reported speech tends to exhibit higher values for intensity and f0 when compared to unreported speech.

Zuckerman 2021 draws attention to the fact that the reported speech is not an exact and faithful enactment of the reported event. Hence, if we analyze them from a gestural perspective, and not only from a semantic or prosodic viewpoint, we can access valuable information about the action itself that is reported but mainly about how the speaker chooses to do so.

In the scope of this paper, we intended to investigate, based on spontaneous speech corpus data, the hypothesis that the shift from the unreported speech to the reported speech, and vice versa, is also marked gesturally.

4. Gestures

Just like speech, gestures can be subdivided into units, the so-called gesture units (GUnits). A GUnit is completed when the speaker removes their hands from a default position, defined as the rest position, performs a series of gestures, and then returns their hands to the rest position. Therefore, GUnits can be defined as «the set of gestures performed between two resting positions.»

The GUnits themselves can be subdivided into gesture phrases (GPhrases). GPhraes are defined by the presence of one gestural nucleus, called stroke (see fig. 4.1). The stroke is one of the gesture phases (GPhases) that constitute a GPhrase and is the only mandatory one. Just as the illocution is the nucleus of a prosodic and informational pattern, the stroke is the nucleus of a GPhrase. Other GPhases can be combined with the stroke to form the GPhrase, and they can be classified in four different ways, depending on the function they perform (fig. 4.1): preparation; retraction; hold, and rest (Ladewig & Brassem 2013).

Figure 4.1 – gesture phases (Ladewig & Brassem 2013, p. 1075)

Studies conducted on spontaneous Italian speech (Cantalini 2018; Cantalini & Moneglia 2020) have shown synchrony between prosodic and gesture units and the tendency of the two nuclei to align with each other (i.e., illocutionary focus and stroke). Furthermore, the authors claimed that GPhrases tend to remain within the same unit of reference (see section 2) and never cross the terminal prosodic boundaries. Barros 2021, while investigating parentheticals in Brazilian Portuguese, also observed that the stroke tends to synchronize with prosodic prominences. Mayberry & Jacques 2000, focusing on stuttering, concluded that interruptions in speech flow had repercussions on gestural patterns, which were interrupted or altered and only were resumed when speech was also resumed fluently.

The concept of gestural pattern considered here encompasses the gestural parameters introduced by Bressem, Ladewig, & Müller 2013, namely those related to movement type, direction, and quality; handshape, orientation, and position (for further details, see Bressem 2013). A change in at least one of these parameters constitutes a change in gestural pattern.

5. The C-ORAL-BGEST

C-ORAL-BGEST (Mello et al, in preparation) is a multimodal corpus of spontaneous speech that is part of the C-ORAL-BRASIL project. By the time this paper was written, the corpus consisted of eleven multimodal texts in Brazilian Portuguese and included video and audio recordings. Altogether, the recordings were 24 minutes and 28 seconds long.

The 11 texts analyzed are informationally tagged according to Language into Act Theory and gesturally labeled according to the guidelines of McNeill 1992, Kendon 2004, and Bressem, Ladewig & Müller 2013. The perceptually-based prosodic segmentation follows the pragmatic-prosodic parameters used in other reference corpora, such as C-ORAL-BRASIL I (Raso & Mello 2012) and C-ORAL-ROM (Cresti & Moneglia 2005) based on the perception of non-terminal boundaries (marked by a single slash, «/») and terminal boundaries (marked by a double slash, «//»). Text, image, and sound were aligned in ELAN software (Sloetjes & Wittenburg 2008), through which qualitative analyses were also done.

6. Results and discussions

By analyzing the 11 texts that make up the C-ORAL-BGEST corpus, eight reported speech constructions were detected. In total, 103 reported prosodic units were found, structured in 35 shifts from utterance level to the reported speech level (and vice versa). According to our observations, 30 of the 35 shifts occurred alongside changes in the gestural pattern.

Regarding the situations in which significant changes in the gestural pattern were not identified, two of them occurred with a speaker who did not gesture profusely throughout the interaction. In the specific passages in which the reported speeches took place, this speaker did not gesture either at the level of the utterance or at the level of the reported speech. In the other cases, the speakers either did not gesticulate in the excerpts of reported speech or only made «incidental movements» (Kendon 2004), i.e., movements that were not aimed at expression, such as scratching the eyes.

Among the changes in the observed gestural pattern, some were concerned with using the body space. In these cases, the speaker used, for instance, his left body space to perform the gestures belonging to the utterance level but used his right body space to gesticulate during the reported speech. In other situations, there were changes in the gestures’ amplitude – from more closed hands to more opened hands when switching levels, for example. There were also instances when speakers changed their head orientation or body position only during the reported speech. Moreover, we identified situations where the speaker stopped gesturing during reported units and resumed gestures when the utterance level was restored.

These results seem to show a relation between the shift between discourse levels and the gestural patterns performed concurrently    .

6.1 Examples    

In this section, we will present several examples that have formed the foundation of our analysis regarding reported speech. In all examples(6.1.1, 6.1.2, 6.1.3, 6.1.4), the meta-illocutionarylevelcorresponds to reported speech, and has been highlighted in bold. It is important to recall that labelsassigned tothis level areappendedwith« _r », where « r » stands for reported.

Example (6.1.1) below was extracted from the multimodal text bgest_004. *PEU is the speaker.

 Figure 6.1.1 – left: unreported speech | right: reported speech

The figure on the left, taken from the initial, unreported part of example (6.1.1) («aí minha mãe me ligou» | «then my mother called me»), shows that, while at the utterance level, *PEU's gestures were mainly characterized by larger movements. At this point, the speaker kept his arms rather open. During the reported speech, as we can see in the picture on the right, *PEU kept his hands closer together while gesturing. Back at the level of the utterance, the speaker did not gesture immediately: his hands remained at rest position for approximately 5 seconds, and *PEU resumed the gestures in another reported speech unit later.

Example (6.1.2) was extracted from the multimodal text bgest_005. The reported parts are marked in bold, and *ZUC is the speaker.

Figure 6.1.2 – left: unreported speech | right: reported speech

In the image on the left, which corresponds to the moment before the reported speech (precisely «aí vem o cara lá do anteprojeto e fala» | «then the guy from the preliminary project says»), *ZUC is leaning on the back of the chair. During the reported speech, however, the speaker moves away from the back of the chair. When resuming the utterance level, *ZUC leans back again on his chair.

Besides, *ZUC also modifies the axis of his gestures when switching levels. Before and after the reported speech, *ZUC's gestures are made along a horizontal line «parallel» to his body, as shown in the picture on the left. During the reported speech, however, the gestures are performed along a horizontal line «perpendicular» to his body (sagittal axis).

Example (6.1.3) was extracted from the multimodal text bgest_006. The reported parts are marked in bold, and *ELI is the speaker.

Figure 6.1.3 – left: unreported speech | right: reported speech

To perform the gestures she made before the reported speech (picture on the left), *ELI uses the central space of her body and her right hand or both of her hands. During the reported speech (picture on the right), she uses her left hand, or both of her hands, in her left space. When resuming the utterance level, she goes back to gesture in the central space of her body.

Example (6.1.4) was extracted from the multimodal text bgest_007. The reported parts are marked in bold, and *CAR is the speaker.

Figure 6.1.4 – left: unreported speech | right: reported speech

For most of the time in this recording, *CAR held a glass; in the excerpt shown in example (6.1.4), she alternated between reported and unreported speech. During all unreported speech units, she held the glass with only one hand and used her other hand to perform several gestures (image on the left). Alternately, during the reported speech units, she kept both her hands on the glass. So, *CAR’s gestural change is consistently linked to the alternation between reported and non-reported speech.

Hence, in example (6.1.4), *CAR gestured only at the level of the utterance – when telling the situation that preceded the reported speech and also in the comments she made about the speech she reported (for instance, «hh tipo / bem [/1] bem chata //» | «like / very [/1] very annoying //»).      

7. Conclusion

In this work, the reported prosodic/informational units were analyzed under the theoretical framework of the Language into Act Theory. The gestures were analyzed based on the studies carried out mostly by McNeill 1992, Kendon 2004, and Bressem, Ladewig & Müller 2013.

According to Good 2015, reported speech can be conceived as an integral part of a whole reported action, which involves various semiotic processes, including prosodic and gestural ones. Thus, we expected to find evidence that the insertion of reported speech into the flow of spontaneous speech was accompanied by marks of its actional nature, both in gestural and prosodic terms.

The results obtained seem to indicate that the differences between the levels of reported and unreported speech tend to be marked not only prosodically but also gesturally. The main changes in gestural patterns that marked the shift from one level to another concerned the use of the body space, the gestures’ amplitude, changes in head orientation or body position, or the alternation between the presence and absence of gesture.

Extending this analysis to the idea of discourse levels, we may assume that the distinction between the utterance level and the meta-illocutionary level is not only prosodic but also gestural. Since Barros 2021 has already pointed out the gestural distinction between the parenthetical level and the utterance level, we can conclude that the gestural pattern is a distinguishing element between utterance, parenthetical and meta-illocutionary levels. For future studies, it would be interesting and relevant to analyze the gestures produced in other types of metaillocution, such as emblematic exemplification and instruction.