Spoken Language Understanding of Human to Human Conversations

The sequence of steps involved in speech metadata extraction is visualized in the above figure. As an example, let us choose a call-centre conversation between an agent and a customer. The details for each of the steps are discussed below,

Call recording

The only input to a speech analytics system is the call recording. Some of the common challenges in call recordings are

• Acoustic challenges

• • Single vs. Stereo channels: Some of the telephony recordings have the agent and customer voices in separate channels. This is an ideal scenario allowing easy identification of where the agent and customer are speaking. However, the majority of the telephony recordings have both agent and customer voices on the same single channel. You will need a diarization system for such single channel recordings to identify different speakers and where they are speaking within the recording.

  • Noisy backgrounds

    • There can be background noise on both the customer and agent sides depending on where they are calling from. Generally, agents have a high side-chatter noise on their end because multiple agents are sitting in a single room and speaking to customers. These kinds of noises can affect the ASR and further the NLU performance.

  • Multiple speakers

    • Some conversations can have multiple agents or customers depending on the complaint. Irrespective of the single or stereo recording, we will need a diarization system to help us figure out how many speakers are present in a call, and when do they speak.

• Transmission and storage challenges: In order to enable faster transmission and reduced storage requirements, call recordings might be compressed, resulting in codec related artefacts which can further affect the SLU performance.

Language identification

The first step of SME is to identify the language of the call, this is especially an important block when the call-center receives calls in multiple languages. Identifying the language of each call enables us to first identify the volume of calls in each language, and further helps us choose the language-specific downstream models of speech-to-text (ASR) and SLU.

Diarization and voice activity detection (VAD)

As discussed in the call recording section above a diarization system is required whenever there are multiple speakers recorded on a single recording channel. The combined system of VAD + diarization together tell us - how many speakers are present, at what time-stamps do they speak, and where are the non-speech or silence regions. The speech regions are processed by downstream models to identify the spoken content. The non-speech regions are used to evaluate different aspects of the call - how noisy is the call? Is the noise from the agent's or the customer's side? what kind of noise is prominent? - This information gives us additional insights for improving the experience for the customers.

Gender identification

For each of the speakers in the call recording, we can identify their gender. This is an additional metadata that can be used for business insights from the call centre.

Speech-to-text or automatic speech recognition (ASR)

Finally, based on the detected language the corresponding ASR system is chosen to recognize the spoken text for each of the speech regions in the call recording.

The above figure visualizes some of the NLU items over the conversational output of SME. We will discuss each of these NLU items in detail below.

Speech turn

A speech turn is one instance of a speaker's dialogue. A conversation is made of multiple speech turns from all the participating speakers. For example, in the above figure, 'Good morning, how may I help you?' is the first speech turn, 'Hello. I am calling regarding my internet connection.' is the second speech turn. There are no duration criteria for speech turns, it can be as short as a single word, or can be multiple sentences long.

Speech dialog act detection

The segmentation of a conversation into speech turns is not really helpful for any application, we need a more meaningful way of segmenting the conversation. In the literature, there have been multiple proposals for this, and one of the popular ways is by using speech dialog acts. According to this, speech turns can be split into multiple meaningful segments based on the following four broad speech acts. Further, each dialog act can be classified into sub-speech acts.

1. 1. Constatives: Sentences that are making a statement

      • • Answering, Claiming, Confirming, Denying, Disagreeing, Stating

   2. Directives: Sentences attempting to get the addressee to do something

      • • Advising, Asking, Forbidding, Inviting, Ordering, Requesting

   3. Commissives: Sentences committing the speaker to future action

      • • Promising, Planning, Vowing, Betting, Opposing

   4. Acknowledgements: Sentences expressing the speaker’s attitude regarding some social action

      • • Apologizing, Greeting, Thanking, Accepting an acknowledgement

Post the speech dialog act detection, every speech turn is split into meaningful sentences, each categorized by the main speech act, and the sub-speech act.

Speech chapters, or keypoints identification

In a conversation, multiple consecutive speech turns can be about the same topic. For example, in the above figure, we see that the complete conversation can be split into five broad topics - opening, information verification, conflict situation, problem resolution, and closing. These topics can vary across calls, however, the total number of such topics is a small finite number. In the literature, this categorization of consecutive turns of the same topic is also referred to as speech chapters or speech keypoint identification. This kind of topic identification can help the NLU layer understand the context of the speech acts. For example, the sentence 'app is not opening', can mean different things based on what chapter it is detected under. For instance, if the detected chapter was 'conflict situation' then the sentence was describing the conflict. However, if the detected chapter was 'closing', it can mean that the customer was simply enquiring about some possibilities beyond the main problem.

Call topics or themes identification

The call topic or theme is a call level metadata. For a given call, we identify the topic of the call. This will help us understand the volume of calls for different call-level topics.