Spoken workout cues with AVSpeechSynthesizer, and when to stay quiet

Synthesized speech is the cheapest way to give a workout app a voice: no recording sessions, every number and exercise name available on demand, every language the device has a voice for. It is also where the documented details bite hardest, because a cue is time-critical in a way a read-aloud article is not. "Rest over" spoken four seconds late is worse than silence.

This page covers AVSpeechSynthesizer and AVSpeechUtterance for coaching cues, and the promptStyle hint Apple publishes for exactly this kind of prompt. The session configuration underneath it is on the category page and the duckOthers page.

The object, and the bug everybody hits first#

Apple's AVSpeechSynthesizer is "An object that produces synthesized speech from text utterances and enables monitoring or controlling of ongoing speech." You create an AVSpeechUtterance with the text and pass it to speak(_:).

The note at the bottom of that page explains the most common first bug, a cue that never plays or stops early: "The system doesn't automatically retain the speech synthesizer, so you need to manually retain it until speech concludes." A synthesizer created as a local variable inside a cue function can go away before it speaks. Hold one synthesizer for the workout, as a property of whatever owns the cue schedule.

final class CueSpeaker: NSObject, AVSpeechSynthesizerDelegate {
    private let synthesizer = AVSpeechSynthesizer()   // retained for the workout
    private var lastUtterance: AVSpeechUtterance?

    override init() {
        super.init()
        synthesizer.delegate = self
    }

    func say(_ text: String) {
        let utterance = AVSpeechUtterance(string: text)   // a new utterance every time
        lastUtterance = utterance
        synthesizer.speak(utterance)
    }
}

The comment on the utterance line is the second bug. Apple's speak(_:) page carries a warning: "Attempting to enqueue the same utterance more than once throws an exception." Caching a "Go!" utterance and replaying it is a crash, not an optimisation.

It is a queue, and late cues have to be cancelled#

Apple: "The speech synthesizer maintains a queue of utterances that it speaks. If the synthesizer isn't speaking, calling speak(_:) begins speaking that utterance either immediately or after pausing for its preUtteranceDelay, if necessary. If the synthesizer is speaking, the synthesizer adds utterances to a queue and speaks them in the order it receives them."

That is right for reading an article and wrong for a workout. If a long cue about the next exercise is still playing when the interval ends, the "go" cue waits behind it, and arrives late. Our rule: a cue that is stale when its moment passes must be cancelled, not queued.

Apple's stopSpeaking(at:) is the tool: "Unlike pausing a speech synthesizer, which can resume after a pause, stopping the synthesizer immediately cancels speech and removes all unspoken utterances from the synthesizer's queue." Its boundary argument, AVSpeechBoundary, has two cases, .immediate and .word, which Apple describes as choosing "whether to stop speech immediately or only after the synthesizer finishes speaking the current word."

// The interval ended: anything still queued is now wrong.
synthesizer.stopSpeaking(at: .word)
synthesizer.speak(AVSpeechUtterance(string: "Go"))

Our judgement on the boundary: .word for a cue that is merely superseded, because a clipped syllable sounds like a fault; .immediate when the user paused the workout and expects silence now.

Who owns the audio session#

This is the decision that determines whether the user's music ducks. Apple's usesApplicationAudioSession is "A Boolean value that specifies whether the app manages the audio session," and: "If you set this value to false, the system creates a separate audio session to automatically manage speech, interruptions, and mixing and ducking the speech with other audio sources."

Apple's page documents what false does; it does not state the default, and we have not verified it, so set the property explicitly either way. If you keep your own session, the delegate callback below is where the activation window closes.

Finishing: the delegate callback that closes the window#

speechSynthesizer(_:didFinish:) "Tells the delegate when the synthesizer finishes speaking an utterance," with a detail worth knowing: "The system ignores the final utterance's postUtteranceDelay and calls this method immediately when speech ends." That makes it the right moment to deactivate a ducking session and give the music back.

func speechSynthesizer(_ synthesizer: AVSpeechSynthesizer,
                       didFinish utterance: AVSpeechUtterance) {
    guard utterance === lastUtterance else { return }   // a later cue is still queued
    try? AVAudioSession.sharedInstance()
        .setActive(false, options: .notifyOthersOnDeactivation)
}

Utterance settings, and when they stop applying#

From Apple's AVSpeechUtterance reference:

• rate is "a decimal representation within the range of AVSpeechUtteranceMinimumSpeechRate and AVSpeechUtteranceMaximumSpeechRate," defaulting to AVSpeechUtteranceDefaultSpeechRate. "Set this property before enqueing the utterance because setting it afterward has no effect."
• volume runs from "0.0 for silent to 1.0 for loudest volume," default 1.0, and setting it after enqueueing "has no effect."
• voice: "If you don't specify a voice, the speech synthesizer uses the system's default voice." AVSpeechSynthesisVoice(language:) takes "A BCP 47 code," and passing nil gives "the default voice for the system's language and region."
• preUtteranceDelay and postUtteranceDelay: with several utterances queued, the synthesizer "pauses a minimum amount of time equal to the sum of the current utterance's postUtteranceDelay and the next utterance's preUtteranceDelay."
• prefersAssistiveTechnologySettings: "A Boolean that specifies whether assistive technology settings take precedence over the property values of this utterance." For a VoiceOver user with a speaking rate they chose, that is worth setting; the wider VoiceOver picture is on the live-metrics page.
• SSML: init(ssmlRepresentation:) returns nil "if you pass an invalid SSML string," and Apple notes that properties affecting prosody, "such as its rate and pitchMultiplier, don't apply to an utterance that uses an SSML representation."

The SSML note is a quiet trap: a team that moves cues to SSML for emphasis loses the user's speech-rate preference if it was applied through rate.

Apple also documents mixToTelephonyUplink, "A Boolean value that specifies whether to send synthesized speech to an active call," which "has no effect when there isn't an active call." Our judgement: a coaching cue has no business in the other person's ear on a phone call, so leave it false.

promptStyle: Apple's signal for when to shut up#

Apple publishes a hint written for navigation and exercise prompts. promptStyle is "A hint to audio sessions that use voice prompt mode to alter the type of prompts they issue in response to other system audio, such as Siri and phone calls." Apple: "Apps that issue voice prompts should observe changes in the prompt style and modify their prompts in response. This property is key-value observable." It applies to sessions in the voicePrompt mode.

"Exercise prompts" is Apple's phrase, not ours. Read the style at the moment of the cue, not when the workout starts, and route the cue to speech, tone or haptic accordingly.

Monday morning#

Make the synthesizer a long-lived property and create a fresh utterance per cue. Put stopSpeaking(at:) in front of every time-critical cue. Decide usesApplicationAudioSession explicitly. Close the ducking window in didFinish. Then observe promptStyle and test it the documented way: start a workout, invoke Siri mid-interval, and confirm the cue becomes a haptic.