NEWS & INSIGHTS

ByteDance’s captioning patent: transcribing separate tracks and resolving overlapping display times

How US11837234B2 processes selected audio tracks and adjusts caption timing, with a practical distinction between patent disclosure and CapCut’s public guidance.

Conceptual cooking-video example showing each of two audio tracks being transcribed separately.
Conceptual cooking-video example showing each of two audio tracks being transcribed separately.

When narration and instructions overlap in a training video, transcription and on-screen timing require separate decisions. ByteDance’s US11837234B2 describes recognising selected audio tracks individually, ordering the resulting text fragments by start time, and shortening an earlier caption when display periods overlap. A hypothetical training video makes the sequence easier to follow.

1. When two voices overlap in a training video

Imagine an assembly tutorial with a presenter explaining the parts and a separately recorded instructor adding short directions. If recognition receives only the mixed soundtrack, simultaneous speech may produce missing or confused words.

An audio track occupies its own row on the video timeline. Keeping narration, instructions and music on separate tracks lets an editor adjust their position and volume independently, and choose which original tracks to transcribe.

US11837234B2, assigned to Beijing ByteDance Network Technology, was granted on December 5, 2023. Its disclosed approach recognises the selected tracks separately and brings their text fragments together for the same video.

2. Recognise each selected audio track

Automatic speech recognition converts recorded speech into text. Figure 1 describes locating audio data for each selected track when subtitle generation is requested; Figure 2 shows selecting one or more tracks and initiating generation.

For example, the editor may select narration and instructions while omitting a separate music track. Each voice is recognised from its own existing track. This process should not be confused with separating speakers from an already mixed recording.

The results are text fragments associated with display start and end times. Even though recognition occurs independently, the fragments must ultimately fit the shared video timeline.

3. Order fragments and shorten an overlapping display period

Simply appending one track’s entire transcript to another would give the wrong chronological order. Figure 3 instead orders fragments from the selected tracks by their display start times.

The system compares an earlier fragment’s end time with the next fragment’s start time. In the illustrated overlapping case, it changes the earlier end to the later start. A gap, or an end already equal to the next start, does not require that shortening.

The adjusted values govern caption display; the original speech stays in place. For fragments starting simultaneously, a separate embodiment retains the one ending latest. An editor who needs every utterance represented must therefore also review the resulting content.

4. A one-second overlap in a hypothetical tutorial

Suppose caption A, “Check the groove in the base first,” runs from 3 to 7 seconds, while caption B, “Stand the side panel upright,” runs from 6 to 9 seconds. They come from different audio tracks and overlap between seconds 6 and 7.

Hypothetical assembly tutorial with separate audio tracks and draft caption intervals.
Hypothetical assembly tutorial with separate audio tracks and draft caption intervals.

Applying the illustrated rule moves A’s end from second 7 to second 6. A now appears from 3 to 6 seconds and B from 6 to 9 seconds. The words and recordings remain unchanged; only the caption’s display interval changes.

Removing the overlap does not establish that the text is readable. A long instruction may not fit comfortably into three seconds. Splitting the text or adjusting its timing remains an editorial task, with the generated intervals serving as a starting point.

5. What CapCut’s public guidance describes

ByteDance lists CapCut among its video-editing products. CapCut’s help explains that automatic captions use speech recognition; on desktop, the user selects an audio source and language, generates captions, and edits their text or timing on the timeline.

The training team can review this draft while watching and listening. A misrecognised word needs a text correction, while a caption remaining after the spoken instruction has ended needs a timing adjustment. These are different editing decisions.

A consistent caption style across videos does not imply identical timing. Faster speech or an additional instruction can require different intervals. Reviewing scene changes, utterances and caption starts together helps identify the necessary edits.

6. Evidence a development team can record

Separate input selection, recognition quality and display quality. Record which tracks were selected and when speech overlaps, then examine missing words and whether each caption’s start and end leave enough reading time.

Compare mixed-audio and per-track recognition on the same video. Include short and long captions, identical start times and non-overlapping cases when assessing timing rules. Record both the corrections needed and the editor’s effort.

For a patent discussion, describe the track-selection criteria, the time values compared between neighbouring fragments, and any readability adjustment after shortening. Results recorded on the same timeline under changed conditions help explain the technical contribution.

The original figure labels and forms are preserved; the text explains the source material.

The training examples are hypothetical. Public product documentation does not establish that a product implements a particular patent claim.

USPTO patent US11837234B2

CapCut: recognising subtitles

ByteDance products and services

iFLYTEK speech recognition: matching meeting terminology to the audio

Samsung’s AI audio editing: preparing controls before separation finishes

Explore IPLEX’s AI projects and consultation materials to discuss speech recognition and caption-timing technology.

Explore our AI and patent practice

Original Tistory article

Read the Korean source

This article reflects the information available when it was published. Contact us to discuss your circumstances.
Discuss this topic ↗All articles

Put your IP strategy into practice.