Home | Resources | Blog

Why high-quality automatic subtitling is harder than it looks

At first glance, automatic subtitle generation seems straightforward. Speech recognition produces text, the text is displayed on screen.  Job done.

In practice, it’s far more complicated. . Subtitles are not simply transcripts placed over a video. They are built the way they are so people can read them easily while still following what is happening on screen. The viewer should never feel like the text is competing with the story.

Getting that balance right turns automatic subtitling into a surprisingly difficult problem.

Subtitles are a visual experience

One of the most underestimated aspects of subtitles is their visual presentation. Viewers read them while watching the video at the same time, so that readability has to be immediate and effortless.

That’s where segmentation comes in.

Segmentation is the way speech is divided into subtitle blocks and lines. It determines:

  • Where one subtitle ends and the next begins
  • How text is split across one or two lines
  • Which words remain together
  • How easily the subtitle can be read in the available time

When segmentation works well, subtitles feel almost invisible. When it does not, viewers slow down, reread lines, or lose the flow of the dialogue.

That’s why segmentation needs to be viewed as a core problem in subtitle generation, rather than a simple formatting step.

Different domains, different expectations

Subtitles should not look the same everywhere. There’s no universal standard, and different platforms and industries expect different styles.

For example:

  • Some applications prefer bottom-heavy subtitles, where the second line carries more text.
  • Others favour balanced lines, where both lines have similar lengths.
  • Some prefer top-heavy subtitles, where the first line contains more content.

These differences aren’t arbitrary. They usually reflect platform guidelines, accessibility requirements, or simply the viewing habits audiences have developed over time.

This means an automatic subtitling system has to adapt to different segmentation profiles while still keeping the text comfortable to read.

Every language has its own typographic rules

The challenge becomes even greater when multiple languages are involved.

Each language comes with its own typographic conventions and segmentation rules, such as:

  • Restrictions on where line breaks may occur
  • Rules about splitting grammatical structures
  • Punctuation placement
  • Handling of clitics, particles, or compounds
  • Natural reading rhythm

A segmentation strategy that works well for English may be inappropriate for German, Spanish, or Japanese. When those linguistic differences are ignored, subtitles often feel awkward or unnatural.

Handling these nuances properly requires both language-specific knowledge and specialised algorithms.

Learning from real subtitle data

Theory alone is not enough to build good subtitle systems.

For example, we study large volumes of subtitle material across many languages and domains. This includes both publicly available datasets and carefully curated private collections.

Looking at real subtitles helps us understand how professional segmentation works in practice and how different domains structure their subtitles.

At the same time, we actively follow academic and industry research in automatic subtitle generation from audio. The field is evolving quickly, and keeping up with that research helps us keep improving our systems.

Combining multiple technologies

One common misconception about automatic subtitling is that a single model can solve the entire problem.

In reality, producing high-quality subtitles requires multiple specialised technologies, each optimised for a specific task.

AI models for fluency and readability

We use advanced AI models to refine subtitles so that the message reaches the reader as smoothly as possible.

These models help:

  • Improve sentence flow
  • Reduce awkward phrasing
  • Adapt spoken language for reading
  • Preserve meaning while enhancing clarity

This step ensures that subtitles feel natural to the viewer, rather than reading like raw transcriptions of speech.

Deep neural networks for speech tagging and localization

Modern deep neural network (DNN) models help us analyse the structure of speech and detect language-specific nuances.

These models support:

  • Accurate alignment between speech and text
  • Identification of linguistic patterns
  • Handling of multiple languages and dialects
  • Adaptation to locale-specific conventions

This allows us to treat subtitles as language-aware content rather than plain text.

Typographic algorithms for precise compliance

AI models provide flexibility, but subtitle formatting also requires precision.

We rely on traditional typographic algorithms that enforce specific subtitle profiles. These algorithms ensure:

  • line length constraints are respected
  • reading speeds remain comfortable
  • segmentation follows typographic rules
  • visual balance matches the desired style (bottom-heavy, balanced, or top-heavy)

By combining AI-driven refinement and deterministic typographic control , we can maintain natural language flow while still meeting strict formatting compliance.

Rigorous evaluation is essential

Improving subtitle systems requires careful evaluation.

We test our systems across multiple domains, languages, and content types to make sure performance holds up in real-world conditions.

To do this, we rely on metrics used by both industry and academia.

Reference-based metrics

One important metric we use is SubER (Subtitle Edit Rate).

SubER measures how much editing would be required to transform our automatically generated subtitles into high-quality human references. It provides a clear indication of segmentation accuracy and formatting quality.

Reference-free quality assessment

We also employ reference-free QA-style metrics that analyse subtitles without relying on human references.

These metrics allow us to examine detailed aspects:

  • Compliance with subtitle profiles
  • Segmentation quality
  • Typographic adherence
  • Formatting consistency

By combining reference-based and reference-free evaluation, we gain a more complete understanding of subtitle quality.

Why the problem is genuinely difficult

Automatic subtitling sits at the intersection of several complex fields:

  • Speech recognition
  • Linguistics
  • Typography
  • Artificial intelligence
  • Human readability research

Each of these areas brings its own challenges. Combining them into a system that produces fluent, readable subtitles across languages and domains is not straightforward.

Turning speech into text is only part of the task. The real challenge is producing subtitles that people can read comfortably, quickly, and without losing track of what is happening on screen.

When subtitles work well, viewers barely notice them… and that’s exactly the point.

Webinar: Watch now!


Semih Altinay, VP AI Solutions at Phrase, guides you through the ins and outs of Phrase Studio.

See Phrase Studio in action and learn how to convert spoken content into high-impact, multilingual assets.

Watch on demand

Keep exploring

Blog (Podcast) Crayola CMO - Borrow and Build - FI

Blog post

How can one global brand stay relevant to different audiences?

How can a global brand maintain a consistent identity when its audiences have different expectations? Former Crayola CMO Victoria Lozano faced this challenge across a business serving children, educators, and international licensing partners. Drawing on Victoria’s borrow-and-build approach, together with independent research, this article examines how businesses can maintain brand consistency while giving individual teams the freedom to respond to their customers.

Victoria Lozano - Crayola CMO - Creative Ecosystem in Colour

Blog post

What business are you really in? How brands find new growth opportunities

Most companies define themselves by the products they sell or the categories they compete in. That definition can narrow where they look for growth. Drawing on the experience of Victoria Lozano, former CMO of Crayola, this article examines how brands can identify new business opportunities by understanding the broader needs they serve. From differentiating Dentyne and Trident at Cadbury to extending Crayola into experiences and licensing, Victoria offers lessons for executives deciding where to invest and how to take a brand into new markets.

Blog post

Our WordPress integration shipped 16 releases this year. Should you build your own?

Most requests I get for a new integration come from teams whose tools we don’t cover yet. Here’s how I’d decide, system by system, between the integrations we maintain, the ones our partners maintain and the ones you build yourself. Version 4.8.6 of our WordPress integration, released on July 17, 2026, fixed a single bug. […]

Blog post

Automated translation workflows that know a tagline from a contract

A tagline, a legal clause and a product description go wrong in different ways, so they shouldn’t share a workflow. Here’s how to decide what to automate for each content type, how to check it, and where a person still needs to be involved. My team looks after Phrase’s own localization, and we don’t use […]

Blog post

How to localize regulated financial content without losing required disclosures

Most translation workflows are optimized for speed and cost. Regulators care about something else: getting it right, and being able to prove it. When a national competent authority asks about a translated Key Information Document (KID), it rarely asks how quickly the translation was turned around. It asks who reviewed the disclosure, when, and on […]