Subtitles & captions

[jminor]

We should support subtitles and captions.

[jminor]

There are several existing formats for subtitles that could be supported via adapters.
WebVTT: https://w3c.github.io/webvtt/
SSA: https://wiki.multimedia.cx/index.php?title=SubStation_Alpha
SRT, etc.

FCP XML supports <title> elements.
AAF probably supports titles too. Someone will need to research this...

Looking at how those formats represent titles, we should be able to model them as a subclass of Marker. Formatting information will likely be hard to normalize across all the formats, but in the spirit of OTIO, we can focus on the content and timing of the titles as that is likely to be the most useful part in practice.

The use cases for subtitles in OTIO seems to relate to swapping in/out localized subtitles, and/or passing subtitles up or downstream in a production pipeline.

[jminor]

Here is some info about how AAF encodes video titles: https://www.amwa.tv/downloads/specifications/AS-05_AAF_Effects_protocol_v1.pdf

I'm not sure if this is the same as a subtitle track. This still needs more research.

[jminor]

Here are some more links for reference, though the license for some of these may not be compatible.
http://www.nikse.dk/SubtitleEdit/
https://github.com/ubershmekel/pytitle
https://github.com/pyahmed/sub2xml

[KarthikRIyer]

I'd like to attempt this. But I'm still trying to understand how different subtitle formats work.

http://www.nikse.dk/SubtitleEdit/ this seems helpful. It has a common representation for subtitles: https://github.com/SubtitleEdit/subtitleedit/blob/master/libse/Subtitle.cs

Will the license be a problem if we refer to this implementation (because we won't be lifting code directly)?

[Jameclarke]

Here is a great article referencing how subtitles are used.
https://jonnyelwyn.co.uk/film-and-video-editing/adding-captions-in-premiere-pro-fix-common-problems/

[apetrynet]

Some of the subtitle formats include positioning so I think this ties into the coordinate system and annotations as well.
#773 #771

[KarthikRIyer]

What are annotations?

Are the positions as coordinates or something like align: left align: center? I saw the alignment approach in webvtt I think. If we use that approach we could leave it up to the adapter or the final consumer of the subtitles to decide the actual position, can't we?

[apetrynet]

Annotations can be drawings, text etc overlaid the video. There's been discussions on how to implement this and that got tied to the position coordinate system.
Yes, those aligns are the ones I was thinking about. It could be up to the adapters, but if we want to have a common ground schema for several subtitle formats it's worth considering to have a way of storing positions.
I don't think waiting for a coordinate system should hold you back from working on this. We can always revisit positioning at a later stage.
Just my two cents (since I looked at this issue my self a few weeks ago :)

[meshula]

@KarthikRIyer Since the license you reference is GPL, one should not reference that code, nor transliterate it.

[reinecke]

Hey @KarthikRIyer, check out TTML2 to get a feeling for what can be expressed in timed text: https://www.w3.org/TR/ttml2/

The Netflix Tech blog has some articles about the complexities of timed text (captions and subtitles). Here are a couple of them:

• https://netflixtechblog.com/implementing-japanese-subtitles-on-netflix-c165fbe61989
• https://netflixtechblog.com/a-scalable-system-for-ingestion-and-delivery-of-timed-text-6f4287a8a600

Hopefully that gives a good jumping off point.

[KarthikRIyer]

Thanks @reinecke ! I'll go through these links.

[KarthikRIyer]

Here's what I thought we could start with, based on what I understood from TTML2 and with how I've used captions in Premiere Pro. Please let me know any suggestions/changes/improvements.

• I think storing a Style object with each TimedText is wasteful, but decided to have this approach because we don't have id system in otio yet. In TTML2 there's a section to specify all styles and then refer then in each TimedText. With the current approach conversion from otio to say TTML2 won't be so direct because we don't have a single set of styles. One other thing I thought of was to have a list of styles inside Subtitle and then have a string id in each TimedText. But then each TimedText object will have no style meaning independently/outside the Subtitle. Thoughts?

• What are Markers used for? Each marker has a markedRange. When we trim an Item or change the sourceRange shouldn't the markedRange of the markers at the start or end of the Item also change? I couldn't find anything that does something like this. Or is this handled in another way? Why I'm asking is if we trim the Subtitle or change its sourceRange, the ranges of TimedTexts at the start or end might change.