Skip to content
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/_static/edit/01_overwrite.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/edit/02_insert.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/edit/03_trim.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/edit/04_slice.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/edit/05_slip.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/edit/06_slide.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/edit/07_ripple.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/edit/08_pointedit.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/edit/09_roll.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
354 changes: 354 additions & 0 deletions docs/design/editorial_design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,354 @@
Designing an Edit API
=====================

# Design Overview
The industry is full of timeline management software but nearly all of them include a fundamental set of tools for manipulation. The most well known being:

- Overwrite
- Trim
- Cut
- Slip
- Slide
- Ripple
- Roll
- Fill (3/4 Point Edit)

OpenTimelineIO, while not seeking to become a fully realized NLE, could benefit from having a simple toolkit to help automate building/modifying timelines based on time rather than index alone.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be rephrased to be closer to OpenTimelineIO's scope... maybe something like:

OpenTimelineIO includes a suite of tools for operating on objects as raw data structures (append, etc), but is missing a library for performing higher level operations familiar to people from the editing world (Slip, 3/4 point edit, etc). This proposal presents an API for working with OpenTimelineIO objects in those ways.

I don't think they need to be enumerated here, since you're going to be covering them in more detail later.


## Structure
The underlying implementation for these edit commands will be in C++, possibly under a new library called `openedit`. This has the benefit of connecting with other languages more simply (Python, Swift, C, etc.) and, hopefully, we only need expose the functionality to the bindings.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean that you'd want a separate c++ DSO? To sit next to OpenTime.so and OpenTimelineIO.so? I kind of think it just belongs in the regular OpenTimelineIO library, but with its own headers. Or are you saying you'd just wrap this subset of the library and not need bindings to OpenTimelineIO?

I would probably just put them under OpenTimelineIO, possibly combined with other operations into an 'algorithms' folder:

- opentimelineio/
         edit/
or 
         algo/

Things like flatten are likely peers of this and I'd like to avoid too many namespaces for people to hunt through.

In python, I was thinking a similar arrangement:
opentimelineio.algorithms.slip()
etc.

Curious to hear your thoughts.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I completely agree with this. I didn't want to make this my first move without larger consensus but this will work super well and clean up a lot of clunky code I was sketching out. As @reinecke suggested, it'll be worth pointing out that no new schemas should be allowed under algo/

```
OpenTimelineIO/
`- src/
`- opentime/
`- opentimelineio/
`- openedit/
`- ...
```

The benefit for the extra library is the edit suite will inevitably grow and will begin crowding the core model, which ultimately should be able to run without it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should assert here that opentime and opentimelineio encapsulate the stored data model and openedit implements operations on that data model but should not define any new schema objects.
This will hopefully create clarity around how their responsibilities are separate.


## Dependencies / Namespaces
The `openedit` library will have to communicate with both `opentime` and `openedit`.

This will warrant either a new namespace `namespace openedit { ... }` or using `opentimelineio` again. The latter makes some of the bellow comments less challenging but again, might pollute an otherwise clean namespace.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo 'below'


### ErrorStatus
This will present challenges when interfacing with the `ErrorStatus` of the different namespaces. That said, with a simple wrapper for this, we could have the instances auto convert as required.

### Retainer
Ideally, we continue to use the `Retainer<>` where possible to make sure editing is reference counted. Namespaces might make the names a bit clunky, seeing `SerializedObject::Retainer<>` all over:

```
namespace openedit { OPENEDIT_VERSION {

using namespace opentimelineio::OPENTIMELINEIO_VERSION;
auto item = SerializedObject::Retainer<Item>(...);

} }
```

So perhaps establishing some common aliases early might help?
```
using RetainedItem = SerializedObject::Retainer<Item>;
using RetainedComposition = SerializedObject::Retainer<Composition>;
...
```
Comment on lines +40 to +35

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RetainedItem seems like a reasonable alias, however, reading this feels like it should all be in the same namespace. The edit library is going to require both OpenTimelineIO and opentime to function, and you'll need constructors, classes, etc from those to setup objects and metadata-- it seems like this might be more analogous to the split in imath between for example ImathBox.h and ImathBoxAlgo.h:
https://github.com/AcademySoftwareFoundation/openexr/blob/master/IlmBase/Imath/ImathBox.h
https://github.com/AcademySoftwareFoundation/openexr/blob/master/IlmBase/Imath/ImathBoxAlgo.h


# Editing Commands
Let's go through some of the most common commands and what the python API might look like:

## Overwrite

![Overwrite](../_static/edit/01_overwrite.png)
- Anything overlapping is destroyed on contact. This may split an item.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to specify things more pedantically using the nomenclature we've established in the recent Allen's Interval Algebra commit. eg.

for all items i
   if e contains i, or i begins e, or e ends i : delete i
   if i contains e : split i at e.start, yielding {i0, i1}. Split i1 at en.end, yielding {i2, i3}, delete i, i1 and i2, return i0 and i3.
  if i overlaps e...
  else skip 

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've specified the rest of the API more formally. I felt, given our new algebraic nomenclature, overlapping wasn't specific enough, also destroyed on contact was open to a bit of interpretation..

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably also worth clarifying intent for the audiences of this. I'm expecting it will be consumed by:

  1. The implementer(s) of the API
  2. Future users of the API - TDs and DCC developers
  3. Future maintainers (not necessarily the people from No. 1)

I think the formal definitions will be key for the implementers and useful reference for maintainers. The human-friendly descriptions should be targeted at being approachable for API consumers and provide a framing of the mental model for future maintainers.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - I've got ASCII art and more explicit instructions per-command on the way (taking in the additional notes where possible). I'll add the audience information to the top along with the replaced "elevator pitch" that @ssteinbach recommended which sums it up beautifully 👍


#### API
```py
"""
Args:
item: Item that we're going to place onto the track
track: Track that this item will belong to afterwards
track_time: RationalTime of our track to put this item at
fill_template: Optional[Item] that will be cloned where required
to fill in the event that this overwrite extends the end of the
composition's limit
"""
otio.algorithms.overwrite(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may help code clarity if the function names are a bit more explicit like overwrite_in_track.

item: Item,
track: Track,
track_time: RationalTime,
fill_template: Item = None, # Default to Gap
)
```
- In the image, Clip `C` is `item` and `track` is the what `A` and `B` are on.

----

## Insert
![Insert](../_static/edit/02_insert.png)
Comment thread
meshula marked this conversation as resolved.
- Items past the in point are shifted by the new items duration.
- Item may be split if required.

#### API
```py
"""
Args:
<same as overwrite>
"""
otio.algorithms.insert(
item: Item,
track: Track,
composition_time: RationalTime,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be track_time to match overwrite?

fill_template: Item = None, # Default to Gap
)
```

---

## Trim
![Trim](../_static/edit/03_trim.png)
- Adjust a single item's start time or duration.
- Do not affect other clips.
- Fill now-empty time with gap or template.
- Clamps source_range to non-gap boundary (can only overwrite gaps)

Comment on lines +110 to +184

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if source range isn't set, does the available range become the source range before modifying the values?

This is assuming the item is in track, is it an error condition if the item has no parent?

If the item is directly in a stack, is a track inserted wrapping the item so that gap can be put before or after the item?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope that the common commands are as simple as possible - with explicitly defined implementation or course. Part of that would be, possibly, having it assume the source_range is the available_range if not set before. Implementing it as an error would totally work too but do we gain anything with that?

#### API
```py
"""
Args:
item: Item to apply trim to
delta_in: RationalTime that the item's source_range().start_time()
will be adjusted by
delta_out: RationalTime that the item's
source_range().end_time_exclusive() will be adjusted by
"""
otio.algorithms.trim(
item: Item,
delta_in: RationalTime = None, #< Duration of change
delta_out: RationalTime = None, #< Duration of change

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be useful to also document error case behavior (e.x. delta_in + delta_out > clip duration)?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - Error cases are something I definitely need to focus on more. I'll try to get more of the edge cases documented for my next push

fill_template: Item = None, #< Default to Gap
)
```
> Not convinced on the argument names for this

---

## Slice
![Slice](../_static/edit/04_slice.png)
- Slice an item, generating a clone of self and augment both item's source_range.

#### API
```py
"""
Args:
item: otio.schema.Item - to cut
at_time: otio.schema.RationalTime - time, based on the
coordinates to slice at
coordinates: Enumerator for the variable calculations that
can be done to the at_time.

Example:
Let item = A
let at_time = 25 @ 24fps

0 N
| ------------------------------------------- |
| [0 GAP 20][0 A 50] |
| ------------------------------------------- |

if coordinates == Local:
| ------------------------------------------- |
| [0 GAP 20][0 A 25][26 A 50] |
| ------------------------------------------- |

if coordinates == Parent:
| ------------------------------------------- |
| [0 GAP 20][0 A 5][6 A 50] |
| ------------------------------------------- |

if coordinates == Global:
- This only matters if the parent is a track in a compound
stack, at which point we might be better off having the
user just convert from one time to the other
"""
otio.algorithms.slice(
item: Item,
at_time: RationalTime,
coordinates: otio.algorithms.Coordinates.(Local|Parent|Global*)
)
```
> I'm not sure if we have a coordinate enumerator already?

---


## Slip
![Slip](../_static/edit/05_slip.png)
- Adjust the start_time of an item's source_range.
- Do not affect surrounding items.
- Clamp to available_range of media (if available)

#### API
```py
"""
Args:
item: Item that we're slipping
delta: RationalTime to adjust the range by.
"""
otio.algorithms.slip(
item: Item,
delta: RationalTime,
)
```
- In the example image, `C` is `item` and `delta` is the arrow's vector.

> Effectively this is already quite simple in the core API but would be nice to have here.
Comment on lines +187 to +348

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect the value of some of these functions might be that they hide the underlying complexity of whether or not source_range is set.

Maybe you split these functions into a "maybe" and "Definitely" pile? that will help prioritize them and see if people come out with concerns or use cases.

This might be a controversial position, but I think I'd also prefer to not have things named too close even if they do mean a specific thing (slip/slide for example)... unless that verb means the same thing in all major editing packages.


---


## Slide
![Slide](../_static/edit/06_slide.png)
- Adjust start time of item and trim adjacent items to fill
- Do not change main item's duration, only adjacent
- Clamp to available range of adjacent items (if available)

#### API
```py
"""
Args:
item: The item to push and pull around, which may in turn affect the
items around it.
delta: The delta that we're looking to push this item about.
"""
otio.algorithms.slide(
item: Item,
delta: RationalTime, # Relative like slide
)
```
- In the example image, `C` is `item` and `delta` is the arrow's vector. The edit command does the rest of the item management for us.

---

## Ripple
![Ripple](../_static/edit/07_ripple.png)
- Adjust a source_range without adjusting any other items
- This effectively shifts all currently adjacent items to stay at the edges
- No items _before_ the item are moved/affected
- (Impl detail for otio, this is the same as adjusting the source_range of the item)

#### API
```py
"""
Args:
item: The item to initiate the edit on
delta_in: see trim()
delta_out: see trim()
"""
otio.algorithms.ripple(
item: Item,
delta_in: RationalTime = None, #< Duration of change
delta_out: RationalTime = None, #< Duration of change
)
```
- In the example image, `A` is item and `delta_out` is a negative RationalTime. The edit command shifts `B` without adjusting it's source_range.

> Again, no crazy about the argument names

---


## Roll
![Roll](../_static/edit/09_roll.png)
- Any trim-like action results in adjacent items source_range being adjusted to fit
- No new items are ever created
- Clamped to available media (if available)

#### API
````py
"""
Args:
item: The item to initiate the edit on
delta_in: see trim()
delta_out: see trim()
"""
otio.algorithms.roll(
item: Item,
delta_in: RationalTime = None, #< Duration of change
delta_out: RationalTime = None, #< Duration of change
)
````
- In the example image, `A` is the item and `delta_out` is a negative RationalTime. The edit command augments `B` without changing it's `source_range().end_time_exclusive()`
- If `delta_in` is supplied, the item to the left of `A` would be modified (if any) - otherwise we may need to add a `Gap` / fill template.

---

## 3/4 Point Edit
![3_4 Point Edit](../_static/edit/08_pointedit.png)
- The most complex - this "fills" a gap based on a source in/out point _or_ track in/out point
- Often used to patch in items as edit is built
- Note: This can be accomplished by a conjunction of commands above

### API
```py
"""
Args:
item: The item to place onto the track
track: Track that will now own this item
replace: Item (or RT?) that we will effectively replace
reference_point: For 4 point editing, the reference point dictates what
transform to use when running the fill.

Options:
- Source : Don't modify the source, overwrite if required
- Sequence : Don't modify the sequence, trim item if required
- Fit : Apply Time Effect
"""
otio.algorithms.fill(
item: Item,
track: Track,
replace: Item = None, # Alternatively, we could have a parent-coord

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like this should maybe be a TimeRange within the track. If there is just a single item to operate on the caller can use range_in_parent to get the value they want to use.

# RationalTime to find the item for us
reference_point: otio.algorithms.ReferencePoint.Source
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one's a bit tricky!
My instinct is that this may be trying to do too much and could yield unexpected results for users. The reference_point being set to Source is effectively the same as just using overwrite, right?
Also there is some clarification needed about whether it's most important to keep the in point or out point of the inserted item.
I wonder if maybe this operation can be broken down into a few more explicit variants instead (e.x. overwrite at time range trimming item, overwrite at time range retiming item, etc.)

```
- This may still require some additional tinkering.


# Expanded Implementation

## Proposal
For editing commands, we should strive to do all validation and assert that everything "works" before committing anything on the timeline. A simple atomic command structure will provide that level of sophistication.

```cpp
class EditEvent {
public:
EditEventKind kind; // e.g. Insert, Append, Remove, Modify
Retainer<Composition> parent;
Retainer<Item> composable;

// ... Additional fields to execute the above as required

bool run(ErrorStatus* error_status);
bool revert();
};
```
An event is atomic in nature and does "one thing" to an `Item`. Each edit maneuver (e.g. `otio.algorithms.overwrite(...)`) would generate a vector of these events that can be played forward and, possibly, backward. The result of them collectively is the commands result.
Comment on lines +320 to +612

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets factor this out into a second PR. It is an orthogonal concept and also relevant to other non-edit api operations in OTIO, so we should design that as its own piece.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally see it being used more widely. I was hoping to build the common commands on top of this system. A command like overwrite will potentially do quite a few events - something like "sub-edits"

e.g.
- set A.duration = F
- set B.duration = B
- insert C into T' at index 3

If the general feeling is they shouldn't use it, then I'll write them as direct procedures.


```cpp
for (EditEvent &event: events) {
event.run(error_status);
if (*error_status) {
for (auto it = completed_events.rbegin(); /*...*/)
(*it).revert();
break;
}
completed_events.push_back(event);
}
```

## Overall
Many of the commands mentioned have common code paths and math that is required. We can streamline many of the placement commands into a single call with different options. We can then expose that as a raw edit platform while giving users the common algorithms for ease-of-use.
Comment thread
meshula marked this conversation as resolved.