build and reconcile a cut

A Cut is the edit; a CutItem is one clip in it. Cut is addressed at /entity/cuts and CutItem at /entity/cut_items, both project-scoped, both requiring only project on create and both generating code as New Cut <id> / New Cut Item <id> when it is omitted. Neither has an entity-type card yet. On the probed site both types hold zero rows, so everything below was measured on rows this probe wrote.

The write order

Four stages, because each one needs ids from the one before and a batch cannot reference an id it creates (recipes/002).

In a batch, entity is the singular schema name, and the two other spellings 400: "cut_item" gives Invalid entity type: entity type [cut_item] does not exist. and "CutItems" the same message with [CutItems]. A CutItem create needs project even when cut is sent: 400 API create() missing 'project' attribute: {"code" => "...", "cut" => {"type" => "Cut", "id" => 19}}.

The fields that place an item on the timeline

Six numbers and four strings, on two axes: where the clip sits in the cut, and where it was taken from in the source. The server fills none of them and relates none of them to each other.

Neither type has a field of data type timecode; the four timecode_*_text fields are text and validate nothing, so the millisecond integer of field_types/timecode is not in play here. The frame rate is Cut.fps, a float that reads back as the string "24.0" and is null when unset. A client therefore needs three things of its own beside the frame numbers: the rate, whether the timecode is drop frame, and the cut's start timecode. None of the three is derivable from a CutItem.

Call

import json
import sys

sys.path.insert(0, "src")                       # or PYTHONPATH=src
from sg_groundtruth.client import FPT           # adds the bearer token and the /api/v1 prefix
from sg_groundtruth.env import load

c = FPT.from_env(load("."))                     # FPT_API_SITE_URL, FPT_API_SCRIPT_NAME, FPT_API_API_KEY
JSON = {"Content-Type": "application/json"}
ARR = {"Content-Type": "application/vnd+shotgun.api3_array+json"}   # array filters need it (probe 004)

PROJECT_ID = 1180                               # the caller supplies these four
CUT_CODE, FPS, START = "reel1", 24.0, "01:00:00:00"

# One tuple per clip, in order: code, first and last frame on the cut timeline, first and last frame
# in the source media. "last frame" is this client's convention. The server stores the six numbers
# and relates none of them to each other.
EDIT_A = [("reel1_sh010", 0, 47, 86400, 86447),
          ("reel1_sh020", 48, 119, 12000, 12071),
          ("reel1_sh030", 120, 167, 400, 447)]
EDIT_B = [("reel1_sh010", 0, 47, 86400, 86447),      # unchanged
          ("reel1_sh020", 48, 95, 12000, 12047),     # retimed, 24 frames shorter
          ("reel1_sh040", 108, 155, 900, 947)]       # new, after 12 frames of black
MADE = []                                           # (slug, id) of every row this run created


def call(method, path, **kw):
    r = c.request(method, path, **kw)
    if not r.ok:
        raise SystemExit(json.dumps(r.json()["errors"], indent=2))
    return r.json() if r.content else {}


def batch(reqs):
    """One atomic call, one result row per request, in request order (recipe 002). Chunk at ~200."""
    return [row.get("data", row)["id"]
            for row in call("POST", "/entity/_batch", json={"requests": reqs})["data"]]


def search(slug, filters, fields, **kw):
    """A cut of 500 items needs paging: page until `data` is empty, never on a missing
    `links.next` (probe 006)."""
    return call("POST", f"/entity/{slug}/_search", headers=ARR,
                json={"filters": filters, "fields": fields, "page": {"size": 500}, **kw})["data"]


def tc(frame, fps, start="00:00:00:00"):
    """Frames to HH:MM:SS:FF, non drop frame. No CutItem field states the rate, so this is the
    client's own arithmetic over a rate it stored on Cut.fps."""
    r = int(round(fps))
    h, m, s, f = (int(x) for x in start.split(":"))
    t = frame + f + r * (s + 60 * m + 3600 * h)
    return "%02d:%02d:%02d:%02d" % (t // (3600 * r), t // (60 * r) % 60, t // r % 60, t % r)


def pairs(rows, code_of):
    """Nothing on CutItem is unique, and cut_order is exactly what a recut changes, so items pair
    on the clip name plus its occurrence, scoped to one Cut."""
    seen, out = {}, {}
    for row in rows:
        code = code_of(row)
        seen[code] = seen.get(code, 0) + 1
        out[(code, seen[code])] = row
    return out


IN_PROJECT = ["project", "is", {"type": "Project", "id": PROJECT_ID}]

# 1. The Cut. Only `project` is required, `code` is not unique, and `revision_number` is a plain
#    number the client sets, so "the current cut" is a sort, not a lookup.
cuts = search("cuts", [IN_PROJECT, ["code", "is", CUT_CODE]], ["code", "revision_number", "fps"],
              sort="-revision_number")
if cuts:
    CUT_ID = cuts[0]["id"]
    FPS = float(cuts[0]["attributes"]["fps"])   # a float reads back as a string (field_types/float)
else:
    CUT_ID = call("POST", "/entity/cuts", headers=JSON, json={
        "project": {"type": "Project", "id": PROJECT_ID}, "code": CUT_CODE, "revision_number": 1,
        "fps": FPS, "timecode_start_text": START})["data"]["id"]
    MADE.append(("cuts", CUT_ID))


def sync(edit):
    # 2. Shots, one per distinct clip name. A Shot code is not unique, so read before creating or a
    #    re-run doubles them (entity_types/Shot).
    codes = sorted({e[0] for e in edit})
    shots = {d["attributes"]["code"]: d["id"]
             for d in search("shots", [IN_PROJECT, ["code", "in", codes]], ["code"])}
    todo = [x for x in codes if x not in shots]
    new = batch([{"request_type": "create", "entity": "Shot",
                  "data": {"project": {"type": "Project", "id": PROJECT_ID}, "code": x}}
                 for x in todo]) if todo else []
    MADE.extend(("shots", i) for i in new)
    shots.update(zip(todo, new))

    # 3. Versions. Each one needs step 2's Shot id for `entity`, and a batch cannot reference an id
    #    it creates (recipe 002), so this is a second call and not the same one.
    by_code = {d["attributes"]["code"]: d["id"]
               for d in search("versions", [IN_PROJECT, ["code", "in", [f"{x}_v001" for x in codes]]],
                               ["code"])}
    vers = {x: by_code[f"{x}_v001"] for x in codes if f"{x}_v001" in by_code}
    todo = [x for x in codes if x not in vers]
    new = batch([{"request_type": "create", "entity": "Version",
                  "data": {"project": {"type": "Project", "id": PROJECT_ID}, "code": f"{x}_v001",
                           "entity": {"type": "Shot", "id": shots[x]}}} for x in todo]) if todo else []
    MADE.extend(("versions", i) for i in new)
    vers.update(zip(todo, new))

    def data(order, e):
        code, edit_in, edit_out, src_in, src_out = e
        return {"project": {"type": "Project", "id": PROJECT_ID}, "code": code,
                "cut": {"type": "Cut", "id": CUT_ID},
                "shot": {"type": "Shot", "id": shots[code]},
                "version": {"type": "Version", "id": vers[code]},
                "cut_order": order,
                "edit_in": edit_in, "edit_out": edit_out,
                "cut_item_in": src_in, "cut_item_out": src_out,
                "cut_item_duration": edit_out - edit_in + 1,
                "timecode_edit_in_text": tc(edit_in, FPS, START),
                "timecode_edit_out_text": tc(edit_out + 1, FPS, START),
                "timecode_cut_item_in_text": tc(src_in, FPS),
                "timecode_cut_item_out_text": tc(src_out + 1, FPS)}

    # 4. Pair the edit against what this Cut already holds. Reading the rows filtered on `cut` is
    #    the linked-Cut check: an id from anywhere else, a cached ingest map or a search on `code`,
    #    can sit on another Cut, and updating it corrupts that Cut instead of this one. Re-checking
    #    `cut.Cut.id` costs nothing here and is the guard to keep when the ids come from elsewhere.
    have = pairs(search("cut_items", [["cut", "is", {"type": "Cut", "id": CUT_ID}]],
                        ["code", "cut.Cut.id"], sort="cut_order"),
                 lambda d: d["attributes"]["code"])
    have = {k: d for k, d in have.items() if d["attributes"]["cut.Cut.id"] == CUT_ID}
    want = pairs(list(enumerate(edit, 1)), lambda p: p[1][0])

    # 5. CutItems last: each needs the Cut, the Shot and the Version. Update, create and delete go
    #    in one batch, which applies whole or not at all.
    reqs = [{"request_type": "update", "entity": "CutItem", "record_id": have[k]["id"],
             "data": data(*want[k])} for k in want if k in have]
    reqs += [{"request_type": "create", "entity": "CutItem", "data": data(*want[k])}
             for k in want if k not in have]
    reqs += [{"request_type": "delete", "entity": "CutItem", "record_id": have[k]["id"]}
             for k in have if k not in want]
    out = batch(reqs)
    MADE.extend(("cut_items", i) for r, i in zip(reqs, out) if r["request_type"] == "create")
    for r, i in zip(reqs, out):
        if r["request_type"] == "delete":
            MADE.remove(("cut_items", i))
    return [f"{r['request_type']} {i}" for r, i in zip(reqs, out)]


# 6. Read the cut back. The order is `cut_order` and nothing else: Cut.cut_items is returned sorted
#    by the item's display name. A gap is not stored, and neither is an overlap: both are the
#    difference between one item's edit_out and the next item's edit_in.
def timeline():
    items = search("cut_items", [["cut", "is", {"type": "Cut", "id": CUT_ID}]],
                   ["code", "cut_order", "edit_in", "edit_out", "cut_item_duration",
                    "timecode_edit_in_text"], sort="cut_order")
    out = []
    for prev, this in zip([None] + items, items):
        a = this["attributes"]
        d = None if prev is None else a["edit_in"] - (prev["attributes"]["edit_out"] + 1)
        out.append(f"{a['cut_order']} {a['code']} {a['edit_in']}-{a['edit_out']} "
                   f"dur {a['cut_item_duration']} {a['timecode_edit_in_text']} "
                   + ("" if not d else f"GAP {d}" if d > 0 else f"OVERLAP {-d}"))
    return out


print("cut", CUT_ID, "| A:", sync(EDIT_A))
print("\n".join("  " + x for x in timeline()))
print("B:", sync(EDIT_B))
print("\n".join("  " + x for x in timeline()))

# 7. Teardown, deleting only what this run made, CutItems before their Cut: deleting a Cut leaves
#    its CutItems behind with `cut` null, findable only with [["cut", "is", None]].
for slug, i in reversed(MADE):
    call("DELETE", f"/entity/{slug}/{i}")

Response

cut 22 | A: ['create 59', 'create 60', 'create 61']
  1 reel1_sh010 0-47 dur 48 01:00:00:00
  2 reel1_sh020 48-119 dur 72 01:00:02:00
  3 reel1_sh030 120-167 dur 48 01:00:05:00
B: ['update 59', 'update 60', 'create 62', 'delete 61']
  1 reel1_sh010 0-47 dur 48 01:00:00:00
  2 reel1_sh020 48-95 dur 48 01:00:02:00
  3 reel1_sh040 108-155 dur 48 01:00:04:12 GAP 12

The reconcile is one batch of 4 requests answering [59, 60, 62, 61], in request order: two updates, then the create, then the delete.

A separate run that prints each stage, with the four clip names created up front:

1. POST /entity/cuts -> 201 id=19 code='reel1' cached_display_name='reel1 v001' fps='24.0'
2. batch create Shot x4    -> {'sh010': 7576, 'sh020': 7577, 'sh030': 7578, 'sh040': 7579}
3. batch create Version x4 -> {'sh010': 31681, 'sh020': 31682, 'sh030': 31683, 'sh040': 31684}
4. batch create CutItem x3 -> [46, 47, 48]

Gaps and overlaps

Nothing between two items is stored. A boundary is next.edit_in - (this.edit_out + 1) under the convention that edit_out is the last frame; a client that treats edit_out as one past the end drops the + 1. Six items in one Cut, read back sorted on cut_order:

Every anomaly writes at 200 and reads back exactly as sent, so a client that does not test for one never learns of it:

The frame pair and the timecode pair are two independent stores of one boundary. Write both from one source of truth in the same call and prefer the frames on read: they are integers the API type-checks (field_types/number), while the timecode strings accept anything. When only the strings are populated, parse them with Cut.fps, and treat a disagreement as corrupt data rather than picking a winner.

Pairing an item across two edits

Matched key means update, unmatched wanted key means create, unmatched existing key means delete. What counts as "the same clip retimed" versus "a different clip" beyond that is the caller's policy; the API offers code, shot, version and the six numbers to decide it on, and enforces none of them.

Notes

• The four stages exist because a batch cannot use an id it creates (recipes/002). A CutItem needs a Cut id, a Shot id and a Version id at once, and a Version needs a Shot id, so the graph is four levels deep and each level is its own call. Within a level, batch, and chunk at around 200 requests.
• An id alone does not say which Cut a row is on. code repeats across Cuts: [["code", "is", "reel1_sh010"]] returned items (46, cut 19) and (53, cut 20). A blind PUT /entity/cut_items/53 with no cut key answered 200, left cut at 20, and overwrote that Cut's metadata.
• Before updating, confirm the row's Cut: filter on cut when reading, or ask for cut.Cut.id in fields and drop every id that does not match. A dotted read through this single entity field works, unlike one through a multi_entity field (probe 016).
• Sending cut in an update moves the item. PUT with {"cut": {"type": "Cut", "id": other}} answers 200 and the item leaves its old Cut. So does the other side: PUT /entity/cuts/<a> with {"cut_items": {"multi_entity_update_mode": "add", "value": [...]}} answered 200 and left the item's former Cut holding [].
• CutItem.cut is single-valued, so an add on the parent is a re-parent, not an addition.
• Cut.cut_items is not the running order. It is returned sorted by the item's display name: ['aaa_last', 'sh010', 'sh020', 'sh030', 'sh030_gap', 'sh030_overlap'] against cut_order 1, 2, 3, 4, 5, 6 on the same six rows.
• Read the items with POST /entity/cut_items/_search, [["cut", "is", {"type": "Cut", "id": N}]], sort: "cut_order". A null cut_order sorts last in both directions.
• No frame rate is reachable from a CutItem. Cut.fps is the only rate on either type, it is null until someone writes it, and no CutItem field points at the Cut's value. Read Cut.fps once and pass it down; float reads back as a string, so float() it (field_types/float).
• Drop frame is expressible only inside the text fields, which validate nothing, so the client owns that flag too.
• Deleting a Cut does not delete its CutItems. DELETE /entity/cuts/<id> answered 204 and the item survived with cut null, reachable only through [["project", "is", ...], ["cut", "is", None]]. Delete the items first. A delete inside a batch is not idempotent: a second delete of the same id 404s and takes the whole batch with it (recipes/002).
• Nothing about a Cut is unique either. Three Cuts created with the same code all answered 201. revision_number is a plain number the client maintains, and the display name the server builds from it is code plus v%03d: reel1 v001, reel1 v002, and bare reel1 when revision_number is null. "The current cut" is sort: "-revision_number" over a code filter.
• Cut.entity accepts ['Sequence', 'Scene', 'Episode', 'Reel'] and Cut.version a Version, whose reverse Version.cuts fills in on the same write.