Download pipeline¶
How segments get from googlevideo to disk. Three pieces in services/streaming/:
| Module | Role |
|---|---|
registry.py |
SegmentRegistry: the only place that knows whether a segment is missing, queued, downloading or done |
downloader.py |
DownloadManager (priority heap + worker threads) and fetch_segment() (one verified HTTP download) |
service.py |
glue: _run_download() with retries, ensure_segment() for request handlers |
The segment registry¶
One lock, one entry per (song_id, index) key (index -1 = init segment):
stateDiagram-v2
[*] --> MISSING
MISSING --> QUEUED: request()
ERROR --> QUEUED: request()
QUEUED --> QUEUED: request() with newer epoch<br/>or better priority (re-rank)
QUEUED --> MISSING: unqueue() (other song started, stop)
QUEUED --> IN_FLIGHT: claim()
MISSING --> IN_FLIGHT: claim() by a request handler
IN_FLIGHT --> DONE: complete(ok)
IN_FLIGHT --> ERROR: complete(error)
DONE --> MISSING: evicted()
| Method | Behaviour |
|---|---|
request(key, priority, epoch) |
no-op if DONE or IN_FLIGHT. Re-ranks a QUEUED entry if the request is from a newer epoch or has a better priority. Queues MISSING / ERROR. Returns whether a heap item must be pushed |
claim(key, priority, epoch) |
workers: succeeds only if their heap item is still the current one for that key. Request handlers (no priority): claim anything not DONE / IN_FLIGHT, jumping the queue |
complete(key, ok, error) |
IN_FLIGHT → DONE / ERROR, sets a threading.Event that wakes waiters |
wait(key, timeout) |
block on that event |
snapshot(song_id) |
states for /chunks, from memory: no filesystem calls |
/chunks reports pending (MISSING, QUEUED), downloading (IN_FLIGHT), downloaded (DONE),
error.
Priorities and the heap¶
class Priority(IntEnum):
CRITICAL = 0 # init + the segment at the playhead
HIGH = 1 # the next 2 segments
NORMAL = 2 # the rest of the ~30 s window
LOW = 3 # prefetch of the next song
The heap is ordered by (priority, newer epoch first, request order). It only holds hints:
when a segment is re-ranked, a new item is pushed and the old one stays in the heap. Workers pop
items and call claim() with the item's (priority, epoch); a stale item fails the claim and is
skipped. No item is ever removed from the middle of the heap.
The epoch is a per-song counter bumped on every seek. Requests made after a seek outrank everything queued before it. It's a priority, not a cancellation: a segment already downloading finishes, because its bytes are still valid.
fetch_segment(): one verified download¶
The robustness-critical path, in downloader.py:
GET urlwithRange: bytes=start-endand the CDN headers (theCookiewhen signed in).- Check the response:
- status must be 206: a 200 means the range was ignored, and is never saved;
Content-Rangemust start withbytes start-end/;- the byte count must equal
end − start + 1.
- Write to a unique temp file
tempfile.mkstemp(dir=song_dir, prefix="NNNN.", suffix=".tmp"),fsync, thenatomic_replace()(os.replace, retried briefly on WindowsPermissionError; if an identical-size file already won the race, ours is dropped).
A crash mid-download can't leave a partial file that looks complete, and two writers never share a temp path.
| Response | Classified as | Handling |
|---|---|---|
| 403 | _Auth403Error |
the signed URL expired: refresh it once and retry |
| 429, 5xx, timeout, short read | _TransientError / other |
back off 0.5 / 1 / 2 s (+ up to 0.3 s jitter), up to MAX_RETRIES attempts |
| 200, other 4xx, Content-Range mismatch | _PermanentError |
ERROR, no retry |
403 refresh, in detail
_rebuild_after_403(song_id, stale_url) runs in the stream-info SingleFlight. If the
cached table's URL already differs from stale_url, another worker has rebuilt it and that
table is reused. Otherwise it calls StreamExtractor.refresh() and rebuilds the segment
table; the layout may change, in which case the cache identity
check purges the old segments. A CDN 403 doesn't touch the sign-in state: it means the
URL expired, not the YouTube session, so the refresh still uses your cookies.
After a successful download: the DB row and the cache size map are updated
(CacheStore.record_segment), the key is completed, and CacheStore.maybe_evict() runs (O(1)
while under the limit). If the song's layout_id changed while the bytes were in flight, the
file is deleted and the key marked ERROR.
Workers¶
DownloadManager.start() launches DOWNLOAD_WORKERS (4) threads. Each loops:
task = queue.get(timeout=1)
if not registry.claim((task.song_id, task.index), task.priority, task.epoch):
continue # superseded, claimed by a handler, or unqueued
handler(task) # StreamingService._run_download → always completes the key
stop() pushes one sentinel per worker and shuts the pool down.
ensure_segment(): request handlers¶
The audio endpoint needs a segment now. StreamingService.ensure_segment(song_id, index):
| Registry state | Action |
|---|---|
| DONE (file present) | bump its LRU time, return the path |
| DONE but file gone | mark MISSING, continue below |
| IN_FLIGHT (another thread) | wait on its event, up to SEGMENT_SYNC_TIMEOUT_S (15 s) |
| QUEUED / MISSING / ERROR | claim it and download in this thread; the queued heap item is skipped later |
On failure or timeout it raises SegmentUnavailable. Nothing polls: waiting is always on the
registry's event. Only the thread holding the claim downloads, so a worker and a handler never
fetch the same segment at the same time.
Who requests what¶
| Trigger | Requests |
|---|---|
/play |
buffer_plan from segment 0 at the current epoch; other songs' queued segments are unqueued |
| position report (every 2 s) | buffer_plan from the playhead segment, only when the playhead entered a new segment |
/seek |
bump epoch, buffer_plan from the target segment |
/resume |
buffer_plan from the current segment |
/buffer?mode=prefetch |
init + first 2 segments of another song at LOW, session untouched |
/stop |
unqueue the song's queued segments (in-flight ones finish) |
/audio |
ensure_segment for each segment the Range covers |
buffer_plan(info, start_index, window_ms) (playback.py) returns init + the playhead segment as
CRITICAL, the next two as HIGH, then NORMAL until the window is covered, measured from the
playhead, not the song start.