twitch-batch-downloader Code
Automate the download of entire Twitch.tv channels
Brought to you by:
dragomerlin
Menu
▾
▴
Tree [ed6d9d] master / History
Read Me
## 0. TABLE OF CONTENTS
- [1. DESCRIPTION](#1-description)
- [1.1. State of things](#11-state-of-things)
- [1.2. Required software](#12-required-software)
- [1.3. Characters and portability](#13-characters-and-portability)
- [1.3.1. Example 1](#131-example-1)
- [1.3.2. Example 2](#132-example-2)
- [1.3.3. Example 3](#133-example-3)
- [1.3.4. Conclusion](#134-conclusion)
- [2. MAIN SCRIPTS](#2-main-scripts)
- [2.1. check-requirements.sh](#21-check-requirementssh)
- [2.2. dump-urls-from-html-lynx.sh](#22-dump-urls-from-html-lynxsh)
- [2.2.1. Notes about the html file](#221-notes-about-the-html-file)
- [2.2.2. Structure of `csv-file`](#222-structure-of-csv-file)
- [2.3. download-twitch-video.sh](#23-download-twitch-videosh)
- [2.3.1. Notes about live streams and past broadcasts](#231-notes-about-live-streams-and-past-broadcasts)
- [2.3.2. Reading the technical makeup of multimedia files](#232-reading-the-technical-makeup-of-multimedia-files)
- [2.3.3. Notes on using `ls` to sort files by date/time](#233-notes-on-using-ls-to-sort-files-by-date-time)
- [2.3.4. Notes on streams with initialization segments](#234-notes-on-streams-with-initialization-segments)
- [2.3.5. Codecs and formats in each type of stream](#235-codecs-and-formats-in-each-type-of-stream)
- [2.3.6. Notes on concatenating noncontiguous segments](#236-notes-on-concatenating-noncontiguous-segments)
- [2.4. download-all.sh](#24-download-allsh)
- [2.5. convert-all.sh](#25-convert-allsh)
- [2.6. check-all.sh](#26-check-allsh)
- [2.7. recode-defective-all.sh](#27-recode-defective-allsh)
- [2.8. combine-multiple-ids.sh](#28-combine-multiple-idssh)
- [2.9. overlay-chat.sh](#29-overlay-chatsh)
- [2.10. monitor-ffmpeg-progress.sh](#210-monitor-ffmpeg-progresssh)
- [2.11. detect-ffmpeg-encoder.sh](#211-detect-ffmpeg-encoder-sh)
- [3. HELPER SCRIPTS](#3-helper-scripts)
- [3.1. restore-segments.sh](#31-restore-segmentssh)
- [3.2. check-titles-online.sh](#32-check-titles-onlinesh)
- [3.2.1. Notes about TwitchTracker](#321-notes-about-twitchtracker)
- [3.3. rename-all-directories-dateid.sh](#33-rename-all-directories-dateidsh)
- [3.4. update-dates-in-csv.sh](#34-update-dates-in-csvsh)
- [3.5. extract-ids-thumbs.py](#35-extract-ids-thumbspy)
- [3.5.1. Externally managed environments](#351-externally-managed-environments)
- [3.6. extract-thumbnail-from-id.sh](#36-extract-thumbnail-from-idsh)
- [3.6.1. Notes about thumbnails resolutions](#361-notes-about-thumbnails-resolutions)
- [3.6.2. Notes about local or remote thumb URLs in plain HTML](#362-notes-about-local-or-remote-thumb-urls-in-plain-html)
- [3.6.3. Notes about thumbnail types and names](#363-notes-about-thumbnail-types-and-names)
- [3.7. update-thumbs-from-html.sh](#37-update-thumbs-from-htmlsh)
- [3.8. update-thumbs-from-json.sh](#38-update-thumbs-from-jsonsh)
- [3.8.1. Notes about thumbnails](#381-notes-about-thumbnails)
- [3.9. test-with-ffmpeg.sh](#39-test-with-ffmpegsh)
- [3.9.1. Samples](#391-samples)
- [3.9.1.1. sample1.ts](#3911-sample1ts)
- [3.9.1.2. sample2.ts](#3912-sample2ts)
- [3.9.1.3. sample3.ts](#3913-sample3ts)
- [3.9.1.4. sample4.ts](#3914-sample4ts)
- [3.9.1.5. sample5.ts](#3915-sample5ts)
- [3.9.1.6. sample6.mp4](#3916-sample6mp4)
- [3.9.1.7. sample7.mp4](#3917-sample7mp4)
- [3.9.1.8. sample8.mp4](#3918-sample8mp4)
- [3.9.1.9. sample9.mp4](#3919-sample9mp4)
- [3.9.2. Conversion statistics](#392-conversion-statistics)
- [3.10. chat_json_timeline_comparison.py](#310-chat-json-timeline-comparison-py)
- [4. OTHER SOFTWARE AND COMMANDS](#4-other-software-and-commands)
- [4.1. SOFTWARE / PROJECTS](#41-software--projects)
- [4.1.1. twitch-dlp-2](#411-twitch-dlp-2)
- [4.1.2. livestream_scripts](#412-livestream_scripts)
- [4.1.3. Download only a section or portion of a VOD](#413-download-only-a-section-or-portion-of-a-vod)
- [4.1.3.1. twitch-dl](#4131-twitch-dl)
- [4.1.3.2. TwitchDownloader](#4132-twitchdownloader)
- [4.1.3.3. twitch-dlp](#4133-twitch-dlp)
- [4.2. USEFUL COMMANDS](#42-useful-commands)
- [4.2.1. Download all playlists in all qualities](#421-download-all-playlists-in-all-qualities)
- [4.2.2. Sort script by exit code](#422-sort-script-by-exit-code)
- [4.2.3. Regenerate thumbnail from video](#423-regenerate-thumbnail-from-video)
- [4.2.4. Subshells, pipes and exit codes](#424-subshells-pipes-and-exit-codes)
- [4.2.5. Render subtitles in video](#425-render-subtitles-in-video)
- [4.2.6. Render Twitch.tv chat in video and concatenate several IDs](#426-twitchtv-chat-in-video-and-concatenate-several-ids)
- [4.2.7. Get information about Twitch chat JSONs](#427-get-information-about-twitch-chat-jsons)
- [5. TWITCH](#5-twitch)
- [5.1. Twitch VOD/ID structure](#51-twitch-vod-id-structure)
- [5.2. Transport Stream files](#52-transport-stream-files)
- [5.3. Highlight creation](#53-highlight-creation)
- [5.4. Defective streams](#54-defective-streams)
- [5.5. Retirement of the :BibleThumb: emote](#55-retirement-of-the-biblethumb-emote)
- [5.6. Obtaining the Twitch user OAUTH (auth-token)](#56-obtaining-the-twitch-user-oauth-auth-token)
- [5.7. Out of the ordinary VODs or streams](#57-Out-of-the-ordinary-vods-or-streams)
- [5.7.1. VODs where the type or number of tracks in some streams do not match the declared values](#571-vods-where-the-type-or-number-of-tracks-in-some-streams-do-not-match-the-declared-values)
- [5.7.2. VODs with several streams with the same resolution and/or framerate](#572-vods-with-several-streams-with-the-same-resolution-and-or-framerate)
- [5.7.3. VODs with BANDWIDTH=0 for all streams](573-vods-with-bandwidth-0-for-all-streams)
- [5.7.4. VODs with placeholders for quality and URL][574-vods-with-placeholders-for-quality-and-url]
- [5.7.5. VODs with resolution and framerate missing for most streams](575-vods-with-resolution-and-framerate-missing-for-most-streams)
- [5.7.6. VODs with and without html encoded entities in the jsons](576-vods-with-and-without-html-encoded-entities-in-the-jsons)
- [5.8. Geoblocked and OAuth-gated content by Twitch and not the Streamer](#58-geoblocked-and-oauth-gated-content-by-twitch-and-not-the-streamer)
- [5.8.1. Entire geoblocked VODs](#581-entire-geoblocked-vods)
- [5.8.2. Renditions being geo-blocked and auth-walled](#582-renditions-being-geo-blocked-and-auth-walled)
- [5.8.3. Summary table about geoblocking and content fencing](#583-summary-table-about-geoblocking-and-content-fencing)
## 1. DESCRIPTION
The `twitch-batch-downloader` project is a set of POSIX shell scripts aimed to automate the download of all the videos available in one or more Twitch channels. Sets of IDs can alse be given. Each script does a separate thing to achieve this. The scripts require external commands, the shell alone is not enough.
Additional documentation is in the [docs/](https://sourceforge.net/projects/twitch-batch-downloader/files/docs/) directory.
### 1.1. State of things
There are already programs to download Twitch videos, but each one has its caveats:
- Download only one video at a time
- Don't allow to keep all the .ts/.mp4 segments, or the original format of the stream
- Software outdated
- Software with only graphical interface, not useful for batch processing
- Software only available for certain desktop or mobile operating systems or specific hardware
- Software based on web services which can expire anytime, or don't offer advanced features
- Software using high level programming languages, requiring complicated dependencies, setup and space requirements
- Difficult to manage the files once they are downloaded
Since the tracking of videos is done by its Twitch video ID (which is a number except for Clips), it's possible to have several Twitch channels combined in the current working directory where the scripts are, being transparent to the user. Each channel can be stored in a separated csv file or they can be mixed in any combination.
### 1.2. Required software
The only special software required is [TwitchDownloader](https://github.com/lay295/TwitchDownloader), used only to download and render the chat files. The rest of the scripts are already included in this project.
The list of all the required commands to run any script from this project is checked by `check-requirements.sh`. All scripts are designed to be POSIX-compliant, any shell should be able to run them, unless they are stripped down versions designed for mobile or embedded devices. Tested shells are: `bash`, `zsh`, `ksh`, `dash`. Other shells native to BSD variants are not tested, because some of the required software is not available in those platforms. All variants of the commands (UNIX/BSD/APPLE/GNU), like awk, sed, etc. should work too, if all the options and arguments are properly formatted to be cross-compatible. Some commands like `date` are so different, that the command has to be crafted differently depending on the installed version. Other commands like `echo` have loose implementations, `printf` is more predictable.
### 1.3. Characters and portability
Use only alphanumeric characters for all the names of files and directories, and also for all the arguments given to commands, whenever possible, unless required otherwise. Ideally, treat them as case-insensitive, that is, if a file or directory exists, don't try to read it or create another just the same but changing the case of any letter(s). For numbers doesn't apply, but that's just a coincidence that they don't exist upper/lower-case. Alphanumeric characters are the following, sorted in ascending order according to their ASCII value: `0123456789`, `ABCDEFGHIJKLMNOPQRSTUVWXYZ` and `abcdefghijklmnopqrstuvwxyz`. The reasons to use these only are many, but the most important:
1. Case sensitivity:
-`Microsoft Windows` is case-insensitive by default, due to the API, although certain builds allow to change it per-directory basis, using `fsutil.exe`. The filesystem `NTFS` is case-sensitive however, so if it's only going to be used with Linux/UNIX based systems, can be treated as such all the time.
- `macOS` also allows its filesystems `HFS+` and `APFS` to be case-sensitive or insensitive.
- Linux's `Ext4` can be configured to be case-insensitive (but only the metadata), and it's not a common setting. Everybody expects BSD/Linux filesystems to be case-sensitive all the time.
3. Special filenames
- Linux filenames (and directories) can contain any character except forward slash `/` (0x2F) and NULL byte (0x00). One is used to separate directories and the other as file/directory name terminator in the filesystem lookup table.
- `\/:*?<>|`: None of these characters is allowed by Windows, although the NTFS filesystem technically could store them in Unicode. Linux allows most of them but should not be used.
- Linux and Windows cannot contain inside a directory, a file and a folder/directory with the same name, unless it's case-sensitive: `touch file && mkdir FiLe`.
- Windows does not allow to create files or folders with these exact names (and case doesn't matter even if the folder was set as case-sensitive): `CON`, `PRN`, `AUX`, `NUL`, `COM1` to `COM9`, `LPT1` to `LPT9`. These names are still allowed by NTFS, so with Linux can be created in any combination.
3. ASCII characters used as semi-control characters:
POSIX-compliant shells, and other command-line interfaces, like `CMD`, `PowerShell` or `Fish`, or even `Python` or `Perl`, use certain characters as "semi-control" characters, that is, not always pure data but still printable. For example: `$#+%&{}^*\"; |>'<!~?`, but there are others. The special case is the forward slash `/`, which is neither a control character, nor pure data (can't be part of a filename), but printable. The semi-control characters are used for shell expansion, wildcards, command execution, command grouping or separation or substitution, process handling, pipe and redirection, arithmetic operations, etc. So whenever the shell or any external command encounters such a character, it doesn't know whether to treat it by face value, or execute the operation it can represent. Keeping it literal would require to escape it for each situation. That and in combination with expansions and substitutions, makes difficult to just use any special character in file/directory names. May be technically possible.
Some characters in file/directory names:
- Underscore `_`: is the most safe character to use for anything after the alphanumeric characters, even alone.
- Comma `,`: it's safe to use in Windows and Linux in any combination, even just one alone, and doesn't need to be escaped. It can't be used in URLs however, needs to be URL encoded as "%2C".
- Dot `.`: Windows does not allow to create any file with only any number of dots, or trailing dots. Linux allows 3 or more like `...`, and with other characters in any combination. In Linux if a dot is leading another dots or characters means hidden file. In Windows the hidden feature is a flag in the filesystem not the file itself.
- Round, square and curly brackets `()[]{}`: They can be used in any combination in Windows and Linux, and people usually use them to organize files by categories. In Linux they need to be escaped. In Windows it depends.
- Unicode characters for languages and emojis are allowed in modern operating systems and filesystems, and they are not mistaken with control or semi-control characters. They don't need to be escaped: `echo '$content' > 文件🏉`.
- Space ` `: Linux allows any number of spaces in any combination, even just one: `touch ' '`. Windows does not allow only spaces, and trims those at both ends of file/directory names. NTFS allows any of that however. Spaces should not be used even in the middle of strings, because several commands or shells use spaces and tabs as fields separators instead of newlines by default. Special attention is required to change the `IFS` every time.
- Hyphen `-`: this is the most controversial and misunderstood character, specially because commands use it directly more than anything else, as a semi-control character even for the most basic tasks. Most commands (if not all), don't have a fixed position for arguments, not even a fixed position for the parameter after a valid option that requires it. This causes the programs to try to interpret everything as everything. They usually try to parse arguments starting with hyphens as options, when not always has to be the case. Some programs have the flag `--` to stop interpreting the rest of the arguments after that as options, so they are used only as parameters. For paths starting with hyphens, they can be prevented from being interpreted as options converting them to absolute paths, either against the current directory (with `./`) or against the root (`/`).
A command is usually structured this way (everything is an argument, it's the top level and the number is the index):
```
command0 arg1 `arg2` "arg3" "$VAR arg 4" '--arg 5' '-arg6' arg\ 7 "$(arg8)" etc.
```
or this way:
```
command [-/--flag(s)] [-/--option [value(s)/parameter(s)]] [argument(s)]
```
Each argument can be optional or not, depending on the program, and its index position can also be fixed or not, also depending on the program, and some options can require value(s)/parameter(s) or not. Most of the programs don't have most of those configurations fixed, so when calling the program, it spends half of the time figuring out which is which. The more flexible the command format can be, the less flexible the user can, so the arguments must be provided in a determinist way. This is where paths, parameters or values, can be mistaken by options or flags.
###### 1.3.1. Example 1:
```
$ TwitchDownloaderCLI2 videodownload -u 1923916260 -o output.mp4 --temp-path '--temp-path'
TwitchDownloaderCLI 1.54.2
Copyright (c) lay295 and contributors
ERROR(S):
Option 'temp-path' is defined multiple times.
```
In this example, `TwitchDownloaderCLI(2)` is trying to parse the 7th argument as an option, instead of a parameter, even if the argument before is already a valid option, and the value is enclosed in single quotes. So instead of just parsing the `n+1` argument as a paramater for the valid `n` option (which the option requires following), it tries to parse it independently. So the only way the program has to know what type of argument it is, it's just by the string itself, and not by its position. The arguments have to be carefully crafted so the command can't mistake them. The culprit in this case, like in most cases, is that the command uses the arbitrary `-` (dash) to tell options apart from arguments. It also doesn't follow the convention of using the argument `--` to stop interpreting any following argument as an option (the flag `--` can only be used once per command):
```
$ TwitchDownloaderCLI2 videodownload -u 1923916260 -o output.mp4 --temp-path -- '--temp-path'
```
Since paths can be formatted with different characters to signal the same effective path, this can be fixed easily by converting all files and directories to absolute paths (against cwd or root), like in these 2 cases (1 file and 1 directory), so any hyphen can be used leading the name, like `./--temp-path` or `./'--temp-path'`:
```
$ TwitchDownloaderCLI videodownload -u 1923916260 -o ./-output.mp4 --temp-path "$(pwd)/--temp-path"
```
For the `bandwidth` option, it seems to works correctly, but probably it's a bug:
```
$ TwitchDownloaderCLI videodownload -u 1923916260 --bandwidth -1 -o output.mp4
```
The dash `-` can't be escaped, so using `\-`, will be literal if the operating system allows it.
###### 1.3.2. Example 2:
The dash is used also by several GNU/UNIX/BSD commands:
```
$ touch -file
touch: invalid option -- 'i'
Try 'touch --help' for more information.
```
So the use of leading dash as option flag has to be disabled, or convert to absolute path:
```
$ touch -- -file
$ touch ./-file
```
In the case of paths, the fix is easy, because the same unique path can have almost unlimited string representations. However, for other types of parameters different than paths, this could not be possible, so the double dash `--` has to be supported by the command, or pass the argument indirectly by other means:
```
$ pgrep -a -- '--my-process'
```
There are 6 types of hyphens, only the one used to designate options has to be "escaped", the others are just like letters. Shouldn't be used since are visually similar to each other, and keyboards don't have physical keys for all of them:
- Hyphen-Minus (-): Unicode: U+002D, used for options when leading the argument
- En Dash (–): Unicode: U+2013
- Em Dash (—): Unicode: U+2014
- Minus Sign (−): Unicode: U+2212
- Figure Dash (‒): Unicode: U+2012
- Horizontal Bar (―): Unicode: U+2015
This is a sample to create and delete the 6 types:
```
touch – — − ‒ ― ./- && rm -- – — − ‒ ― -
mkdir – — − ‒ ― -- - && rmdir – — − ‒ ― -- -
```
###### 1.3.3. Example 3:
Special characters have to be escaped or enclosed in single quotes, like spaces, etc. For example to save to output file `-_ &$file[]%()¿.mp4` using temp folder `-_ &$my[]}+%()¿.tmp`, it can be done quoting in single quotes the strings, or escaping every special character. If any starts with hyphen, has to be converted to absolute path forcibly:
```
$ TwitchDownloaderCLI videodownload -u 1923916260 -o ./'-_ &$file[]%()¿.mp4' --temp-path './-_ &$my[]}+%()¿.tmp'
$ TwitchDownloaderCLI videodownload -u 1923916260 -o ./-_\ \&\$file\[\]%\(\)¿.mp4 --temp-path ./-_\ \&\$my\[\]\}+%\(\)¿.tmp/
```
###### 1.3.4. Conclusion:
To make the handling of non-alphanumeric characters reliable and simple, when used for files/directories names, ideally the shell itself, alongside all the external programs (lbrynet, TwitchDownloader2, curl, jq, lynx, ffmpeg, ffprobe, etc.), would have to be coded into a single program, so the reading and writing to the filesystem takes place always by the same application, keeping consistency and handling internally all the nuances without any room for mistake.
And even in that case, sharing online pictures or videos, may force to rename the files. Servers can impose restrictions to filenames regarding allowed characters and length. In best cases, the filenames could be formatted using URL Encoding, so no information is lost. Still, that doesn't assure that the file will keep the original name once the server stores it locally, or serves it to the public.
### 1.2. Licensing
All the scripts are under the [GNU General Public License, version 2](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html). The rest of the software has its own license.
## 2. MAIN SCRIPTS
The scripts are intended to be compatible with common shells: `ksh`, `zsh`, `bash`, `dash`, etc, and to be POSIX compliant. They don't use built-ins from specific shells.
Also, the commands called by the scripts, like `sed`, `awk`, etc. can be the GNU or the UNIX/BSD/macOS versions, since this is another point that reduces scripts portability.
The scripts are described below in the order they are meant to be used, and they are 9 main scripts to do the job. There are other additional scripts.
When any script fails before completing the steps (exit code ≠ 0), all temporary files generated will remain in place, to facilitate debugging issues. They have to be remove manually if not desired.
### 2.1. check-requirements.sh
This script simply checks that all needed commands or shell built-ins, required to run all scripts, are available in `$PATH`, using the command `command`. Outputs a message beginning with the number of missing commands, whether it's zero or more. The exit code of the script will also be that number.
The script does not require arguments, so has the following format:
```
sh check-requirements.sh
```
List of commands and/or builtins which are tested for:
```
awk basename bc cat cd command cp curl cut date dirname dwebp exit export ffmpeg ffprobe find head grep jq kill lbrynet lynx magick mkdir mktemp mv parallel pcre2grep
printf ps python3 read readlink realpath return rm rmdir sed seq set sha384sum shift shuf sleep sort tail test touch tr TwitchDownloaderCLI uname uniq wc wget which xmllint xmlstarlet
```
### 2.2. dump-urls-from-html-lynx.sh
This script command has the following format, so 4 arguments must be provided:
```
sh dump-urls-from-html-lynx.sh <--url=|-u=>[URL] <all-videos-html> <csv-file> <json-file>
```
A URL can be downloaded and saved to specified html file, instead of using the currently existing one, if the field is not empty, for example:
```
sh dump-urls-from-html-lynx.sh --url='https://www.twitch.tv/twitch/videos?filter=all&sort=time' twitch.html twitch.csv twitch.json
```
The URL has to be enclosed in single or double quotes to avoid any shell interpretation: `--url="$URLVAR"` or `--url='http://someurl'`. To avoid fetching any url but use the existing html file, leave the value of the paramater empty. The argument must exist:
```
sh dump-urls-from-html-lynx.sh -u= twitch.html twitch.csv twitch.json
```
Reads given html file in 2 ways (saves only IDs for past broadcasts, highlights and uploads, not clips.):
1) The content is in plain html tags:
`all-videos-html` argument must be any html file containing Twitch links starting with `https://www.twitch.tv/videos/`. Those links are extracted by the program `lynx` outputting one link per line.
Then the scripts checks if the `ID` of that video is present in the `csv-file`, and if not, appends if at the end of the file in a new line, with a comma preceding the ID.
So if the following URL is present:
```
https://www.twitch.tv/videos/1234567890
```
The following line may be appended to csv-file:
```
,1234567890
```
2) The content is in LD+JSON (JavaScript Object Notation Linked Data):
Extracts the LD+JSON block from the html file and formats it in tree view (html comes in single line). The json is saved to `json-file`. The IDs are extracted and saved to `csv-file` after a comma, if they don't exist yet.
#### 2.2.1. Notes about the html file
A valid HTML file can contain only HTML tags, only LD+JSON, or both. A sample webpage downloaded with Firefox contains 30 links in html tags and 5 in LD+JSON format (but these 5 are duplicates of the other ones).
The proper way to get a valid html, is to use a desktop web browser, and load the channel feed with the filter set to load `All Videos`. Since only the a few are loaded by default, it's required to keep scrolling down so all of them are loaded.
A sample URL is like this (official Twitch.tv channel):
[https://www.twitch.tv/twitch/videos?filter=all&sort=time](https://www.twitch.tv/twitch/videos?filter=all&sort=time)
Once all (or wanted) are loaded, save the page as html.
Sometimes the html file does not contain neither the `ld+json` script structure, nor any `href="https://www.twitch.tv/videos/` tag, only CSS or JavaScript code. That happens sometimes when downloading with `wget` or `curl`. In that case an error is issued since nothing can be extracted. Retrying usually downloads a valid html on second successive attempt. Can happen also with `Save As...` in desktop web browsers.
`wget` usually downloads the file in ld+json format; while a full desktop web browser like `firefox` or `chrome` usually saves the file with the valid html tags or with the ld+json also.
The number of videos displayed in the feed and when saving a web page vary depending if the user is logged in to Twitch or not, or if it's a subscriber or not to that channel. For command line tools, using cookies can be used, they are not retrieved automatically from the desktop web browser profile.
When using curl/wget, the json embedded in the simplified html file, is not up to date with the content available in Twitch servers at the very moment of the download. It's only refreshed every number of hours, so new content may take a while to appear. Only the last IDs are listed since last refresh. If more than 1 video is available since last sync, usually all of them are updated at once in the JSON and the older are discarded.
According to several tests, the JSON is refreshed between 16.6-25.3 hours since the last stream started or 14.7-22.5 hours since the last stream ended. Live streams are not included, only VODs available at the moment of refresh.
The number of IDs included vary a lot, between 0 and 6, and only for the last 7 days. And even with more than 6 the number can be lower. For example, `SOLIDFPS` has at the moment or writing, 12 VODs since last week, and 1 currently live, 13 in time order:
2184763802, 2187097195, 2188130081, 2188586057, 2188586059, 2188586060, 2188586061, 2188586062, 2188586063, 2188586066, 2188608295, 2188737320, 2190407892
But the JSON in HTML downloaded with curl only includes 4:
2188586057, 2188586059, 2188608295, 2188737320
The best way to get all the updated IDs and all of them available, is to save the DOM object of the page using a desktop web browser, which is not the same as save the page from the browser menu File > Save Page As (Ctrl+S, Command+S). To save the DOM there's no Menu link or keyboard shortcut, it has to be done from `More Tools` > `Developer Tools`/`Web Developer Tools`, select the `<html class=*>` opening line in the `Inspector`/`Elements` tab > `Copy` > `Outer HTML`. Then paste in any text editor and save as html file. This will save the html as exactly shown in the browser UI and is more precise than saving the predefined way. Web scrolling has to be done too before saving so all the cards are loaded on demand.
TLDR, 3 ways to save a web page:
- Use curl/wget/spider anonymously: not updated in real time. Some IDs can missing. Limited number. Only last week.
- Save As using a desktop web browser: the html is not exactly as displayed, IDs may be missing if JavaScript code is instead of actual html, but allows to save pictures and other content. Second best.
- DOM: the most accurate to what is displayed, but requires lots of manual intervention. HTML only, not images or attachments.
#### 2.2.2. Structure of `csv-file`
The `csv-file` aims to contain 2 fields per line, separated by a comma and no extra spaces:
```
creation_date,Twitch_video_ID
```
The creation date is the "created_at" or "Created at:". There's also the "published_at", which most of the times is the creation date, and when not, it's later in time, can't be before. Both values for a given ID can be obtained using the official Twitch API.
TwitchDownloaderCLI(2) only gets the "created_at", which is usually the most important.
Only the second field in the csv is mandatory (no comma required if the first field is missing), because that alone already unambiguously defines which video it is. Knowing only the date and time, does not tell which/whiches ID it is, since several videos could have been created at the same time. The date and time will be calculated later if missing, placing the comma if missing too.
The Twitch IDs are only numbers. Oldest IDs are 8-digit long, but the number of digits has been increasing as needed.
The date and time is the first field, added later by `download-twitch-video.sh`, and has the following format based on the Gregorian calendar and [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601):
```
YYYYMMDDTHHMMSSZ
```
* `YYYY` means 4-digit year, like 2023
* `MM` means 2-digit month, like 12
* `DD` means 2-digit day, like 24
* `T` separated the date from the time
* `HH` means hours in 24h format
* `MM` means minutes
* `SS` means seconds
* `Z` means Coordinated Universal Time (UTC), so it's not the local time of the streamer but Greenwich's one.
It's important to know that this time is when the video was first started to stream, not the last time it was (re)published. Since some copyright may apply later on to existing videos, Twitch or maybe the uploader, can edit them to remove audio parts which match copyrighted songs.
The last time the video was modified can be seen on mouse over the thumbnail of the video, whilst "first aired" / "created at" can be obtained from other methods (usually CLI downloaders or the Twitch API can provide this info as well).
Some videos require to be logged in or to be a paid subscriber to actually view the content, but for just listing them, with titles and thumbnails included (at least on my experience) it's not required, so the browsing can be done anonymously.
The html can be downloaded with web crawlers (cURL, Wget, HTTrack, etc.), but if the scripts in the page (like JavaScript) are not processed to a consumer format, the file will contain the links in different formats, like JSON. This requires further processing and is not currently supported.
### 2.3. download-twitch-video.sh
This script command has the following format, so 10 arguments must be provided:
```
sh download-twitch-video.sh
<OAUTH>
<csv-file>
<csv-line-number>
<[--select-stream=]skip|number>
<[--find-stream=]skip|best/chunked/source|worst|audio|high|medium|low|mobile|[^][Nx]Np[N][$]/[^]NxN[p[N]][$]/[^]pN[$]/^p/p$]>
<[--keep-segments=]yes|no>
<[--concurrent-downloads=]number>
<[--loop-accordingly=]yes|no>
<[--decode-html-entities=]true/yes|false/no>
<shell>
```
* `<OAUTH>` is the [Twitch OAuth Access Token](https://dev.twitch.tv/docs/authentication/getting-tokens-oauth/). Since this script allows to download subscriber-only VODs too, a valid token is required. It's not mandatory to have any active subscription, unless intending to download content that requires it, so free accounts are valid too.
* `<csv-file>` is the file containing the ID(s) of the videos. It can use the one generated by `dump-urls-from-html-lynx.sh`, or be created manually.
* `<csv-line-number>` is an integer, which means which line (starting from 1) of the csv-file to read the ID from. This script only downloads one ID.
* `<[--select-stream=]number>`: select `skip` to not provide any stream number. The stream number to download, which goes from 1 to n within each VOD, sorted by quality from best to worst (chunked gets the 1 and audio_only the n when they exist). To view all `STREAM-NUMBER`: `bash twitchdownloader-shell.sh info --id $ID`. This parameter is optional and default is 1 (best).
* `<[--find-stream=]skip|best/chunked/source|worst|audio|high|medium|low|mobile|[^][Nx]Np[N][$]/[^]NxN[p[N]][$]/[^]pN[$]/^p/p$]>`: select `skip` to not use the filter. If this is set, overrides "--select-stream=". Choose a keyword for quality, or a regular expression for resolution and/or framerate. Several values can be provided separated by comma, so in case one doesn't fit, tries with the next until the last. This allows to narrow qualities when downloading several VODs at the same time. This parameter is optional and default is no filter.
* `<[--keep-segments=]yes|no>`: choose whether to keep or not the segments (parts).
* `<[--concurrent-downloads=]number>`: how many segments will be downloaded in parallel for each ID. Not how many IDs at the same time, because it's only one (for that, run the script multiple times with different csv files, so the IDs don't ovelap). The number of concurrent downloads will be limited by the number of segments in each ID. It doesn't do multithreaded downloading, that is several connections per segment (the server has to support it anyway). Use accordingly to server politics of fair use, to avoid OAUTH's account or IP denial of service. The recommended value is 1, with 4 fails often.
* `<[--loop-accordingly=]yes|no>`: if the option is `no`, save to output only the unique segments in the same order as they appear. This is the recommended option for all videos. If the option is `yes`, include all segments as listed in `index-dvr.m3u8`, even if they repeat. This can lead to huge files, for no reason other than view the same video several times in a row as a single file. Some playlists can extend duration of the output by means of repeating the same segments a certain amount of times, ideally full loops of the entire video (this is checked in the script, and if it's not exactly that way will warn and exit). Looping is rare, and I've only found it in custom Twitch uploads, since the [Twitch Video Producer](https://dashboard.twitch.tv/u/USERNAME/content/video-producer) doesn't allow to duplicate the same segment in the timeline when creating a Highlight. See [5. TWITCH](#5-twitch). If the loops is performed (yes), the duration of the output file will be N times the duration of the non-loop file (yet may result in a broken file). Still the chapters in the metadata.txt will not be updated by this script (the 2 samples tested reflect the duration of the non-loop because this is how it's stored in Twitch, which reflects it's not meant to be looped anyway). So use yes only for testing purposes, because if concatenating this file with `combine-multiple-ids.sh` it may fail when performing the duration checks.
* `<[--decode-html-entities=]true/yes|false/no>`: Some JSONs returned by Twitch do HTML entities encoding, so instead of returning `'` they return `'` instead (and others alike). Enabling this option reverts back the change in all cases, so all HTML entities are unescaped. I think the problem is, that Twitch used to save special characters in the server as encoded entities for safety reasons, but now stores them as literal because the software is now better and doesn't trip. The consequence is that now it's impossible to know whether the user introduced `'` in the title, description or chapters, or `'` as-is (without meaning the `'`). Since people usually don't intend to write literal strings like `&`, `<`, `>`, `"` or `'`, it's reasonable to thing that in 99.9% of the cases the user/streamer/channel meant the unescaped character. This is why it's recommended to use the value `true`.
* `<shell>`: the shell that will be used to call the script `twitchdownloader-shell.sh` when selected in `--downloader=`. It can be just the filename like `dash` or the relative or absolute path like `/bin/ksh`.
The `csv-file` (only for this script), can contain the ID as just the number, or the video URL (for easier pasting from the web browser). It will be simplified afterwards to just the ID. The date/time is optional and will be fetched too. If any line doesn't contain both fields (date and ID/URL), only the ID/URL is expected, the comma is not needed and will be added afterwards. So all these lines are valid before being processed:
```
20110612T044615Z,38158674
https://www.twitch.tv/videos/38158674
,https://www.twitch.tv/videos/38158674
file://localhost/videos/38158674
,file://localhost/videos/38158674
http://www.twitch.tv/videos/38158674?filter=all&sort=time
,http://www.twitch.tv/videos/38158674?filter=all&sort=time
```
The TLDR is, this script generates a directory at script's level, in case the ID doesn't exist already, with all files inside, and udpates the `csv-file` accordingly.
The script operates roughly as follows:
1. All operations take place in the `$TMP` temporary directory (can be changed in the script, and can be in a different storage device than where the operating system is). By default it's in the CWD where the scripts are. If the script fails at any point, the temporary files remain, and a new iteration will create new ones.
2. Reads the specified line in the csv file, to extract the date/time and the video ID.
3. Checks if a folder with same ID exists, if so exits.
4. Calls the script `twitchdownloader-shell.sh`, which just downloads the `metadata.txt` and the playlist for selected quality as `index-dvr.m3u8` (contains custom tag). Other downloaders should generate the same information.
5. If the date and time for the given ID is not set in the specified line in the csv file, extracts such value from `metadata.txt`, and writes it back to the set line in the csv file, so now all the info for that ID is complete. The file `metadata.txt` must exist with correct information.
6. Downloads the segments/parts listed in the m3u8 playlist with `parallel` and `wget`, downloading one ore more segments at a time, depending on the parameters given to script and the number of segments per current ID. These segments comprise the whole stream, in the order as they are listed in the m3u8.
7. Create the file `parts.txt`, which lists each segment's size in bytes followed by its name on each line. The segments should be listed as many times and in the same order as they appear in the output file (and not necessarily exactly as listed in the m3u8 file), even if they repeat. The size and segment name are separated by a comma, which serves as the field separator for these two fields.
8. Combine the segments using the playlist. The resulting `output.EXTENSION` is a raw concatenation of all of them in order as they are listed. Also checks that all the segments have the same extension, and that will be the extension of the output file. Only `ts` and `mp4` extensions are allowed.
9. Extracts a frame of the video, saving it to `thumb-ffmpeg.webp`, at a random timestamp, and saves it. It substracts 3 seconds from the total duration to give room, since the measured duration is not accurate. Tries 3 times just in case with more room. Ffmpeg can throw exit code (!=0) here (v6.0.1), but does not mean the frame was not extracted, so it's checked (see error below). That was fixed later (v6.1.1).
10. Moves the folder from the TMP dir to working directory and renames it with this format: `$DATE,$ID`, being DATE the `date and time` in the ISO format mentioned previously. So the name of the folder matches the file in the csv.
11. Checks if the webp picture is defective. Sometimes Ffmpeg generates a few bytes image (also defective) and with exit code 0.
12. Generates and saves the SHA-384 hash for all the 5 files inside the folder (excluding the hash file itself) to `hash-sha384.txt`. The segments are not hashed.
13. Deletes all segments if requested.
14. Exists the script.
The m3u/m3u8 playlist file is downloaded from servers by the downloader, with an additional tag in second line to give more information about the segments, so such tag should go immediately after `#EXTM3U`. The first field is separated by `:` and the rest by `,`. Example tag:
```
#CUSTOM-DOWNLOADER:TwitchDownloaderCLI2,Twitch.tv,chunked,https://idaddress/chunked/index-dvr.m3u8
```
OR
```
#CUSTOM-DOWNLOADER:twitchdownloader-shell.sh,Twitch.tv,1920x1080p60,https://idaddress/chunked/index-dvr.m3u8
```
* `#CUSTOM-DOWNLOADER`: it's a custom tag for m3u8 files to be used by this project. It's not standard.
* `TwitchDownloaderCLI2`: it's the downloader used to fetch the playlist from remote server.
* `Twitch.tv`: it's the server, address or IP where the playlist was downloaded from.
* `chunked` or know quality like `1920x1080p60` or `mobile`: it's the name or tag which the server uses to define the quality of the stream in the playlist. Ideally all the segments should have same quality and format. This field can be a word like `chunked`, `audio_only`, or a specific value, like `720p30`. In Twitch it's the last part in the URL before the name of the playlist. This values are assigned by the server, and the technical specifications of the stream don't have to match always to it, it's a shorthand. Some downloaders grab the parent directory of the URL resource, while other downloaders may parse the M3U headers to get specific values even if the URL says chunked.
* `URL`: the URL for the specific quality. There are at least 2 quality playlists, one audio only and other with video too, but usually there are many of them being only one audio.
Example URL (first Twitch's official channel video): [https://www.twitch.tv/videos/38158674?filter=all&sort=time](https://www.twitch.tv/videos/38158674?filter=all&sort=time)
The folders generated in the current directory will have a structure like this example (excluding all the files and scripts from this project):
```
$ tree -F */
20110612T044615Z,38158674//
├── hash-sha384.txt
├── index-dvr.m3u8
├── metadata.txt
├── output.ts
├── parts.txt
└── thumb-ffmpeg.webp
TMP//
└── TwitchDownloader/
3 directories, 6 files
```
The `csv-file` will contain this:
```
20110612T044615Z,38158674
```
Sample error during step 7 (extracting a frame), this uses a `screencap.jpg` JPEG frame instead of a WebP:
```
$ ffmpeg -y -ss 4840 -i output.ts -frames:v 1 -q:v 25 screencap.jpg
ffmpeg version 6.0.1 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13 (GCC)
configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --enable-chromaprint --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libdrm --enable-libjack --enable-libjxl --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libilbc --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-libplacebo --enable-librsvg --enable-librav1e --enable-librubberband --enable-libsmbclient --enable-version3 --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libshaderc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-avfilter --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libvpl --enable-runtime-cpudetect
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
[NULL @ 0x5644b36e64c0] illegal reordering_of_pic_nums_idc 16
Last message repeated 1 times
Input #0, mpegts, from 'output.ts':
Duration: 01:27:08.01, start: 1258.015000, bitrate: 6462 kb/s
Program 1
Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 168 kb/s
Stream #0:1[0x102]: Data: timed_id3 (ID3 / 0x20334449)
Stream #0:2[0x101]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, unknown/bt470bg/unknown, progressive), 1920x1080, 60 fps, 60 tbr, 90k tbn
Stream mapping:
Stream #0:2 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[NULL @ 0x5644b36e64c0] illegal reordering_of_pic_nums_idc 16
[h264 @ 0x5644b3968800] cabac_init_idc 13 overflow
[h264 @ 0x5644b3968800] decode_slice_header error
[h264 @ 0x5644b3968800] no frame!
[h264 @ 0x5644b3793100] deblocking_filter_idc 6 out of range
[h264 @ 0x5644b3793100] decode_slice_header error
[h264 @ 0x5644b3793100] no frame!
[h264 @ 0x5644b3ac4980] illegal modification_of_pic_nums_idc 16
[h264 @ 0x5644b3ac4980] decode_slice_header error
[h264 @ 0x5644b3ac4980] no frame!
Error while decoding stream #0:2: Invalid data found when processing input
[NULL @ 0x5644b36e64c0] illegal reordering_of_pic_nums_idc 16
Error while decoding stream #0:2: Invalid data found when processing input
[h264 @ 0x5644b3968800] reference count overflow
[h264 @ 0x5644b3968800] decode_slice_header error
[h264 @ 0x5644b3968800] no frame!
[h264 @ 0x5644b3793100] illegal modification_of_pic_nums_idc 27
[h264 @ 0x5644b3793100] decode_slice_header error
[h264 @ 0x5644b3793100] no frame!
Error while decoding stream #0:2: Invalid data found when processing input
Last message repeated 1 times
[h264 @ 0x5644b3ac4980] illegal modification_of_pic_nums_idc 16
[h264 @ 0x5644b3ac4980] decode_slice_header error
[h264 @ 0x5644b3ac4980] no frame!
Error while decoding stream #0:2: Invalid data found when processing input
[h264 @ 0x5644b3968800] reference count overflow
[h264 @ 0x5644b3968800] decode_slice_header error
[h264 @ 0x5644b3968800] no frame!
Error while decoding stream #0:2: Invalid data found when processing input
Last message repeated 1 times
[swscaler @ 0x5644b3986880] deprecated pixel format used, make sure you did set range correctly
Last message repeated 3 times
Output #0, image2, to 'screencap.jpg':
Metadata:
encoder : Lavf60.3.100
Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn
Metadata:
encoder : Lavc60.3.100 mjpeg
Side data:
cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
[image2 @ 0x5644b38ee800] The specified filename 'screencap.jpg' does not contain an image sequence pattern or a pattern is invalid.
[image2 @ 0x5644b38ee800] Use a pattern such as %03d for an image sequence or use the -update option (with -frames:v 1 if needed) to write a single image.
frame= 1 fps=0.0 q=25.0 Lsize=N/A time=00:00:00.00 bitrate=N/A dup=1 drop=0 speed= 0x
video:68kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!
```
```
$ echo $?
69
```
This build after `6.1.1` still displays the error, but now throws exit code 0 instead of 69, which is accurate:
```
~/bin/ffmpeg -y -ss 4840 -i output.ts -frames:v 1 -q:v 25 screencap.jpg
ffmpeg version N-68334-ga87a52ed0b-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 8 (Debian 8.3.0-6)
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg
libavutil 58. 36.101 / 58. 36.101
libavcodec 60. 37.100 / 60. 37.100
libavformat 60. 20.100 / 60. 20.100
libavdevice 60. 4.100 / 60. 4.100
libavfilter 9. 17.100 / 9. 17.100
libswscale 7. 6.100 / 7. 6.100
libswresample 4. 13.100 / 4. 13.100
libpostproc 57. 4.100 / 57. 4.100
[NULL @ 0xceaec40] illegal reordering_of_pic_nums_idc 16
Last message repeated 1 times
Input #0, mpegts, from 'output.ts':
Duration: 01:27:08.01, start: 1258.015000, bitrate: 6462 kb/s
Program 1
Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 168 kb/s
Stream #0:1[0x102]: Data: timed_id3 (ID3 / 0x20334449)
Stream #0:2[0x101]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, unknown/bt470bg/unknown, progressive), 1920x1080, 60 fps, 60 tbr, 90k tbn
Stream mapping:
Stream #0:2 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[h264 @ 0xd032680] cabac_init_idc 13 overflow
[NULL @ 0xceaec40] illegal reordering_of_pic_nums_idc 16
[h264 @ 0xd032680] decode_slice_header error
[h264 @ 0xd032680] no frame!
[h264 @ 0xcfb74c0] deblocking_filter_idc 6 out of range
[h264 @ 0xcfb74c0] decode_slice_header error
[h264 @ 0xcfb74c0] no frame!
[h264 @ 0xcf27cc0] illegal modification_of_pic_nums_idc 16
[vist#0:2/h264 @ 0xced5f40] Error submitting packet to decoder: Invalid data found when processing input
[NULL @ 0xceaec40] illegal reordering_of_pic_nums_idc 16
[h264 @ 0xcf27cc0] decode_slice_header error
[h264 @ 0xcf27cc0] no frame!
[vist#0:2/h264 @ 0xced5f40] Error submitting packet to decoder: Invalid data found when processing input
[h264 @ 0xd032680] reference count overflow
[h264 @ 0xd032680] decode_slice_header error
[h264 @ 0xd032680] no frame!
[vist#0:2/h264 @ 0xced5f40] Error submitting packet to decoder: Invalid data found when processing input
[h264 @ 0xcfb74c0] illegal modification_of_pic_nums_idc 27
[h264 @ 0xcfb74c0] decode_slice_header error
[h264 @ 0xcfb74c0] no frame!
[vist#0:2/h264 @ 0xced5f40] Error submitting packet to decoder: Invalid data found when processing input
[h264 @ 0xcf27cc0] illegal modification_of_pic_nums_idc 16
[h264 @ 0xcf27cc0] decode_slice_header error
[h264 @ 0xcf27cc0] no frame!
[vist#0:2/h264 @ 0xced5f40] Error submitting packet to decoder: Invalid data found when processing input
[h264 @ 0xd032680] reference count overflow
[h264 @ 0xd032680] decode_slice_header error
[h264 @ 0xd032680] no frame!
[vist#0:2/h264 @ 0xced5f40] Error submitting packet to decoder: Invalid data found when processing input
Last message repeated 1 times
[swscaler @ 0xf2b92d00] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to 'screencap.jpg':
Metadata:
encoder : Lavf60.20.100
Stream #0:0: Video: mjpeg, yuvj420p(pc, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn
Metadata:
encoder : Lavc60.37.100 mjpeg
Side data:
cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
[image2 @ 0xd022ec0] The specified filename 'screencap.jpg' does not contain an image sequence pattern or a pattern is invalid.
[image2 @ 0xd022ec0] Use a pattern such as %03d for an image sequence or use the -update option (with -frames:v 1 if needed) to write a single image.
[out#0/image2 @ 0xceae500] video:68kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame= 1 fps=0.0 q=25.0 Lsize=N/A time=00:00:00.01 bitrate=N/A speed=0.128x
```
```
$ echo $?
0
```
#### 2.3.1. Notes about live streams and past broadcasts
IDs are not expected to be reused by Twitch, not even when a live stream becomes a VOD at the same time. VOD means that can be viewed from start to finish at any later moment, with the ability to seek the timeline. VODs may be deleted eventually depending on the type of VOD and the type of streamer account.
If a stream if live, and the account has enabled `Store past broadcasts` and `Always Publish VODs`, a new ID for the VOD of that stream will be created. In such case, as the live stream goes on, the M3U8 for the VOD will be updated at certain time intervals to include the new segments, until live ends. Going to the `Videos` section of any streamer, and filtering by `Past Broadcasts` while the stream is live, such VOD will appear there. The icon will be a question mark `?`, and once the live stream ends, a proper thumbnail will be generated.
If a VOD for a live stream is downloaded (with `download-twitch-video.sh` or with `TwitchDownloaderCLI(2)`) while the stream is live, only the stored portion in Twitch servers of that stream will be downloaded, and the whole process will succeed, giving the false impression that the whole VOD has been downloaded. For that reason, it's advisable to download the VOD only after the live stream has ended, and eventually wait for a few minutes until the m3u8 is synced, or download VODs other than the last one. According to my experience, the playlist update to the VOD is immediate as soon as the live ends, still there are reports online saying that it takes some minutes (probably that was the case but now it's faster).
To download live streams as they go, and even rewind to get all the parts from the start also (when possible), and continue monitoring, there are other tools specialized in live capture, like [twitch-dlp-2](https://sourceforge.net/projects/twitch-dlp-2/). This project (`twitch-batch-downloader`) is designed to download only VODs (Past Broadcasts, Uploads and Highlights, not Clips), since it's intended for archiving purposes.
`TwitchDownloaderCLI(2)` can download clips by itself, but it's not integrated to work like that in this project.
There's other archiving software like [ganymede](https://github.com/Zibbp/ganymede), which works in a more abstracted way and it's aimed for automation, with prevalece of the user interface. It has the biggest downside, that relies too much on the correct behaviour of third party commands and closed specifications. It requires too many dependencies, it's only recommended for investigation/debugging purposes.
#### 2.3.2. Reading the technical makeup of multimedia files
These are the sample files, used to test how several programs read their technical makeup, and compare which software is accurate and under which conditions. I want to test which way can I find out the real duration of the video inside any multimedia file.
The `mediainfo --version` is `MediaInfo Command line, MediaInfoLib - v24.06`.
- file1: normal output.ts stream.
- file2: the same output.ts once converted to mp4 in copy mode (-c:a copy -c:v copy), similar to using `convert-all.sh`.
- file3: the ID 637343547, ts format and no loop.
- file4: the ID 637343547, ts format and loop.
- file5: the ID 637388605, mp4 format with init file and no loop.
- file6: the ID 637388605, mp4 format with init file and loop.
These are the tests to be performed on each file:
- test1: get value in `END=` of last chapter (`grep "^END=" metadata.txt | awk -F'=' '{print $2}' | tail -n 1`). That equals the duration of all chapters, since the first chapter starts at 0 seconds.
- test2: output of `ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 fileN`.
- test3: fully process the file discarding the output, with `ffmpeg -i "$FILE" -f null - 2>&1 | grep "time=" | awk -F'time=' '{print $2}' | awk '{print $1}' | tail -n 1`.
- test4: result of `mediainfo --Inform="General;%Duration%" fileN`
- test5: result of `mediainfo --ParseSpeed=1.0 --Inform="General;%Duration%" fileN`
- test6: result of `mediainfo --Inform="Menu;%Duration%" fileN`
- test7: result of `mediainfo --ParseSpeed=1.0 --Inform="Menu;%Duration%" fileN`
- test8: result of `mediainfo --Inform="Audio;%Duration%" fileN`
- test9: result of `mediainfo --ParseSpeed=1.0 --Inform="Audio;%Duration%" fileN`
- test10: result of `mediainfo --Inform="Video;%Duration%" fileN`
- test11: result of `mediainfo --ParseSpeed=1.0 --Inform="Video;%Duration%" fileN`
- test12: result of `mediainfo --Inform="Audio;%BitRate%" "$FILE"`
- test13: result of `mediainfo --ParseSpeed=1.0 --Inform="Audio;%BitRate%" "$FILE"`
- test14: result of `mediainfo --Inform="Video;%BitRate%" "$FILE"`
- test15: result of `mediainfo --ParseSpeed=1.0 --Inform="Video;%BitRate%" "$FILE"`
- test16: real duration. Duration of the file in seconds, by fully converting the input file with `ffmpeg -i "$FILE" -f null -`, like in test3. The time/duration would be what's after `time=` once the conversion ends, because all frames have to be parsed. If the file loops, run the ffmpeg command on the same ID but without loop, and multiply N times to get the real duration (the whole looped file can't be used since ffmpeg doesn't detect it correctly<sup>**</sup>). All the values in this fow have been manually revised to be exact, the others are just the output of the commands.
Here's the code to get all the lines for each column, except test16 which is hand crafted:
```
FILE="$(find . -type f -name 'output.*' | head -n 1)"
printf "Starting:\n"
printf "line 1: "; grep "^END=" metadata.txt | awk -F'=' '{print $2}' | tail -n 1
printf "line 2: "; ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$FILE"
printf "line 3: "; ffmpeg -i "$FILE" -f null - 2>&1 | grep "time=" | awk -F'time=' '{print $2}' | awk '{print $1}' | tail -n 1
printf "line 4: "; mediainfo --Inform="General;%Duration%" "$FILE"
printf "line 5: "; mediainfo --ParseSpeed=1.0 --Inform="General;%Duration%" "$FILE"
printf "line 6: "; mediainfo --Inform="Menu;%Duration%" "$FILE"
printf "line 7: "; mediainfo --ParseSpeed=1.0 --Inform="Menu;%Duration%" "$FILE"
printf "line 8: "; mediainfo --Inform="Audio;%Duration%" "$FILE"
printf "line 9: "; mediainfo --ParseSpeed=1.0 --Inform="Audio;%Duration%" "$FILE"
printf "line 10: "; mediainfo -f --Inform="Video;%Duration%" "$FILE"
printf "line 11: "; mediainfo -f --ParseSpeed=1.0 --Inform="Video;%Duration%" "$FILE"
printf "line 12: "; mediainfo --Inform="Audio;%BitRate%" "$FILE"
printf "line 13: "; mediainfo --ParseSpeed=1.0 --Inform="Audio;%BitRate%" "$FILE"
printf "line 14: "; mediainfo --Inform="Video;%BitRate%" "$FILE"
printf "line 15: "; mediainfo --ParseSpeed=1.0 --Inform="Video;%BitRate%" "$FILE"
```
| TABLE1 | file1<sup>*</sup> | file2 | file3| file4 | file5 | file6 |
|---------------|---------------|---------------|---------------|---------------|---------------|---------------|
| test1 | 5288000 | 5288000 | 840000 | 840000 | 840000 | 840000 |
| test2 | 5228.006600 | 5288.006372 | 840.724889 | 840.724889 | 840.733333 | 840.733333 |
| test3 | 01:28:08.00 | 01:28:08.00 | 00:14:00.70 | 24:45:12.51 | 00:14:00.73 | 00:14:00.73<sup>**</sup> |
| test4 | 5228015.429928 | 5288007 | 840686.518681 | 840683.033195 | 840733 | 89117736 |
| test5 | 5297994.373078 | 5288007 | 840686.518681 | 89113064.519804 | 840733 | 89117736 |
| test6 | 5228001.000000 | 5288000 | 840683.000000 | 840683.000000 | null | null |
| test7 | 5297979.750000 | 5288000 | 840683.000000 | 89113061.001111 | null | null |
| test8 | 5227996 | 5288007 | 840771 | 840771 | null | null |
| test9 | 5287996 | 5288007 | 840771 | 89121762 | null | null |
| test10 | 5228001 | 5288001 | 840684 | 840684 | 840733 | 89117736 |
| test11 | 5228001 | 5288001 | 840684 | 840684 | 840733 | 89117736 |
| test12 | null | 160530 | null | null | null | null |
| test13 | 162942 | 160530 | 132300 | 132300 | null | null |
| test14 | null | 6000083 | null | null | 7812947 | 7812947 |
| test15 | 6000000 | 6000083 | 5285100 | 560220582 | 7812947 | 7812947 |
| test16 | 5288 | 5288 | 840.7 | 89114.2 | 840.7 | 89114.2 |
<sup>*</sup> When playing ts files with `mpv` it finds the correct duration after some time of playback (file1), by means of instantly changing the current timestamp to a negative value. So in reality there are 60 seconds more of duration: (5228 - (-60)) instead of (end - start) being (5228 - 0):
```
$ mpv output.ts
mpv 0.38.0 Copyright © 2000-2024 mpv/MPlayer/mplayer2 projects
(+) Video --vid=1 (h264 1920x1080)
(+) Audio --aid=1 (aac 2ch 44100Hz)
AO: [coreaudio] 44100Hz stereo 2ch floatp
VO: [gpu] 1920x1080 yuv420p
[ffmpeg/demuxer] mpegts: Packet corrupt (stream = 2, dts = 113936940).
Invalid audio PTS: 8.021118 -> -51.978878
Reset playback due to audio timestamp reset.
VO: [gpu] 1920x1080 yuv420p
AV: -00:00:28 / 01:27:08 (0%) A-V: 0.000
Exiting... (Quit)
```
<sup>*</sup> When re-encoding the file with `ffmpeg`, the fast parse just upon opening the file shows one duration (this is ffprobe), and the conversion log shows another duration once it completes. `Duration: 01:27:08.01` is listed below as `time=01:28:08.00`, which is the real duration of the video:
```
$ ffmpeg -benchmark -i output.ts -f null -
ffmpeg version 7.0.2 Copyright (c) 2000-2024 the FFmpeg developers
Input #0, mpegts, from 'output.ts':
Duration: 01:27:08.01, start: 1258.015000, bitrate: 6462 kb/s
Program 1
Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 130 kb/s
Stream #0:1[0x102]: Data: timed_id3 (ID3 / 0x20334449)
Stream #0:2[0x101]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, unknown/bt470bg/unknown, progressive), 1920x1080, 60 fps, 60 tbr, 90k tbn
Stream mapping:
Stream #0:2 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Stream #0:0 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf61.1.100
Stream #0:0: Video: wrapped_avframe, yuv420p(tv, unknown/bt470bg/unknown, progressive), 1920x1080, q=2-31, 200 kb/s, 60 fps, 60 tbn
Metadata:
encoder : Lavc61.3.100 wrapped_avframe
Stream #0:1: Audio: pcm_s16le, 44100 Hz, stereo, s16, 1411 kb/s
Metadata:
encoder : Lavc61.3.100 pcm_s16le
[mpegts @ 0x13de059c0] Packet corrupt (stream = 2, dts = 113936940).
[in#0/mpegts @ 0x600001090100] corrupt input packet in stream 2
[aist#0:0/aac @ 0x13df04e00] timestamp discontinuity (stream id=256): -59999997, new offset= 59999997
[vf#0:0 @ 0x600001e94000] Reconfiguring filter graph because video parameters changed to yuv420p(tv, bt470bg), 1920x1080
[out#0/null @ 0x600001994000] video:136331KiB audio:910940KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=317280 fps=1071 q=-0.0 Lsize=N/A time=01:28:08.00 bitrate=N/A speed=17.9x
bench: utime=1199.852s stime=76.223s rtime=296.216s
bench: maxrss=161447936KiB
```
<sup>**</sup> Ffmpeg fails here to succeessfully detect the real duration of the video in the file, because drops duplicate frames, which shouldn't do. This is the mp4 with loop (ID 637388605):
```
$ ffmpeg -y -i output.mp4 -c:v libx264 -crf 51 -preset ultrafast -c:a aac -b:a 16k -movflags +faststart -vf scale=-2:120,format=yuv420p -nostdin -f mp4 /dev/null
ffmpeg version 7.0.2 Copyright (c) 2000-2024 the FFmpeg developers
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'output.mp4':
Metadata:
major_brand : isom
minor_version : 0
compatible_brands: iso8mp41dashav01cmfc
creation_time : 2020-07-16T19:53:10.000000Z
Duration: 00:14:00.73, start: 0.000000, bitrate: 829024 kb/s
Stream #0:0[0x1](und): Video: av1 (libdav1d) (Main) (av01 / 0x31307661), yuv420p(tv), 2560x1440, 828160 kb/s, 120 fps, 120 tbr, 15360 tbn (default)
Metadata:
creation_time : 2020-07-16T19:53:10.000000Z
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : AOM Coding
[out#0/mp4 @ 0x600003648000] Codec AVOption b (set bitrate (in bits/s)) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream.
Stream mapping:
Stream #0:0 -> #0:0 (av1 (libdav1d) -> h264 (libx264))
[libx264 @ 0x128f05ac0] using SAR=320/321
[libx264 @ 0x128f05ac0] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x128f05ac0] profile Constrained Baseline, level 2.1, 4:2:0, 8-bit
[libx264 @ 0x128f05ac0] 264 - core 164 r3108 31e19f9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=4 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=51.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, mp4, to '/dev/null':
Metadata:
major_brand : isom
minor_version : 0
compatible_brands: iso8mp41dashav01cmfc
encoder : Lavf61.1.100
Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p(tv, progressive), 214x120 [SAR 320:321 DAR 16:9], q=2-31, 120 fps, 15360 tbn (default)
Metadata:
creation_time : 2020-07-16T19:53:10.000000Z
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.3.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x128e059c0] DTS 0 < 12913536 out of order= 35.0kbits/s speed=2.74x
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x128e059c0] DTS 0 < 12913536 out of order= 34.8kbits/s dup=0 drop=100306 speed=1.36x
(103 lines omitted here...)
[mp4 @ 0x128f04fc0] Starting second pass: moving the moov atom to the beginning of the filerop=10580125 speed=0.0316x
[out#0/mp4 @ 0x600003648000] video:3766KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.001271%
frame=101254 fps=3.8 q=-1.0 Lsize= 3766KiB time=00:14:03.78 bitrate= 36.6kbits/s dup=0 drop=10580275 speed=0.0316x
[libx264 @ 0x128f05ac0] frame I:406 Avg QP:49.64 size: 361
[libx264 @ 0x128f05ac0] frame P:100848 Avg QP:50.46 size: 37
[libx264 @ 0x128f05ac0] mb I I16..4: 100.0% 0.0% 0.0%
[libx264 @ 0x128f05ac0] mb P I16..4: 2.4% 0.0% 0.0% P16..4: 5.7% 0.0% 0.0% 0.0% 0.0% skip:92.0%
[libx264 @ 0x128f05ac0] coded y,uvDC,uvAC intra: 10.0% 53.3% 1.1% inter: 0.6% 1.8% 0.0%
[libx264 @ 0x128f05ac0] i16 v,h,dc,p: 42% 28% 23% 7%
[libx264 @ 0x128f05ac0] i8c dc,h,v,p: 91% 6% 3% 0%
[libx264 @ 0x128f05ac0] kb/s:36.56
```
According to all the results, several conclusions can be drawn:
* `ffmpeg` is the most accurate command to get the real duration, but only after processing the full file. Only fails with loop in mp4 files. Is the slowest method by far.
* `ffprobe` is not as accurate as ffmpeg with full processing. Fails in some .ts files that do not loop, and fails always in files (ts/mp4) that loop. The fast parse of ffmpeg `ffmpeg -i "$FILE"`, with or without conversion, is just the output of ffprobe.
* Even if the playlist includes loops, in reality it's not intended to do that, so unless explicitly stated, the download and conversion must not loop. The loop is for developing purposes but it's not intended to be used in real life.
* `mediainfo` is poorly programmed, and is consistently broken. The full output `mediainfo -f --ParseSpeed=1.0 "$FILE"` is not determinist, since the same attribute/parameter has several values (some the same and some different).
* To get audio bitrates, a full parsing is required (with either ffmpeg or mediainfo), since the audio usually is in variable bitrate.
* Same for video bitrates, specially if the duration is not detected correctly, and the bitrate is obtained by dividing video size / duration.
* Get exact values of audio and video bitrates is even trickier than obtaining the duration, due to overheads, and if the command allows to fully parse the file or not.
#### 2.3.3. Notes on using `ls` to sort files by date/time
This is the command used previously to sort the downloaded .ts files by last modification date, and perform hash on the binary concatenation of them. The file can be displayed with `git show cde3f93838497d892f351d3f8a59708d5f03a451:download-twitch-video.sh`:
```
ls -1tr | grep '.ts$' | grep -v '^output.ts$' | while IFS= read -r line ; do cat -- "$line" ; done | sha384sum -b --
```
This command (there are other `ls` in the script, used up to that specific commit), rely on sorting files in the filesystem by last modification date. This is an attribute of the filesystem, it's not related to the contents or the filename of the file itself. This usually does not pose a problem, since modern filesystems allow a high precision. For example `NTFS stores timestamps in ticks` (granularity of 100 nanoseconds (ns)), and the segments are not downloaded so fast that multiple files will have the same `mtime` up to the same tick, so they can be told apart from each other and concatenated in the expected order.
However, using certain filesystems on certain operating systems, or depending on filesystem mounting options or driver limitations, the `mtime` precision can be drastically reduced. This happens for example when using [Microsoft NTFS for Mac by Paragon](https://www.paragon-software.com/home/ntfs-mac/), to read/write to a `NTFS` partition from `macOS` 14.6.1 with Apple Silicon chip. In this case the precision decreases from ticks to seconds, so multiple files have the same exact mtime: files can't be sorted correctly, and each operating system will pick a different sorting order. This is why all `ls` commands were replaced by using a filelist where possible.
This is the driver causing the low mtime precision:
```
# kextstat | grep -i ntfs
Executing: /usr/bin/kmutil showloaded
No variant specified, falling back to release
237 0 0 0x373a 0x373a com.paragon-software.filesystems.ntfs (186.1.16) 550E8400-E29B-41D4-A716-446655440000 <7 5 4 1>
```
```
# kmutil showloaded --bundle-identifier com.paragon-software.filesystems.ntfs
No variant specified, falling back to release
Index Refs Address Size Wired Name (Version) UUID <Linked Against>
237 0 0 0x373a 0x373a com.paragon-software.filesystems.ntfs (186.1.16) 550E8400-E29B-41D4-A716-446655440000 <7 5 4 1>
```
```
# plutil -p /Library/Extensions/ufsd_NTFS.kext/Contents/Info.plist
{
"BuildMachineOSBuild" => "14A389"
"CFBundleDevelopmentRegion" => "English"
"CFBundleExecutable" => "ufsd_NTFS"
"CFBundleIdentifier" => "com.paragon-software.filesystems.ntfs"
"CFBundleInfoDictionaryVersion" => "6.0"
"CFBundleName" => "ufsd_NTFS"
"CFBundlePackageType" => "KEXT"
"CFBundleShortVersionString" => "16.1.186"
"CFBundleVersion" => "186.1.16"
"DTCompiler" => "com.apple.compilers.llvm.clang.1_0"
"DTPlatformBuild" => "6A1052d"
"DTPlatformVersion" => "GM"
"DTSDKBuild" => "14A382"
"DTSDKName" => "macosx10.10"
"DTXcode" => "0610"
"DTXcodeBuild" => "6A1052d"
"NSHumanReadableCopyright" => "Copyright © 2023 Paragon. All rights reserved."
"OSBundleAllowUserLoad" => 1
"OSBundleLibraries" => {
"com.apple.kpi.bsd" => "8.0.0b1"
"com.apple.kpi.libkern" => "8.0.0b1"
"com.apple.kpi.mach" => "8.9.9"
"com.apple.kpi.unsupported" => "8.9.9"
}
}
```
1) This is the output of `GNU ls` on `macOS`, for the ID 2195510359, with which the files were created:
```
$ gls -latr --time-style=full-is
total 10382876
-rw-r--r-- 1 mcuser staff 8035496 2024-07-20 00:18:41.000000000 +0000 0.ts
drwxr-xr-x 0 mcuser staff 4096 2024-07-20 00:18:41.000000000 +0000 ..
-rw-r--r-- 1 mcuser staff 8030796 2024-07-20 00:18:42.000000000 +0000 5.ts
-rw-r--r-- 1 mcuser staff 8034556 2024-07-20 00:18:42.000000000 +0000 4.ts
-rw-r--r-- 1 mcuser staff 8001844 2024-07-20 00:18:42.000000000 +0000 3.ts
-rw-r--r-- 1 mcuser staff 8042264 2024-07-20 00:18:42.000000000 +0000 2.ts
-rw-r--r-- 1 mcuser staff 8008612 2024-07-20 00:18:42.000000000 +0000 1.ts
-rw-r--r-- 1 mcuser staff 8022336 2024-07-20 00:18:43.000000000 +0000 9.ts
-rw-r--r-- 1 mcuser staff 8069900 2024-07-20 00:18:43.000000000 +0000 8.ts
-rw-r--r-- 1 mcuser staff 8038128 2024-07-20 00:18:43.000000000 +0000 7.ts
-rw-r--r-- 1 mcuser staff 8012184 2024-07-20 00:18:43.000000000 +0000 6.ts
-rw-r--r-- 1 mcuser staff 8017824 2024-07-20 00:18:43.000000000 +0000 10.ts
```
When reading again in `Windows` 11 the same ID in the `NTFS` partition with `PowerShell`, the decimals are zeroed. Is not that they were not viewable but they were not created by macOS`s Paragon:
```
$ Get-ChildItem -File | Sort-Object LastWriteTime | ForEach-Object { $_.LastWriteTime.ToString("yyyy-MM-dd HH:mm:ss.ffffff") + " " + $_.Name }
2024-07-20 00:18:41.000000 0.ts
2024-07-20 00:18:42.000000 1.ts
2024-07-20 00:18:42.000000 4.ts
2024-07-20 00:18:42.000000 5.ts
2024-07-20 00:18:42.000000 2.ts
2024-07-20 00:18:42.000000 3.ts
2024-07-20 00:18:43.000000 10.ts
2024-07-20 00:18:43.000000 6.ts
2024-07-20 00:18:43.000000 7.ts
2024-07-20 00:18:43.000000 8.ts
2024-07-20 00:18:43.000000 9.ts
```
2) This is another ID 2200907544 created in the same partition, but with `Windows` as host operating system. The mtime is of the highest precision:
```
PS D:\> TwitchDownloaderCLI.exe videodownload -u 2200907544 -o 2200907544.mp4 -q best --temp-path '.\TMP\' -t 1
```
```
PS D:\TMP\TwitchDownloader\2200907544_638607137162872395> Get-ChildItem -File | Sort-Object LastWriteTime | ForEach-Object { $_.LastWriteTime.ToString("yyyy-MM-dd HH:mm:ss.ffffff") + " " + $_.Length + " " + $_.Name }
2024-08-31 17:08:38.626165 9854396 0.ts
2024-08-31 17:08:38.991977 9840860 1.ts
2024-08-31 17:08:39.286144 9773368 2.ts
2024-08-31 17:08:39.596530 9839356 3.ts
2024-08-31 17:08:39.911079 9803260 4.ts
2024-08-31 17:08:40.269625 9828828 5.ts
2024-08-31 17:08:40.552398 9810968 6.ts
2024-08-31 17:08:40.873316 9811908 7.ts
2024-08-31 17:08:41.281105 9820180 8.ts
2024-08-31 17:08:41.644851 9819804 9.ts
2024-08-31 17:08:41.957129 9809276 10.ts
2024-08-31 17:08:42.326012 9850636 11.ts
2024-08-31 17:08:42.651340 9781640 12.ts
```
Reading the partition from `Linux` 6.10.7, using the module `fs-ntfs3` (ntfs3.ko.zst) by Konstantin Komarov, gets mtime with nanoseconds precision rounded to ticks:
```
# ls -latr --time-style=full-is
total 124712
drwxr-xr-x 1 root root 0 2024-08-31 17:08:38.301072000 +0000 ..
-rwxr-xr-x 1 root root 9854396 2024-08-31 17:08:38.626165500 +0000 0.ts
-rwxr-xr-x 1 root root 9840860 2024-08-31 17:08:38.991977100 +0000 1.ts
-rwxr-xr-x 1 root root 9773368 2024-08-31 17:08:39.286144400 +0000 2.ts
-rwxr-xr-x 1 root root 9839356 2024-08-31 17:08:39.596530500 +0000 3.ts
-rwxr-xr-x 1 root root 9803260 2024-08-31 17:08:39.911079200 +0000 4.ts
-rwxr-xr-x 1 root root 9828828 2024-08-31 17:08:40.269625500 +0000 5.ts
-rwxr-xr-x 1 root root 9810968 2024-08-31 17:08:40.552398800 +0000 6.ts
-rwxr-xr-x 1 root root 9811908 2024-08-31 17:08:40.873316300 +0000 7.ts
-rwxr-xr-x 1 root root 9820180 2024-08-31 17:08:41.281105100 +0000 8.ts
-rwxr-xr-x 1 root root 9819804 2024-08-31 17:08:41.644851800 +0000 9.ts
-rwxr-xr-x 1 root root 9809276 2024-08-31 17:08:41.957129200 +0000 10.ts
-rwxr-xr-x 1 root root 9850636 2024-08-31 17:08:42.326012400 +0000 11.ts
-rwxr-xr-x 1 root root 9781640 2024-08-31 17:08:42.651340200 +0000 12.ts
drwxr-xr-x 1 root root 8192 2024-08-31 17:08:57.309453900 +0000 .
```
Reading the ID with macOS and the same Paragon NTFS driver as before, rounds it to seconds:
```
# gls -latr --time-style=full-is
total 124712
-rwxr-xr-x 1 mcuser staff 9840860 2024-08-31 17:08:38.000000000 +0000 1.ts
-rwxr-xr-x 1 mcuser staff 9854396 2024-08-31 17:08:38.000000000 +0000 0.ts
drwxr-xr-x 0 mcuser staff 0 2024-08-31 17:08:38.000000000 +0000 ..
-rwxr-xr-x 1 mcuser staff 9803260 2024-08-31 17:08:39.000000000 +0000 4.ts
-rwxr-xr-x 1 mcuser staff 9839356 2024-08-31 17:08:39.000000000 +0000 3.ts
-rwxr-xr-x 1 mcuser staff 9773368 2024-08-31 17:08:39.000000000 +0000 2.ts
-rwxr-xr-x 1 mcuser staff 9811908 2024-08-31 17:08:40.000000000 +0000 7.ts
-rwxr-xr-x 1 mcuser staff 9810968 2024-08-31 17:08:40.000000000 +0000 6.ts
-rwxr-xr-x 1 mcuser staff 9828828 2024-08-31 17:08:40.000000000 +0000 5.ts
-rwxr-xr-x 1 mcuser staff 9819804 2024-08-31 17:08:41.000000000 +0000 9.ts
-rwxr-xr-x 1 mcuser staff 9820180 2024-08-31 17:08:41.000000000 +0000 8.ts
-rw-r--r-- 1 mcuser staff 9809276 2024-08-31 17:08:41.000000000 +0000 10.ts
-rwxr-xr-x 1 mcuser staff 9781640 2024-08-31 17:08:42.000000000 +0000 12.ts
-rw-r--r-- 1 mcuser staff 9850636 2024-08-31 17:08:42.000000000 +0000 11.ts
drwxr-xr-x 0 mcuser staff 8192 2024-08-31 17:08:57.000000000 +0000 .
```
Reading the ID with macOS and the `ntfs-3g-mac` version 2022.10.3 driver, shows the most precise timestamps, like with Microsoft driver in Windows:
1. Download, install [macFUSE](https://osxfuse.github.io/) (in use 4.8.0) and reboot
2. Install the brew formula driver with `brew install ntfs-3g-mac`
3. Mount the partition with `sudo ntfs-3g /dev/disk6s1 /Volumes/NTFS -olocal -oallow_other`. The mount point directory will be created automatically
4. For cleanup, umount with `sudo umount /Volumes/NTFS/`. The directory `/Volumes/NTFS/` should be deleted automatically
```
$ gls -latr --time-style=full-is
total 124712
drwxrwxrwx 1 root wheel 0 2024-08-31 17:08:38.301072000 +0000 ..
-rwxrwxrwx 1 root wheel 9854396 2024-08-31 17:08:38.626165500 +0000 0.ts
-rwxrwxrwx 1 root wheel 9840860 2024-08-31 17:08:38.991977100 +0000 1.ts
-rwxrwxrwx 1 root wheel 9773368 2024-08-31 17:08:39.286144400 +0000 2.ts
-rwxrwxrwx 1 root wheel 9839356 2024-08-31 17:08:39.596530500 +0000 3.ts
-rwxrwxrwx 1 root wheel 9803260 2024-08-31 17:08:39.911079200 +0000 4.ts
-rwxrwxrwx 1 root wheel 9828828 2024-08-31 17:08:40.269625500 +0000 5.ts
-rwxrwxrwx 1 root wheel 9810968 2024-08-31 17:08:40.552398800 +0000 6.ts
-rwxrwxrwx 1 root wheel 9811908 2024-08-31 17:08:40.873316300 +0000 7.ts
-rwxrwxrwx 1 root wheel 9820180 2024-08-31 17:08:41.281105100 +0000 8.ts
-rwxrwxrwx 1 root wheel 9819804 2024-08-31 17:08:41.644851800 +0000 9.ts
-rwxrwxrwx 1 root wheel 9809276 2024-08-31 17:08:41.957129200 +0000 10.ts
-rwxrwxrwx 1 root wheel 9850636 2024-08-31 17:08:42.326012400 +0000 11.ts
-rwxrwxrwx 1 root wheel 9781640 2024-08-31 17:08:42.651340200 +0000 12.ts
drwxrwxrwx 1 root wheel 8192 2024-08-31 17:08:57.309453900 +0000 .
```
3) If the partition is formatted in `APFS`, `macOS` shows the precision in nanoseconds. The ID is 2195510359:
```
$ gls -latr --time-style=full-is
total 3651764
drwxr-xr-x 4 mcuser staff 128 2024-07-20 00:38:01.532632519 +0000 ..
-rw-r--r-- 1 mcuser staff 8035496 2024-07-20 00:38:01.756572912 +0000 0.ts
-rw-r--r-- 1 mcuser staff 8008612 2024-07-20 00:38:01.987052932 +0000 1.ts
-rw-r--r-- 1 mcuser staff 8042264 2024-07-20 00:38:02.213174270 +0000 2.ts
-rw-r--r-- 1 mcuser staff 8001844 2024-07-20 00:38:02.441518802 +0000 3.ts
-rw-r--r-- 1 mcuser staff 8034556 2024-07-20 00:38:02.663182753 +0000 4.ts
-rw-r--r-- 1 mcuser staff 8030796 2024-07-20 00:38:02.891482494 +0000 5.ts
-rw-r--r-- 1 mcuser staff 8012184 2024-07-20 00:38:03.139969185 +0000 6.ts
-rw-r--r-- 1 mcuser staff 8038128 2024-07-20 00:38:03.374738513 +0000 7.ts
-rw-r--r-- 1 mcuser staff 8069900 2024-07-20 00:38:03.603988481 +0000 8.ts
-rw-r--r-- 1 mcuser staff 8022336 2024-07-20 00:38:03.829879033 +0000 9.ts
-rw-r--r-- 1 mcuser staff 8017824 2024-07-20 00:38:04.052912116 +0000 10.ts
```
The conclusion is that each filesystem, driver and operating sytem can implement filesystem timestamps with different degrees of precision, so it's not something to be relied upon. Also, any of the `atime`, `mtime` or `ctime`, may not be supported or be disabled.
#### 2.3.4. Notes on streams with initialization segments
Some streams (Twitch IDs) contain an initialization file, listed in the m3u8 playlist, which is declared by a line starting with `#EXT-X-MAP:URI=`. Such initialization file does not contain any audio or video by itself, it's purely metadata, and the declaration goes before any actual payload data (segments which do not start with a comment `#`).
However, some defective streams may contain multiple initialization files, for example when the stream was started with some defective configuration or there was any codec error, so a new initialization file is provided in the m3u8 for the next segments. Each initialization file only applies to all segments after it, and until another initialization file is declared or the end of the playlist.
Check [4.2.1. Download all playlists in all qualities](#421-download-all-playlists-in-all-qualities) for specific shell code.
Some Past Broadcasts by the artist [Rodney](https://www.twitch.tv/rodney/videos?filter=archives&sort=time) (renamed channel/artist, originally [r0dn3y](https://www.twitch.tv/r0dn3y/videos?filter=archives&sort=time)) suffer from having 2 initialization segments. The first initialization segment only applies to first payload segment (in this case), and the second to the rest. The currently affected IDs are (with one or more qualities): 2178091266, 2189874110, 2191047631, 2203436959, 2223719525. As example the ID's `2178091266` playlist with quality 1080p60:
```
#EXTM3U
#EXT-X-VERSION:6
#EXT-X-TARGETDURATION:10
#ID3-EQUIV-TDTG:2024-06-21T22:31:49
#EXT-X-PLAYLIST-TYPE:EVENT
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TWITCH-ELAPSED-SECS:0.000
#EXT-X-TWITCH-TOTAL-SECS:4863.267
#EXT-X-MAP:URI="init-0.mp4"
#EXTINF:2.000,
0.mp4
#EXT-X-MAP:URI="init-1.mp4"
#EXTINF:10.000,
1.mp4
#EXTINF:10.000,
2.mp4
[...]
#EXTINF:1.267,
487.mp4
#EXT-X-ENDLIST
```
In all the 5 IDs listed above, there are 2 initialization segments present in some playlists (not all qualities are affected, depends on the codec in use), at lines 9 and 12 without counting the `#CUSTOM-DOWNLOADER:` tag (but having 2 may not be the case for all artists or streams, probably it's a result of a bad software configuration on r0dn3y's side).
All payload segment(s) need to be concatenated after its own initialization segment. For example, in the ID `2178091266`, the first payload segment `0.mp4` needs to go after `init-0.mp4`, and the same for the others:
```
$ cat init-0.mp4 0.mp4 > first_piece_of_stream.mp4
$ cat init-1.mp4 $(seq -f '%g.mp4' 1 487) > second_piece_of_stream.mp4
```
The first piece of the stream is just an audio track with intro music, which continues in the second piece of the stream (in others IDs this first piece has no audible audio but it's still present):
```
$ ffprobe -hide_banner -i first_piece_of_stream.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'first_piece_of_stream.mp4':
Metadata:
major_brand : mp42
minor_version : 1
compatible_brands: isommp42dashavc1iso6hlsf
Duration: 00:01:14.10, start: 72.119333, bitrate: 5 kb/s
Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 4 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
```
After remuxing it without conversion, the duration is shorter and exact:
```
$ ffmpeg -loglevel warning -i first_piece_of_stream.mp4 -c copy first_piece_of_stream_remuxed.mp4
$ ffprobe -hide_banner -i first_piece_of_stream_remuxed.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'first_piece_of_stream_remuxed.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf61.1.100
Duration: 00:00:01.98, start: 0.000000, bitrate: 163 kb/s
Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 159 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
```
The second piece of the stream contains the actual video with its audio, and has no errors:
```
$ ffprobe -hide_banner -i second_piece_of_stream.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'second_piece_of_stream.mp4':
Metadata:
major_brand : mp42
minor_version : 1
compatible_brands: isommp42dashavc1iso6hlsf
Duration: 01:22:15.32, start: 74.099000, bitrate: 8083 kb/s
Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 157 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 7855 kb/s, 59.80 fps, 60 tbr, 1000k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
```
After remuxing it contains the real duration:
```
$ ffmpeg -loglevel warning -i second_piece_of_stream.mp4 -c copy second_piece_of_stream_remuxed.mp4
$ ffprobe -hide_banner -i second_piece_of_stream_remuxed.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'second_piece_of_stream_remuxed.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf61.1.100
Duration: 01:21:01.25, start: 0.000000, bitrate: 8153 kb/s
Stream #0:0[0x1](und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 7975 kb/s, 59.80 fps, 60 tbr, 1000k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 159 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
```
The `metadata.txt` obtained with `TwitchDownloaderCLI2` shows 3 chapters, with a total duration of `4863` seconds, which is `1:21:03`, and the duration of the second piece once remuxed is `01:21:01.25`:
```
;FFMETADATA1
title=[QSV AV1 1440p] Shadow Gambit, Warframe & Tiny Glade (2178091266)
artist=r0dn3y
date=2024
genre=Shadow Gambit: The Cursed Crew
comment=All renditions encoded using QuickSync (QSV) on an Intel Arc A750.\
* 1440p60 AV1 @ 12 Mbps\
* 1080p60 HEVC @ 8 Mbps\
* 720p60 H.264 @ 3.5 Mbps\
* 480p30 H.264 @ 1 Mbps\
------------------------\
Created at: 2024-06-21 21:10:44Z\
Video id: 2178091266\
Views: 306
[CHAPTER]
TIMEBASE=1/1000
START=0
END=4022000
title=Shadow Gambit: The Cursed Crew
[CHAPTER]
TIMEBASE=1/1000
START=4022000
END=4599000
title=Warframe
[CHAPTER]
TIMEBASE=1/1000
START=4599000
END=4863000
title=Tiny Glade
```
The first chapter would have to have its duration reduced in about 2 seconds to match within one second of margin of error, so other scripts don't complain (but this is just skipped since does not affect currently a lot of IDs and it's a small duration).
Ideally, the 2 stream pieces could be concatenated with ffmpeg in copy mode by remuxing them (and no need to edit the metadata file), but since one piece has audio only and the other audio and video, the first piece would require to be recoded with an empty black video track, so the same number and type of streams match. This step is not important since the missed content is a couple of seconds at the beginning where there's only an intro.
The difference in duration for the piece(s) once remuxed, shows that all streams with initialization segment(s), provide realiable technical specifications only after they are remuxed. This is why even some streams being already downloaded as `output.mp4` and `ouptut.ts` should still be `remuxed` to `$DATE_$ID.mp4`, so `ffmpeg` regenerates correctly the multimedia container.
When there are 2 initialization segments (r0dn3y-style), the first one and the first segment immediately after will be saved to subdirectory "$DATE,$ID/unused-segments/" for archiving purposes, but they don't go into final output.
#### 2.3.5. Codecs and formats in each type of stream
Twitch.tv currently accepts 4 different codecs for audio/video:
- Codec `AV1` is reported as `av1` by ffprobe and as `av01` by Twitch.
- Codec `H.265` is reported as `hevc` by ffprobe and as `hvc1` by Twitch.
- Codec `H.264` is reported as `h264` by ffprobe and as `avc1` by Twitch.
- Codec `AAC` is reported as `aac` by ffprobe and as `mp4a` by Twitch.
1) Streams in TS / Transport Stream
Playlists (M3U8) in .ts format, none has initialization segment.
The codecs found until now are only `H.264` for video and `AAC` for audio:
```
$ ffprobe -v error -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 ./twitch_1080p60/20240629T140613Z,2184776763/output.ts
aac
h264
timed_id3
aac
h264
timed_id3
```
`timed_id3` is a metadata track which can include subtitles (currently they are English AI generated and have some delay).
2) Streams in MP4
Playlists (M3U8) in .mp4 format, all have an initialization segment.
Video codecs can be: av1, hevc and h264. Audio is aac.
The stream `2223719525` contains all possible audio and video codecs currently known, depending on chosen quality, and all are in .mp4 container:
```
$ ffprobe -v error -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 ./r0dn3y_1440p60_chunked/20240814T012631Z,2223719525/output.mp4
aac
av1
$ ffprobe -v error -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 ./r0dn3y_1080p60/20240814T012631Z,2223719525/output.mp4
aac
hevc
$ ffprobe -v error -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 ./r0dn3y_720p60/20240814T012631Z,2223719525/output.mp4
aac
h264
$ ffprobe -v error -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 ./r0dn3y_480p30/20240814T012631Z,2223719525/output.mp4
aac
h264
$ ffprobe -v error -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 ./r0dn3y_audio_only/20240814T012631Z,2223719525/output.mp4
aac
```
#### 2.3.6. Notes on concatenating noncontiguous segments
1) Streams in TS
Example: stream with 761 segments: (0-760).ts
Any .ts segment / part is independent on its own, contains all the headers needed to be played in a standalone way: `mpv 600.ts`.
They can also be concatenated in any contiguous/noncontiguous combination, and the file structure is still correct:
```
cat 150.ts 600.ts 601.ts 602.ts 755.ts > clip.ts
```
Although each segment duration is detected correctly by `ffprobe`, the concatenated clip (or even full output.ts sometimes) may not display the correct duration:
```
$ for file in 150.ts 600.ts 601.ts 602.ts 755.ts clip.ts output.ts; do
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$file" ; done
10.000667
10.000667
10.000667
10.000667
10.000667
6060.000667
7602.381667
```
Fixed easily by repackaging / reparsing the streams without encoding (even to .ts again, no need for .mp4):
```
$ ffmpeg -hide_banner -i clip.ts -c copy clip_reparsed.ts
$ ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 clip_reparsed.ts
50.001000
```
2) Streams in MP4
Example: stream with 482 segments: init-0.mp4 + (0-480).mp4
```
$ ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 output.mp4
4867.614333
$ ffmpeg -loglevel warning -i output.mp4 -c copy output_reparsed.mp4
$ ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 output_reparsed.mp4
4803.564666
$ head -n 20 index-dvr.m3u8
#EXTM3U
#CUSTOM-DOWNLOADER:TwitchDownloaderCLI2,Twitch.tv,chunked,https://d2nvs31859zcd8.cloudfront.net/da2e2817cb4121d6c5d9_djclancy_44845615675_1727048740/chunked/index-dvr.m3u8
#EXT-X-VERSION:6
#EXT-X-TARGETDURATION:10
#ID3-EQUIV-TDTG:2024-09-23T01:05:52
#EXT-X-PLAYLIST-TYPE:EVENT
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TWITCH-ELAPSED-SECS:0.000
#EXT-X-TWITCH-TOTAL-SECS:4803.534
#EXT-X-MAP:URI="init-0.mp4"
#EXTINF:10.000,
0.mp4
#EXTINF:10.000,
1.mp4
#EXTINF:10.000,
2.mp4
#EXTINF:10.000,
3.mp4
#EXTINF:10.000,
4.mp4
$ tail -n 5 index-dvr.m3u8
#EXTINF:10.000,
479.mp4
#EXTINF:3.534,
480.mp4
#EXT-X-ENDLIST
```
2.1) Concatenating any number of segments WITHOUT the initialization file will render an invalid (totally broken) clip, and can't be fixed, not even by re-encoding it (even less in copy mode):
```
$ cat 300.mp4 350.mp4 400.mp4 > clip.mp4
$ ffprobe -hide_banner -i clip.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x13ce067e0] could not find corresponding trex (id 1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x13ce067e0] could not find corresponding track id 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x13ce067e0] trun track id unknown, no tfhd was found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x13ce067e0] error reading header
clip.mp4: Invalid data found when processing input
$ ffmpeg -hide_banner -i clip.mp4 clip_recoded.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x155e06d30] could not find corresponding trex (id 1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x155e06d30] could not find corresponding track id 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x155e06d30] trun track id unknown, no tfhd was found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x155e06d30] error reading header
[in#0 @ 0x60000152ca00] Error opening input: Invalid data found when processing input
Error opening input file clip.mp4.
Error opening input files: Invalid data found when processing input
```
2.2) Concatenating WITH the init file with noncontiguous segments, renders also the clip damaged / partially broken, and can't be fixed either not even by re-encoding:
```
$ cat init-0.mp4 100.mp4 250.mp4 400.mp4 > clip_init.mp4
$ ffmpeg -hide_banner -i clip_init.mp4 clip_init_recoded.mp4
$ ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 clip_init_recoded.mp4
3010.016000
```
Here in `clip_init_recoded.mp4`:
The 3 segments are still present and playable in the generated file, separated each other the time they were in output.mp4.
The timeline before the first one and the after the last one is trimmed.
The timeline in between in unplayable, so the segments are not placed adjacent.
Ffmpeg parameters `-fflags +genpts+igndts` don't have any effect.
Payload segments in `output.mp4` (the init file is omitted since has no real data):
```
-------100.mp4----250.mp4----400.mp4-------
```
Payload segments in `clip_init_recoded.mp4`:
```
100.mp4----250.mp4----400.mp4
```
The concatenated clip which is not re-encoded,` clip_init.mp4`, has also the segments separated each other the same time as in output.mp4, except that in this case the time is trimmed only after the last segment:
```
$ ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 clip_init.mp4
4074.126000
```
```
-------100.mp4----250.mp4----400.mp4
```
This happens because these segments retain their original timestamps, causing gaps when concatenated.
Trying to use other concatenation methods or filters fails directly, because treats each input file as if had its own header:
```
$ cat list.txt
file 'init-0.mp4'
file '100.mp4'
file '250.mp4'
file '400.mp4'
$ ffmpeg -f concat -safe 0 -i list.txt -loglevel warning -c copy clip_list_reparsed.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x14ee06bc0] could not find corresponding trex (id 1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x14ee06bc0] could not find corresponding track id 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x14ee06bc0] trun track id unknown, no tfhd was found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x14ee06bc0] error reading header
[concat @ 0x14ee06960] Impossible to open '100.mp4'
[mp4 @ 0x14ef050f0] track 1: codec frame size is not set
[in#0/concat @ 0x600000474500] Error during demuxing: Input/output error
[out#0/mp4 @ 0x600000d70000] Output file is empty, nothing was encoded
$ ffmpeg -f concat -safe 0 -i list.txt -loglevel warning clip_list_recoded.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x156004400] could not find corresponding trex (id 1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x156004400] could not find corresponding track id 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x156004400] trun track id unknown, no tfhd was found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x156004400] error reading header
[concat @ 0x156004080] Impossible to open '100.mp4'
[in#0/concat @ 0x600003b48000] Error during demuxing: Input/output error
[aost#0:1/aac @ 0x154749ef0] No filtered frames for output stream, trying to initialize anyway.
[vost#0:0/libx264 @ 0x154606d10] No filtered frames for output stream, trying to initialize anyway.
[out#0/mp4 @ 0x600003240480] Output file is empty, nothing was encoded(check -ss / -t / -frames parameters if used)
$ ffmpeg -i init-0.mp4 -i 100.mp4 -i 250.mp4 -i 400.mp4 \
-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0][3:v:0][3:a:0]concat=n=4:v=1:a=1[outv][outa]" \
-map "[outv]" -map "[outa]" -c copy -loglevel warning clip_filter_remuxed.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x15480a350] could not find corresponding trex (id 1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x15480a350] could not find corresponding track id 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x15480a350] trun track id unknown, no tfhd was found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x15480a350] error reading header
[in#1 @ 0x600003990400] Error opening input: Invalid data found when processing input
Error opening input file 100.mp4.
Error opening input files: Invalid data found when processing input
$ ffmpeg -i init-0.mp4 -i 100.mp4 -i 250.mp4 -i 400.mp4 \
-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0][3:v:0][3:a:0]concat=n=4:v=1:a=1[outv][outa]" \
-map "[outv]" -map "[outa]" -loglevel warning clip_filter_recoded.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x14ae0d0f0] could not find corresponding trex (id 1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x14ae0d0f0] could not find corresponding track id 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x14ae0d0f0] trun track id unknown, no tfhd was found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x14ae0d0f0] error reading header
[in#1 @ 0x60000103c800] Error opening input: Invalid data found when processing input
Error opening input file 100.mp4.
Error opening input files: Invalid data found when processing input
```
The only correct way to extract portions of the whole stream, when there's an initialization file, is to have first generated `output.mp4`, which consists in the raw concatenation of all the segments in the correct order with the initialization file.
This is done by `download-twitch-video.sh`.
Then each segment / set of contiguous segments, can be extracted being self-contained, by splitting correctly by keyframes (full re-encoding is required otherwise):
```
$ ffmpeg -ss 1000 -t 10 -i output.mp4 -c copy -avoid_negative_ts 1 segment100.mp4
$ ffmpeg -ss 2500 -t 10 -i output.mp4 -c copy -avoid_negative_ts 1 segment250.mp4
$ ffmpeg -ss 4000 -t 10 -i output.mp4 -c copy -avoid_negative_ts 1 segment400.mp4
$ cat segments.txt
file 'segment100.mp4'
file 'segment250.mp4'
file 'segment400.mp4'
$ ffmpeg -f concat -safe 0 -i segments.txt -c copy -loglevel warning clip_segments_concat.mp4
$ ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 clip_segments_concat.mp4
30.267998
```
Here they are joined using the concate demuxer, and according to index-dvr.m3u8 each segment except last one durates 10 seconds.
There are still some frames dropped during playback, probably due to cut not done exactly by keyframe, or ffmpeg being buggy, but all the content audio+video is there and plays perfectly fine (let video play entirely):
```
$ mpv --geometry=400x320+20+30 clip_segments_concat.mp4
● Video --vid=1 (hevc 2560x1440 59.7716 fps) [default]
● Audio --aid=1 (aac 2ch 48000 Hz 159 kbps) [default]
AO: [coreaudio] 48000Hz stereo 2ch floatp
VO: [gpu] 2560x1440 yuv420p
AV: 00:00:30 / 00:00:30 (100%) A-V: 0.000 ct: 0.091 Dropped: 2
Exiting... (End of file)
```
Another way is to use a filter, but filtering and streamcopy cannot be used together:
```
ffmpeg -i output.mp4 -filter_complex "\
[0:v]trim=start=1000:end=1010,setpts=PTS-STARTPTS[v0]; \
[0:a]atrim=start=1000:end=1010,asetpts=PTS-STARTPTS[a0]; \
[0:v]trim=start=2500:end=2510,setpts=PTS-STARTPTS[v1]; \
[0:a]atrim=start=2500:end=2510,asetpts=PTS-STARTPTS[a1]; \
[0:v]trim=start=4000:end=4010,setpts=PTS-STARTPTS[v2]; \
[0:a]atrim=start=4000:end=4010,asetpts=PTS-STARTPTS[a2]; \
[v0][a0][v1][a1][v2][a2]concat=n=3:v=1:a=1[outv][outa]" \
-map "[outv]" -map "[outa]" -c copy -loglevel warning clip_segments_filter_reparsed.mp4
[vost#0:0 @ 0x123e0c5d0] Streamcopy requested for output stream fed from a complex filtergraph. Filtering and streamcopy cannot be used together.
Error opening output file clip_segments_filter_reparsed.mp4.
Error opening output files: Invalid argument
```
Performing a re-encoding with filter (from output.mp4), will usually avoid skipped/dropped frames during playback (let video play entirely). The framerate must be supplied manually:
```
ffmpeg -i output.mp4 -filter_complex "\
[0:v]trim=start=1000:end=1010,setpts=PTS-STARTPTS[v0]; \
[0:a]atrim=start=1000:end=1010,asetpts=PTS-STARTPTS[a0]; \
[0:v]trim=start=2500:end=2510,setpts=PTS-STARTPTS[v1]; \
[0:a]atrim=start=2500:end=2510,asetpts=PTS-STARTPTS[a1]; \
[0:v]trim=start=4000:end=4010,setpts=PTS-STARTPTS[v2]; \
[0:a]atrim=start=4000:end=4010,asetpts=PTS-STARTPTS[a2]; \
[v0][a0][v1][a1][v2][a2]concat=n=3:v=1:a=1[outv][outa]" \
-map "[outv]" -map "[outa]" -c:v libx264 -crf 23 -preset slower \
-c:a aac -b:a 192k -r 60 clip_segments_filter_recoded.mp4
```
```
$ mpv clip_segments_filter_recoded.mp4
● Video --vid=1 (h264 2560x1440 60 fps) [default]
● Audio --aid=1 (aac 2ch 48000 Hz 193 kbps) [default]
AO: [coreaudio] 48000Hz stereo 2ch floatp
VO: [gpu] 2560x1440 yuv420p
AV: 00:00:29 / 00:00:30 (100%) A-V: 0.000
Exiting... (End of file)
```
Once re-coding, cutting points don't have to match a keyframe, and the resulting file plays perfectly with no framedrop:
```
$ time ffmpeg -i output.mp4 -filter_complex "\
[0:v]trim=start=1002:end=1016,setpts=PTS-STARTPTS[v0]; \
[0:a]atrim=start=1002:end=1016,asetpts=PTS-STARTPTS[a0]; \
[0:v]trim=start=2499:end=2512,setpts=PTS-STARTPTS[v1]; \
[0:a]atrim=start=2499:end=2512,asetpts=PTS-STARTPTS[a1]; \
[0:v]trim=start=4002:end=4013,setpts=PTS-STARTPTS[v2]; \
[0:a]atrim=start=4002:end=4013,asetpts=PTS-STARTPTS[a2]; \
[v0][a0][v1][a1][v2][a2]concat=n=3:v=1:a=1[outv][outa]" \
-map "[outv]" -map "[outa]" -c:v libx264 -crf 23 -preset slower \
-c:a aac -b:a 192k -r 60 -loglevel warning clip_unaligned.mp4
real 9m0,339s
user 56m42,536s
sys 0m59,775s
```
```
$ mpv --geometry=400x320+20+30 clip_unaligned.mp4
● Video --vid=1 (h264 2560x1440 60 fps) [default]
● Audio --aid=1 (aac 2ch 48000 Hz 193 kbps) [default]
AO: [coreaudio] 48000Hz stereo 2ch floatp
VO: [gpu] 2560x1440 yuv420p
AV: 00:00:37 / 00:00:38 (100%) A-V: 0.000
Exiting... (End of file)
```
3) Conclusion
It's almost always mandatory to reparse all output.(ts|mp4), so the duration and other parameters are re-adjusted correctly. Also for noncontiguous .ts (not noncontiguous .mp4 which will be broken). This fixes warnings about `corrupt input packet in stream 1` and `timestamp discontinuity (stream id=256): 4490005333, new offset= -4490005333` which are very common in TS streams when playing with `mpv`.
If there's an initialization file, the segments must be concatenated all of them and in the m3u8 playlist order, and with the initialization file, to generate the well-known output.mp4. To extract clips or portions it must be done from that file which must be generated beforehand.
### 2.4. download-all.sh
This script command has the following format, so 9 arguments must be provided:
```
sh download-all.sh
<OAUTH>
<csv-file>
<shell>
<[--select-stream=]skip|number>
<[--find-stream=]skip|best/chunked/source|worst|audio|high|medium|low|mobile|[^][Nx]Np[N][$]/[^]NxN[p[N]][$]/[^]pN[$]/^p/p$]>
<[--keep-segments=]yes|no>
<[--concurrent-downloads=]number>
<[--loop-accordingly=]yes|no>
<[--decode-html-entities=]true/yes|false/no>
```
Reads all the lines in the `<csv-file>` sequentially, and for each line calls the script `download-twitch-video.sh` (with arguments alike), so downloads only 1 ID at a time.
If a file named `stop` exists in current directory, it will only run a full iteration, after reading the first line of the csv file. If the `stop` file is created during the process, it will exit after all operations for current ID are completed. This can be used to download only the first video or to exit the script as soon as possible without leaving incomplete operations.
The `<shell>` is the command or full path of the shell that will be used to call the script `download-twitch-video.sh` on each ID iteration (and consequently `twitchdownloader-shell.sh` too when used). It calls the `which` command to find the full path of the given shell, allowing to use the same or different shell for each script. This has to be done, because the scripts called by another script don't use the same shell automatically, so it either has to be hard coded, try to detect it with commands (unreliable), or be provided as an argument:
```
/opt/homebrew/bin/bash <csv-file> download-all.sh ksh <..>
dash download-all.sh <csv-file> /usr/bin/zsh <..>
```
All the arguments are passed directly to `download-twitch-video.sh`.
### 2.5. convert-all.sh
This script command has the following format, so 1 argument must be provided:
```
sh convert-all.sh <csv-file>
```
Looks for all the directories containing the IDs listed in the csv file, sequentially. Inside each directory, uses ffmpeg in copy mode, to generate `$DATE_$ID.mp4` (the format explained before) from `output.(ts|mp4)` (only one output of these two can be present), keeping both input and output files. Also adds to each mp4 the corresponding `metadata.txt` as metadata channel. The reason to use an underscore `_`, is that the comma `,` is not allowed in URLs, which would be a problem later on when uploading the video to any server. The hyphen-minus `-` is not accepted either by some servers like Odysee.
If the `$DATE_$ID.mp4` file already exists, just skips that conversion. After each conversion, adds the `SHA-384` of the mp4 file to `hash-sha384.txt`, or replaces the existing one. If there's any conversion error will exit.
Some IDs like 637388605 have no audio, so the copy mode skips it. This can leave with some IDs having audio and others not, but this is how this comes from Twitch. It has to be managed later on.
If a file named `stop` exists in current directory, will stop after 1 iteration. If there's any conversion warning or error, will not stop because of that. Each conversion generates its own `ffmpeg-convert-error.txt` log file for that ID, which will be empty if there were no conversion errors according to ffmpeg.
It's important to use always the latest or most stable version of `ffmpeg`, since there are known bugs which very infrequently affect the conversion. The mp4 files should be checked for playback errors before deleting the original output files, using for example the script `check-all.sh`.
### 2.6. check-all.sh
This script command has the following format, so 1 argument must be provided:
```
sh check-all.sh <csv-file>
```
Purpose: detect if all given `$DATE_$ID.mp4` files have any potential playback issue, acording to ffmpeg. This is safeguard step if `output.(ts|mp4)` files are going to be deleted, or be shared or uploaded to any streaming service like Odysee. The section [3.9.2. Conversion statistics](#392-conversion-statistics) explains why `output.ts` is not checked also or instead.
Looks for all the folders containing the IDs listed in the csv file. For each folder, if the file has not been checked before, uses ffmpeg in re-encode mode to generate another mp4, file which is discarded to `/dev/null`. The ffmpeg loglevel mode is `warning` and stderr is saved to file `ffmpeg-check-error.txt`.
The loglevel warning implies that some `ffmpeg-check-error.txt` files may not be empty, and still the video file be in perfect shape. If the multimedia file has no audio there isn't any further message because of that. For example, these IDs generate some warning log, but play perfectly fine all the way through with mpv:
20200531T145012Z,637343547 (avoid the warning by adding the filter `-vf "zscale=rangein=full:range=limited"`):
```
[swscaler @ 0x76a988012c40] deprecated pixel format used, make sure you did set range correctly
```
20200531T155805Z,637388605 (avoid the warning by not setting audio bitrate `-b:a xx` and making the audio track optional `-map 0:a?`, for those IDs that don't have audio):
```
[out#0/mp4 @ 0x56a526b29cc0] Codec AVOption b (set bitrate (in bits/s)) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream.
```
These messages don't appear in loglevel error even without the command modification. The "deprecated format" message should be classified as info and not warning, since no format gets deprecated ever, it's how people did things back in the day. No supporting it is different from being deprecated, specially when reading from and not encoding to.
If a file named `stop` exists in current directory, will stop after 1 iteration. If there's any conversion warning or error, will still continue throughout the csv-file. If the `ffmpeg-check-error.txt` file already exists will skip the check of that ID.
### 2.7. recode-defective-all.sh
This script command has the following format, so 1 argument must be provided:
```
sh recode-defective-all.sh <csv-file> <[--exit-onconverterror=]yes|no>
```
Purpose: fully re-encode those IDs which `$DATE_$ID.mp4` files which have errors according to the previous script. Since the copy mode didn't seem to work, the full reencoding is the second attempt.
Looks for all the folders containing the IDs listed in the csv file. For each folder, if the file `ffmpeg-check-error.txt` has a size bigger than 0 (means `check-all.sh` detected issues), does the following:
1. Read audio and video bitrates from the `$DATE_$ID.mp4` file, which must exist previously. The bitrates could be read from output.(ts|mp4) files, but that's a chore (see [2.3.2. Reading the technical makeup of multimedia files](#232-reading-the-technical-makeup-of-multimedia-files)).
2. If `output.(ts|mp4)` exists, use it as input file, otherwise use the `$DATE_$ID.mp4` file.
3. Fully encode the input file to a temporary file `nometadata.mp4`, without metadata in it, using the same audio/video codecs from the input. The supported audio codec is aac, and the supported video codecs are: h264, hevc and av1. Detects if the input file has audio or not and keeps output the same way. Some codecs like AV1 can be very slow and should be done in hardware.
4. Rename the existing `$DATE_$ID.mp4` to `$DATE_$ID.mp4.bak.n`, being n a number for a filename that does not exist already.
5. Apply the metadata to the temporary file and save it as the mp4 with the same pattern as the original file did: `$DATE_$ID.mp4`.
6. Delete the temporary file. The `.bak.n` file is not deleted.
7. Check the newly created file for errors, like `check-all.sh` does.
8. Update the hash in `hash-sha384.txt`.
9. During the process check for any errors. Exit if there's any, unless `--exit-onconverterror=no`, which will prevent exiting if the first conversion has errors. The rest of the checks are not affected. This option is useful because some conversions will inevitably generate errors due to input file condition, but still the output file could have no issues from the playback standpoint (some content may result damaged or lost still during conversion if the input file was bad enough).
Example of `ffmpeg-convert-error.txt`, where the first/main conversion always generates error (input file is defective and has some content lost), but the output file has no playback issues after recoding:
```
[h264 @ 0x151f0df20] reference picture missing during reorder
[h264 @ 0x151f0df20] Missing reference picture, default is 65740
[h264 @ 0x151f21930] illegal short term buffer state detected
[h264 @ 0x151f29fd0] mmco: unref short failure
[h264 @ 0x151f4c5d0] mmco: unref short failure
```
Most common errors found by FFmpeg 7.0 in `ffmpeg-check-error.txt`:
Case 1:
```
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x156e05d20] st: 1 edit list: 1 Missing key frame while searching for timestamp: 3041
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x156e05d20] st: 1 edit list 1 Cannot find an index entry before timestamp: 3041.
```
Case 2:
```
[vost#0:0/libx264 @ 0x139706960] More than 1000 frames duplicated
```
### 2.8. combine-multiple-ids.sh
This script command has the following format, so 9 arguments must be provided:
```
sh combine-multiple-ids.sh <csv-file-tocombine> <[--match-title=]yes|no> <[--ask-titlesdiff=]yes|no> <[--add-chapters=]yes|no>
<[--bitrate-mode=]min|max|avg> <[--resolution-mode=]min|max|same> <[--framerate-mode=]min|max|same> <[--backup-importantfiles=]yes|no> <[--continue-onoutputerror=]yes|no>
```
The purpose of this script is to concatenate 2 or more video IDs, by re-encoding them into a new video file (can be slow).
Description of each argument. All of them are mandatory and in the same order as described, since are positional:
* `<csv-file-tocombine>`: the text file in csv format, listing the IDs in the order they will be concatenated. Can be made of lines extracted from the csv generated by `download-twitch-video.sh` or `download-all.sh`. Only the commas and the IDs are needed per line inside the csv, like the rest of the scripts.
* `<[--match-title=]yes|no>`: when `yes`, will only continue if all the videos have the same title (removing the ID) in the metadata file. This script is primarily designed for those "single" `Twitch streams which are split into differents IDs` for different reasons: due to stream interruption and re-connect; made on purpose by the streamer instead of using chapters; or avoid big files, but the different IDs are conceptually the same ID, so all codec parameters should be the same. If the titles don't match when this option is set, check then `--ask-titlesdiff`. When `no`, the first title will be used as is for the output title.
* `<[--ask-titlesdiff=]yes|no>`: if `--match-title` is set to yes, but titles don't match, either ask user for the new title in console (option `yes`) or exit program with exit code (option `no`).
* `<[--add-chapters=]yes|no>`: when `yes`, reads all the chapters from all the metadata files, and shifts all the timestamps accordingly to the duration of the output file, which is the sum of the duration of all the input files. Also with this option, the duration of each `$DATE_$ID.mp4` is checked to match the duration of the chapters in its own metadata file. When the option is `no`, all chapters are discarded and no duration is checked.
* `<[--bitrate-mode=]min|max|avg>`: use for the output file the lowest, highest or average bitrate of the input files (audio and video separately). The average does not account for the duration of each segment, it's a simple arithmetic among all bitrates.
* `<[--resolution-mode=]min|max|same>`: when `same`, will continue only if all the video files have the same resolution. Otherwise will use the highest or the lowest in terms of pixel count, for all videos (no cropping or letterboxing).
* `<[--framerate-mode=]min|max|same>`: when `same`, will continue only if all the video files have the same framerate. Otherwise will use the highest or the lowest in floating point number for all videos.
* `<[--backup-importantfiles=]yes|no>`: when `yes`, copy all `metadata.txt`, `index-dvr.m3u8`, `parts.txt` and `output.(ts|mp4)` files (but not the mp4s) to destination directory, with new format `{filename}_$N.{extension}`, where `$N` is the index number of the ID in the csv file (starting at 1). Also copies its hashes, does not recalculate them. Avoids duplicates. The reason to copy .ts and not .mp4, is because .ts files may include data streams, which are not easy to be parsed and adapted to the MP4 format, so it's just for archiving purposes. If the any of the source videos are in mp4 format, and they contain metadata, such metadata is supposed to be in their corresponding metadata.txt, to avoid having to read the mp4 each time.
* `<[--continue-onoutputerror=]yes|no>`: continue until the end of the script or exit immediately if the output file has any ffmpeg warning/error. Sometimes the log file `ffmpeg-check-error.txt` is not empty because contains only warning information, that does not relate to any issue in the file, for example: `[swscaler @ 0x1194d0000] No accelerated colorspace conversion found from yuv420p to bgr24.`. These cases are best ignored.
The mp4 file(s) with format `$DATE_$ID.mp4` are mandatory, to read from them resolutions, bitrates and duration, since reading those values from a .ts file is a chore (see [2.3.2. Reading the technical makeup of multimedia files](#232-reading-the-technical-makeup-of-multimedia-files)). However, for those `output.(ts|mp4)` files that exist, such will be used as input files for re-encoding, instead of the `$DATE_$ID.mp4`.
Both audio and video tracks are mandatory for all input files. When there's no audio: `bitrate=null`.
This script recodes all input files, to another new files into the destination directory, with a unified format for compatibility. Then concatenates them using the [Ffmpeg concat demuxer](https://trac.ffmpeg.org/wiki/Concatenate#demuxer) and adds the metadata. This have to be done because, combining different audio/video codecs and files with/without audio track, may generate defective or partially unplayable output files.
Recoding each input file separately also helps fix plausible audio/video sync issues, that if don't appear by each file alone, could appear if concatenating without recoding (provided that specs are already the same and recoding wouldn't be necessary apparently).
The audio codec for the output(s) is `AAC` and the video codec is `H.264`, with preset slow. The same bitrates are kept for each individual part.
Each time the script is run, will save the new files to a new directory in `pool-combined-ids/$CURRENTDATE/$ORIGINALDATE,$ID/`, where the current date is the system date when the script is run, and the original date is the airing date / creation date of the first ID (the directory will have the same name like if the first ID folder were copied over). This allows to have duplicates in several combinations having the first segment as the first ID, and the resulting video takes the airing date and ID from the first video in the list.
The resulting metadata and video files include information of all the IDs (with its creation date), and the descriptions also when are present. The goal is to preserve all the metadata except the data streams from TS files, which require intricate parsing.
None of the input files is modified or removed. This script is not designed to re-combine videos that have already been combined, metadata may get messed up. It will work but only the first ID and date of each one will be kept.
### 2.9. overlay-chat.sh
This script command has the following format, so 9 arguments must be provided:
```
sh overlay-chat.sh
<csv-file>
<[--ranges-position=]RANGE-RANGE-POSITION[,RANGE-RANGE-POSITION …]>
<[--allow-overlap=]yes|no>
<[--chats-location=]download|local>
<[--cleanup-tempdir=]yes|no>
<[--check-output=]yes|no>
<[--move-output-back=]yes|no>
<[--recode-input-file=]yes|no>
<[--require-input-checked=]yes|no>
<[--recode-audio-overlay=]yes|no>
<shell>
```
The purpose of this script is to overlay the Twitch chat over the corresponding video of the same ID. Since involves recoding several times can be very slow.
Description of each argument. All of them are mandatory and in the same order as described, since are positional:
* `<csv-file>`: the text file in csv format, listing the ID(s). Can be made of lines extracted from the csv generated by `download-twitch-video.sh` or `download-all.sh`. Only the commas and the IDs are needed per line inside the csv, like the rest of the scripts. The videos for the IDs must exist previously.
* `<[--ranges-position=]RANGE-RANGE-POSITION[,RANGE-RANGE-POSITION …]>`: set the expression(s), separated by comma, for each chat overlay. Each expression is like this: RANGE-RANGE-POSITION. Each range has the pattern: `0|inf[inite|[[hh:]mm]ss[.ms]|.ms`. The values can exceed the clock allowed values (mm>59 and ss>59), and the milliseconds can have any number of decimals. This doesn't matter because the values are converted to `SS.ms` with 3 decimal places. 0 means from the start and infinite until the video ends. The positions can only be right, left and center (where to place horizontally the chat over the main video). The main video is expected to be 1080p, the overlaid chat is 480x780, and the margin is 100 pixels to side borders and unless it's in the center. The chat is centered vertically. If the input video were of different resolution, the chat dimensions and margins would require to be adjusted for each case to fit in. Two or more overlays can be displayed at the same time also (and not only in the same video), if the timestamps overlap, and the positions are different: `0-inf-left,10:0-65:140-center,1800-inf-right`.
* `<[--allow-overlap=]yes|no>`: if the overlays can overlap each other, so there's more than one chat being overlaid in the same time range of the timeline. This is useful to place the chat in various positions at the same time, but can clutter too much the scene.
* `<[--chats-location=]download|local>`: where to get the `chat.json` file for each ID: either existing alongside with video in the `*,$ID` directory, or download from Twitch. Some images / pictures / gifs / emojis / emotes may still need to be downloaded from the internet when rendering the chat, even if the json file is already present (depending if all the content is stored in the json or not as data themselves or just in URL format). Media removed from the servers may be missing in the final video.
* `<[--cleanup-tempdir=]yes|no>`: remove or not each temporary directory created inside `pool-combined-ids` dir, each time the script is run and for each ID. Option `no` is recommended to keep all files, unless `--move-output-back=` is set to `yes` (otherwise is like running in test mode).
* `<[--check-output=]yes|no>`: if to check the final `$DATE_$ID.mp4` for each ID for errors, with ffmpeg.
* `<[--move-output-back=]yes|no>`: when `yes` is selected: moved the output `$DATE_$ID.mp4` with the chat already burnt-in, to the original directory where it came from (`*,$ID`); update the new hash; and rename previously the existing `$DATE_$ID.mp4` to `$DATE_$ID.mp4.bak.n`, being n a number for a filename that does not exist already. This option leaves the original directory like if nothing happened but with the chat overlaid one or multiple times in one or more positions (multiple runs over the same csv-file).
* `<[--recode-input-file=]yes|no>`: whether to recode the input `$DATE_$ID.mp4` file or not, before overlaying the chat. If the file has not been re-encoded before (`convert-all.sh` does not re-encode so doesn't count), the overlaid chat can get out of sync more and more increasingly as the timeline progresses. This is likely due to some bug or how the input file is arranged internally. So the strongly advised value here is `yes`, unless the id has been re-encoded already (like with `recode-defective-all.sh`, or manually for any other reason). Both audio and video should be recoded to get a proper interleaving.
* `<[--require-input-checked=]yes|no>`: whether or not the file `ffmpeg-check-error.txt` will be required to exist and be empty. This is just to prevent taking input files which can be potentially broken so they are taken care before.
* `<[--recode-audio-overlay=]yes|no>`: whether or not to recode the audio stream of the input file during the overlay process. This is because recoding audio with aac codec in ffmpeg can introduce audio artifacts, so is best avoided when unnecessary. If the audio was already recoded (`--recode-input-file=yes` or `recode-defective-all.sh`) or other means then it's best to select no, and recode only the video which is mandatory.
* `<shell>`: the shell command or path to which will be used to call the scripts `monitor-ffmpeg-progress.sh` and `detect-ffmpeg-encoder.sh`.
Each time the script is run, will save all the temporary files to a new directory in `pool-combined-ids/$CURRENTDATE/`, where the current date is the system date when the script is run and for each ID, so running the script several times, over the same csv-file, prevents overwritting of temporary files.
Before actually overlaying the chats, each ID is optionally recoded to the temporary directory `pool-combined-ids/$CURRENTDATE/`, to fix plausible shifts between audio and video. Shifts in the chat could also occur, even of several minutes of difference in an hour (compared to input file) increasingly as the timeline moves along, so recoding fixes it in all tested cases. Shifts happen occasionally unless recoding, even if the input file already had empty `ffmpeg-convert-error.txt` and `ffmpeg-check-error.txt` files.
### 2.10. monitor-ffmpeg-progress.sh
This script command has the following format, so 3 arguments must be provided and 1 is optional:
```
sh monitor-ffmpeg-progress.sh
<reference-file>
<growing-file>
<[--sleep-period=]VALUE>
[[--pid=]ffmpeg-PID]
```
The purpose of this script is to monitor the second (growing) file against the first (reference) file. It displays the size in bytes of both files and the percentage with one decimal value. If the output file can't be read correctly display a waiting text. The input file is read only once.
Description of each argument. When provided, they are positional:
* `<reference-file>`: The file taken as reference for its size, read only once.
* `<growing-file>`: The file to check periodically at sleep time interval. Its size can change in any direction or be missing, doesn't have to just grow.
* `<[--sleep-period=]VALUE>`: Every how many seconds check the size of the growing file. It supports decimals and can be also 0 (continuously).
* `[[--pid=]ffmpeg-PID]`: When a process id is provided, it keeps checking the growing file size until the process doesn't exist anymore, then exits the script. This parameter is designed to check the input and output files of any ffmpeg process while the conversion is happening, but any valid process id can be provided. This parameter is optional, and if it's not provided in will keep checking forever until the script is killed with Control+C.
### 2.11. detect-ffmpeg-encoder.sh
This script has the following format, and requires 1 argument:
```
sh detect-ffmpeg-encoder.sh <[--detection-mode=]auto|software/universal>
```
The purpose of this script is to find the best ffmpeg encoder for h264, when called by other scripts. Prints to stdout the best or selected encoder, otherwise prints `error`.
Description of each argument:
* `<[--detection-mode=]auto|software/universal>`: `software` selects the most compatible encoder, which is `libx264`. If `auto` is selected, searchs for `h264_videotoolbox` and `h264_nvenc`; if not possible prints `libx264`.
## 3. HELPER SCRIPTS
These scripts are useful for further checks and operations, and are 9 in total.
### 3.1. restore-segments.sh
This script command has the following format, so 3 arguments must be provided:
```
sh restore-segments.sh <csv-file> <[--check-hash=]none|output|parts.txt|both>
```
The purpose of this script is to divide `output.(ts|mp4)` (can only be one or will exit) back into all segments which where downloaded from Twitch.tv, same name, size and number of them.
Existing parts with overwritten.
The segments are placed next to the output file, which is not deleted. Other files are not modified.
All arguments are mandatory and in the order as described:
* `<csv-file>`: the text file in csv format, listing the ID(s) to be segmented. Only the commas and the IDs are needed per line inside the csv, like the rest of the scripts.
* `<[--check-hash=]none|output|parts.txt|both>`: whether to check or not the hashes of certain input files before splitting.
### 3.2. check-titles-online.sh
This script command has the following format, so 3 arguments must be provided:
```
sh check-titles-online.sh <csv-file> <[--service-provider=]twitchtracker> <[--extractor=]xmllint|pcre2grep> <[--max-minutes-off=]integer>
```
The purpose of this script is to compare the title of each ID, as it is stored in its metadata.txt file, against the title stored in certain websites, specialized in storing information about past Twitch streams. The list of IDs to be compared is provided by the csv file.
All arguments are mandatory and in the order as described:
* `<csv-file>`: the text file in csv format, listing the ID(s). Only the commas and the IDs are needed per line inside the csv. Will read the title stored in the metadata file inside matching directory.
* `<[--service-provider=]twitchtracker>`: the online service (website) which will be used to find past streams titles and dates. For now, only TwitchTracker is supported.
* `<[--extractor=]xmllint|pcre2grep>`: the command which will be used to extract the dates and the titles, from the html downloaded from the selected service provider. It can be `xmllint` or `pcre2grep`, but the result should be exactly the same.
* `<[--max-minutes-off=]integer>`: 0 to any whole positive number. Extracts titles in given range from the streamlist. Compares the date/time in metadata to all dates/times in the streamlist, checking from offset 0 (minutes) up to value provided, in steps of 1 and for positive and negative increments. 0 means exact minute match (current service providers don't include seconds). Since TwitchTracker is usually up to 3 minutes off, a value of 3 covers -3,-2,-1,0,1,2,3 minutes off. This allows to extract titles that are not stored accurately by the service provider, or the times are rounded up/down. A value high enough will check even different days, and since uses the `date` command, the result is exact, covering leap years and other cases. All the dates are operated in UTC.
The script operates as follows:
1. Reads line by line the csv file, to extract the ID and find the associated directory for each one. Skips missing directories.
2. Parse each metadata file to extract the original airing date/time, artist and title for each video ID.
3. Downloads the streams' html page for the specific artist/streamer from selected service provider, if hasn't been downloaded already during the current session. Saves it to each artist and service file/directory, like `tracker-twitchtracker/artist.html`. Each session is each time the script is run, so existing html files will be overwritten. This allows to work with more than one artist in the csv-file.
4. Extracts the stream list with date/time and title, to a text file (using selected extractor), next to the html file, like `tracker-twitchtracker/artist-streamlist.txt`, if hasn't been downloaded already for current session.
5. For each title in the streamlist file, unescapes certain html characters which are escaped in the html tag: `&`, `>`, `<`, `"`, `'`/`'`/`'` to `&`, `>`, `<`, `"`, `'` respectively. Does not unescape all possible cases because other characters are not escaped.
6. Compares the title in each ID's metadata file, with the title(s) of the streams in the streamlist file, within the range of time according to `max-minutes-off` parameter value. If there's more than 1 extracted title from the streamlist file, exits the script. If the title doesn't match, or there's none for that range, prints some DEBUG info, to manually help check what's the issue. Since TwitchTracker is missing some streams, with enough offset value, it may end up matching the closest stream with same title, instead of the correct one.
#### 3.2.1. Notes about TwitchTracker
[TwitchTracker](https://twitchtracker.com/) is the best tracking service provider for several reasons:
* Allows to download all the streams from all years in a single page, one for each Twitch streamer: `https://twitchtracker.com/$ARTIST/streams`.
* The single html can be downloaded with `curl` or `wget`.
* Does not require signing in with an account.
* Does not require methods like POST/GET.
* Does not require a desktop web browser to do CloudFlare or "I'm not a robot" checks.
* Does not require paying for API access.
* All the needed information is in static HTML, easy to parse.
### 3.3. rename-all-directories-dateid.sh
This script command has the following format, 1 argument is required:
```
sh rename-all-directories-dateid.sh <run>
```
Description: this script renames any directory in `.` not named "TMP" ("TMP/") to the `DATE,ID` format according to each `metadata.txt`.
Explanation of arguments:
* `<run>`: runs the renaming loop. This argument is just there to avoid running the script when there are no arguments (print usage in that case).
Lists all the directories (folders) in current directory only (depth 1, no recursive), and for each one (except "TMP/"):
* If the file `metadata.txt` exists inside, extract the video ID and date/time of first aired / created at.
* If the previous condition is met, and the DATE and ID values are none empty, and the directory's name doesn't match the format `$DATE,$ID/` already, rename the selected directory.
### 3.4. update-dates-in-csv.sh
This script command has the following format, so 1 argument must be provided:
```
sh update-dates-in-csv.sh <csv-file>
```
* Reads all the lines in the csv file, and for each:
* If the date/time field is empty (string before the comma),
* Read the directory's name in working dir ending with same ID
* Replace the current line in the csv with the directory's name as is in the filesystem
This script takes for granted that the script `rename-all-directories-dateid.sh` has been executed before, so the directories have well formatted names.
### 3.5. extract-ids-thumbs.py
This `python3` script parses given html and json file(s).
Reads all the video IDs and its associated thumbnail location from all input files.
Prints to each line: the ID of the video followed by relative/absolute local path or URL of its thumbnail. Both separated by a comma.
Can also place line numbers.
Results can be saved to csv text file instead of printing them to stdout.
This script has the following format:
```
extract-ids-thumbs.py [-h|--help] [-c|--check] [-m|--html <html-file> ...] [-j|--json <json-file> ...] [-o|--output <output-file>] [-s|--sort] [-f|--fullhd]
```
The arguments are:
* No argument: print the custom help (version and usage).
* `-h|--help` prints the automatic help generated by Python.
* `-c|--check` checks for `BeautifulSoup4` and `json` modules, with exit code and info messages.
* `--m|--html` must be followed by the html filename(s) to parse.
* `-j|--json` must be followed by the json filename(s) to parse.
* `-o|--output <output-file>` must be followed by the filename to save the results as text file (csv formatted).
* `-s|--sort` sorts the output by video ID before printing or saving.
* `-f|--fullhd` replaces lower resolution remote URLs (80x45,320x180,640x360) with 'thumb0-1920x1080.jpg'.
* `-n|--linenumbers` places a line number (starting at 1) and a comma, before each line.
Example:
```
$ if python3 extract-ids-thumbs.py -c >/dev/null ; then python3 extract-ids-thumbs.py -m *.html -j *.json -s -f -n ; fi
```
#### 3.5.1. Externally managed environments
This script requires the [beautifulsoup4](https://pypi.org/project/beautifulsoup4/) (bs4) library/module, which is only compatible with Python version 3. Depending how `python3` (and `pip3`) was installed, the module can be installed directly with pip3 or not. In some systems, using `pip` instead works, because is a link to `pip3`, it's not that the module is compatible with Python 2 (in such case wouldn't work).
1- If python3 was installed in a way that's not externally managed, for example using the official installer for macOS (python-3.12.2-macos11.pkg), the module can be installed (user/system)-wide, and used by any script:
```
$ pip3 install beautifulsoup4
Collecting beautifulsoup4
Using cached beautifulsoup4-4.12.3-py3-none-any.whl.metadata (3.8 kB)
Requirement already satisfied: soupsieve>1.2 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from beautifulsoup4) (2.5)
Using cached beautifulsoup4-4.12.3-py3-none-any.whl (147 kB)
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.12.3
```
The commands are installed to:
```
/usr/local/bin/pip3
/usr/local/bin/python3
```
2- However, if some package manager like [Homebrew](https://brew.sh/) is used, in this case the Homebrew packages are: `python` and `brew-pip`:
```
/opt/homebrew/bin/pip3
/opt/homebrew/bin/python3
```
The module can't be installed by pip, not even user-wide:
```
$ pip3 install bs4 --user
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
xyz, where xyz is the package you are trying to
install.
If you wish to install a non-brew-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip.
If you wish to install a non-brew packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
```
and it's not available as a brew package either:
```
$ brew search bs4
Error: No formulae or casks found for "bs4".
$ brew search beautifulsoup
Error: No formulae or casks found for "beautifulsoup".
```
The only way to get it working without breaking the system, is to create a Python virtual environment inside a folder. It simulates a clean (user/system)-wide/unmanaged install only there, and doesn't modify the rest of the system. It can be created anywhere and be used for several modules at once, or one for each module for testing:
```
$ mkdir ~/venv
$ python3 -m venv ~/venv
$ source ~/venv/bin/activate
$ pip install bs4
Collecting bs4
Downloading bs4-0.0.2-py2.py3-none-any.whl.metadata (411 bytes)
Collecting beautifulsoup4 (from bs4)
Using cached beautifulsoup4-4.12.3-py3-none-any.whl.metadata (3.8 kB)
Collecting soupsieve>1.2 (from beautifulsoup4->bs4)
Using cached soupsieve-2.5-py3-none-any.whl.metadata (4.7 kB)
Downloading bs4-0.0.2-py2.py3-none-any.whl (1.2 kB)
Using cached beautifulsoup4-4.12.3-py3-none-any.whl (147 kB)
Using cached soupsieve-2.5-py3-none-any.whl (36 kB)
Installing collected packages: soupsieve, beautifulsoup4, bs4
Successfully installed beautifulsoup4-4.12.3 bs4-0.0.2 soupsieve-2.5
```
It's not required to cd the directory.
The virtual environment has to be loaded for each new shell session.
Finally test for all required modules:
```
$ python3 extract-ids-thumbs.py -c
CSV module is available.
JSON module is available.
BeautifulSoup module is available.
```
To exit from the virtual environment to the (user/system)-wide one, just deactivate it. It doesn't delete the venv directory:
```
$ deactivate
```
Since brew.sh works itself as an isolated environment, installs also its own versions of python when other packages/casks have it as dependency, regardless if the same version of python is already installed by other means (like official packages). This can be forced not to happen, but depending on how these packages are configured (or what parts of the software are actually used) they could work or not (to automatically detect and use the other python installs).
Both official Python pkg and homebrew versions can be installed at the same time (they go to different folders in filesystem), but due to how directories take priority alongside `$PATH`, the brew commands will be listed first and called by default (also happens to any other software). Using the full path, mimics that only one version is installed:
```
/usr/local/bin/pip3 install bs4
/usr/local/bin/python3 extract-ids-thumbs.py
```
Calling the `./script.py` directly may cause issues, if the default (higher priority) command is not the right one.
Some Linux distributions, like Fedora 39, also install some individual python modules as distro packages (externally managed):
```
$ dnf search beautifulsoup4
python3-beautifulsoup4.noarch : HTML/XML parser for quick-turnaround applications like screen-scraping
$ pip3 uninstall beautifulsoup4
Found existing installation: beautifulsoup4 4.12.3
ERROR: Cannot uninstall beautifulsoup4 4.12.3, RECORD file not found. Hint: The package was installed by rpm.
```
### 3.6. extract-thumbnail-from-id.sh
This script prints the official Twitch's thumbnail remote URL or absolute local path, corresponding to a video ID, reading it from the given html or json. If there's more thumb associated to the same ID, than one decide which one is best to print. If it's a strip prints it too. Otherwise prints an error word instead so can be processed accordingly by the other 2 scripts.
This script command has the following format, so 3 arguments must be provided:
```
sh extract-thumbnail-from-id.sh <html-json-file> <video-id> <html|json>
```
Filetype is `html`:
1. Check if the video URL with the given ID exists in current html before continuing.
2. Extract the values of all the `href=` and `src=` html tags in the order they are, so both videos and thumbs are listed. Each video URL is followed by the realtive path, or URL, of some thumbnail or strip.
3. Filter in all the possible URLs/paths that contain the string `-320x180` or `-strip-`, each time the given video ID is present, since the ID can be present more than once and each thumb src goes afterwards and before the next video ID URL.
4. Depending on how many thumbs are, or what's its filename, decide any specific error return value when error is due.
5. Return URL as is or return absolute path if local. Otherwise return specific error.
Filetype is `json`:
1. Check if the video ID is present in the json file in the first place.
2. Check if the video ID has any `thumbnailUrl` at all.
3. Check if any of the thumbnail URLs ends with `-640x360.` and the extension.
4. Echo the element, or specific error word instead.
Supported extensions are png, jpg, jpeg.
#### 3.6.1. Notes about thumbnails resolutions
The JSON+LD references all thumbs as remote URLs (https://), and for `videos` can be 3 in different resolutions: `80x45`, `320x180` and `640x360`.
For `clips` the thumbnails can be `86x45`, `260x147` and `480x272` (for future reference).
The HTML file must be static/plain html, containing all needed information for offline processing (src tags for local/remote assets, or embedded ld+json with same). The thumbs can be referenced as local or remote files, and all of them with `320x180` resolution.
The thumbnail for the same video ID is the same, regardless if it's extracted from the `href` tag in the html or the `thumbnailUrl` field in the json. The only difference is the resolution of the image which correlates with filename. The base URL (path) before the filename is exactly the same when referring to it on Twitch servers for all thumbnails of the same ID. So just changing the resolution in the filename of the remote URL, any resolution up to Full HD can be retrieved, just by knowing the URL of one of them.
1080p is currently the maximum resolution offered by Twitch, both for video and thumbs. Obtaining thumbs of higher resolution works, but the pictures just have black bars around. The difference in quality is noticeable if the original thumbnail was uploaded at the highest resolution.
For example, some of these URLs were extracted from the plain html, and others from the json, so extrapolating, the last one is obtained:
```
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/451df8cd372cc7862394_twitch_39746374069_1688892304//thumb/thumb0-80x45.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/451df8cd372cc7862394_twitch_39746374069_1688892304//thumb/thumb0-320x180.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/451df8cd372cc7862394_twitch_39746374069_1688892304//thumb/thumb0-640x360.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/451df8cd372cc7862394_twitch_39746374069_1688892304//thumb/thumb0-1280x720.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/451df8cd372cc7862394_twitch_39746374069_1688892304//thumb/thumb0-1920x1080.jpg
```
#### 3.6.2. Notes about local or remote thumb URLs in plain HTML
When a video feed webpage (https://www.twitch.tv/CHANNELNAME/videos?filter=all&sort=time) is saved as html (using the `File` > `Save Page As...` included in most web browsers), sometimes the web browser does not download all remote files to a local path, so the html code still refers to the online file (and the path is still a URL like http(s)). Other times the browser does not save the additional folder at all where all other files than html should be (even when selecting to save web page complete instead of HTML only). Browser's bugs. In these cases, the thumbnail must be retrieved while it's online (and same for the video), because most Twitch.tv content is deleted after a while. In any case, usually all the remote thumbnails are downloaded and referenced locally at `320x180` resolution.
To save the web page as plain html (DOM) and keep all references to thumbnails in remote URL format (like when browsing the website), so save only the HTML page without any image:
1. Load the feed webpage in the desktop browser. Other Twitch pages should work too.
2. Login with Twitch account to access subscriber's only content. Reload in such case.
3. Scroll down slowly the webpage until all videos are loaded.
4. In `Mozilla Firefox`, open Application menu > More tools > `Web Developer Tools`: `Ctrl+Shift+I` on Windows/Linux or `⌥⌘I` on macOS.
5. In the `Inspector` tab (leftmost one, selected by default), select the row starting with `<html`. It's the second one just after `<!DOCTYPE html>`
6. Right click with the mouse > Copy > Outer HTML.
7. Paste the HTML code in any text editor and save with `.html` extension. This is DOM code, the same way as it's represented in the web browser. It's not the same as when saving using the menu, or viewing source.
8. In `Google Chrome` it's the same but with `Developer Tools`.
Now all the thumbnail URLs inside `preview-card-thumbnail__image` classes are remote, so thumb files can be retrieved from servers with higher resolution than referenced by default. This html file is called DOM.
On the other hand, saving the HTML provided by Application menu > More tools > `View Page Source` (`⌘U`, `Ctrl+U`), does not work, because there shows only the JavaScript (dynamic) code.
The third way is to use `Save Page As...`, which sometimes will have useful information and other times only JavaScript code, so must be retried several times. The advantage of this method is that it saves also the folder `*_files` where all thumbnails are relocated locally, so can be retrieved later on at any point, unlike the DOM model.
The JSON always refers to absolute remote URLs.
#### 3.6.3. Notes about thumbnail types and names
Thumbnails can be in jpg, jpeg or png format. The samples in png which were tested are really jpg files inside with the extension changed.
Some thumbnail files can be strips of 10 thumbnails stacked vertically, so the strip has 320x1800 resolution. They have the string `-strip-` in the filename, and all checked files contain the ID leading, like `ID-strip-0.jpg`. For those that are just the thumbnail, there can be several characters before the resolution (found only alphanumeric + hyphen (U+002D)) `-320x180`, and before the extension (parentheses, numbers, underscore).
JSON files only have remote thumbnails `http(s)`, and no strips.
HTML files can contain local or remote thumbnails, except when it's a strip which all cases tested are remote.
Some examples of local and remote paths for thumbnails and strips are,
in JSON (thumbnailUrl):
```
https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/ecab313960e96e506063_fauxre_52802182973_1735419012/thumb/custom-a5d843d3-dc0b-4645-b546-e5551a6febda-80x45.png
https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/ecab313960e96e506063_fauxre_52802182973_1735419012/thumb/custom-a5d843d3-dc0b-4645-b546-e5551a6febda-320x180.png
https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/ecab313960e96e506063_fauxre_52802182973_1735419012/thumb/custom-a5d843d3-dc0b-4645-b546-e5551a6febda-640x360.png
https://static-cdn.jtvnw.net/cf_vods/d1m7jfoe9zdc1j/ee66ecf572ae467d2e07_fauxre_52770852365_1735146030//thumb/thumb0-80x45.jpg
https://static-cdn.jtvnw.net/cf_vods/d1m7jfoe9zdc1j/ee66ecf572ae467d2e07_fauxre_52770852365_1735146030//thumb/thumb0-320x180.jpg
https://static-cdn.jtvnw.net/cf_vods/d1m7jfoe9zdc1j/ee66ecf572ae467d2e07_fauxre_52770852365_1735146030//thumb/thumb0-640x360.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/c76a0aab2f2e007f59ff_danucd_43459174920_1735388146//thumb/thumb0-80x45.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/c76a0aab2f2e007f59ff_danucd_43459174920_1735388146//thumb/thumb0-320x180.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/c76a0aab2f2e007f59ff_danucd_43459174920_1735388146//thumb/thumb0-640x360.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/5b5e8d22e0bcd81d43f2_danucd_54037775427_9349002654//thumb/thumb2337379155-80x45.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/5b5e8d22e0bcd81d43f2_danucd_54037775427_9349002654//thumb/thumb2337379155-320x180.jpg
https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/5b5e8d22e0bcd81d43f2_danucd_54037775427_9349002654//thumb/thumb2337379155-640x360.jpg
```
in HTML (img src tag):
```
https://d1m7jfoe9zdc1j.cloudfront.net/3a67676fb37bbb6fb356_fauxre_52456076637_1732042914/storyboards/2305934471-strip-0.jpg
./FAUXRE_files/custom-15e1eb5b-5973-4315-8756-2652fb92c16d-320x180.jpeg
./FAUXRE_files/custom-942e24e7-bcf1-426a-a669-670a4f818b88-320x180.png
./FAUXRE_files/custom-a5d843d3-dc0b-4645-b546-e5551a6febda-320x180.png
./FAUXRE_files/custom-af931e9a-d127-488e-8520-6cdbc5becfdd-320x180.png
./FAUXRE_files/custom-bbb8a757-0057-4fb2-bbc4-d14827c6572a-320x180.jpeg
./FAUXRE_files/thumb0-320x180(1).jpg
./FAUXRE_files/thumb0-320x180(2).jpg
./FAUXRE_files/thumb0-320x180(3).jpg
./FAUXRE_files/thumb0-320x180(4).jpg
./FAUXRE_files/thumb0-320x180(5).jpg
./FAUXRE_files/thumb0-320x180.jpg
./FAUXRE_files/thumb1-320x180(1).jpg
./FAUXRE_files/thumb1-320x180(2).jpg
./FAUXRE_files/thumb1-320x180.jpg
./FAUXRE_files/thumb2-320x180(1).jpg
./FAUXRE_files/thumb2-320x180(2).jpg
./FAUXRE_files/thumb2-320x180(3).jpg
./FAUXRE_files/thumb2-320x180(4).jpg
./FAUXRE_files/thumb2-320x180(5).jpg
./FAUXRE_files/thumb2-320x180(6).jpg
./FAUXRE_files/thumb2-320x180(7).jpg
./FAUXRE_files/thumb2-320x180.jpg
./FAUXRE_files/thumb2322849006-320x180.jpg
./sodapoppin_files/custom-704658ea-ec25-4310-b884-9618084d6285-320x180.jpeg
./sodapoppin_files/thumb0-320x180(740).jpg
./sodapoppin_files/thumb0-320x180.jpg
./sodapoppin_files/thumb1-320x180.jpg
./sodapoppin_files/thumb1135184000-320x180.jpg
./sodapoppin_files/thumb916689305-320x180.jpg
./DanucD_files/custom-fd597407-5354-4636-a1ce-71d1ef2133a2-320x180.jpeg
./DanucD_files/index-0000000000-320x180.jpg
./DanucD_files/thumb0-320x180(1).jpg
https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/bbddcd644ffa7b4f128d_danucd_96010104464_9996990197//thumb/thumb787862693-320x180.jpg
./DanucD_files/thumb792992064-320x180.jpg
```
### 3.7. update-thumbs-from-html.sh
This script calls the script `extract-thumbnail-from-id.sh` to batch read all the IDs from the given csv file, obtains the corresponding thumbnail from the html, and download/copy it to each video folder. `<shell>` is the binary name or full path to the shell which will be used to call `extract-thumbnail-from-id.sh` for each valid ID and can be different from the one used to call this script.
This script command has the following format, so 4 arguments must be provided:
```
sh update-thumbs-from-html.sh <html-file> <csv-file> <shell> <[--from-strip=]yes|no>
```
1. Read one ID per line from the csv file.
2. Detect if there's any folder in current working directory (CWD) likely to be named like the ID.
3. Check if such directory exists in the first place.
4. Extract the thumbnail's URL or local absolute path from given html, using `extract-thumbnail-from-id.sh`. Handle any return error message instead.
5. If it's a local file, copy it as is; otherwise force download from server at 1080p resolution.
6. If it's a strip, extract randomly one thumb from it, if `--from-strip=yes`. Otherwise skip it.
7. Save as `thumb-180p.jpg` due to resolution 320x180 (copied) or `thumb-1080p.jpg` (downloaded) inside the likely ID's folder. Save always as jpg extension even if came from a jpeg or png, to make it simpler to use with other scripts.
8. Add or update hash for picture to `hash-sha384.txt`. In case the hash file doesn't have a size bigger than 0 will exit, but still keep the thumbnail.
9. Notify to stdout about info and errors.
The thumbnails extracted from several html samples were found to be all of them `320x180` in resolution, containing `-320x180` in the filename, and extension `.jpg`, `.jpeg` or `.png`.
The thumbnail path echoed by `extract-thumbnail-from-id.sh` can be a local absolute path or remote path (URL). The filename (string after the last forward slash in the path) can have 2 kind of naming formats:
- Filename matching `1234567890-strip-0.jpg`: this is not a valid thumbnail. The number before the first hyphen is the ID of the video, and the 0 after the last hyphen is a number, probably only 0. This is a vertical strip of 10 thumbnails with total size of `320x1800` pixels, all from the same video. Those are the pictures that play in a loop like a GIF when hovering the mouse over the video on the website (`mouseenter`) and stop when not (`mouseleave`) to return to original thumbnail. But none of the 10 match such one that appears statically, which is the one most desired, since can be custom made and is the first that appears. When the thumb has this filename, they are all in URL format, and there's no local thumb saved for that ID besides that strip. Most likely a bug introduced by the web browser, and usually it's fixed by re-saving the webpage. Searching for filename containing `-strip-` should be enough to know it's not a valid thumbnail.
- Filename containing `-320x180`: these are valid thumbnails:
- `thumb0-320x180.jpg`: when the path is URL or the thumbnails are still stored on Twitch's server, all the thumbnails for all the videos have this exact filename. Hence the resolution is `320x180` pixels. When saved locally by a web browser, one of the thumbnails will have this filename.
- `thumb0-320x180_xxx.jpg`: For those thumbnails saved locally, after `thumb0-320x180.jpg` exists, `Mozilla Firefox` starts adding an underscore and a 3-digit number at the end of the name, before the extension, so the thumbs are not overwritten. It goes from `thumb0-320x180_002`.
- `thumb0-320x180(n).jpg`: For those thumbnails saved locally, once `thumb0-320x180.jpg` exists, several web browsers start adding a natural number counter between parentheses at the end of the name, before the extension, so the thumbs are not overwritten. It goes from `thumb0-320x180(1)`. These web browsers are: `Google Chrome`, `Opera`, `Microsoft Edge`, `Brave`.
- `custom-e24427fd-1ee9-1741-15b9914a3d632-320x180.jpeg`. Values in the middle are picture ids and are different for each one.
- `index-0000000000-320x180.jpg`
- `index-0000000000-320x180(3).jpg`
- `index-0000000000-320x180_004.jpg`
- `thumb1234567890-320x180.jpg`: the number after `thumb` is the ID of the video.
- All examples above in [3.6.3.](#363-notes-about-thumbnail-types-and-names)
### 3.8. update-thumbs-from-json.sh
This script command has the following format, so 3 arguments must be provided:
```
sh update-thumbs-from-json.sh <json-file> <csv-file> <shell>
```
This script calls the script `extract-thumbnail-from-id.sh` to batch read all the IDs from the given csv file, obtain the corresponding thumbnail URL from the json, force download from server at 1080p resolution, and save it to each video folder. Notifies to stdout about info and errors, and saves thumb as `thumb-1080p.jpg` inside each corresponding directory according to ID.
Also adds or updates hash for picture to `hash-sha384.txt`. In case the hash file doesn't have a size bigger than 0 will exit, but still keep the thumbnail.
`<shell>` is the binary name or full path to the shell which will be used to call `extract-thumbnail-from-id.sh` for each valid ID and can be different from the one used to call this script.
#### 3.8.1. Notes about thumbnails
After using the corresponding scripts, it's possible to choose between 3 thumbnails: `screencap.jpg`, `thumb-html.jpg` or `thumb-json.jpg`. The first one extracted as frame from the video, and the others downloaded from Twitch's servers.
Sometimes it's important to download and keep the original Twitch's (or other sites) thumbnails, because can be custom made by the uploader, and provide further information that is not available in the video.
Streaming services that allow to manually upload a thumbnail, do not require it to be a frame from the video. Thus sometimes the thumbnail's appearance does not match the content of the video; or can even be misleading to lure viewer's attention, to increase video popularity at the expense of the viewer's time. Sometimes this is mentioned in the video comments made by the viewers.
### 3.9. test-with-ffmpeg.sh
This script command has the following format, so 1 argument must be provided:
```
sh test-with-ffmpeg.sh <multimedia-file>
```
This script converts the input file to 2 file formats: `mp4` and `null`, and for each with loglevel `warning` and `error`. So in total does 4 operations.
The standard error file descriptor is saved to a file in same working dir as the file. The converted files (output) are discarded to `/dev/null`.
The purpose is to find out according to logs, if `output.ts`/`output.mp4` files have any issue in practical terms (according to what `ffmpeg` logs and doesn't log), reasons being:
* Some streams provided by `Twitch.tv` are defective, incurring or not data loss for part of the stream. This could be due to Twitch stream processing or transmission being temporarily interrupted during live streams.
* Some conversions made by `ffmpeg` from `TS` to `MP4`, randomly introduce codec errors to the output file, without generating any error (or logging it to `ffmpeg-convert-error.txt`) during conversion.
* Determine if having conversion warnings or errors (or not) could mean data missing or not (which is the most important thing).
Depending on ffmpeg conversion warnings or errors, some conclusions could be drawn, to determine if the multimedia file:
* Has any playback error
* Where the error was introduced
* The error happens always or happens randomly
* Which multimedia players can handle the warnings/errors properly
* Is any content missing (data loss)
* Data loss is present in the source file or was produced during conversion
To view at what point in the timeline the message appears, run ffmpeg without loglevel (or `info` which is the default).
#### 3.9.1. Samples
These samples are either `output.ts` (downloaded directly from Twitch servers at best quality), or `output.mp4` (generated from `output.ts` by `convert-all.sh`). The reason for conversion is, `MP4` can embed more metadata, and video hosting sites prefer the mp4 format (specially those that can reproduce the file directly on the fly to user without converting it).
##### 3.9.1.1. sample1.ts
* `sample1.ts_warning_null.txt`:
```
[null @ 0x13a608b20] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3 >= 3
[null @ 0x13a608b20] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 7 >= 7
[...]
```
* `sample1.ts_error_null.txt`:
```
[null @ 0x1587075f0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3 >= 3
[null @ 0x1587075f0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 7 >= 7
[...]
```
* `sample1.ts_warning_mp4.txt`:
```
[swscaler @ 0x118618000] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x138418000] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x129788000] deprecated pixel format used, make sure you did set range correctly
Last message repeated 1 times
```
* `sample1.ts_error_mp4.txt`: empty file
Results: `sample1.ts` plays flawlessly and doesn't have any forward jump or content missing. When loglevel is set to error it shouldn't output any information, which does if format is set to null.
I classify this as a `False Positive`, because ffmpeg claims that the file has errors when doesn't. For loglevel set to warning is more reasonable and is accurate.
##### 3.9.1.2. sample2.ts
* `sample2.ts_warning_null.txt`:
```
[mpegts @ 0x150605300] Packet corrupt (stream = 2, dts = 113936940).
[in#0/mpegts @ 0x60000287c000] corrupt input packet in stream 2
[aist#0:0/aac @ 0x150706390] timestamp discontinuity (stream id=256): -59999997, new offset= 59999997
```
* `sample2.ts_error_null.txt`: empty file
* `sample2.ts_warning_mp4.txt`:
```
[mpegts @ 0x146605300] Packet corrupt (stream = 2, dts = 113936940).
[in#0/mpegts @ 0x600002290000] corrupt input packet in stream 2
[aist#0:0/aac @ 0x146704940] timestamp discontinuity (stream id=256): -59999997, new offset= 59999997
```
* `sample2.ts_error_mp4.txt`: empty file
Results: with `mpv`, the timestamp at second `8` jumps to second `-49` while keeps playing without jumping the video. There's a micro-stutter which lasts hundredths of a second while it makes the transition, but actually the video is in perfect condition, no parts are missing. That micro-stutter is due to timestamp adjustment. The duration is listed as `1:27:08`.
After converting it to `output.mp4`, the duration is listed as `1:28:08`. Twitch.tv lists its duration as `1:28:08` too.
I classify this as a `True Negative`, because the loglevel error in both cases generates empty files, and really the file is not missing any audio/video information, it's just that there's an adjustment in timestamp.
##### 3.9.1.3. sample3.ts
* `sample3.ts_warning_null.txt`:
```
[mpegts @ 0x14e605300] Packet corrupt (stream = 2, dts = 54716940).
[mpegts @ 0x14e605300] Packet corrupt (stream = 2, dts = 54716940).
[in#0/mpegts @ 0x600001f00000] corrupt input packet in stream 2
[vist#0:2/h264 @ 0x14e7214c0] timestamp discontinuity (stream id=257): -59982667, new offset= 59982667
[mpegts @ 0x14e605300] Packet corrupt (stream = 2, dts = 1009618470).
[in#0/mpegts @ 0x600001f00000] corrupt input packet in stream 2
[vist#0:2/h264 @ 0x14e7214c0] timestamp discontinuity (stream id=257): 60017333, new offset= -34666
[null @ 0x14e729510] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 640377 >= 640377
```
* `sample3.ts_error_null.txt`:
```
[null @ 0x147913f50] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 640377 >= 640377
```
* `sample3.ts_warning_mp4.txt`:
```
[mpegts @ 0x1459041c0] Packet corrupt (stream = 2, dts = 54716940).
[mpegts @ 0x1459041c0] Packet corrupt (stream = 2, dts = 54716940).
[in#0/mpegts @ 0x6000001c0000] corrupt input packet in stream 2
[vist#0:2/h264 @ 0x145815dc0] timestamp discontinuity (stream id=257): -59982667, new offset= 59982667
[mpegts @ 0x1459041c0] Packet corrupt (stream = 2, dts = 1009618470).
[in#0/mpegts @ 0x6000001c0000] corrupt input packet in stream 2
[vist#0:2/h264 @ 0x145815dc0] timestamp discontinuity (stream id=257): 60017333, new offset= -34666
```
* `sample3.ts_error_mp4.txt`: empty file
Results: the timestamp at second `3` jumps to second `-54` while keeps playing with a micro-stutter, without jumping the video. The duration is listed as `2:57:55`. At time `2:56:52`, the timestamp instantly jumps to `2:57:53` without jumping the video, and plays until the end. The file is not missing any content, only issues with timestamps.
After converting it to `output.mp4`, the duration is listed as `2:57:55` too. Twitch.tv lists its duration as `2:57:56`.
I classify this as a `False Positive` because of the same reason as `sample1.ts`, the `non monotonically increasing dts to muxer` shouldn't be there in the error loglevel to null format.
##### 3.9.1.4. sample4.ts
* `sample4.ts_warning_null.txt`:
```
[mpegts @ 0x156e05300] Packet corrupt (stream = 1, dts = 588238470).
[in#0/mpegts @ 0x600002a98000] corrupt input packet in stream 1
[vist#0:1/h264 @ 0x156f18bc0] timestamp discontinuity (stream id=257): 60017333, new offset= -60017333
[null @ 0x156f0d330] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 388202 >= 388202
[mpegts @ 0x156e05300] Packet corrupt (stream = 1, dts = 593816940).
[in#0/mpegts @ 0x600002a98000] corrupt input packet in stream 1
[vist#0:1/h264 @ 0x156f18bc0] timestamp discontinuity (stream id=257): 57050333, new offset= -117067666
[null @ 0x156f0d330] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 310657216 >= 310655952
[null @ 0x156f0d330] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 310657216 >= 310656976
[mpegts @ 0x156e05300] Packet corrupt (stream = 1, dts = 599036940).
[in#0/mpegts @ 0x600002a98000] corrupt input packet in stream 1
[vist#0:1/h264 @ 0x156f18bc0] timestamp discontinuity (stream id=257): -59982667, new offset= -57084999
```
* `sample4.ts_error_null.txt`:
```
[null @ 0x131711d90] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 388202 >= 388202
[null @ 0x131711d90] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 310657216 >= 310655952
[null @ 0x131711d90] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 310657216 >= 310656976
```
* `sample4.ts_warning_mp4.txt`:
```
[mpegts @ 0x12f606130] Packet corrupt (stream = 1, dts = 588238470).
[in#0/mpegts @ 0x6000010a4000] corrupt input packet in stream 1
[vist#0:1/h264 @ 0x12f6207a0] timestamp discontinuity (stream id=257): 60017333, new offset= -60017333
[mpegts @ 0x12f606130] Packet corrupt (stream = 1, dts = 593816940).
[in#0/mpegts @ 0x6000010a4000] corrupt input packet in stream 1
[vist#0:1/h264 @ 0x12f6207a0] timestamp discontinuity (stream id=257): 57050333, new offset= -117067666
[aac @ 0x12f66acb0] Queue input is backward in time
[mp4 @ 0x12f619d60] Non-monotonic DTS in output stream 0:1; previous: 310657216, current: 310655952; changing to 310657217. This may result in incorrect timestamps in the output file.
[mp4 @ 0x12f619d60] Non-monotonic DTS in output stream 0:1; previous: 310657217, current: 310656976; changing to 310657218. This may result in incorrect timestamps in the output file.
[mpegts @ 0x12f606130] Packet corrupt (stream = 1, dts = 599036940).
[in#0/mpegts @ 0x6000010a4000] corrupt input packet in stream 1
[vist#0:1/h264 @ 0x12f6207a0] timestamp discontinuity (stream id=257): -59982667, new offset= -57084999
```
* `sample4.ts_error_mp4.txt`: empty file
Log file from `mpv`:
```
mpv sample4.ts
(+) Video --vid=1 (h264 1920x1080 60.000fps)
(+) Audio --aid=1 (aac 2ch 48000Hz)
AO: [coreaudio] 48000Hz stereo 2ch floatp
VO: [libmpv] 1920x1080 yuv420p
[ffmpeg/demuxer] mpegts: Packet corrupt (stream = 1, dts = 588238470).
Invalid audio PTS: 6470.058667 -> 6530.058667
Reset playback due to audio timestamp reset.
[ffmpeg/video] h264: co located POCs unavailable
[ffmpeg/demuxer] mpegts: Packet corrupt (stream = 1, dts = 593816940).
[ffmpeg/demuxer] mpegts: Packet corrupt (stream = 1, dts = 599036940).
Invalid audio PTS: 6532.064000 -> 6589.066667
Reset playback due to audio timestamp reset.
[ffmpeg/video] h264: co located POCs unavailable
[ffmpeg/video] h264: co located POCs unavailable
Exiting... (Quit)
```
Results: at timestamp `1:47:49` jumps to `1:48:50` while keeps playing with a micro-stutter, without jumping the video. Then at timestamp `1:48:51` the displayed time jumps to `1:48:50`, while the video jumps forward a bit, probably a minute, so in this case that portion of the video is lost. It's not only an adjustment of the timestamp, it's noticeable that the streamer is doing something very different and by the chat too. Then keeps playing smoothly and displayed time changes from `1:48:50` to `1:48:52` without going through the second `51`. The duration is listed as `1:59:08`.
After converting it to `output.mp4`, the duration is listed as `1:58:10`, and it's missing a part of the video which corresponds where jumps forward.
Twitch.tv lists its duration as `1:58:11`, and behaves like if it were playing the mp4, with that part of the video missing. That's even when selecting `Quality: Source (Source)`, which should be the original TS and not a transcoded version.
I classify this as a `False Negative` because the loglevel error when converting to mp4 does not display any error in the TS file, despite a part of the stream being missing, and the error when converting to null is not related to the missing data either. For some reason, `ffmpeg` is not able to detect this. Look like the source file is damaged beyond repair.
##### 3.9.1.5. sample5.ts
* `sample5.ts_warning_null.txt`:
```
[mpegts @ 0x151e05460] Packet corrupt (stream = 1, dts = 676186470).
[in#0/mpegts @ 0x60000269c000] corrupt input packet in stream 1
[vist#0:1/h264 @ 0x151f0fa30] timestamp discontinuity (stream id=257): 59999333, new offset= -59999333
```
* `sample5.ts_error_null.txt`: empty file
* `sample5.ts_warning_mp4.txt`:
```
[mpegts @ 0x120f04950] Packet corrupt (stream = 1, dts = 676186470).
[in#0/mpegts @ 0x600002bc4900] corrupt input packet in stream 1
[vist#0:1/h264 @ 0x120f0b160] timestamp discontinuity (stream id=257): 59999333, new offset= -59999333
```
* `sample5.ts_error_mp4.txt`: empty file
Results: at timestamp `2:04:10` jumps to second `2:05:11` while keeps playing without jumping the video, then immediately quits because the video ends. The duration is listed as `2:05:12`.
After converting it to `output.mp4`, the duration is listed as `2:04:12`. Twitch.tv lists its duration as `2:04:12` too.
I classify this as a `True Negative`, because the loglevel error in both cases generates empty files, and really the file is not missing any audio/video information, it's just that there's a transition in timestamp. It's the same as `sample2.ts`.
##### 3.9.1.6. sample6.mp4
This file has 2 issues. Logs for the entire timeline:
* `sample6.mp4_warning_null.txt`:
```
[h264 @ 0x158f22df0] error while decoding MB 79 25, bytestream -56
[vist#0:0/h264 @ 0x158f04c50] corrupt decoded frame
```
* `sample6.mp4_error_null.txt`:
```
[h264 @ 0x13df23aa0] error while decoding MB 79 25, bytestream -56
```
* `sample6.mp4_warning_mp4.txt`:
```
[h264 @ 0x10431f900] error while decoding MB 79 25, bytestream -56
[vist#0:0/h264 @ 0x10ff041d0] corrupt decoded frame
[vost#0:0/libx264 @ 0x10ff05710] More than 1000 frames duplicated
[vost#0:0/libx264 @ 0x10ff05710] More than 10000 frames duplicated
```
* `sample6.mp4_error_mp4.txt`:
```
[h264 @ 0x131622920] error while decoding MB 79 25, bytestream -56
```
* `sample6.mp4_info_mp4.txt`:
```
ffmpeg -y -i "$1" -c:v libx264 -crf 51 -preset ultrafast -c:a aac -b:a 16k -movflags +faststart -vf scale=-2:120,format=yuv420p -loglevel info -f mp4 /dev/null 2>"$1_info_mp4.txt"
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
Output #0, mp4, to '/dev/null':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf60.16.100
Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p(tv, unknown/bt470bg/unknown, progressive), 214x120, q=2-31, 60 fps, 15360 tbn (default)
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 16 kb/s (default)
frame= 0 fps=0.0 q=0.0 size= 0kB time=00:00:00.07 bitrate= 4.8kbits/s speed=5.57x
...
frame=35649 fps=879 q=52.0 size= 2304kB time=00:09:54.30 bitrate= 31.8kbits/s dup=3 drop=0 speed=14.6x
[h264 @ 0x121623780] error while decoding MB 79 25, bytestream -56
[h264 @ 0x121623780] concealing 5130 DC, 5130 AC, 5130 MV errors in I frame
[vist#0:0/h264 @ 0x12160e3f0] corrupt decoded frame
[vost#0:0/libx264 @ 0x12160f890] More than 1000 frames duplicated
frame=48656 fps=1171 q=59.0 size= 2560kB time=00:13:31.08 bitrate= 25.9kbits/s dup=12602 drop=0 speed=19.5x
...
frame=107508 fps=937 q=52.0 size= 6656kB time=00:29:51.96 bitrate= 30.4kbits/s dup=12602 drop=0 speed=15.6x
[vost#0:0/libx264 @ 0x12160f890] More than 10000 frames duplicated
frame=116605 fps=1012 q=50.0 size= 6912kB time=00:32:23.56 bitrate= 29.1kbits/s dup=21405 drop=0 speed=16.9x
...
frame=424307 fps=911 q=51.0 size= 27904kB time=01:57:51.94 bitrate= 32.3kbits/s dup=21406 drop=0 speed=15.2x
[mp4 @ 0x12160edf0] Starting second pass: moving the moov atom to the beginning of the file
[out#0/mp4 @ 0x6000022b0480] video:11503kB audio:16466kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000171%
frame=424493 fps=911 q=-1.0 Lsize= 27969kB time=01:57:55.00 bitrate= 32.4kbits/s dup=21408 drop=0 speed=15.2x
[libx264 @ 0x12160fb00] frame I:1698 Avg QP:47.40 size: 810
[libx264 @ 0x12160fb00] frame P:422795 Avg QP:49.38 size: 25
[libx264 @ 0x12160fb00] mb I I16..4: 100.0% 0.0% 0.0%
[libx264 @ 0x12160fb00] mb P I16..4: 1.2% 0.0% 0.0% P16..4: 2.0% 0.0% 0.0% 0.0% 0.0% skip:96.8%
[libx264 @ 0x12160fb00] coded y,uvDC,uvAC intra: 14.2% 75.7% 41.4% inter: 0.2% 1.2% 0.2%
[libx264 @ 0x12160fb00] i16 v,h,dc,p: 54% 30% 12% 4%
[libx264 @ 0x12160fb00] i8c dc,h,v,p: 68% 19% 12% 1%
[libx264 @ 0x12160fb00] kb/s:13.32
[aac @ 0x12165bc50] Qavg: 0.244
```
* `sample6.mp4_info_mp4.txt`: omitting because is not useful.
Results for first issue: with `mpv`, at timestamp `10:00`, there's a forward jump of `3:32` (from `10:00` to `13:32`), but only the audio keeps playing then, the video gets stuck with a fixed frame. If any of the seek backward/forward key is pressed: (left/right arrow), then the video unstucks, and the timestamp moves to either `9:55` or `13:33`, respectively. If moving forward, then is not possible to get below the `13:32` timestamp using the left arrow, the slider has to be used instead.
So in this sample, this portion of the video is lost due to corruption, which was caused during conversion by ffmpeg from `output.ts` to `output.mp4`. If the conversion is repeated several times, the timestamp at which the error happens varies, but most of the times there's no error at all.
The current version in use is `ffmpeg 6.1.1`, but this corrupt sample was probably generated with `ffmpeg 6.0` or `ffmpeg 5.1.2`.
The source file `output.ts` is in perfect condition the whole duration, so it's not the cause of the problem.
I classify this first issue as a `True Positive`, because the 4 logs give error.
There's a second error in the file. At timestamp `29:55`, the video and audio jump to `32:22` (showing the timestamp accordingly) and keeps playing. `mpv` does show the errors while playing:
```
mpv sample6.mp4
(+) Video --vid=1 (*) (h264 1920x1080 56.974fps)
(+) Audio --aid=1 (*) (aac 2ch 44100Hz)
AO: [coreaudio] 44100Hz stereo 2ch floatp
VO: [libmpv] 1920x1080 yuv420p
AV: 00:00:00 / 01:57:55 (0%) A-V: 0.000
...
AV: 00:09:59 / 01:57:55 (8%) A-V: 1.584
Invalid audio PTS: 600.988866 -> 810.967007
AV: 00:09:59 / 01:57:55 (8%) A-V: 1.584
Reset playback due to audio timestamp reset.
AV: 00:09:59 / 01:57:55 (8%) A-V: 1.584
[ffmpeg/video] h264: error while decoding MB 79 25, bytestream -56
AV: 00:09:59 / 01:57:55 (8%) A-V: 1.584
(...) AV: 00:00:00 / 01:57:55 (0%) A-V: 0.000
AV: 00:10:00 / 01:57:55 (8%) A-V: 0.000 ct: 20.999
[lavf] Too many packets in the demuxer packet queues:
AV: 00:10:00 / 01:57:55 (8%) A-V: 0.000 ct: 20.999
[lavf] video/0: 12062 packets, 157301872 bytes
AV: 00:10:00 / 01:57:55 (8%) A-V: 0.000 ct: 20.999
[lavf] audio/1: 0 packets, 0 bytes
AV: 00:10:00 / 01:57:55 (8%) A-V: 0.000 ct: 20.999
AV: 00:13:30 / 01:57:55 (11%) A-V: 0.000 ct: 21.006
...
AV: 00:29:56 / 01:57:55 (25%) A-V: 0.000
Invalid audio PTS: 1796.259342 -> 1942.986009
AV: 00:29:56 / 01:57:55 (25%) A-V: 0.000
Reset playback due to audio timestamp reset.
AV: 00:29:56 / 01:57:55 (25%) A-V: 0.000
[ffmpeg/video] h264: co located POCs unavailable
AV: 00:29:56 / 01:57:55 (25%) A-V: 0.000
(...) AV: 00:29:46 / 01:57:55 (25%) A-V: 0.000
AV: 00:32:22 / 01:57:55 (27%) A-V: 0.000
...
```
Then using the left or down arrows to seek backwards can't be moved before `32:24`, clicking the OSD timeline directly with the mouse has to be used.
`VLC 3.0.20` freezes complete and doesn't output anything in the Messages window.
This second error is reported by `ffmpeg` only in loglevel warning and saving in MP4 format, not PCM: `More than 10000 frames duplicated`. The source file `output.ts` contains all the content missing here as well.
I classify this second issue as a `False Negative`, because an important error is not reported in 3/4 cases, only warning mp4 is accurate. FFmpeg doesn't know if those 10,000 frames are meant to be duplicated or not; `mpv` reports the error by default which is the desired result.
##### 3.9.1.7. sample7.mp4
* `sample7.mp4_warning_null.txt`:
```
[...]
[null @ 0x147e06e10] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 525000 >= 525000
[h264 @ 0x147e400a0] error while decoding MB 90 8, bytestream -6
[vist#0:0/h264 @ 0x147e062f0] corrupt decoded frame
[null @ 0x147e06e10] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 525120 >= 525120
[...]
```
* `sample7.mp4_error_null.txt`:
```
[...]
[null @ 0x1240052e0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 525000 >= 525000
[h264 @ 0x12263f8e0] error while decoding MB 90 8, bytestream -6
[null @ 0x1240052e0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 525120 >= 525120
[...]
```
* `sample7.mp4_warning_mp4.txt`:
```
[h264 @ 0x155e3e6b0] error while decoding MB 90 8, bytestream -6
```
* `sample7.mp4_error_mp4.txt`:
```
[h264 @ 0x10e920140] error while decoding MB 90 8, bytestream -6
```
Results: with `mpv`, at timestamp `2:25:51` the displayed time freezes alongside with the video, while the audio instantly jumps 4 seconds forward and keeps playing a little bit. Then the timestamp jumps to `2:25:56` after 3 of audio playing and everything continues playing as normal. In total there's about 5 seconds of lost content, both audio and video.
The source file `output.ts` does not have any content missing.
I classify this as a `True Positive`, because the 4 logs give error, which is what is expected. However, the warning in null gives more errors than should (non monotonically increasing...).
##### 3.9.1.8. sample8.mp4
* `sample8.mp4_warning_null.txt`:
```
[h264 @ 0x150e3dc90] error while decoding MB 9 49, bytestream -9
[vist#0:0/h264 @ 0x150f06270] corrupt decoded frame
```
* `sample8.mp4_error_null.txt`:
```
[h264 @ 0x14d63e4d0] error while decoding MB 9 49, bytestream -9
```
* `sample8.mp4_warning_mp4.txt`:
```
[h264 @ 0x137f3c3c0] error while decoding MB 9 49, bytestream -9
[vist#0:0/h264 @ 0x137e06080] corrupt decoded frame
[vost#0:0/libx264 @ 0x137e07610] More than 1000 frames duplicated
```
* `sample8.mp4_error_mp4.txt`:
```
[h264 @ 0x134e3fa30] error while decoding MB 9 49, bytestream -9
```
Results: same case as `sample6.mp4`, but in this case the error content is lost between timestamps `2:48:46` and `2:52:26`.
The source file `output.ts` does not have any content missing.
I classify this as a `True Positive`, because the 4 logs give error, which is what is expected.
##### 3.9.1.9. sample9.mp4
* `sample9.mp4_warning_null.txt`:
```
[null @ 0x148e06a10] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 537 >= 537
[h264 @ 0x129d16670] error while decoding MB 101 50, bytestream -9
```
* `sample9.mp4_error_null.txt`:
```
[null @ 0x125504580] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 537 >= 537
[h264 @ 0x12552b6f0] error while decoding MB 101 50, bytestream -9
```
* `sample9.mp4_warning_mp4.txt`:
```
[h264 @ 0x11e52ab80] error while decoding MB 101 50, bytestream -9
[vist#0:0/h264 @ 0x11ce049d0] corrupt decoded frame
[vost#0:0/libx264 @ 0x11ce05fc0] More than 1000 frames duplicated
```
* `sample9.mp4_error_mp4.txt`:
```
[h264 @ 0x13ee2bd30] error while decoding MB 101 50, bytestream -9
```
Results: same case as `sample6.mp4`, but in this case the error content is lost between timestamps `42:30` and `43:33`.
The source file `output.ts` does not have any content missing.
I classify this as a `True Positive`, because the 4 logs give error, which is what is expected.
#### 3.9.2. Conversion statistics
| TABLE2 | Data integrity | null/warning | null/error | mp4/warning | mp4/error | Classification |
|---------------|---------------|---------------|---------------|---------------|---------------|---------------|
| sample1.ts | ✓ | ✗ | ✗ | ✗ | ✓ | False Positive |
| sample2.ts | ✓ | ✗ | ✓ | ✗ | ✓ | True Negative |
| sample3.ts | ✓ | ✗ | ✗ | ✗ | ✓ | False Positive |
| sample4.ts | ✗ | ✗ | ✓ | ✗ | ✓ | False Negative |
| sample5.ts | ✓ | ✗ | ✓ | ✗ | ✓ | True Negative |
| sample6.mp4 (1st issue) | ✗ | ✗ | ✗ | ✗ | ✗ | True Positive |
| sample6.mp4 (2nd issue) | ✗ | ✓ | ✓ | ✗ | ✓ | False Negative |
| sample7.mp4 | ✗ | ✗ | ✗ | ✗ | ✗ | True Positive |
| sample8.mp4 | ✗ | ✗ | ✗ | ✗ | ✗ | True Positive |
| sample9.mp4 | ✗ | ✗ | ✗ | ✗ | ✗ | True Positive |
Trying to detect errors in input `.ts` files is not realiable, because `ffmpeg` sometimes gives false positives and other false negatives. None of the 4 columns warning/error or mp4/null match the data integrity status in all cases, from sample 1 to 5.
The files `ffmpeg-convert-error.txt` generated by `convert-all.sh` are also affected.
When the input file is `.mp4`, the only column that matches the 5 cases (sample 6 to 9) is the `mp4/warning`.
Conclusion: it's only reliable to check for integrity in `mp4` files, not `ts` ones, and only when converting them to another `mp4` with loglevel `warning` while using `ffmpeg`.
### 3.10. chat_json_timeline_comparison.py
This script command has the following format, so 2 arguments must be provided:
```
python3 chat_json_timeline_comparison.py <chat_1.json> <chat_2.json>
```
The purpose of this script, is to compare the comments between 2 json files generated by `TwitchDownloaderCLI chatdownload`, and know exactly in which parts of the timeline each comment matches or differ, based on the id and the timestamp.
This can be be useful to compare the chat downloaded just after the stream ended with another one donwloaded time later, usually some comments disappear for various reasons.
Also serves to compare chat files download amid live stream with other also amid live stream or once it has ended, and see if the comments are inserted consecutively or interleaved.
Sometimes the chat file downloaded about 15 seconds after the live stream ends is not complete, and the comments don't have from the beginning of the stream, they are picked from various parts of it.
The reason to download the chat file so fast after the stream ends, is to catch the chat as fast as possible in case the VOD is deleted or set to unpublished. Ideally the chat should be downloaded multiple times along the live stream, then combine them when the most complete version is available.
Sometimes comments are deleted for a reason or due to some policy infringement and this is a good way to catch them.
The script does not take into account the whole duration of the VOD, since that doesn't help to know if any comment(s) are missing in both json files, so it takes as beginning and end the first and last comments among both files respectively.
Sample comparison:
```
$ python3 chat_json_timeline_comparison.py 1.json 2.json
2024-01-01T12:00:00+00:00 <
2024-01-01T12:00:10+00:00 <
2024-01-01T12:00:20+00:00 <
2024-01-01T12:00:30+00:00 <
2024-01-01T12:00:30+00:00 >
2024-01-01T12:00:30+00:00 >
2024-01-01T12:00:35+00:00 <
2024-01-01T12:00:35+00:00 >
2024-01-01T12:00:50+00:00 >
2024-01-01T12:00:55+00:00 >
2024-01-01T12:01:00+00:00 >
2024-01-01T12:01:05+00:00 >
2024-01-01T12:01:10+00:00 <>
2024-01-01T12:01:15+00:00 <>
2024-01-01T12:01:20+00:00 >
2024-01-01T12:01:25+00:00 >
<··········<···········<··········≠·····≠·················>·····>····>·····>·····=·····=·····>·····>
timeline (time)
<·····<······<·····<······>······>·····<······>·····>······>······>·····>······=·····=······>······>
timeline (count)
```
The ouptut prints each comment in a line, alongside with in which json such comment is located by a symbol: `<`, `<>` or `>`. All comments appear, because they are identified by their id, not only the timestamp (which can be exactly the same for multiple).
There are 2 timelines, each divided in 100 slices, each representing in each bar:
- The 1% of the duration between the earliest and latest comment across both logs
- The 1% of the number of unique comment IDs after the two logs have been combined
## 4. OTHER SOFTWARE AND COMMANDS
### 4.1. SOFTWARE / PROJECTS
Copies of original versions available in the [third-party-repos](https://sourceforge.net/projects/twitch-batch-downloader/files/third-party-repos/) section of this project.
### 4.1.1. twitch-dlp-2
Update on 20250723: the project `twitch-dlp-2` is no longer active, since the original project `twitch-dlp` has received lots of updates later on, and has superseded it for the most part. The only part where the fork is still better, is that checks for createdAt ±1 because sometimes it's not second accurate, at least as provided by Twitch. Some users have already reported this feature request to the developer but it's still not implemented.
[twitch-dlp-2](https://sourceforge.net/projects/twitch-dlp-2/) uses `npm`, `pnpm` or `yarn v2+` from [Node.js](https://nodejs.org/) v20 or newer, to downloads content from Twitch.tv.
It's a fork of [twitch-dlp](https://github.com/DmitryScaletta/twitch-dlp) with some improvements.
It's much more complete than `livestream_scripts`. This is the recommended program to actively monitor a Twitch channel, saving all streams to local storage as they appear. It has other advanced features too.
Also can download currently ongoing streams from the beginning, even if the stream capture is started amid stream: recovers the past segments and appends to them the ongoing live stream (in some cases this rewind feature is just not possible for any program, because Twitch unpublishes immediate past segments right away, giving HTTP 403 status code upon petition).
Can download also other types of hidden/private streams without requiring oauth or to be a subscriber, but may require the additional use of a desktop web browser to avoid Cloudflare blocks or other CAPTCHAs.
### 4.1.2. livestream_scripts
The [livestream_scripts](https://github.com/mrwnwttk/livestream_scripts) repository has 4 python scripts. The most important is `save_livestream.py`, which scans the argument URL every x seconds, checking if the channel is live.
If it's live, starts saving the stream as .ts until ends. Once the stream ends, keeps scanning again periodically for the next live stream, the cycle repeats continuously.
Allows to pass the Twitch oauth token to streamlink, the same oauth used in the scripts:
```
python3 save_livestream.py --author-name "SOLIDFPS" --downloader-args "--twitch-api-header 'Authorization=OAuth "$OAUTH"'" 'https://twitch.tv/solidfps'
```
Description and notes:
- The purpose of this script, is to save to a local file all the streams of a given channel, continuously unattended.
- More than one channel can be monitored, running separate commands at once.
- Not all live streams become a VOD, so the only way to keep a record of those is to save them while they are live.
- The channel scan is performed repeatedly according to values `MIN_WAIT` and `MAX_WAIT` set inside the python script, which can be changed.
- If a stream starts immediately just after the scan, will wait until next one to start recording, so at the beginning of each stream a number of seconds up to `MAX_WAIT` will not be recorded.
- If the script is started amid live stream, only the part from there going forward will be saved, this program does not have the rewind feature.
- The values of `MIN_WAIT` and `MAX_WAIT` can be set to very low values in seconds, but if Twitch regards that as abuse, the local IP or Account could get banned.
- The default quality is `best`, but can be changed inside the script.
- Twitch has [Go Live Notifications](https://help.twitch.tv/s/article/how-to-use-go-live-notifications?language=en_US), which is a native way to inform viewers using email and push notifications. Regarded as unreliable: notifications being received up to half an hour late, unsubscribing viewers automatically due to inactivity. Parsing that to start and stop `save_livestream.py` automatically based on that could be possible, but depends on too many variables.
- Some streamers have dedicated `Discord` / `Twitch` / `X` / `Mastodon` / `Telegram` / `WhatsApp` / `Facebook` channels or profiles, that automatically notify when a new stream has just started. They can do it manually but it's more accurate when done with a bot. If the stream stops and a new one starts immediately after (due to connection issues), the notifications are send too according to Twitch and how bots work.
- Some streamers display a logo or loop animation (audio/video) at the start of all their live streams, so no critical information is given during the very first seconds of the stream and let everybody join in time.
### 4.2. USEFUL COMMANDS
#### 4.2.1. Download all playlists in all qualities
This code download all playlists in all qualities, for all IDs listed in the csv-file. Oauth may be added if required. It also checks which playlists contain 2 or more initialization segments, for debugging purposes.
The script could be called `get-all-playlists.sh`, but the code can be copied directly to shell and hit enter:
```
#!/usr/bin/env sh
mkdir -p playlists 2>/dev/null
awk -F',' '{print $NF}' videos.txt | while IFS= read -r ID; do
printf 'Checking %s...\n' "$ID"
output=$(TwitchDownloaderCLI2 info --log-level Error -u "$ID" -f Raw --oauth "$OAUTH")
printf '%s\n' "$output" | grep -E 'https?://' | while IFS= read -r url; do
quality=$(printf '%s\n' "$url" | sed -n 's#.*/\([^/]*\)/index-dvr.m3u8#\1#p')
filename="playlists/${ID}-${quality}.m3u8"
if [ ! -s "$filename" ]; then
curl -s "$url" -o "$filename"
fi
if [ -s "$filename" ]; then
map_count=$(grep -c '^#EXT-X-MAP:URI=' "$filename")
if [ "$map_count" -ge 2 ]; then
printf 'File %s contains %d lines starting with #EXT-X-MAP:URI=\n' "$filename" "$map_count"
fi
fi
done
done
```
#### 4.2.2. Sort script by exit code
This command extracts only the lines containing the exit codes, then sorts and prints it. Does not modify any file.
This command is designed to check how many exit codes are in a script and if there's anyone missing or repeated.
All exit commands inside the script must include an exit code number, and such number be the last string of its line, otherwise it will not be shown:
```
FILE="download-twitch-video.sh"
echo && grep -n 'exit' "$FILE" | awk '{
match($0, /exit[ ]*[0-9]+/);
if (RSTART > 0) {
num = substr($0, RSTART + 4, RLENGTH - 4);
line = $0;
gsub(/exit[ ]*[0-9]+/, "\033[31m&\033[0m", line);
print num, line;
}
}' | sort -n | cut -d' ' -f2-
```
#### 4.2.3. Regenerate thumbnail from video
Commands to extract a frame from the video (any ts/mp4), overwritting existing screenshot, and also update the hash file.
Useful when the thumbnail extracted automatically by `download-twitch-video.sh` is not representative:
```
FILE="output.ts" && \
SECS="$(ffprobe -i "$FILE" -show_format -v quiet | sed -n 's/duration=//p' | cut -d'.' -f 1)" && \
START="$(shuf -i 0-"$SECS" -n 1)" && \
ffmpeg -y -ss "$START" -i "$FILE" -frames:v 1 -q:v 25 -nostdin -loglevel panic "thumb-ffmpeg.webp" && \
(pcre2grep -v " \*thumb-ffmpeg.webp$" hash-sha384.txt 2>/dev/null | sponge -- hash-sha384.txt || true) && \
(dwebp -quiet thumb-ffmpeg.webp -o /dev/null && sha384sum -b thumb-ffmpeg.webp >> hash-sha384.txt) || \
printf 'BAD PICTURE, TRY AGAIN!\n'
```
#### 4.2.4. Subshells, pipes and exit codes
Running this whole code in a subshell forces that the last command in the pipe is also run in a subshell, so the `exit 99` only exits that subshell and not the entire script. This way the exit code can be captured, and the `Checkpoint` message will always be printed then just exit the program:
```
OUTSEGMENTLIST='melon.ts
banana.ts
apple.ts'
(
printf "%s\n" "$OUTSEGMENTLIST" | while IFS= read -r LINE; do
rm -- "$LINE" || exit 99
done
)
EXITCODE=$?
if [ "$EXITCODE" -ne 0 ]; then
printf "%s\n" "Checkpoint reached. Exiting." && exit "$EXITCODE"
fi
```
However, if there's no subshell surrounding the whole pipe, some shells will run the `while` in a subshell but others not (here the script will exit immediately and no checkpoint message in that case):
```
OUTSEGMENTLIST='melon.ts
banana.ts
apple.ts'
printf "%s\n" "$OUTSEGMENTLIST" | while IFS= read -r LINE; do
rm -- "$LINE" || exit 99
done
EXITCODE=$?
if [ "$EXITCODE" -ne 0 ]; then
printf "%s\n" "Checkpoint reached. Exiting." && exit "$EXITCODE"
fi
```
* `bash` and `dash` will reach the checkpoint, then exit the script.
* `ksh` and `zsh` will not reach the checkpoint, because the last command in a pipe will not be in a subshell and the whole script will exit immediately.
* In all cases, the `while` will stop as soon as the first `rm` fails because of the OR condition.
* If the exit code or break is nested within one or more subshells, it will propagate to the outter shell, but will not exist immediately the script.
If a `break` is used instead, it will equal to `exit 0` like the while being within a subshell. The code flow will pass over the checkpoint and won't exit the script in any case, regardless if it's in a subshell or not:
```
OUTSEGMENTLIST='melon.ts
banana.ts
apple.ts'
#(
printf "%s\n" "$OUTSEGMENTLIST" | while IFS= read -r LINE; do
rm -- "$LINE" || break
done
#)
EXITCODE=$?
echo "EXITCODE=$EXITCODE"
if [ "$EXITCODE" -ne 0 ]; then
printf "%s\n" "Checkpoint reached. Exiting."
exit "$EXITCODE"
fi
```
This code will behave a bit differently depending on shell:
```
echo "No declared subshells:"
LINE="aaa" ; LINE2="111" ; LINE3="&&&" ; LINE4="KKK" ; LINE5='!!!'
LINE="bbb" | LINE2="222" | LINE3="###"
echo "LINE=$LINE" ; echo "LINE2=$LINE2" ; echo "LINE3=$LINE3"
LINE4="TTT" || echo "here"
false || LINE5='$$$'
echo "LINE4=$LINE4" ; echo "LINE5=$LINE5"
echo "Declared subshells:"
LINE="aaa" ; LINE2="111" ; LINE3="&&&" ; LINE4="KKK" ; LINE5='!!!'
(LINE="bbb" | LINE2="222" | LINE3="###")
echo "LINE=$LINE" ; echo "LINE2=$LINE2" ; echo "LINE3=$LINE3"
(LINE4="TTT" || echo "here")
(false || LINE5='$$$')
echo "LINE4=$LINE4" ; echo "LINE5=$LINE5"
```
Results from `bash` and `dash`:
```
No declared subshells:
LINE=aaa
LINE2=111
LINE3=&&&
LINE4=TTT
LINE5=$$$
Declared subshells:
LINE=aaa
LINE2=111
LINE3=&&&
LINE4=KKK
LINE5=!!!
```
Results from `ksh` and `zsh`:
```
No declared subshells:
LINE=aaa
LINE2=111
LINE3=###
LINE4=TTT
LINE5=$$$
Declared subshells:
LINE=aaa
LINE2=111
LINE3=&&&
LINE4=KKK
LINE5=!!!
```
This is another way to use the `while` loop without involving pipes:
```
OUTSEGMENTLIST="melon.ts
banana.ts
EOF
apple.ts"
while IFS= read -r LINE; do
echo "$LINE"
done <<EOF
$OUTSEGMENTLIST
EOF
echo here
```
The `EOF` in the contents of the variable does not cause the loop to stop, and it works well in all shells:
```
melon.ts
banana.ts
EOF
apple.ts
here
```
Another way without involving pipes, would be reading the variable line by line with sed by using a counter. Empty lines with a newline are counted too. Careful with `wc -l` and other commands, because if the variable is empty / null / not initialized, the number of lines will still be `1` instead of `0` when there's only a newline in the input (check manually):
```
OUTSEGMENTLIST="melon.ts
banana.ts
apple.ts"
if [ -z "$OUTSEGMENTLIST" ]; then
LINE_COUNT=0
else
LINE_COUNT="$(printf "%s\n" "$OUTSEGMENTLIST" | wc -l)"
fi
if [ "$LINE_COUNT" -gt 0 ]; then
i=1
while [ "$i" -le "$LINE_COUNT" ]; do
LINE=$(printf "%s\n" "$OUTSEGMENTLIST" | sed -n "${i}p")
echo "Line $i: $LINE"
i="$((i + 1))"
done
else
echo "OUTSEGMENTLIST is empty."
fi
echo "here"
```
Result:
```
Line 1: melon.ts
Line 2: banana.ts
Line 3:
Line 4: apple.ts
here
```
Script to test break behaviour in loops, subshells and main script:
```
(
echo aaaa
while true; do
echo bbb
break
echo ccc
done
echo ddd
break
echo eee
)
break
echo fff
```
Results with `dash` and `ksh`:
```
aaaa
bbb
ddd
eee
fff
```
Results with `bash`:
```
aaaa
bbb
ddd
script.sh: line 9: break: only meaningful in a `for', `while', or `until' loop
eee
script.sh: line 12: break: only meaningful in a `for', `while', or `until' loop
fff
```
Results with `zsh`:
```
aaaa
bbb
ddd
script.sh:break:9: not in while, until, select, or repeat loop
script.sh:break:12: not in while, until, select, or repeat loop
```
Conclusions:
* If there's no pipe involved (only assign `VAR=xxx`), the command is always executed in current shell context. Otherwise would be impossible to assign any variable.
* When there's a pipe (at least one `|`), `bash` and `dash` execute all commands as a separate process (subshell), from first to last, while `ksh` and `zsh` execute only the last command in current shell context (no subshell).
* Commands connected by logical operators like `||` and `&&` are always executed in the current shell context and not in subshells, unless explicitly stated `()`.
* Nested subshells propagate the exit code outwards.
* Built-in `break` is only designed to work inside `for`, `while` or `until` loops, otherwise is not reliable and depends on the shell. It propagates exit code 0 when in a loop and when not it depends on the shell.
* To assure portability and avoid unexpected bugs: run pipes in a subshell (either the whole pipe or just the last command); do not use break outside loops.
#### 4.2.5. Render subtitles in video
To burn-in (toast, render or hard code) subtitles on a video (without being an attachment like in .mkv format), a simple command can be used:
```
ffmpeg -i nometadata.mp4 -vf "subtitles=subs.srt:force_style=FontName='Lobster Two,PrimaryColour=&H00005AFF,FontSize=12'" \
-c:v libx264 -b:v 3800k -c:a aac -b:a 160k -movflags +faststart -metadata title="Title" -metadata artist="Artist" -metadata date="2024" \
-metadata comment="Created at: 2024-01-18 11:12:07Z" -metadata genre="Just Chatting" -metadata chapter="0:00:00.000-0:05:00.000:Just Chatting" \
-preset slower -nostdin output_with_subtitles.mp4
```
The metadata is manually baked into the output without reading it from metadata.txt, but that file could be used instead. If there's already any metadata in the video file should be removed first to avoid overlaps.
It's important to make sure the subtitles have enough duration. They can just be the Twitch chat converted from TXT/HTML/JSON to SRT. A portion of the chat or the whole of it can be downloaded.
Downloading the chat can be done in .txt format with `TwitchDownloaderCLI chatdownload` (ignoring all emojis and pictures which are difficult to parse).
The color for the subtitles can be specified with the parameter `PrimaryColour=&H00005AFF`, and they are in format ABGR where:
- `H00` is the alpha (transparency). `H00` is fully opaque, and `HFF` is fully transparent/invisible.
- `005AFF` is the BGR code for the color. Is in reverse order of HTML (most common). In HTML (RGB) would be `FF5A00`.
- `&H00005AFF` then is a dark orange opaque subtitle.
- Some subtitle formats like ASS let fully customize the subtitles aspect, so even each letter or character can have its own style.
- To burn in the Twitch emojis, emotes, avatars, GIFs, profile pictures, (or any other image), is a more complicated process.
#### 4.2.6. Render Twitch.tv chat in video and concatenate several IDs
Example: the script `example1-chatrender-3subs.sh` downloads and concatenates 3 Twitch videos and overlays (burns-in) its chats:
* [315016672](https://www.twitch.tv/videos/315016672): render corresponding chat starting at 2210 seconds of this ID (00:36:50)
* [2174725499](https://www.twitch.tv/videos/2174725499): render corresponding chat all the way through this ID
* [2122384875](https://www.twitch.tv/videos/2122384875): render corresponding chat up to 894 seconds of this ID (00:14:54)
The chats are downloaded whole for each ID using `TwitchDownloaderCLI` and the videos using `download-all.sh`.
This scripts takes about 7 hours 45 minutes to complete in an Apple Silicon M1, and the log to terminal (including the call to the script) is 1041 lines.
All the files will be written to a new cloned directory called `./twitch-batch-downloader_chatrender/`, next to the script `example1-chatrender-3subs.sh`.
The purpose of this script is to confirm that concatenating files as downloaded from Twitch, and/or burning-in chat as overlay, requires sometimes to re-encode some Twitch stream(s) (this is why all IDs must be recoded just in case). Despite the streams being a correct file on its own, when performing operations directly on them, like overlaying chat and/or concatenating them, it can lead to audio/video desync issues, or the chat being sped up compared to the video and loose sync across the timeline. This is due to how the audio and video are interleaved together and how timestamps operate. A contributing factor is that ffmpeg has several bugs, doesn't hand those cases correctly (it should but current ffmpeg developers don't see it the same way). The ffmpeg version used for these tests is 7.1.
A new directory `./twitch-batch-downloader_chatrender/subs_chatrender/` is created with 25 files and 1 directory inside that are for chats specifically:
- `1_original.mp4`, `2_original.mp4` and `3_original.mp4`: the IDs from Twitch after the script `convert-all.sh`, so they are converted in copy mode from ts to mp4.
- `metadata.txt`: the medatata file which would be generated if using `combine-multiple-ids.sh`.
- `TMP/`: the directory where TwitchDownloaderCLI stores the emojis, emotes and badges.
- `1_315016672.json`, `2_2174725499.json`, `3_2122384875.json`: the chat files for each ID with as much multimedia content as possible.
- `1_chat.mp4`, `1_chat_mask.mp4`, `2_chat.mp4`, `2_chat_mask.mp4`, `3_chat.mp4`, `3_chat_mask.mp4`: the json chats rendered as video files with its masks by TwitchDownloaderCLI. The masks are required because mp4 files don't support transparency, so the whole chat is not overlaid as a block but only the text and pictures. The chat videos have lower resolution than the streams so they are overlaid on the right side of them at certain positions.
- `1_original_recoded.mp4`, `2_original_recoded.mp4`, `3_original_recoded.mp4`: the first files but re-encoded with same bitrates, etc. to fix subtle parameters that can break ffmpeg down the line.
- `burned1.mp4`, `burned2.mp4`, `burned3.mp4`: the 3 original_recoded files with its corresponding chats overlaid in certain position and for certain duration according to each `-filter_complex`. A full recoding is done separately for each file.
- `list.txt`: the ffmpeg file list containing the 3 burned files.
- `final_concat.mp4`: the 3 burned files concatenated using the concat demuxer (copy mode). This is the best possible outcome, because each burned file can have its own bitrates (the other parameters like resolution, codec, tbn, fps, etc. must be the same). It's also the simplest, because generating each burned file is a simple filter, and each burned file can be checked for errors independently. Also generating the file list for the concat demuxer is straightforward, and scales the same with any number of burned files to be concatenated. This is the recommended and the best method to concatenate different Twitch IDs, with or without also burning-in chats.
- `final_singletake.mp4`: Read the 3 original recoded files and craft a complex filter, to overlay its chats and concatenate them in a single take (a single ffmpeg command). This avoids having to generate the intermediate burned files, but has several serious downsides: 1) The complex filter and input arguments grow larger with the number of files, probably reaching eventually a length limit or out of memory exception; 2) Crafting the ffmpeg command gets overly complex in case it has to be generated inside a shell script; 3) The audio and video bitrate apply to the entire output file, making it inadequate when the input files have very different bitrates, and the best ratio of size/quality has to be achieved without doing 2 passes; 4) If there's any warning or error during the transcoding, the elapsed time when it happend is much longer.
- `concat_filter_recoded_good.mp4`: this is like "final_singletake.mp4" but with ultrafast preset, and does not overlay any chat. Just transcodes and concatenates the 3 original recoded files to demonstrate that the output file is correct and there isn't any warning or error in ffmpeg log. This file is for testing purposes only.
- `concat_filter_original_broken.mp4`: the same as "concat_filter_recoded_good.mp4", but using as input files the original not recoded (like was intended in the first place). This demonstrates that the second input file (2_original.mp4) generates a corrupt output file: only the fist input and a few seconds of video of the second are present in the output, while the audio is for the entire duration. There's 01:03:50 of video and 01:35:05 of audio. This file is for testing purposes only.
This tests demonstrate that is mandatory to convert all input files to an intermediate format, so videos can be edited in fine detail, and exported avoiding unnecessary errors. Most video editors do this automatically or at least recommend it.
#### 4.2.7. Get information about Twitch chat JSONs
The chat.json files generated with `TwitchDownloaderCLI chatdownload -E -u $ID -o chat.json` can be read with `jq` to obtain information.
4.2.7.1. - Count the number of comments in a json file:
```
$ jq '.comments | length' chat_stream_just_ended.json
320
```
```
$ jq '.comments | length' chat_hours_later.json
1972
```
In this previous example, the first json was obtained 15 seconds after the live stream ended, and has 320 comments. The second json was obtained a few days later and has 1972 comments. This shows that the JSON for the VOD chat can take some time to be filled entirely by Twitch. The comments of the smaller json don't have to be the first consecutive of the whole VOD, they can be interleaved, in a timeline it would be:
```
$ python3 chat_json_timeline_comparison.py chat_stream_just_ended.json chat_hours_later.json
>>>>>>>===>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>==>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>============
timeline (time)
>>>==>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>==>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>===========
timeline (count)
```
4.2.7.2. - Compare differences in comments between 2 json files, specially if they are from the same VOD but obtained at different moments.
Can serve to check several things:
- If some comments were deleted by moderation, even after the stream has ended.
- If the account was deleted or sanctioned or banned. This can affect all the comments of the user in all the VODs of Twitch at once.
- If one of the jsons was obtained too fast after the stream ended. Sometimes it's still not complete and takes a while to be filled by Twitch.
To compare jsons from the same VODs, the field `updated_at` has to be ignored, because is updated constantly for several reasons. That doesn't mean that the comment has changed its contents (messages in chat can't be edited, only removed). It's mainly due to changes in the profile of the commenter.
```
$ diff -u <(jq -S --indent 2 'del(..|.updated_at?) | .comments' old.json) <(jq -S --indent 2 'del(..|.updated_at?) | .comments' new.json)
--- /dev/fd/63 2025-06-14 04:28:49
+++ /dev/fd/62 2025-06-14 04:28:49
@@ -4970,40 +4970,6 @@
}
},
{
- "_id": "YwgfO2IYJBcq7A",
- "channel_id": "30383713",
- "commenter": {
- "_id": "41542680",
- "bio": null,
- "created_at": "2013-03-20T22:06:55.587454Z",
- "display_name": "kenpetto",
- "logo": "https://static-cdn.jtvnw.net/user-default-pictures-uv/ead5c8b2-a4c9-4724-b1dd-9f00b46cbd3d-profile_image-300x300.png",
- "name": "kenpetto"
- },
- "content_id": "1642308858",
- "content_offset_seconds": 477,
- "content_type": "video",
- "created_at": "2022-11-03T14:09:32.2115624Z",
- "message": {
- "bits_spent": 0,
- "body": "holy shit",
- "emoticons": [],
- "fragments": [
- {
- "emoticon": null,
- "text": "holy shit"
- }
- ],
- "user_badges": [
- {
- "_id": "no_audio",
- "version": "1"
- }
- ],
- "user_color": null
- }
- },
- {
"_id": "shsfO2IYJBdc9A",
"channel_id": "30383713",
"commenter": {
```
#### 4.1.3. Download only a section or portion of a VOD
This project `twitch-batch-downloader`, at least up to version `20250721.2`, only downloads entire VODs (no Clips), not sections of them. The reason, is that it's designed for archival purposes, so the idea is to get the entire VOD at the highest quality. Still needs to implement the ability to unmute segments when possible (#TODO) (use `twitch-dlp` for that for now). To download a section, means too to download only the required segments, not download the entire VOD then trim it: it's mostly useful to avoid downloading and storing big files needlessly, but it's not something that can't be done already by having the whole VOD. Also, downloading a section of the VOD, requires to re-adjust the chapters and maybe the createdAt time, shifting it the amount of time that was trimmed at the beginning. All the other scripts related to chat and so on may require adjustment too.
The ffmpeg version used in this section is: ffmpeg version 7.1.1 built with Apple clang version 17.0.0 (clang-1700.0.13.3). The examples below try to download sections and trim in the middle of segments, to see if the program and/or ffmpeg can handle it or what is its behaviour.
The operating systems used for testing is: macOS 15.5 with Apple M1.
3 VODs will be tested with various programs (Clips are not tested since its duration is short already):
* 2510401818, which is in .ts format, and is made of 3071 segments of 10 seconds exactly each (even the last one).
* 2507267603, which is in .mp4 format, made of an init file, and 1609 segments of 10 seconds each exactly, no chapters.
* 797302800, which is in .ts format, made of 4273 segments of various durations, ranging from 10.000 to 14.966 seconds, and the last one which is 4.967. This makes more complicated to perform operations later on the timestamps.
Projects that have the download-section feature:
##### 4.1.3.1. twitch-dl
Tested version: [twitch-dl](https://github.com/ihabunek/twitch-dl/) 3.1.0 in .pyz format.
This program can be downloaded as `pipx` (venv recommended), or standalone executable in a zipped format `pyz` (recommended).
Allows to download:
* Specific chapters, with flag `-c` (starting at number 1).
* A section of a VOD (not for clips) with flags `--start` and/or `--end`. When using these flags, `-c` is disabled.
Roughly this is how it works:
1. Downloads only the needed whole segments.
2. Creates a custom playlist called `playlist_downloaded.m3u8`, listing only the needed segments.
3. Calls ffmpeg with `-ss` and `-t` flags against the playlist, to trim the fist and last segments (if needed) to match exactly the timeline of the chapter or section in whole seconds.
Here are 3 examples of 3 different VODs, one in MP4 format and the other two in Transport Stream format:
1. Example 1, with VOD 2510401818:
```
$ ./twitch-dl.3.1.0.pyz info 'https://www.twitch.tv/videos/2510401818?filter=all&sort=time'
Fetching video...
Fetching access token...
Fetching playlists...
Fetching chapters...
Video 2510401818
🌸COMMUNITY NIGHT!🌸| !Youtube !Social
Aliythia playing Just Chatting
Published 2025-07-12 @ 11:21:59 Length: 8 h 31 min
https://www.twitch.tv/videos/2510401818
Playlists:
Name Group Resolution URL
-------------- ---------- ---------- ----------------------------------------------------------------------------------------------------------------------------------
1080p60 source chunked 1920x1080 https://d2nvs31859zcd8.cloudfront.net/ea829a554588c5ffa471_aliythia_314349571579_1752319314/chunked/index-muted-7Q43UYJSYK.m3u8
720p60 720p60 1280x720 https://d2nvs31859zcd8.cloudfront.net/ea829a554588c5ffa471_aliythia_314349571579_1752319314/720p60/index-muted-7Q43UYJSYK.m3u8
480p 480p30 852x480 https://d2nvs31859zcd8.cloudfront.net/ea829a554588c5ffa471_aliythia_314349571579_1752319314/480p30/index-muted-7Q43UYJSYK.m3u8
360p 360p30 640x360 https://d2nvs31859zcd8.cloudfront.net/ea829a554588c5ffa471_aliythia_314349571579_1752319314/360p30/index-muted-7Q43UYJSYK.m3u8
160p 160p30 284x160 https://d2nvs31859zcd8.cloudfront.net/ea829a554588c5ffa471_aliythia_314349571579_1752319314/160p30/index-muted-7Q43UYJSYK.m3u8
Audio Only audio_only None https://d2nvs31859zcd8.cloudfront.net/ea829a554588c5ffa471_aliythia_314349571579_1752319314/audio_only/index-muted-7Q43UYJSYK.m3u8
Chapters:
00:00:00 Just Chatting (01:22:00)
01:22:00 Call of Dragons (01:25:54)
02:47:54 Just Chatting (03:42:02)
06:29:56 Marbles on Stream (28:42)
06:58:38 Call of Dragons (01:24:48)
08:23:26 Just Chatting (08:24)
Placeholder Value
--------------- ------------------------------
{channel} Aliythia
{channel_login} aliythia
{date} 2025-07-12
{datetime} 2025-07-12T11:21:59Z
{format} mp4
{game} Just Chatting
{game_slug} just_chatting
{id} 2510401818
{time} 11:21:59Z
{title} COMMUNITY NIGHT Youtube Social
{title_slug} community_night_youtube_social
```
```
$ ./twitch-dl.3.1.0.pyz download -c 4 -q source 'https://www.twitch.tv/videos/2510401818?filter=all&sort=time'
Looking up video...
Found video: 🌸COMMUNITY NIGHT!🌸| !Youtube !Social by Aliythia playing Just Chatting (08:31:50)
Target: 2025-07-12_2510401818_aliythia_community_night_youtube_social.mp4
Fetching chapters...
Chapter selected: Marbles on Stream
Fetching access token...
Fetching playlists...
Fetching playlist...
Downloading to cache: /Users/abc/Library/Caches/twitch-dl/videos/2510401818/chunked
Downloading 173 VODs using 10 workers
Downloaded 171/173 VODs 99% of ~1.1GB at 45.2MB/s ETA 00:00
Joining files...
ffmpeg -i /Users/abc/Library/Caches/twitch-dl/videos/2510401818/chunked/playlist_downloaded.m3u8 -ss 00:06 -i /Users/abc/Library/Caches/twitch-dl/videos/2510401818/chunked/metadata.txt -map_metadata 1 -t 28:42 -c copy -stats -loglevel warning file:2025-07-12_2510401818_aliythia_community_night_youtube_social.mp4
[in#1/ffmetadata @ 0x6000007d0000] could not seek to position 6.000
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2193478470).5091.9kbits/s speed=1.08e+03x
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2193476940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2194378470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2194376940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2195278470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2195276940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2196178470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2196176940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2197078470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2197076940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2198878470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2198876940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2199778470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2199776940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2200678470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2200676940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2201578470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2201576940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2202478470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2202476940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2203378470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2203376940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2204278470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2204276940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2205178470).e=5104.4kbits/s speed=1.04e+03x
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2205176940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2206078470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2206076940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2207878470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2207876940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
[mpegts @ 0x14d706e20] Packet corrupt (stream = 1, dts = 2208778470).
[hls @ 0x14d706bb0] Packet corrupt (stream = 1, dts = 2208776940).
[in#0/hls @ 0x6000007d8a00] corrupt input packet in stream 1
frame=103319 fps=60294 q=-1.0 Lsize= 1077138KiB time=00:28:42.08 bitrate=5124.0kbits/s speed=1e+03x
Deleting cached files...
Downloaded: 2025-07-12_2510401818_aliythia_community_night_youtube_social.mp4
```
```
$ ffprobe -hide_banner -i 2025-07-12_2510401818_aliythia_community_night_youtube_social.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2025-07-12_2510401818_aliythia_community_night_youtube_social.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : 🌸COMMUNITY NIGHT!🌸| !Youtube !Social
artist : Aliythia
date : 2025-07-12
encoder : Lavf61.7.100
show : Just Chatting
Duration: 00:28:42.03, start: 0.000000, bitrate: 5124 kb/s
Chapters:
Chapter #0:0: start 0.000000, end 1716.000000
Metadata:
title : Marbles on Stream
Chapter #0:1: start 1716.000000, end 1722.033667
Metadata:
title : Call of Dragons
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 5003 kb/s, 60 fps, 60 tbr, 90k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 105 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:2[0x3](eng): Data: bin_data (text / 0x74786574)
Metadata:
handler_name : SubtitleHandler
Unsupported codec with id 98314 for input stream 2
```
* Downloaded segments: 173, from 02339.ts to 02511.ts including them.
* Requested section: chapter 4 "Marbles on Stream", which goes from 23396 seconds to 25118 seconds, that is 1722 seconds.
* Resulting section: goes from 23390 seconds to 25112 seconds, that is 1722 seconds.
* Issues found: The timeline is shifted rearward to match the beginning of the first segment, instead of trimming it by the value in "-ss". Running the command with `--start 6:29:56 --end 6:58:38`, instead of `-c 4`, runs exactly the same ffmpeg command and yields exactly the same results. Another issue is that the chapters were not adjusted.
2. Example 2, with VOD 2507267603:
```
$ ./twitch-dl.3.1.0.pyz info 'https://www.twitch.tv/videos/2507267603?filter=all&sort=time'
Fetching video...
Fetching access token...
Fetching playlists...
Fetching chapters...
Video 2507267603
WATCHING PRO SCRIMS 🏆 PETERBOT vs. POLLO OFF SPAWN ⚔️
AussieAntics playing Fortnite
Published 2025-07-08 @ 21:37:34 Length: 4 h 28 min
https://www.twitch.tv/videos/2507267603
Playlists:
Name Group Resolution URL
---------- ---------- ---------- --------------------------------------------------------------------------------------------------------------------------------------
1080p60 1080p60 1920x1080 https://d2nvs31859zcd8.cloudfront.net/10828224dae0b4dd2c0b_aussieantics_314497094135_1752010648/1080p60/index-muted-W6C5P2TRJ6.m3u8
720p60 720p60 1280x720 https://d2nvs31859zcd8.cloudfront.net/10828224dae0b4dd2c0b_aussieantics_314497094135_1752010648/720p60/index-muted-W6C5P2TRJ6.m3u8
480p 480p30 852x480 https://d2nvs31859zcd8.cloudfront.net/10828224dae0b4dd2c0b_aussieantics_314497094135_1752010648/480p30/index-muted-W6C5P2TRJ6.m3u8
360p 360p30 640x360 https://d2nvs31859zcd8.cloudfront.net/10828224dae0b4dd2c0b_aussieantics_314497094135_1752010648/360p30/index-muted-W6C5P2TRJ6.m3u8
Audio Only audio_only None https://d2nvs31859zcd8.cloudfront.net/10828224dae0b4dd2c0b_aussieantics_314497094135_1752010648/audio_only/index-muted-W6C5P2TRJ6.m3u8
Placeholder Value
--------------- ------------------------------------------------
{channel} AussieAntics
{channel_login} aussieantics
{date} 2025-07-08
{datetime} 2025-07-08T21:37:34Z
{format} mp4
{game} Fortnite
{game_slug} fortnite
{id} 2507267603
{time} 21:37:34Z
{title} WATCHING PRO SCRIMS PETERBOT vs. POLLO OFF SPAWN
{title_slug} watching_pro_scrims_peterbot_vs_pollo_off_spawn
```
```
$ ./twitch-dl.3.1.0.pyz download --start 23:46 --end 24:46 --quality 1080p60 'https://www.twitch.tv/videos/2507267603?filter=all&sort=time'
Looking up video...
Found video: WATCHING PRO SCRIMS 🏆 PETERBOT vs. POLLO OFF SPAWN ⚔️ by AussieAntics playing Fortnite (04:28:10)
Target: 2025-07-08_2507267603_aussieantics_watching_pro_scrims_peterbot_vs_pollo_off_spawn.mp4
Fetching chapters...
Fetching access token...
Fetching playlists...
Fetching playlist...
Downloading to cache: /Users/abc/Library/Caches/twitch-dl/videos/2507267603/1080p60
Downloading init section init-0.mp4...
Downloading 0 VODs using 10 workers
Joining files...
ffmpeg -i /Users/abc/Library/Caches/twitch-dl/videos/2507267603/1080p60/playlist_downloaded.m3u8 -i /Users/abc/Library/Caches/twitch-dl/videos/2507267603/1080p60/metadata.txt -map_metadata 1 -c copy -stats -loglevel warning file:2025-07-08_2507267603_aussieantics_watching_pro_scrims_peterbot_vs_pollo_off_spawn.mp4
[hls @ 0x127e06720] Empty segment [/Users/abc/Library/Caches/twitch-dl/videos/2507267603/1080p60/playlist_downloaded.m3u8]
[out#0/mp4 @ 0x60000035c480] Output file does not contain any stream
Error opening output file file:2025-07-08_2507267603_aussieantics_watching_pro_scrims_peterbot_vs_pollo_off_spawn.mp4.
Error opening output files: Invalid argument
Error: Joining files failed
```
* Downloaded segments: 1, only init-0.mp4 with no "content" segments.
* Requested section: from 1426 seconds to 1486 seconds, that is 60 seconds.
* Resulting section: no output file, failure.
* Issues found: Downloads only the init file "init-0.mp4" and the playlist "playlist_downloaded.m3u8" contains only tags, no segments at all.
3. Example 3, with VOD 797302800:
```
$ ./twitch-dl.3.1.0.pyz info 'https://www.twitch.tv/videos/797302800?filter=all&sort=time'
Fetching video...
Fetching access token...
Fetching playlists...
Fetching chapters...
Video 797302800
AC Valhalla | Follow @shroud on socials
shroud playing Assassin's Creed Valhalla
Published 2020-11-09 @ 19:33:05 Length: 14 h 12 min
https://www.twitch.tv/videos/797302800
Playlists:
Name Group Resolution URL
------------- ---------- ---------- ------------------------------------------------------------------------------------------------------------------
936p60 source chunked 1664x936 https://d2nvs31859zcd8.cloudfront.net/ab7fabc76258d11b77d3_shroud_40411683838_1604950366/chunked/index-dvr.m3u8
720p60 720p60 1280x720 https://d2nvs31859zcd8.cloudfront.net/ab7fabc76258d11b77d3_shroud_40411683838_1604950366/720p60/index-dvr.m3u8
720p 720p30 1280x720 https://d2nvs31859zcd8.cloudfront.net/ab7fabc76258d11b77d3_shroud_40411683838_1604950366/720p30/index-dvr.m3u8
480p 480p30 852x480 https://d2nvs31859zcd8.cloudfront.net/ab7fabc76258d11b77d3_shroud_40411683838_1604950366/480p30/index-dvr.m3u8
360p 360p30 640x360 https://d2nvs31859zcd8.cloudfront.net/ab7fabc76258d11b77d3_shroud_40411683838_1604950366/360p30/index-dvr.m3u8
160p 160p30 284x160 https://d2nvs31859zcd8.cloudfront.net/ab7fabc76258d11b77d3_shroud_40411683838_1604950366/160p30/index-dvr.m3u8
Audio Only audio_only None https://d2nvs31859zcd8.cloudfront.net/ab7fabc76258d11b77d3_shroud_40411683838_1604950366/audio_only/index-dvr.m3u8
Placeholder Value
--------------- ------------------------------------
{channel} shroud
{channel_login} shroud
{date} 2020-11-09
{datetime} 2020-11-09T19:33:05Z
{format} mp4
{game} Assassin's Creed Valhalla
{game_slug} assassins_creed_valhalla
{id} 797302800
{time} 19:33:05Z
{title} AC Valhalla Follow shroud on socials
{title_slug} ac_valhalla_follow_shroud_on_socials
```
```
$ ./twitch-dl.3.1.0.pyz download --start 1:58:50 --end 1:59:16 --quality 936p60 'https://www.twitch.tv/videos/797302800?filter=all&sort=time'
Looking up video...
Found video: AC Valhalla | Follow @shroud on socials by shroud playing Assassin's Creed Valhalla (14:12:19)
Target: 2020-11-09_797302800_shroud_ac_valhalla_follow_shroud_on_socials.mp4
Fetching chapters...
Fetching access token...
Fetching playlists...
Fetching playlist...
Downloading to cache: /Users/abc/Library/Caches/twitch-dl/videos/797302800/chunked
Downloading 3 VODs using 10 workers
Downloaded 2/3 VODs 94% of ~36.8MB at 58.8MB/s ETA 00:00
Joining files...
ffmpeg -i /Users/abc/Library/Caches/twitch-dl/videos/797302800/chunked/playlist_downloaded.m3u8 -ss 00:04 -i /Users/abc/Library/Caches/twitch-dl/videos/797302800/chunked/metadata.txt -map_metadata 1 -t 00:26 -c copy -stats -loglevel warning file:2020-11-09_797302800_shroud_ac_valhalla_follow_shroud_on_socials.mp4
[in#1/ffmetadata @ 0x600001910700] could not seek to position 4.000
frame= 1562 fps=0.0 q=-1.0 Lsize= 26482KiB time=00:00:26.03 bitrate=8333.1kbits/s speed= 595x
Deleting cached files...
Downloaded: 2020-11-09_797302800_shroud_ac_valhalla_follow_shroud_on_socials.mp4
```
```
$ ffprobe -hide_banner -i 2020-11-09_797302800_shroud_ac_valhalla_follow_shroud_on_socials.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2020-11-09_797302800_shroud_ac_valhalla_follow_shroud_on_socials.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : AC Valhalla | Follow @shroud on socials
artist : shroud
date : 2020-11-09
encoder : Lavf61.7.100
show : Assassin's Creed Valhalla
Duration: 00:00:26.03, start: 0.000000, bitrate: 8333 kb/s
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1664x936, 8157 kb/s, 60 fps, 60 tbr, 90k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
```
* Downloaded segments: 3, from 00579.ts to 00581.ts including them.
* Requested section: from 7130 seconds to 7156 seconds, that is 26 seconds.
* Resulting section: goes from 7125.317 seconds to 7151.317 seconds, that is 26 seconds.
* Issues found: The timeline is shifted rearward to match the beginning of the first segment, instead of trimming it by the value in "-ss". The floating point absolute timestamp `7125.317` is where `578.ts` and `579.ts` touch, which is where Ffmpeg jumps to. Another issue is the approximation made to calculate "-ss", which should be `-ss 00:04.683` instead of `-ss 00:04`.
Conclusions about `twitch-dl`:
* Efficient: downloads only the required segments.
* Fails completely to download certain types of VODs.
* Has important issues with the chapters.
* When downloading .ts files, does not trim the first downloaded segment to discard its beginning (when needed). Instead, includes it completely.
* The project has serious bugs and is not updated frequently. It's not recommended to use in the current state, since the outcome is very unpredictable.
##### 4.1.3.2. TwitchDownloader
Tested version (TwitchDownloaderCLI only): [TwitchDownloader](https://github.com/lay295/TwitchDownloader) 1.56.1+9240cea217da04394e863990c5bea321c4959f9c.
This program can be downloaded as standalone executable, or compiled from source code for latest commit (recommended).
Allows to download:
* A section of a VOD (not for clips) with flags `-b, --beginning` and/or `-e, --ending`, up to milliseconds precision.
Roughly this is how it works:
1. Downloads only the needed whole segments.
2. Creates a concatenation file called `concat.txt`, listing only the needed segments.
3. Calls ffmpeg with `-ss` and `-t` flags against the concatenation file, to trim the fist and last segments (if needed) to match exactly the timeline of the section.
Here are 3 examples of 3 different VODs, one in MP4 format and the other two in Transport Stream format:
1. Example 1, with VOD 2510401818:
```
$ TwitchDownloaderCLI info --use-utf8 true --banner false -u 2510401818
[STATUS] - Fetching Video Info [1/1]
Video Info
┌─────────────┬────────────────────────────────────────┐
│ Key │ Value │
├─────────────┼────────────────────────────────────────┤
│ Streamer │ Aliythia │
│ Title │ 🌸COMMUNITY NIGHT!🌸| !Youtube !Social │
│ Length │ 8:31:50 │
│ Category │ Just Chatting │
│ Views │ 14,567 │
│ Created at │ 2025-07-12 11:21:59 UTC │
│ Description │ - │
└─────────────┴────────────────────────────────────────┘
Video Streams
┌────────────┬────────────┬─────┬────────────────────────┬──────────┬───────────┐
│ Name │ Resolution │ FPS │ Codecs │ Bitrate │ File size │
├────────────┼────────────┼─────┼────────────────────────┼──────────┼───────────┤
│ 1080p60 │ 1920x1080 │ 60 │ avc1.64002A, mp4a.40.2 │ 5805kbps │ ~20.76GiB │
│ 720p60 │ 1280x720 │ 60 │ avc1.4D4020, mp4a.40.2 │ 3244kbps │ ~11.60GiB │
│ 480p30 │ 852x480 │ 30 │ avc1.4D401F, mp4a.40.2 │ 1488kbps │ ~5.32GiB │
│ 360p30 │ 640x360 │ 30 │ avc1.4D401E, mp4a.40.2 │ 728kbps │ ~2.60GiB │
│ 160p30 │ 284x160 │ 30 │ avc1.4D400C, mp4a.40.2 │ 287kbps │ ~1.03GiB │
│ Audio Only │ - │ - │ mp4a.40.2 │ 268kbps │ ~984.5MiB │
└────────────┴────────────┴─────┴────────────────────────┴──────────┴───────────┘
Video Chapters
┌───────────────────┬─────────────┬─────────┬─────────┬─────────┐
│ Category │ Type │ Start │ End │ Length │
├───────────────────┼─────────────┼─────────┼─────────┼─────────┤
│ Just Chatting │ GAME_CHANGE │ 0:00 │ 1:22:00 │ 1:22:00 │
│ Call of Dragons │ GAME_CHANGE │ 1:22:00 │ 2:47:54 │ 1:25:54 │
│ Just Chatting │ GAME_CHANGE │ 2:47:54 │ 6:29:56 │ 3:42:02 │
│ Marbles on Stream │ GAME_CHANGE │ 6:29:56 │ 6:58:38 │ 28:42 │
│ Call of Dragons │ GAME_CHANGE │ 6:58:38 │ 8:23:26 │ 1:24:48 │
│ Just Chatting │ GAME_CHANGE │ 8:23:26 │ 8:31:50 │ 8:24 │
└───────────────────┴─────────────┴─────────┴─────────┴─────────┘
```
```
$ TwitchDownloaderCLI videodownload --banner false --quality best --threads 1 --id 2510401818 --beginning 6:29:56 --ending 6:58:38 --log-level All --output 2510401818_TwitchDownloaderCLI.mp4
[STATUS] - Fetching Video Info [1/4]
[STATUS] - Downloading 100% [2/4]
[STATUS] - Verifying Parts 100% [3/4]
[STATUS] - Finalizing Video 0% [4/4]
[VERBOSE] - Running "ffmpeg" in "/var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2510401818_638890516570542670" with args: -ss 6.000 -t 1722 -stats -y -avoid_negative_ts make_zero -analyzeduration 2147483647 -probesize 2147483647 -f concat -max_streams 2147483647 -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2510401818_638890516570542670/concat.txt -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2510401818_638890516570542670/metadata.txt -map_metadata 1 -c copy /TwitchDownloader/20250712T112159Z,2510401818/2510401818_TwitchDownloaderCLI.mp4
[STATUS] - Finalizing Video 100% [4/4]
```
```
$ ffprobe -hide_banner -i 2510401818_TwitchDownloaderCLI.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2510401818_TwitchDownloaderCLI.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : 🌸COMMUNITY NIGHT!🌸| !Youtube !Social (2510401818)
artist : Aliythia
date : 2025
encoder : Lavf61.7.100
comment : Created at: 2025-07-12 11:21:59Z
: Video id: 2510401818
: Views: 14567
genre : Just Chatting
Duration: 00:28:42.03, start: 0.000000, bitrate: 5124 kb/s
Chapters:
Chapter #0:0: start 0.000000, end 0.000000
Metadata:
title : Just Chatting
Chapter #0:1: start 0.000000, end 1722.000000
Metadata:
title : Marbles on Stream
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 5003 kb/s, 60 fps, 60 tbr, 90k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 105 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:2[0x3](eng): Data: bin_data (text / 0x74786574)
Metadata:
handler_name : SubtitleHandler
Unsupported codec with id 98314 for input stream 2
```
The command that TwitchDownloaderCLI called to combine the segments is this:
```
$ ffmpeg -ss 6.000 -t 1722 -stats -y -avoid_negative_ts make_zero -analyzeduration 2147483647 -probesize 2147483647 -f concat -max_streams 2147483647 -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2510401818_638890516570542670/concat.txt -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2510401818_638890516570542670/metadata.txt -map_metadata 1 -c copy /TwitchDownloader/20250712T112159Z,2510401818/2510401818_TwitchDownloaderCLI.mp4
```
* Downloaded segments: 173, from 2339.ts to 2511.ts including them.
* Requested section: from 23396 seconds to 25118 seconds, that is 1722 seconds.
* Resulting section: goes from 23396 seconds to 23396 seconds, that is 1722 seconds.
* Issues found: the video trimming works exactly as expected for both audio and video tracks. One minor issue is that there's the previous chapter with duration 0 which should not be there.
2. Example 2, with VOD 2507267603:
```
$ TwitchDownloaderCLI info --use-utf8 true --banner false -u 2507267603
[STATUS] - Fetching Video Info [1/1]
Video Info
┌─────────────┬───────────────────────────────────────────────────────┐
│ Key │ Value │
├─────────────┼───────────────────────────────────────────────────────┤
│ Streamer │ AussieAntics │
│ Title │ WATCHING PRO SCRIMS 🏆 PETERBOT vs. POLLO OFF SPAWN ⚔️ │
│ Length │ 4:28:10 │
│ Category │ Fortnite │
│ Views │ 88,185 │
│ Created at │ 2025-07-08 09:37:34 UTC │
│ Description │ - │
└─────────────┴───────────────────────────────────────────────────────┘
Video Streams
┌────────────┬────────────┬─────┬────────────────────────┬──────────┬───────────┐
│ Name │ Resolution │ FPS │ Codecs │ Bitrate │ File size │
├────────────┼────────────┼─────┼────────────────────────┼──────────┼───────────┤
│ 1080p60 │ 1920x1080 │ 60 │ avc1.64002A, mp4a.40.2 │ 6235kbps │ ~11.68GiB │
│ 720p60 │ 1280x720 │ 60 │ avc1.640020, mp4a.40.2 │ 2735kbps │ ~5.12GiB │
│ 480p30 │ 852x480 │ 30 │ avc1.64001F, mp4a.40.2 │ 1210kbps │ ~2.27GiB │
│ 360p30 │ 640x360 │ 30 │ avc1.4D401E, mp4a.40.2 │ 710kbps │ ~1.33GiB │
│ Audio Only │ - │ - │ mp4a.40.2 │ 203kbps │ ~389.8MiB │
└────────────┴────────────┴─────┴────────────────────────┴──────────┴───────────┘
Video Chapters
┌──────────┬─────────────┬───────┬─────────┬─────────┐
│ Category │ Type │ Start │ End │ Length │
├──────────┼─────────────┼───────┼─────────┼─────────┤
│ Fortnite │ GAME_CHANGE │ 0:00 │ 4:28:10 │ 4:28:10 │
└──────────┴─────────────┴───────┴─────────┴─────────┘
```
```
$ TwitchDownloaderCLI videodownload --banner false --quality best --threads 1 --id 2507267603 --beginning 23:46 --ending 24:46 --log-level All --output 2507267603_TwitchDownloaderCLI.mp4
[STATUS] - Fetching Video Info [1/4]
[VERBOSE] - Downloading header file from 'https://d2nvs31859zcd8.cloudfront.net/10828224dae0b4dd2c0b_aussieantics_314497094135_1752010648/1080p60/init-0.mp4' to '/var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2507267603_638890549438776580/init-0.mp4'
[STATUS] - Downloading 100% [2/4]
[STATUS] - Verifying Parts 100% [3/4]
[STATUS] - Finalizing Video 0% [4/4]
[VERBOSE] - Running "ffmpeg" in "/var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2507267603_638890549438776580" with args: -ss 6.000 -t 60 -stats -y -avoid_negative_ts make_zero -analyzeduration 2147483647 -probesize 2147483647 -f concat -max_streams 2147483647 -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2507267603_638890549438776580/concat.txt -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2507267603_638890549438776580/metadata.txt -map_metadata 1 -c copy /TwitchDownloader/20250708T213734Z,2507267603/2507267603_TwitchDownloaderCLI.mp4
[STATUS] - Finalizing Video 100% [4/4]
```
```
$ ffprobe -hide_banner -i 2507267603_TwitchDownloaderCLI.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2507267603_TwitchDownloaderCLI.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : WATCHING PRO SCRIMS 🏆 PETERBOT vs. POLLO OFF SPAWN ⚔️ (2507267603)
artist : AussieAntics
date : 2025
encoder : Lavf61.7.100
comment : Created at: 2025-07-08 21:37:34Z
: Video id: 2507267603
: Views: 88185
genre : Fortnite
Duration: 00:01:00.07, start: 0.000000, bitrate: 6411 kb/s
Chapters:
Chapter #0:0: start 0.000000, end 60.000000
Metadata:
title : Fortnite
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 6032 kb/s, 60 fps, 60 tbr, 1000k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 161 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:2[0x3](eng): Data: bin_data (text / 0x74786574), 0 kb/s
Metadata:
handler_name : SubtitleHandler
Unsupported codec with id 98314 for input stream 2
```
The command that TwitchDownloaderCLI called to combine the segments is this:
```
$ ffmpeg -ss 6.000 -t 60 -stats -y -avoid_negative_ts make_zero -analyzeduration 2147483647 -probesize 2147483647 -f concat -max_streams 2147483647 -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2507267603_638890549438776580/concat.txt -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/2507267603_638890549438776580/metadata.txt -map_metadata 1 -c copy /TwitchDownloader/20250708T213734Z,2507267603/2507267603_TwitchDownloaderCLI.mp4
```
* Downloaded segments: 7 + init file: from 142.mp4 to 148.mp4 including them, and init-0.mp4. The init file is not used at all.
* Requested section: from 1426 seconds to 1486 seconds, that is 60 seconds.
* Resulting section: goes from 1426 seconds to 1486 seconds, that is 60 seconds.
* Issues found: no issues found, the video trimming works exactly as expected for both audio and video tracks.
3. Example 3, with VOD 797302800:
```
$ TwitchDownloaderCLI info --use-utf8 true --banner false -u 797302800
[STATUS] - Fetching Video Info [1/1]
Video Info
┌─────────────┬─────────────────────────────────────────┐
│ Key │ Value │
├─────────────┼─────────────────────────────────────────┤
│ Streamer │ shroud │
│ Title │ AC Valhalla | Follow @shroud on socials │
│ Length │ 14:12:19 │
│ Category │ Assassin's Creed Valhalla │
│ Views │ 1,797,895 │
│ Created at │ 2020-11-09 07:33:05 UTC │
│ Description │ - │
└─────────────┴─────────────────────────────────────────┘
Video Streams
┌────────────┬────────────┬─────┬────────────────────────┬──────────┬───────────┐
│ Name │ Resolution │ FPS │ Codecs │ Bitrate │ File size │
├────────────┼────────────┼─────┼────────────────────────┼──────────┼───────────┤
│ 936p60 │ 1664x936 │ 60 │ avc1.64002A, mp4a.40.2 │ 8655kbps │ ~51.53GiB │
│ 720p60 │ 1280x720 │ 60 │ avc1.4D401F, mp4a.40.2 │ 3087kbps │ ~18.38GiB │
│ 720p30 │ 1280x720 │ 30 │ avc1.4D401F, mp4a.40.2 │ 2240kbps │ ~13.34GiB │
│ 480p30 │ 852x480 │ 30 │ avc1.4D401E, mp4a.40.2 │ 1422kbps │ ~8.47GiB │
│ 360p30 │ 640x360 │ 30 │ avc1.4D401E, mp4a.40.2 │ 697kbps │ ~4.15GiB │
│ 160p30 │ 284x160 │ 30 │ avc1.4D400C, mp4a.40.2 │ 285kbps │ ~1.70GiB │
│ Audio Only │ - │ - │ mp4a.40.2 │ 214kbps │ ~1.28GiB │
└────────────┴────────────┴─────┴────────────────────────┴──────────┴───────────┘
Video Chapters
┌───────────────────────────┬─────────────┬───────┬──────────┬──────────┐
│ Category │ Type │ Start │ End │ Length │
├───────────────────────────┼─────────────┼───────┼──────────┼──────────┤
│ Assassin's Creed Valhalla │ GAME_CHANGE │ 0:00 │ 14:12:19 │ 14:12:19 │
└───────────────────────────┴─────────────┴───────┴──────────┴──────────┘
```
```
$ TwitchDownloaderCLI videodownload --banner false --quality best --threads 1 --id 797302800 --beginning 1:58:50 --ending 1:59:16 --log-level All --output 797302800_TwitchDownloaderCLI.mp4
[STATUS] - Fetching Video Info [1/4]
[STATUS] - Downloading 100% [2/4]
[STATUS] - Verifying Parts 100% [3/4]
[STATUS] - Finalizing Video 0% [4/4]
[VERBOSE] - Running "ffmpeg" in "/var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/797302800_638890563912517940" with args: -ss 4.683 -t 26 -stats -y -avoid_negative_ts make_zero -analyzeduration 2147483647 -probesize 2147483647 -f concat -max_streams 2147483647 -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/797302800_638890563912517940/concat.txt -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/797302800_638890563912517940/metadata.txt -map_metadata 1 -c copy /TwitchDownloader/20201109T193305Z,797302800/797302800_TwitchDownloaderCLI.mp4
[STATUS] - Finalizing Video 100% [4/4]
```
```
$ ffprobe -hide_banner -i 797302800_TwitchDownloaderCLI.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '797302800_TwitchDownloaderCLI.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : AC Valhalla | Follow @shroud on socials (797302800)
artist : shroud
date : 2020
encoder : Lavf61.7.100
comment : Created at: 2020-11-09 19:33:05Z
: Video id: 797302800
: Views: 1797895
genre : Assassin's Creed Valhalla
Duration: 00:00:26.03, start: 0.000000, bitrate: 7243 kb/s
Chapters:
Chapter #0:0: start 0.000000, end 26.000000
Metadata:
title : Assassin's Creed Valhalla
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1664x936, 8202 kb/s, 60 fps, 60 tbr, 90k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:2[0x3](eng): Data: bin_data (text / 0x74786574), 0 kb/s
Metadata:
handler_name : SubtitleHandler
Unsupported codec with id 98314 for input stream 2
```
The command that TwitchDownloaderCLI called to combine the segments is this:
```
$ ffmpeg -ss 4.683 -t 26 -stats -y -avoid_negative_ts make_zero -analyzeduration 2147483647 -probesize 2147483647 -f concat -max_streams 2147483647 -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/797302800_638890563912517940/concat.txt -i /var/folders/x9/hs7t3ncj84v9qw5mdgtptn_80000gn/T/TwitchDownloader/797302800_638890563912517940/metadata.txt -map_metadata 1 -c copy /TwitchDownloader/20201109T193305Z,797302800/797302800_TwitchDownloaderCLI.mp4
```
* Downloaded segments: 3, from 579.ts to 581.ts including them.
* Requested section: from 7130 seconds to 7156 seconds, that is 26 seconds.
* Resulting section: goes from 7126 seconds to 7152 seconds, that is 26 seconds.
* Issues found: *out-of-keyframe-trim.
*out-of-keyframe-trim: The audio track lasts 26 seconds, and exactly the requested section of the VOD (7130-1256 absolute seconds). The video track however, lasts 22 seconds, starts playing at offset 4 seconds of the mp4 file, so goes from 4-26 (7134-1256 absolute seconds). This issue is due to using the copy method in ffmpeg `-c copy`, which only starts including the video after the first keyframe after the starting trimming point. So the trim is requested to start at `1:58:50`, while surrounding keyframes are at `01:58:49.434` and `01:58:53.600`. So what happens is that the video starts being trimmed at or after the immediate keyframe, so the video is included only after `01:58:53.600`. The audio is included correctly because doesn't work like that. As a result, the output file has the tracks like this (v-video and a-audio, each letter or symbol is half a second):
```
········vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
```
Placing the seek before the input file instead of after after it, when doing it over the already downloaded whole VOD, makes the audio and video tracks to be included correctly:
```
$ ffmpeg -ss 7130 -to 7156 -i 20201109T193305Z_797302800.mp4 -c copy 797302800_audio_and_video_tracks_ok.mp4
$ ffmpeg -i 20201109T193305Z_797302800.mp4 -ss 7130 -to 7156 -c copy 797302800_video_track_only_after_next_keyframe_audio_ok.mp4
```
This demonstrates that under certain conditions, it's possible to trim a file with audio+video accurately outside of a keyframe.
Note: about multimedia players, video editors, information/metadata extractors, etc.: when the audio and video tracks do not match exactly the same duration inside the file or the same timeline (shifted one to each other), certain software may not display the values correctly or skip parts of it; in those cases extracting each track and parsing it separately can help. Some software may display approximate values when the bitrate is variable, and the full file is not parsed (like .aac with no header/container).
Conclusions about `TwitchDownloader`:
* Efficient: downloads only the required segments.
* Calculates accurately "-ss" values down to the millisecond.
* Has cosmetic issues with the chapters.
* Trims VODs starting at either the exact requested timestamp or from next keyframe.
* The project has a fair quality, but some bugs need to be ironed out, specially: if the next keyframe is way too far from the requested seek point, an important part of the video track can be missing. The fact that does not include the init file in mp4 formats can cause failure if the stream needs it.
##### 4.1.3.3. twitch-dlp
Tested version: [twitch-dlp](https://github.com/DmitryScaletta/twitch-dlp) v0.5.18.
Allows to download:
* A section of a VOD with flag `--download-sections`. Clips claim to be included.
Roughly this is how it works:
1. Downloads whole segments.
2. Concatenates the segments with Node+Ffmpeg or Ffmpeg only. No trimming is done.
Here are 3 examples of 3 different VODs, one in MP4 format and the other two in Transport Stream format:
1. Example 1, with VOD 2510401818:
```
$ node twitch-dlp.js -F 'https://www.twitch.tv/videos/2510401818?filter=all&sort=time'
Downloading video access token
Downloading video metadata
Downloading video manifest
┌─────────┬──────────────┬──────────────┬──────┬───────────────┬────────┐
│ (index) │ format_id │ resolution │ fps │ total_bitrate │ source │
├─────────┼──────────────┼──────────────┼──────┼───────────────┼────────┤
│ 0 │ 'Audio_Only' │ 'audio only' │ null │ '263k' │ null │
│ 1 │ '160p' │ '284x160' │ 30 │ '281k' │ null │
│ 2 │ '360p' │ '640x360' │ 30 │ '712k' │ null │
│ 3 │ '480p' │ '852x480' │ 30 │ '1454k' │ null │
│ 4 │ '720p60' │ '1280x720' │ 60 │ '3169k' │ null │
│ 5 │ '1080p60' │ '1920x1080' │ 60 │ '5669k' │ true │
└─────────┴──────────────┴──────────────┴──────┴───────────────┴────────┘
```
```
$ node twitch-dlp.js --keep-fragments --download-sections '*6:29:56-6:58:38' 'https://www.twitch.tv/videos/2510401818?filter=all&sort=time'
Downloading video access token
Downloading video metadata
Downloading video manifest
[unmute] The video is old, not trying to unmute
Downloading playlist
[download] 100.0% of ~ 1.08GB at 27.64MB/s ETA 00:00:00 (frag 174/174)
[mpegts @ 0x10af88870] Packet corrupt (stream = 1, dts = 2193478470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 82794060).
[mpegts @ 0x13f6b20f0] Packet corrupt (stream = 1, dts = 2194378470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 83694060).
[h264 @ 0x141093960] missing picture in access unit with size 6
[mpegts @ 0x13f6b32a0] Packet corrupt (stream = 1, dts = 2195278470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 84594060).
[mpegts @ 0x1411dc2d0] Packet corrupt (stream = 1, dts = 2196178470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 85494060).
[mpegts @ 0x13f6d4f50] Packet corrupt (stream = 1, dts = 2197078470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 86394060).
[mpegts @ 0x13f6e5ce0] Packet corrupt (stream = 1, dts = 2198878470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 88194060).
[h264 @ 0x13f21c110] missing picture in access unit with size 6
[mpegts @ 0x10afdf100] Packet corrupt (stream = 1, dts = 2199778470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 89094060).
[mpegts @ 0x1410b3f40] Packet corrupt (stream = 1, dts = 2200678470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 89994060).
[mpegts @ 0x1410b3f40] Packet corrupt (stream = 1, dts = 2201578470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 90894060).
[mpegts @ 0x1410b3f40] Packet corrupt (stream = 1, dts = 2202478470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 91794060).
[h264 @ 0x1410d08d0] missing picture in access unit with size 6
[mpegts @ 0x13f32b910] Packet corrupt (stream = 1, dts = 2203378470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 92694060).
[mpegts @ 0x13f32b8f0] Packet corrupt (stream = 1, dts = 2204278470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 93594060).
[mpegts @ 0x1410f3f60] Packet corrupt (stream = 1, dts = 2205178470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 94494060).
[mpegts @ 0x1410f3da0] Packet corrupt (stream = 1, dts = 2206078470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 95394060).
[mpegts @ 0x1410f3da0] Packet corrupt (stream = 1, dts = 2207878470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 97194060).
[mpegts @ 0x1410f3da0] Packet corrupt (stream = 1, dts = 2208778470).
Last message repeated 1 times
[concat @ 0x13f605180] Packet corrupt (stream = 520, dts = 98094060).
Input #0, concat, from '/twitch-dlp/🌸COMMUNITY NIGHT!🌸｜ !Youtube !Social [v2510401818].mp4-ffconcat.txt':
Duration: 00:29:00.00, start: 0.000000, bitrate: 0 kb/s
Stream mapping:
Output #0, mp4, to '/twitch-dlp/🌸COMMUNITY NIGHT!🌸｜ !Youtube !Social [v2510401818].mp4':
Metadata:
encoder : Lavf61.7.100
Press [q] to stop, [?] for help
[in#0/concat @ 0x600001014900] corrupt input packet in stream 520trate=5093.5kbits/s speed= 307x
[in#0/concat @ 0x600001014900] corrupt input packet in stream 520trate=5099.1kbits/s speed= 305x
Last message repeated 12 times
[in#0/concat @ 0x600001014900] corrupt input packet in stream 520trate=5102.6kbits/s speed= 306x
Last message repeated 1 times
[out#0/mp4 @ 0x600001918540] video:1062435KiB audio:23314KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.259106%
frame=104400 fps=18240 q=-1.0 Lsize= 1088562KiB time=00:28:59.96 bitrate=5125.1kbits/s speed= 304x
[stats] Fragments
┌──────────────────────────┬────────┐
│ (index) │ Values │
├──────────────────────────┼────────┤
│ Total │ 174 │
│ Downloaded │ 174 │
│ Muted │ 18 │
│ Unmuted total │ 0 │
│ Unmuted (same format) │ 0 │
│ Unmuted (replaced audio) │ 0 │
└──────────────────────────┴────────┘
```
```
$ ffprobe -hide_banner -i '🌸COMMUNITY NIGHT!🌸｜ !Youtube !Social [v2510401818].mp4'
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '🌸COMMUNITY NIGHT!🌸｜ !Youtube !Social [v2510401818].mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Duration: 00:29:00.01, start: 0.000000, bitrate: 5124 kb/s
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 5002 kb/s, 60 fps, 60 tbr, 90k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 107 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
```
* Downloaded segments: 174, from 2339.ts to 2512.ts including them.
* Requested section: from 23396 seconds to 25118 seconds, that is 1722 seconds.
* Resulting section: goes from 23390 seconds to 25130 seconds, that is 1740 seconds.
* Issues found: No chapters at all. The downloaded section is just the concatenation of all the segments that include any part of the desired timeline, plus one more at the end (2512.ts) which is already out of the requested section and totally unnecessary.
2. Example 2, with VOD 2507267603:
```
$ node twitch-dlp.js -F 'https://www.twitch.tv/videos/2507267603?filter=all&sort=time'
Downloading video access token
Downloading video metadata
Downloading video manifest
┌─────────┬──────────────┬──────────────┬──────┬───────────────┬────────┐
│ (index) │ format_id │ resolution │ fps │ total_bitrate │ source │
├─────────┼──────────────┼──────────────┼──────┼───────────────┼────────┤
│ 0 │ 'Audio_Only' │ 'audio only' │ null │ '198k' │ null │
│ 1 │ '360p' │ '640x360' │ 30 │ '694k' │ null │
│ 2 │ '480p' │ '852x480' │ 30 │ '1182k' │ null │
│ 3 │ '720p60' │ '1280x720' │ 60 │ '2671k' │ null │
│ 4 │ '1080p60' │ '1920x1080' │ 60 │ '6089k' │ true │
└─────────┴──────────────┴──────────────┴──────┴───────────────┴────────┘
```
```
$ node twitch-dlp.js --keep-fragments --download-sections '*23:46-24:46' 'https://www.twitch.tv/videos/2507267603?filter=all&sort=time'
Downloading video access token
Downloading video metadata
Downloading video manifest
[unmute] The video is old, not trying to unmute
Downloading playlist
[download] 100.0% of ~ 59.43MB at 47.92MB/s ETA 00:00:00 (frag 9/9)
WARN: ffconcat merge method is not supported for fMP4 streams. Using append instead
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'file:/twitch-dlp/WATCHING PRO SCRIMS 🏆 PETERBOT vs. POLLO OFF SPAWN ⚔️ [v2507267603].mp4':
Metadata:
major_brand : mp42
minor_version : 1
compatible_brands: isommp42dashavc1iso6hlsf
Duration: 00:26:04.07, start: 1484.065000, bitrate: 318 kb/s
Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 161 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 6001 kb/s, 60 fps, 60 tbr, 1000k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Output #0, mp4, to 'file:/twitch-dlp/WATCHING PRO SCRIMS 🏆 PETERBOT vs. POLLO OFF SPAWN ⚔️ [v2507267603].temp.mp4':
Metadata:
major_brand : mp42
minor_version : 1
compatible_brands: isommp42dashavc1iso6hlsf
encoder : Lavf61.7.100
Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 161 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:1(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 6001 kb/s, 60 fps, 60 tbr, 1000k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Press [q] to stop, [?] for help
[mp4 @ 0x12d713370] Starting second pass: moving the moov atom to the beginning of the file
[out#0/mp4 @ 0x6000018dc000] video:58611KiB audio:1581KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.231231%
frame= 4800 fps=0.0 q=-1.0 Lsize= 60331KiB time=00:01:19.96 bitrate=6180.4kbits/s speed= 159x
[stats] Fragments
┌────────────┬────────┐
│ (index) │ Values │
├────────────┼────────┤
│ Total │ 8 │
│ Downloaded │ 8 │
│ Muted │ 0 │
└────────────┴────────┘
```
```
$ ffprobe -hide_banner -i 'WATCHING PRO SCRIMS 🏆 PETERBOT vs. POLLO OFF SPAWN ⚔️ [v2507267603].mp4'
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'WATCHING PRO SCRIMS 🏆 PETERBOT vs. POLLO OFF SPAWN ⚔️ [v2507267603].mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Duration: 00:01:20.00, start: 0.000000, bitrate: 6177 kb/s
Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 161 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 6001 kb/s, 60 fps, 60 tbr, 1000k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
```
* Downloaded segments: 8 + init file: from 142.mp4 to 149.mp4 including them, and init-0.mp4.
* Requested section: from 1426 seconds to 1486 seconds, that is 60 seconds.
* Resulting section: goes from 1420 seconds to 1500 seconds, that is 80 seconds.
* Issues found: No chapters at all. The downloaded section is just the concatenation of the init file and all the segments that include any part of the desired timeline, plus one more at the end (149.mp4) which is already out of the requested section and totally unnecessary.
3. Example 3, with VOD 797302800:
```
$ node twitch-dlp.js -F 'https://www.twitch.tv/videos/797302800?filter=all&sort=time'
Downloading video access token
Downloading video metadata
Downloading video manifest
┌─────────┬──────────────┬──────────────┬──────┬───────────────┬────────┐
│ (index) │ format_id │ resolution │ fps │ total_bitrate │ source │
├─────────┼──────────────┼──────────────┼──────┼───────────────┼────────┤
│ 0 │ 'Audio_Only' │ 'audio only' │ null │ '210k' │ null │
│ 1 │ '160p' │ '284x160' │ 30 │ '279k' │ null │
│ 2 │ '360p' │ '640x360' │ 30 │ '681k' │ null │
│ 3 │ '480p' │ '852x480' │ 30 │ '1389k' │ null │
│ 4 │ '720p' │ '1280x720' │ 30 │ '2188k' │ null │
│ 5 │ '720p60' │ '1280x720' │ 60 │ '3015k' │ null │
│ 6 │ '936p60' │ '1664x936' │ 60 │ '8452k' │ true │
└─────────┴──────────────┴──────────────┴──────┴───────────────┴────────┘
```
```
$ node twitch-dlp.js --keep-fragments --download-sections '*1:58:50-1:59:16' 'https://www.twitch.tv/videos/797302800?filter=all&sort=time'
Downloading video access token
Downloading video metadata
Downloading video manifest
[unmute] The video is old, not trying to unmute
Downloading playlist
[download] 100.0% of ~ 50.31MB at 15.39MB/s ETA 00:00:00 (frag 4/4)
Input #0, concat, from '/twitch-dlp/AC Valhalla ｜ Follow @shroud on socials [v797302800].mp4-ffconcat.txt':
Duration: 00:00:48.90, start: 0.000000, bitrate: 0 kb/s
Stream mapping:
Output #0, mp4, to '/twitch-dlp/AC Valhalla ｜ Follow @shroud on socials [v797302800].mp4':
Metadata:
encoder : Lavf61.7.100
Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1664x936, q=2-31, 60 fps, 60 tbr, 90k tbn
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 164 kb/s
Press [q] to stop, [?] for help
[out#0/mp4 @ 0x600000440000] video:48819KiB audio:973KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.160206%
frame= 2934 fps=0.0 q=-1.0 Lsize= 49872KiB time=00:00:48.86 bitrate=8360.5kbits/s speed= 170x
[stats] Fragments
┌────────────┬────────┐
│ (index) │ Values │
├────────────┼────────┤
│ Total │ 4 │
│ Downloaded │ 4 │
│ Muted │ 0 │
└────────────┴────────┘
```
```
$ ffprobe -hide_banner -i 'AC Valhalla ｜ Follow @shroud on socials [v797302800].mp4'
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'AC Valhalla ｜ Follow @shroud on socials [v797302800].mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Duration: 00:00:48.91, start: 0.000000, bitrate: 8352 kb/s
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1664x936, 8179 kb/s, 60 fps, 60 tbr, 90k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
```
* Downloaded segments: 3, from 579.ts to 582.ts including them.
* Requested section: from 7130 seconds to 7156 seconds, that is 26 seconds.
* Resulting section: goes from 7125.317 seconds to 7174.217 seconds, that is 48.9 seconds.
* Issues found: No chapters at all. The downloaded section is just the concatenation of the init file and all the segments that include any part of the desired timeline, plus one more at the end (582.ts) which is already out of the requested section and totally unnecessary.
Conclusions about `twitch-dlp`:
* Almost efficient: downloads only the required segments + 1 more at the end that is not needed.
* No chapters at all.
* Does not trim anything, only concatenates downloaded whole segments.
* The project has a good quality. Despite downloading a section wider than requested, prevents missing any requested content by including whole segments. By avoiding any trim operation prevents errors related to it. This is almost like downloading the entire VOD but like if the VOD were shorter to begin with. The most common issue that could arise is due to the init file not working well with segments that are not consecutive (older VODs). Also it includes the init file for concatenating mp4 files, which can be required.
## 5. TWITCH
This information about the way Twitch.tv and its services work, is according to my own deductions.
* Past Broadcasts always expire after 7/14/60 days, according to [Video On Demand](https://help.twitch.tv/s/article/video-on-demand?language=en_US#faq), regardless of how Premium the owner is.
* Highlights and Uploads never expire.
* Uploads require the twitch account to be an Affiliate or Partner.
* Past Broadcasts can be converted entirely to Highlights by the owner/streamer.
* A Live Stream will create immediately also a Past Broadcast stream with another ID, if the account has certain options enabled. Its duration and m3u8 will be synchronized every few minutes, and after the live stream ends.
* The streamer may require to be a (paid) subscriber to watch: Live Stream, Past Broadcast, Upload, Highlight. To be a subscriber it not related to be a follower or registered (with a twitch account).
* If the Live Stream does not require to be a subscriber, the stream can be watched by unregistered users (public) or with a registered account. The streamer doesn't have the option to allow viewers based on that.
* The tag `#EXT-X-DISCONTINUITY` should only appear in the M3U8 of Uploads and Highlights, not Live / Past Broadcasts. This is because if a live stream ends (on purpose or by connection error) the stream immediately ends, and the streamer has to start another, which will have another ID. There's no option to "pause" the stream. This is why Twitch added in 2019 the feature [Disconnect Protection](https://help.twitch.tv/s/article/Disconnect-Protection?language=en_US). When enabled, can hold the stream live for up to 90 seconds in case of connection error.
### 5.1. Twitch VOD/ID structure
Twitch has for each VOD or ID a master playlist (also called "all qualities playlist" or "Video Manifest").
This master playlist can be "muted" (contain some/all segments muted), so there can be "duplicates" of the master playlist for censorship/copyright reasons. Each one placed at a different URL (usually adding "-muted" to the filename).
- The master playlist describes all the streams available, with both headers (lines starting with "#", which act as comments) and the URL to download the playlist for that "quality".
- The quality playlist contains the list of all the segments that comprise the whole stream, with additional metadata in the headers (also starting with "#").
- Each stream can contain one or more "useful" tracks, all of them into a single multimedia file. Currently up to 1 audio track and/or 1 video track inside.
- The TS streams contain an additional track of metadata, and the MP4 files rarely its own metadata in the container.
The master playlist can be something like this, for example ID 1923916260:
```
#EXTM3U
#EXT-X-TWITCH-INFO:ORIGIN="s3",B="false",REGION="REDACTED",USER-IP="REDACTED",SERVING-ID="REDACTED",CLUSTER="cloudfront_vod",USER-COUNTRY="REDACTED",MANIFEST-CLUSTER="cloudfront_vod"
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="1080p60",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=6251644,CODECS="avc1.64002A,mp4a.40.2",RESOLUTION=1920x1080,VIDEO="chunked",FRAME-RATE=58.863
https://d2nvs31859zcd8.cloudfront.net/00ead364f55d62184702_twitch_23923135812_8256104857/chunked/highlight-1923916260.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p60",NAME="720p60",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=3025011,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=1280x720,VIDEO="720p60",FRAME-RATE=58.863
https://d2nvs31859zcd8.cloudfront.net/00ead364f55d62184702_twitch_23923135812_8256104857/720p60/highlight-1923916260.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p30",NAME="720p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=2234064,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=1280x720,VIDEO="720p30",FRAME-RATE=29.826
https://d2nvs31859zcd8.cloudfront.net/00ead364f55d62184702_twitch_23923135812_8256104857/720p30/highlight-1923916260.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="480p30",NAME="480p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=1341635,CODECS="avc1.4D401E,mp4a.40.2",RESOLUTION=852x480,VIDEO="480p30",FRAME-RATE=29.826
https://d2nvs31859zcd8.cloudfront.net/00ead364f55d62184702_twitch_23923135812_8256104857/480p30/highlight-1923916260.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="Audio Only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=177672,CODECS="mp4a.40.2",VIDEO="audio_only"
https://d2nvs31859zcd8.cloudfront.net/00ead364f55d62184702_twitch_23923135812_8256104857/audio_only/highlight-1923916260.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="360p30",NAME="360p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=722806,CODECS="avc1.4D401E,mp4a.40.2",RESOLUTION=640x360,VIDEO="360p30",FRAME-RATE=29.826
https://d2nvs31859zcd8.cloudfront.net/00ead364f55d62184702_twitch_23923135812_8256104857/360p30/highlight-1923916260.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="160p30",NAME="160p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=309679,CODECS="avc1.4D400C,mp4a.40.2",RESOLUTION=284x160,VIDEO="160p30",FRAME-RATE=29.826
https://d2nvs31859zcd8.cloudfront.net/00ead364f55d62184702_twitch_23923135812_8256104857/160p30/highlight-1923916260.m3u8
```
Some values were redacted, specially the user IP, which is the public IP of the computer making the request.
The values in the headers or the URL, most of the time are just placeholders to tell things apart, do not have to match the real values of the content. For example the framerate or the bandwidth. The resolution looks like it's always accurate when disclosed.
A quality playlist looks like this (chunked quality in this case for the same ID):
```
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#ID3-EQUIV-TDTG:2023-08-25T20:39:38
#EXT-X-PLAYLIST-TYPE:EVENT
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TWITCH-ELAPSED-SECS:0.000
#EXT-X-TWITCH-TOTAL-SECS:58.0
#EXTINF:8.145,
1923916260v0-133.ts
#EXT-X-DISCONTINUITY
#EXTINF:10.267,
134.ts
#EXTINF:10.033,
135.ts
#EXTINF:10.066,
136.ts
#EXTINF:10.067,
137.ts
#EXT-X-DISCONTINUITY
#EXTINF:9.422,
1923916260v5-138.ts
#EXT-X-ENDLIST
```
Key points in tree-view mode (made up example):
```
https://d2nvs31859zcd8.cloudfront.net/ ← one of ±14 CloudFront VOD domains
└── 00ead364f55d62184702_twitch_23923135812_8256104857/ ← full-vod-path (hash + channel + stream-id + utcStart)
├── chunked/highlight-1923916260.m3u8 ← variant-playlist for “chunked” (Source)
├── 720p60/highlight-1923916260.m3u8 ← variant-playlist for 720p60
├── 480p30/highlight-1923916260.m3u8 ← …and so on for every quality
│ • playlist file name can vary: • "index-dvr.m3u8" for full VOD
│ – index-dvr.m3u8 (normal archive) • "highlight-*.m3u8" for highlights
│ – highlight-<id>.m3u8 (this example)
│ - index-muted-IU66FWZNAG.m3u8 ← playlists with muted segments contain the word "muted"
├── mobile/ ← (legacy) placeholders may be a predefined word: chunked, audio_only, high, medium, low, mobile
├── 1440p60-av01/ ← (new) placeholders may be a combination or resolution [+ optionally framerate [+ optionally codec]]
└── chunked/ ← quality placeholder = segment-base (prefix)
├── 130.ts ← media segment (may have ?start_offset=…&end_offset=…, may start ≠ 0)
├── 131.ts
├── 1923916260v5-138.ts ← non-consecutive number = mute splice / highlight gap
│ (empty lines are allowed between #EXTINF blocks)
└── 149-muted.ts ← certain segments (or all) can contain muted audio track (not removed)
└── 150.ts ← last segment (can be shorter than TARGETDURATION (the rest of them))
```
- The segments are referenced to same base URL as its quality playlist URL, but that base URL is not included in the quality playlist itself as received from Twitch.
- Segment filenames usually start at filename 0 for whole VODs, but if it's a cut like a highlight then may not.
- Segment filenames may not be consecutive "7.ts..8.ts.." if the timeline was cut for same reason as before (from Twitch Dashboard this can be done).
- All segments have the same duration (example 10 seconds) except the last which has to accommodate.
- Streams in MP4 have an init file, while in TS don't. The extension of the segments is the same for all of them and matches the final file.
- There can be empty lines in the quality playlist.
- The filenames of the playlists can match different patterns.
- Streams in TS (Transport Stream) can be trimmed, fetching only the intermediate and concatenating them. With MP4 streams this is not possible, due to full reliance on each other's contiguous and the header file.
- Some segments or all of them can be muted. This does not mean that the audio tracks is removed. Then the word "muted" appears in the playlists and affected segments' filenames.
- The segments that are source for the muted ones are renamed to "-unmuted" in the original playlists, but are 403, and also the original ones. So only the muted ones can be downloaded after some time.
- The placeholders can be a combination of resolution+p(+framerate(+codec)), or a predefined word.
- Segments can contain query parameters after the filename.
- All VODs tested include the "audio_only" stream but not all include the "chunked" one. The chunked is regarded as the best quality in the VOD.
- Some VODs are public, and the master playlist can be obtained freely; other VODs are subscriber-only, and the endpoint requires to provide the OAuth Access Token to serve the master playlist.
- There are 4 types of VODs: archive (past broadcast), highlight, upload and clip.
### 5.2. Transport Stream files
The .ts files (Transport Stream) served by Twitch have 3 tracks (meaning multimedia streams, not the whole Twitch stream): audio, video and data. They are not always in the same order even for the same Twitch stream (highlight, past broadcast, upload). The tracks are identified uniquely still by the hexadecimal identifier, so concatenating them into `output.ts` renders a valid file. Some programs or media players complain, but it's still a valid file and most can play it fine. Converting output.ts to mp4 just in copy mode is enough fo avoid warnings about that particular issue, but should not be cause of any playback error or data loss.
The default order for `Live Streams` and `Past Broadcasts` is:
```
Stream #0:0[0x100]: Audio
Stream #0:1[0x101]: Video
Stream #0:2[0x102]: Data: timed_id3 (ID3 / 0x20334449)
```
Edited parts/segments (like when creating a Highlight), or muted segments due to copyrighted audio, get renamed to a different filename pattern. Also they get another channel order:
```
Stream #0:0[0x100]: Audio
Stream #0:1[0x102]: Data: timed_id3 (ID3 / 0x20334449)
Stream #0:2[0x101]: Video
```
The Twitch ID 637388605 is Upload type, in .ts format, and follows the same order as broadcasts.
### 5.3. Highlight creation
The [Higlights](https://help.twitch.tv/s/article/creating-highlights-and-stream-markers?language=en_US) can be created in the [Twitch Video Producer](https://dashboard.twitch.tv/u/USERNAME/content/video-producer) by the owner of the original ID. Clicking on `Highlight` over any past stream, will create a new ID specific for that Highlight, which will be unique (different ID from the Past Broadcast ID and the other Highlights, even if they are based on the same past stream), like [Edit new Highlight](https://dashboard.twitch.USERNAME/content/video-producer/highlighter/NEW-ID).
The timeline in the editor, is basically a method to trim out of from the whole past broadcast, the sections which are not wanted, and remain those that are desired as "segments". It's like a "best off" collection. The "segments" which have been already accepted are filled in purple, and in yellow the one which is being currently edited. The cuts are made by choosing `Start segment` and/or `End segment` at the Playhead, with the resulting segment in between. Additional "segments" can be created by moving the slider outide already existing "segments", and clicking `Add new segment` to make another.
Such editor is a very basic linear video editor, with the following limitations:
* The "segment" cut point can only be made at certain points in the timestamp.
* The "segments" can't overlap, and its content can't be moved.
* All the "segments" have to be from the same Past Broadcast, because there's only one source file in the time editor.
* Segments can't be copied and pasted over the timeline, to duplicate them or make the Highlight larger than the original stream.
More details about the highlights:
* There can be any number of Highlights made from each past stream, because each one gets a new ID on its own.
* The "segments" as referred by Twitch, are portions of the timeline, but they don't have to be exactly the same segments which comprise the resulting stream (parts in the m3u8). Some may match file-by-file, but some "segments" can be made of one or more segments/parts if they are large enough.
* Each highlight can have one or more "segments", as much as allowed by the cutting points.
* The Highlight can be made of the entire past stream as a single "segment". This is what some streamers do before past broadcasts expire, to preserve them under a different ID. This is also possible because there's no highlight duration limit. This does not require any additional payment or subscription.
* Converting a whole Past Broadcast to Highlight may still cause it to be of slightly less duration, even when cutting at the very edges. For example a past broadcast of 19.168 seconds (10.034+9.134) turned out to last 19.0 seconds (10.034+8.966) with the `#EXT-X-DISCONTINUITY` tag in the middle of the segments in the m3u8 playlist. The first segment/part `0.ts` had the same hash in both cases (exactly same file). But the last segment/part had different name and hash. In theory could be possible for the whole highlight stream to match exactly the broadcast (same output.ts hash), but the broadcast would have to be made in a way/duration that Twitch likes, probably ending exactly in a keyframe or something like that, so the Video Producer just copies all the parts without any modification up to the last one. It's still weird because the broadcast was supposed to have a valid codec format in the first place.
* The parts that get modified when producing the highlight, have different filename patterns, and the channels order in those ts parts is also changed. While the past broadcast follows the channel order `Audio-Video-Data`, the modified parts follow the order `Audio-Data-Video`, while the others keep the same because are copied directly. This happens also to muted segments. It's unclear why Twitch does this, but a raw concatenation of the parts to `output.ts` still makes a valid file, despite some programs complaining, since channels are idendified correctly in each part. Converting the ts file to mp4 in copy mode is enough to avoid any related playback warning.
### 5.4. Defective streams
Some Twitch.tv streams are defective, which can lead to ffmpeg issues, if those defects are not idendified early on. For example, this HEVC stream (ffmpeg 7.1):
```
$ ffmpeg -y -hide_banner -i 20240720T094852Z_2202491162.mp4 -c:v libx264 -crf 51 -preset ultrafast \
-vf "zscale=rangein=full:range=limited,format=yuv420p,scale=-2:120" -map 0:v -map 0:a? -c:a aac -nostdin -f mp4 /dev/null
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '20240720T094852Z_2202491162.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
title : [NVENC HEVC 1440p] Neon White (2202491162)
artist : Rodney
date : 2024
encoder : Lavf61.7.100
comment : Created at: 2024-07-20 09:48:52Z
: Video id: 2202491162
: Views: 79
genre : Neon White
Duration: 00:52:13.42, start: 0.000000, bitrate: 8203 kb/s
Chapters:
Chapter #0:0: start 0.000000, end 43.000000
Metadata:
title : Neon White
Chapter #0:1: start 43.000000, end 3133.000000
Metadata:
title : Neon White
Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 191 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 7997 kb/s, 59.98 fps, 60 tbr, 1000k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:2[0x3](eng): Data: bin_data (text / 0x74786574)
Metadata:
handler_name : SubtitleHandler
Stream mapping:
Stream #0:1 -> #0:0 (hevc (native) -> h264 (libx264))
Stream #0:0 -> #0:1 (aac (native) -> aac (native))
[libx264 @ 0x14c707010] using SAR=320/321
[libx264 @ 0x14c707010] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x14c707010] profile Constrained Baseline, level 1.3, 4:2:0, 8-bit
[libx264 @ 0x14c707010] 264 - core 164 r3108 31e19f9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=4 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=51.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, mp4, to '/dev/null':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
title : [NVENC HEVC 1440p] Neon White (2202491162)
artist : Rodney
date : 2024
genre : Neon White
comment : Created at: 2024-07-20 09:48:52Z
: Video id: 2202491162
: Views: 79
encoder : Lavf61.7.100
Chapters:
Chapter #0:0: start 0.000000, end 43.000000
Metadata:
title : Neon White
Chapter #0:1: start 43.000000, end 3133.000000
Metadata:
title : Neon White
Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 214x120 [SAR 320:321 DAR 16:9], q=2-31, 60 fps, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.19.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.19.100 aac
frame= 509 fps=0.0 q=42.0 size= 0KiB time=00:00:08.51 bitrate= 0.1kbits/s speed=16.9x
[...]
frame=142580 fps=581 q=56.0 size= 45056KiB time=00:39:36.36 bitrate= 155.3kbits/s speed=9.68x
[hevc @ 0x14c63bdc0] Could not find ref with POC 92
[hevc @ 0x14c63bdc0] Could not find ref with POC 87
[hevc @ 0x14c63bdc0] Could not find ref with POC 82
[hevc @ 0x14c63bdc0] Could not find ref with POC 77
[hevc @ 0x14c63bdc0] Could not find ref with POC 73
[hevc @ 0x14c643240] Could not find ref with POC 92
[hevc @ 0x14c643240] Could not find ref with POC 87
[hevc @ 0x14c643240] Could not find ref with POC 82
[hevc @ 0x14c643240] Could not find ref with POC 77
[hevc @ 0x14c608970] Could not find ref with POC 92
[hevc @ 0x14c608970] Could not find ref with POC 87
[hevc @ 0x14c608970] Could not find ref with POC 82
[hevc @ 0x14c608970] Could not find ref with POC 77
[hevc @ 0x14c610230] Could not find ref with POC 92
[hevc @ 0x14c610230] Could not find ref with POC 87
[hevc @ 0x14c610230] Could not find ref with POC 82
[hevc @ 0x14c610230] Could not find ref with POC 77
[hevc @ 0x14c617860] Could not find ref with POC 92
[hevc @ 0x14c617860] Could not find ref with POC 87
[hevc @ 0x14c617860] Could not find ref with POC 82
[hevc @ 0x14c617860] Could not find ref with POC 77
[hevc @ 0x14c61ebc0] Could not find ref with POC 92
[hevc @ 0x14c61ebc0] Could not find ref with POC 87
[hevc @ 0x14c61ebc0] Could not find ref with POC 82
[hevc @ 0x14c61ebc0] Could not find ref with POC 77
[hevc @ 0x14c626040] Could not find ref with POC 92
[hevc @ 0x14c626040] Could not find ref with POC 87
[hevc @ 0x14c626040] Could not find ref with POC 82
[hevc @ 0x14c62d4c0] Could not find ref with POC 92
[hevc @ 0x14c62d4c0] Could not find ref with POC 87
[hevc @ 0x14c62d4c0] Could not find ref with POC 82
[hevc @ 0x14c634940] Could not find ref with POC 92
[hevc @ 0x14c634940] Could not find ref with POC 87
[hevc @ 0x14c634940] Could not find ref with POC 82
[hevc @ 0x14c63bdc0] Could not find ref with POC 92
[hevc @ 0x14c63bdc0] Could not find ref with POC 87
[hevc @ 0x14c63bdc0] Could not find ref with POC 82
[hevc @ 0x14c643240] Could not find ref with POC 92
[hevc @ 0x14c643240] Could not find ref with POC 87
[hevc @ 0x14c643240] Could not find ref with POC 82
[hevc @ 0x14c608970] Could not find ref with POC 92
[hevc @ 0x14c608970] Could not find ref with POC 87
[hevc @ 0x14c610230] Could not find ref with POC 92
[hevc @ 0x14c610230] Could not find ref with POC 87
[hevc @ 0x14c617860] Could not find ref with POC 92
[hevc @ 0x14c617860] Could not find ref with POC 87
[hevc @ 0x14c61ebc0] Could not find ref with POC 92
[hevc @ 0x14c61ebc0] Could not find ref with POC 87
[hevc @ 0x14c626040] Could not find ref with POC 92
[hevc @ 0x14c626040] Could not find ref with POC 87
[hevc @ 0x14c62d4c0] Could not find ref with POC 92
[hevc @ 0x14c634940] Could not find ref with POC 92
[hevc @ 0x14c63bdc0] Could not find ref with POC 92
[hevc @ 0x14c643240] Could not find ref with POC 92
frame=142838 fps=581 q=52.0 size= 45056KiB time=00:39:41.60 bitrate= 155.0kbits/s speed=9.68x
[hevc @ 0x14c608970] Could not find ref with POC 92
frame=143114 fps=580 q=58.0 size= 45056KiB time=00:39:46.20 bitrate= 154.7kbits/s speed=9.68x
[...]
frame=187656 fps=559 q=42.0 size= 59904KiB time=00:52:08.56 bitrate= 156.9kbits/s speed=9.33x
[out#0/mp4 @ 0x600000f54000] video:9142KiB audio:50902KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 5.702364%
frame=187947 fps=560 q=-1.0 Lsize= 63467KiB time=00:52:13.35 bitrate= 165.9kbits/s speed=9.33x
[libx264 @ 0x14c707010] frame I:752 Avg QP:48.64 size: 462
[libx264 @ 0x14c707010] frame P:187195 Avg QP:49.56 size: 48
[libx264 @ 0x14c707010] mb I I16..4: 100.0% 0.0% 0.0%
[libx264 @ 0x14c707010] mb P I16..4: 4.6% 0.0% 0.0% P16..4: 7.4% 0.0% 0.0% 0.0% 0.0% skip:88.0%
[libx264 @ 0x14c707010] coded y,uvDC,uvAC intra: 6.0% 51.5% 1.4% inter: 0.6% 2.7% 0.0%
[libx264 @ 0x14c707010] i16 v,h,dc,p: 42% 28% 24% 6%
[libx264 @ 0x14c707010] i8c dc,h,v,p: 90% 6% 4% 0%
[libx264 @ 0x14c707010] kb/s:23.90
[aac @ 0x14c64aec0] Qavg: 586.890
```
Others give warnings due to lack of some kind of hardware acceleration, but the videos are fine (ffmpeg 7.1):
```
$ ffmpeg -y -hide_banner -i 20240617T224312Z_2174725499.mp4 -c:v libx264 -crf 51 -preset ultrafast \
-vf "zscale=rangein=full:range=limited,format=yuv420p,scale=-2:120" -map 0:v -map 0:a? -c:a aac -nostdin -f mp4 /dev/null
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '20240617T224312Z_2174725499.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : AV1/HEVC/H.264 - Dan Clancy FULL segment (2174725499)
artist : DJClancy
date : 2024
encoder : Lavf61.7.100
comment : Created at: 2024-06-17 22:43:12Z
: Video id: 2174725499
: Views: 653
genre : Just Chatting
Duration: 00:08:41.01, start: 0.000000, bitrate: 7571 kb/s
Chapters:
Chapter #0:0: start 0.000000, end 521.000000
Metadata:
title : Just Chatting
Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, unknown/bt709/unknown, progressive), 1920x1080, 7429 kb/s, 59.99 fps, 60 tbr, 90k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:2[0x3](eng): Data: bin_data (text / 0x74786574)
Metadata:
handler_name : SubtitleHandler
Stream mapping:
Stream #0:1 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:0 -> #0:1 (aac (native) -> aac (native))
[libx264 @ 0x128f06100] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x128f06100] profile Constrained Baseline, level 1.3, 4:2:0, 8-bit
[libx264 @ 0x128f06100] 264 - core 164 r3108 31e19f9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=4 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=51.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, mp4, to '/dev/null':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : AV1/HEVC/H.264 - Dan Clancy FULL segment (2174725499)
artist : DJClancy
date : 2024
genre : Just Chatting
comment : Created at: 2024-06-17 22:43:12Z
: Video id: 2174725499
: Views: 653
encoder : Lavf61.7.100
Chapters:
Chapter #0:0: start 0.000000, end 521.000000
Metadata:
title : Just Chatting
Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p(tv, unknown/bt709/unknown, progressive), 214x120, q=2-31, 60 fps, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.19.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.19.100 aac
[vf#0:0 @ 0x600002014140] Reconfiguring filter graph because video parameters changed to yuv420p(tv, bt709), 1920x1080
frame= 434 fps=0.0 q=53.0 size= 0KiB time=00:00:07.30 bitrate= 0.1kbits/s speed=14.5x
[...]
frame=30925 fps=776 q=48.0 size= 8960KiB time=00:08:35.48 bitrate= 142.4kbits/s speed=12.9x
[vf#0:0 @ 0x600002014140] Reconfiguring filter graph because video parameters changed to yuv420p(tv, unknown), 1920x1080
[swscaler @ 0x1182e8000] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x118320000] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x118358000] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x118390000] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x1183c8000] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x118400000] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x118438000] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x1191d8000] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x119210000] No accelerated colorspace conversion found from yuv420p to bgr24.
[out#0/mp4 @ 0x60000240c000] video:905KiB audio:8199KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 6.275041%
frame=31256 fps=776 q=-1.0 Lsize= 9675KiB time=00:08:41.00 bitrate= 152.1kbits/s speed=12.9x
[libx264 @ 0x128f06100] frame I:126 Avg QP:47.27 size: 632
[libx264 @ 0x128f06100] frame P:31130 Avg QP:49.99 size: 27
[libx264 @ 0x128f06100] mb I I16..4: 100.0% 0.0% 0.0%
[libx264 @ 0x128f06100] mb P I16..4: 0.8% 0.0% 0.0% P16..4: 3.4% 0.0% 0.0% 0.0% 0.0% skip:95.8%
[libx264 @ 0x128f06100] coded y,uvDC,uvAC intra: 11.3% 83.8% 54.3% inter: 0.1% 2.4% 0.7%
[libx264 @ 0x128f06100] i16 v,h,dc,p: 37% 21% 34% 8%
[libx264 @ 0x128f06100] i8c dc,h,v,p: 71% 13% 14% 2%
[libx264 @ 0x128f06100] kb/s:14.22
[aac @ 0x128f54bc0] Qavg: 459.979
```
### 5.5. Retirement of the :BibleThumb: emote
On September 30, 2024, the iconic [:BibleThumb:](https://web.archive.org/web/20210723145631/https://twitchemotes.com/global/emotes/86) emote was retired, so won't be possible to use it in chats starting October 1, 2024 (the icon is missing in the emote list inside the chat window when trying to send a new message). This removal was done by effectively taking offline the URLs (now are 404 Not Found by nginx) that serve the emote in different sizes and styles `https://static-cdn.jtvnw.net/emoticons/v2/86/*`.
The chats downloaded with `TwitchDownloaderCLI(2) chatdownload` and the online Twitch chat by web, still include the emote as the same reference, like nothing happened, but the fact that the images were taken offline makes the removal total (also retroactive). All chats regardless of the date are affected, even before October. No idea if the license expiration required to do so.
There were [3 proposed new emotes](https://x.com/Twitch/status/1839412204397088833) to replace it: `:BigSad:`, `:UnBearable:` and `:LayersOfSad:`. The winner by public contest in X was `:BigSad:`.
The reasons to remove `:BibleThumb:` were a few: because it was too emotional, had too much personality, had a tangencial link with religion, and the character belongs to a commercial game: [The Binding of Isaac](https://store.steampowered.com/app/113200/The_Binding_of_Isaac/). Twitch wants the chats to provide a more unwound and transient ambience. Also, brands external to Twitch can be a future liability, this is why brand names/logos or any other kind of copyrighted material is now allowed in emotes, not even by agreement.
### 5.6. Obtaining the Twitch user OAUTH (auth-token)
To obtain the Twitch.tv `"$AOTH"` used by some scripts in this project, (auth is required to download some content), login to Twitch.tv using a desktop web browser. The `auth-token` will be a 30-character lowercase alphanumeric string, like `s0qrne3ym0wbvcs4ygbpgfa68mxog0`. It's only for the current username/artist which is performing the log-in action. These oauths are called `User access tokens`, and do not expire: [Do user access tokens expire?](https://discuss.dev.twitch.com/t/do-user-access-tokens-expire/25814). However, if intentionally logging out of the account, the oauth token is not longer valid, and a new one is generated upon logging in again. This is to avoid having permanent auth tokens, in case one is leaked or obtained by brute force or random string generation (collison). Ideally this should apply automatically to all auth/oauth tokens unless the owner sets otherwise.
To get the auth-token from most desktop web browsers (probably could be done by commands from terminal too):
* Google Chrome and related: go to the Chrome Menu or Overflow Menu represented by three vertical dots (⋮) > More Tools > Developer Tools > Application > Storage > Cookies > https://www.twitch.tv > auth-token
* Mozilla Firefox and related: go to the Hamburger Menu > More tools > Web Developer Tools > Storage > Cookies > https://www.twitch.tv > auth-token
The developer frame can also be opened by clicking Inspect on contextual menu over the webpage.
The token can be added to macOS shell environment by running this command in a terminal:
```
printf 'export OAUTH="s0qrne3ym0wbvcs4ygbpgfa68mxog0"\n' >> ~/.zshrc
```
or GNU/Linux or UNIX related systems:
```
printf 'export OAUTH="s0qrne3ym0wbvcs4ygbpgfa68mxog0"\n' >> ~/.bashrc
```
Another way to obtain the oauth is to paste and run JavaScript code in the Console tab of the developer frame. The auth string can be selected by double clicking with a mouse:
```
{
function getCookieValue(name) {
const value = `; ${document.cookie}`;
const parts = value.split(`; ${name}=`);
if (parts.length === 2) return parts.pop().split(';').shift();
}
const authToken = getCookieValue('auth-token');
console.log('Auth Token:', authToken);
}
```
Similar code:
```
function logAuthToken() {
const authToken = document.cookie
.split('; ')
.find(row => row.startsWith('auth-token='))
?.split('=')[1];
console.log('Auth Token:', authToken);
}
logAuthToken();
```
A pop-up can alert can be generated where the line can be selected and copied:
```
javascript:(function(){
var authToken = document.cookie.split('; ').find(row => row.startsWith('auth-token='))?.split('=')[1];
alert('Auth Token: ' + authToken);
})();
```
To generate a copy button over the webpage in top left corner, to copy the oauth directly to clipboard (with just code only is not possible in all browsers to directly copy to RAM):
```
javascript:(function(){
var authToken = document.cookie.split('; ').find(row => row.startsWith('auth-token='))?.split('=')[1];
if (authToken) {
var copyButton = document.createElement('button');
copyButton.textContent = 'Copy Auth Token';
copyButton.style.position = 'fixed';
copyButton.style.top = '10px';
copyButton.style.left = '10px';
copyButton.style.zIndex = '9999';
copyButton.style.padding = '10px';
copyButton.style.backgroundColor = '#007BFF';
copyButton.style.color = 'white';
copyButton.style.border = 'none';
copyButton.style.cursor = 'pointer';
document.body.appendChild(copyButton);
copyButton.addEventListener('click', function() {
var tempInput = document.createElement('input');
document.body.appendChild(tempInput);
tempInput.value = authToken;
tempInput.focus();
tempInput.select();
document.execCommand('copy');
document.body.removeChild(tempInput);
document.body.removeChild(copyButton);
alert('Auth Token copied to clipboard: ' + authToken);
});
} else {
alert('Auth Token not found.');
}
})();
```
Web Browser extensions exist and are more comfortable to use, like [Cookie-Editor](https://cookie-editor.com/) or [EditThisCookie](https://www.editthiscookie.com/).
### 5.7. Out of the ordinary VODs or streams
Some IDs (or VODs) have particularities, they are special cases. Below are some examples that have been found. Further examples may be found in source code comments.
#### 5.7.1. VODs where the type or number of tracks in some streams do not match the declared values
Example ID: 48360849 (highlight). Applies also to 50463362 (highlight).
This VOD has 2 streams only: `chunked` and `audio_only`. Both streams are exactly the same output file:
```
$ sha384sum -b '20130120T040606Z,48360849_chunked/output.ts' '20130120T040606Z,48360849_audio/output.ts'
ee878fc046aaf1c752ebb6bfd6f1ac1b25ee9c510958083a9f8be402784b269f96d70195c917f6187eafa02840548021 *20130120T040606Z,48360849_chunked/output.ts
ee878fc046aaf1c752ebb6bfd6f1ac1b25ee9c510958083a9f8be402784b269f96d70195c917f6187eafa02840548021 *20130120T040606Z,48360849_audio/output.ts
```
The info is as follows:
```
[INFO - VOD INFORMATION JSON]
{
"data": {
"video": {
"title": "NightSky 100% Speedrun in 38:02",
"broadcastType": "HIGHLIGHT",
"thumbnailURLs": [
"https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/677acb5ad5/zfg1_4617235008_4617235008/thumb/thumb0-320x180.jpg"
],
"createdAt": "2013-01-20T04:06:06Z",
"publishedAt": "2013-01-20T04:06:05.670566Z",
"updatedAt": "2016-08-26T23:59:18Z",
"lengthSeconds": 2296,
"owner": {
"id": "8683614",
"displayName": "Zfg1",
"login": "zfg1"
},
"viewCount": 198,
"game": {
"id": "11557",
"displayName": "The Legend of Zelda: Ocarina of Time",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/11557_IGDB-{width}x{height}.jpg"
},
"description": "game crashed omg",
"status": "RECORDED"
}
},
"extensions": {
"durationMilliseconds": 44,
"requestID": ""
}
}
[INFO - VOD ADJUSTED CHAPTERS JSON]
{
"data": {
"video": {
"id": "48360849",
"moments": {
"edges": [
{
"node": {
"id": "",
"type": "GAME_CHANGE",
"positionMilliseconds": 0,
"durationMilliseconds": 2296000,
"description": "The Legend of Zelda: Ocarina of Time",
"subDescription": "",
"thumbnailURL": null,
"moments": null,
"video": null,
"details": {
"game": {
"id": "11557",
"displayName": "The Legend of Zelda: Ocarina of Time",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/11557_IGDB-40x53.jpg"
}
}
}
}
]
}
}
},
"extensions": {
"durationMilliseconds": 38,
"operationName": "VideoPlayer_ChapterSelectButtonVideo",
"requestID": ""
}
}
[INFO - VOD RAW PLAYLIST]
#EXTM3U
#EXT-X-TWITCH-INFO:ORIGIN="s3",B="false",REGION="",USER-IP="",SERVING-ID="",CLUSTER="cloudfront_vod",USER-COUNTRY="",MANIFEST-CLUSTER="cloudfront_vod"
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="Source",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=428608,CODECS="avc1.64001F",RESOLUTION=1280x720,VIDEO="chunked"
https://d2nvs31859zcd8.cloudfront.net/677acb5ad5/zfg1_4617235008_4617235008/chunked/highlight-48360849.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="Audio Only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=428608,VIDEO="audio_only"
https://d2nvs31859zcd8.cloudfront.net/677acb5ad5/zfg1_4617235008_4617235008/audio_only/highlight-48360849.m3u8
[INFO - VOD EXPANDED PLAYLIST JSON]
[
{
"STREAM-NUMBER": "1",
"TYPE": "VIDEO",
"NAME": "Source",
"GROUP-ID": "chunked",
"VIDEO": "chunked",
"CODECS": "avc1.64001F",
"BANDWIDTH": "428608",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "chunked",
"NUMBER-OF-TRACKS": "1",
"AUDIO-CODEC": "",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/677acb5ad5/zfg1_4617235008_4617235008/chunked/highlight-48360849.m3u8",
"RES-FPS": "1280x720p"
},
{
"STREAM-NUMBER": "2",
"TYPE": "VIDEO",
"NAME": "Audio Only",
"GROUP-ID": "audio_only",
"VIDEO": "audio_only",
"CODECS": "",
"BANDWIDTH": "428608",
"RESOLUTION": "",
"QUALITY-BY-URL": "audio_only",
"NUMBER-OF-TRACKS": "0",
"AUDIO-CODEC": "",
"VIDEO-CODEC": "",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/677acb5ad5/zfg1_4617235008_4617235008/audio_only/highlight-48360849.m3u8",
"RES-FPS": ""
}
]
```
Both streams have only 1 track, which is a video one. The codecs reported only match for the audio ones (which is missing in both, so that's correct because there's no audio track). However for the video codec, the `audio_only` does not report any video codec, despite having one track. The ffprobe is as follows:
```
$ ffprobe -hide_banner -i 20130120T040606Z,48360849_audio/output.ts
Input #0, mpegts, from '20130120T040606Z,48360849_audio/output.ts':
Duration: 26:08:58.91, start: 1613.139000, bitrate: 10 kb/s
Program 1
Stream #0:0[0x101]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1280x720, 29.92 tbr, 90k tbn
Stream #0:1[0x102]: Data: timed_id3 (ID3 / 0x20334449)
Unsupported codec with id 98313 for input stream 1
```
```
$ ffprobe -hide_banner -i '20130120T040606Z,48360849/20130120T040606Z_48360849.mp4'
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '20130120T040606Z,48360849/20130120T040606Z_48360849.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : NightSky 100% Speedrun in 38:02 (48360849)
artist : Zfg1
date : 2013
encoder : Lavf61.7.100
comment : game crashed omg
: ------------------------
: Created at: 2013-01-20 04:06:06Z
: Video id: 48360849
: Views: 194
genre : The Legend of Zelda: Ocarina of Time
Duration: 00:38:16.66, start: 0.000000, bitrate: 394 kb/s
Chapters:
Chapter #0:0: start 0.000000, end 2296.000000
Metadata:
title : The Legend of Zelda: Ocarina of Time
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1280x720, 390 kb/s, 29.99 fps, 29.92 tbr, 90k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:1[0x2](eng): Data: bin_data (text / 0x74786574)
Metadata:
handler_name : SubtitleHandler
Unsupported codec with id 98314 for input stream 1
```
This demonstrates that the codecs reported by Twitch may not be accurate, not even the type of tracks each stream has (none, audio, video, audio+video), so scripts have to take care detecting this before any operation.
#### 5.7.2. VODs with several streams with the same resolution and/or framerate
Some VODs have 2 or more streams with the same resolution; the framerates may be missing or be different. When there are 3 with same resolution one of them is chunked, if there are 2 one can be chunked or not, if the chunked has higher resolution for example.
Note that I'm referring to declared values in the headers or the URL; actual stream/track values (reported by `ffprobe`) can mismatch (since most of the time they are placeholders to tell streams apart).
Example 1: 3 streams with same resolution but all of different framerate. ID 437346960:
"RES-FPS" : 1280x720p61 (chunked), 1280x720p60, 1280x720p30, ...
```
[INFO - VOD INFORMATION JSON]
{
"data": {
"video": {
"title": "Ocarina of Time 100% speedrun in 3:52:15",
"broadcastType": "HIGHLIGHT",
"thumbnailURLs": [
"https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812//thumb/thumb437346960-320x180.jpg"
],
"createdAt": "2019-06-11T00:43:28Z",
"publishedAt": "2019-06-11T00:43:28Z",
"updatedAt": "2019-06-11T19:07:46Z",
"lengthSeconds": 14164,
"owner": {
"id": "8683614",
"displayName": "Zfg1",
"login": "zfg1"
},
"viewCount": 6535,
"game": {
"id": "11557",
"displayName": "The Legend of Zelda: Ocarina of Time",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/11557_IGDB-{width}x{height}.jpg"
},
"description": "",
"status": "RECORDED"
}
},
"extensions": {
"durationMilliseconds": 46,
"requestID": ""
}
}
[INFO - VOD ADJUSTED CHAPTERS JSON]
{
"data": {
"video": {
"id": "437346960",
"moments": {
"edges": [
{
"node": {
"id": "",
"type": "GAME_CHANGE",
"positionMilliseconds": 0,
"durationMilliseconds": 14164000,
"description": "The Legend of Zelda: Ocarina of Time",
"subDescription": "",
"thumbnailURL": null,
"moments": null,
"video": null,
"details": {
"game": {
"id": "11557",
"displayName": "The Legend of Zelda: Ocarina of Time",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/11557_IGDB-40x53.jpg"
}
}
}
}
]
}
}
},
"extensions": {
"durationMilliseconds": 47,
"operationName": "VideoPlayer_ChapterSelectButtonVideo",
"requestID": ""
}
}
[INFO - VOD RAW PLAYLIST]
#EXTM3U
#EXT-X-TWITCH-INFO:ORIGIN="s3",B="false",REGION="",USER-IP="",SERVING-ID="",CLUSTER="cloudfront_vod",USER-COUNTRY="",MANIFEST-CLUSTER="cloudfront_vod"
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="720p60",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=3713464,CODECS="avc1.640020,mp4a.40.2",RESOLUTION=1280x720,VIDEO="chunked",FRAME-RATE=61.000
https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/chunked/highlight-437346960.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p60",NAME="720p60",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=3100000,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=1280x720,VIDEO="720p60",FRAME-RATE=60.000
https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/720p60/highlight-437346960.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p30",NAME="720p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=2100000,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=1280x720,VIDEO="720p30",FRAME-RATE=30.000
https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/720p30/highlight-437346960.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="480p30",NAME="480p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=1200000,CODECS="avc1.4D401E,mp4a.40.2",RESOLUTION=852x480,VIDEO="480p30",FRAME-RATE=30.000
https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/480p30/highlight-437346960.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="Audio Only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=96685,CODECS="mp4a.40.2",VIDEO="audio_only"
https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/audio_only/highlight-437346960.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="360p30",NAME="360p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=630000,CODECS="avc1.4D401E,mp4a.40.2",RESOLUTION=640x360,VIDEO="360p30",FRAME-RATE=30.000
https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/360p30/highlight-437346960.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="160p30",NAME="160p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=230000,CODECS="avc1.4D400C,mp4a.40.2",RESOLUTION=284x160,VIDEO="160p30",FRAME-RATE=30.000
https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/160p30/highlight-437346960.m3u8
[INFO - VOD EXPANDED PLAYLIST JSON]
[
{
"STREAM-NUMBER": "1",
"TYPE": "VIDEO",
"NAME": "720p60",
"GROUP-ID": "chunked",
"VIDEO": "chunked",
"CODECS": "avc1.640020+mp4a.40.2",
"BANDWIDTH": "3713464",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "chunked",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "61.000",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/chunked/highlight-437346960.m3u8",
"RES-FPS": "1280x720p61"
},
{
"STREAM-NUMBER": "2",
"TYPE": "VIDEO",
"NAME": "720p60",
"GROUP-ID": "720p60",
"VIDEO": "720p60",
"CODECS": "avc1.4D401F+mp4a.40.2",
"BANDWIDTH": "3100000",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "720p60",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "60.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/720p60/highlight-437346960.m3u8",
"RES-FPS": "1280x720p60"
},
{
"STREAM-NUMBER": "3",
"TYPE": "VIDEO",
"NAME": "720p",
"GROUP-ID": "720p30",
"VIDEO": "720p30",
"CODECS": "avc1.4D401F+mp4a.40.2",
"BANDWIDTH": "2100000",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "720p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "30.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/720p30/highlight-437346960.m3u8",
"RES-FPS": "1280x720p30"
},
{
"STREAM-NUMBER": "4",
"TYPE": "VIDEO",
"NAME": "480p",
"GROUP-ID": "480p30",
"VIDEO": "480p30",
"CODECS": "avc1.4D401E+mp4a.40.2",
"BANDWIDTH": "1200000",
"RESOLUTION": "852x480",
"QUALITY-BY-URL": "480p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.775",
"FRAME-RATE": "30.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/480p30/highlight-437346960.m3u8",
"RES-FPS": "852x480p30"
},
{
"STREAM-NUMBER": "5",
"TYPE": "VIDEO",
"NAME": "360p",
"GROUP-ID": "360p30",
"VIDEO": "360p30",
"CODECS": "avc1.4D401E+mp4a.40.2",
"BANDWIDTH": "630000",
"RESOLUTION": "640x360",
"QUALITY-BY-URL": "360p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "30.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/360p30/highlight-437346960.m3u8",
"RES-FPS": "640x360p30"
},
{
"STREAM-NUMBER": "6",
"TYPE": "VIDEO",
"NAME": "160p",
"GROUP-ID": "160p30",
"VIDEO": "160p30",
"CODECS": "avc1.4D400C+mp4a.40.2",
"BANDWIDTH": "230000",
"RESOLUTION": "284x160",
"QUALITY-BY-URL": "160p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.775",
"FRAME-RATE": "30.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/160p30/highlight-437346960.m3u8",
"RES-FPS": "284x160p30"
},
{
"STREAM-NUMBER": "7",
"TYPE": "VIDEO",
"NAME": "Audio Only",
"GROUP-ID": "audio_only",
"VIDEO": "audio_only",
"CODECS": "mp4a.40.2",
"BANDWIDTH": "96685",
"RESOLUTION": "",
"QUALITY-BY-URL": "audio_only",
"NUMBER-OF-TRACKS": "1",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/e8710b36e2b0fb865ceb_zfg1_37190126351_6345795812/audio_only/highlight-437346960.m3u8",
"RES-FPS": ""
}
]
```
Example 2: 3 streams with same resolution and one of them with different framerate. ID 453932468:
"RES-FPS": 1280x720p60 (chunked), 1280x720p60, 1280x720p30, ...
Example 3: 2 streams with same resolution but different framerate, chunked exists with higher resolution. ID 2436469260:
"RES-FPS": 1280x720p60, 1280x720p30, ...
```
[INFO - VOD INFORMATION JSON]
{
"data": {
"video": {
"title": "Majora's Mask Actor Randomizer, No Starting Items, 1 hit KO",
"broadcastType": "ARCHIVE",
"thumbnailURLs": [
"https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640//thumb/thumb0-320x180.jpg",
"https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640//thumb/thumb1-320x180.jpg",
"https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640//thumb/thumb2-320x180.jpg",
"https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640//thumb/thumb3-320x180.jpg"
],
"createdAt": "2025-04-18T20:20:45Z",
"publishedAt": "2025-04-18T20:20:45Z",
"updatedAt": "2025-04-18T23:02:14Z",
"lengthSeconds": 9651,
"owner": {
"id": "8683614",
"displayName": "Zfg1",
"login": "zfg1"
},
"viewCount": 10608,
"game": {
"id": "12482",
"displayName": "The Legend of Zelda: Majora's Mask",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/12482_IGDB-{width}x{height}.jpg"
},
"description": "",
"status": "RECORDED"
}
},
"extensions": {
"durationMilliseconds": 42,
"requestID": ""
}
}
[INFO - VOD ADJUSTED CHAPTERS JSON]
{
"data": {
"video": {
"id": "2436469260",
"moments": {
"edges": [
{
"node": {
"id": "",
"type": "GAME_CHANGE",
"positionMilliseconds": 0,
"durationMilliseconds": 9651000,
"description": "The Legend of Zelda: Majora's Mask",
"subDescription": "",
"thumbnailURL": null,
"moments": null,
"video": null,
"details": {
"game": {
"id": "12482",
"displayName": "The Legend of Zelda: Majora's Mask",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/12482_IGDB-40x53.jpg"
}
}
}
}
]
}
}
},
"extensions": {
"durationMilliseconds": 43,
"operationName": "VideoPlayer_ChapterSelectButtonVideo",
"requestID": ""
}
}
[INFO - VOD RAW PLAYLIST]
#EXTM3U
#EXT-X-TWITCH-INFO:ORIGIN="s3",B="false",REGION="",USER-IP="",SERVING-ID="",CLUSTER="cloudfront_vod",USER-COUNTRY="",MANIFEST-CLUSTER="cloudfront_vod"
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="1080p60",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=6398399,CODECS="avc1.64002A,mp4a.40.2",RESOLUTION=1920x1080,VIDEO="chunked",FRAME-RATE=60.000
https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/chunked/index-muted-WUJEZQDQV3.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p60",NAME="720p60",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=3010005,CODECS="avc1.4D4020,mp4a.40.2",RESOLUTION=1280x720,VIDEO="720p60",FRAME-RATE=60.000
https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/720p60/index-muted-WUJEZQDQV3.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p30",NAME="720p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=2212932,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=1280x720,VIDEO="720p30",FRAME-RATE=30.000
https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/720p30/index-muted-WUJEZQDQV3.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="480p30",NAME="480p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=1392936,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=852x480,VIDEO="480p30",FRAME-RATE=30.000
https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/480p30/index-muted-WUJEZQDQV3.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="Audio Only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=217634,CODECS="mp4a.40.2",VIDEO="audio_only"
https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/audio_only/index-muted-WUJEZQDQV3.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="360p30",NAME="360p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=686585,CODECS="avc1.4D401E,mp4a.40.2",RESOLUTION=640x360,VIDEO="360p30",FRAME-RATE=30.000
https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/360p30/index-muted-WUJEZQDQV3.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="160p30",NAME="160p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=287134,CODECS="avc1.4D400C,mp4a.40.2",RESOLUTION=284x160,VIDEO="160p30",FRAME-RATE=30.000
https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/160p30/index-muted-WUJEZQDQV3.m3u8
[INFO - VOD EXPANDED PLAYLIST JSON]
[
{
"STREAM-NUMBER": "1",
"TYPE": "VIDEO",
"NAME": "1080p60",
"GROUP-ID": "chunked",
"VIDEO": "chunked",
"CODECS": "avc1.64002A+mp4a.40.2",
"BANDWIDTH": "6398399",
"RESOLUTION": "1920x1080",
"QUALITY-BY-URL": "chunked",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "60.000",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/chunked/index-muted-WUJEZQDQV3.m3u8",
"RES-FPS": "1920x1080p60"
},
{
"STREAM-NUMBER": "2",
"TYPE": "VIDEO",
"NAME": "720p60",
"GROUP-ID": "720p60",
"VIDEO": "720p60",
"CODECS": "avc1.4D4020+mp4a.40.2",
"BANDWIDTH": "3010005",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "720p60",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "60.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/720p60/index-muted-WUJEZQDQV3.m3u8",
"RES-FPS": "1280x720p60"
},
{
"STREAM-NUMBER": "3",
"TYPE": "VIDEO",
"NAME": "720p",
"GROUP-ID": "720p30",
"VIDEO": "720p30",
"CODECS": "avc1.4D401F+mp4a.40.2",
"BANDWIDTH": "2212932",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "720p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "30.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/720p30/index-muted-WUJEZQDQV3.m3u8",
"RES-FPS": "1280x720p30"
},
{
"STREAM-NUMBER": "4",
"TYPE": "VIDEO",
"NAME": "480p",
"GROUP-ID": "480p30",
"VIDEO": "480p30",
"CODECS": "avc1.4D401F+mp4a.40.2",
"BANDWIDTH": "1392936",
"RESOLUTION": "852x480",
"QUALITY-BY-URL": "480p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.775",
"FRAME-RATE": "30.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/480p30/index-muted-WUJEZQDQV3.m3u8",
"RES-FPS": "852x480p30"
},
{
"STREAM-NUMBER": "5",
"TYPE": "VIDEO",
"NAME": "360p",
"GROUP-ID": "360p30",
"VIDEO": "360p30",
"CODECS": "avc1.4D401E+mp4a.40.2",
"BANDWIDTH": "686585",
"RESOLUTION": "640x360",
"QUALITY-BY-URL": "360p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "30.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/360p30/index-muted-WUJEZQDQV3.m3u8",
"RES-FPS": "640x360p30"
},
{
"STREAM-NUMBER": "6",
"TYPE": "VIDEO",
"NAME": "160p",
"GROUP-ID": "160p30",
"VIDEO": "160p30",
"CODECS": "avc1.4D400C+mp4a.40.2",
"BANDWIDTH": "287134",
"RESOLUTION": "284x160",
"QUALITY-BY-URL": "160p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.775",
"FRAME-RATE": "30.000",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/160p30/index-muted-WUJEZQDQV3.m3u8",
"RES-FPS": "284x160p30"
},
{
"STREAM-NUMBER": "7",
"TYPE": "VIDEO",
"NAME": "Audio Only",
"GROUP-ID": "audio_only",
"VIDEO": "audio_only",
"CODECS": "mp4a.40.2",
"BANDWIDTH": "217634",
"RESOLUTION": "",
"QUALITY-BY-URL": "audio_only",
"NUMBER-OF-TRACKS": "1",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/db88dda7d7ce616f22b9_zfg1_319775354236_1745007640/audio_only/index-muted-WUJEZQDQV3.m3u8",
"RES-FPS": ""
}
]
```
Example 4: 2 streams with same resolution and framerate. ID 3923803:
"RES-FPS": 1088x612p30 (chunked), 1088x612p30, ...
Example 5: 2 streams with same resolution but one has framerate information and the other not. ID 127818852:
"RES-FPS": 1280x720p (chunked), 1280x720p26, ...
Example 6: 3 streams with same resolution but 2 have framerate information and 1 doesn't. ID 275664455:
"RES-FPS": 1280x720p (chunked), 1280x720p60, 1280x720p30, ...
#### 5.7.3. VODs with BANDWIDTH=0 for all streams
Some VODs, like ID 1944355271 and 1990806299, contain "BANDWIDTH": "0" for all streams:
```
[INFO - VOD INFORMATION JSON]
{
"data": {
"video": {
"title": "Destacado: || TIER LIST EMPANADAS Y GASEOSAS || !redes !cafecito !paypal",
"broadcastType": "HIGHLIGHT",
"thumbnailURLs": [
"https://static-cdn.jtvnw.net/cf_vods/d1m7jfoe9zdc1j/7c197635ba7ade27a084_soydairi_51493578584_3425947261//thumb/thumb1944355271-320x180.jpg"
],
"createdAt": "2023-10-06T21:21:39Z",
"publishedAt": "2023-10-06T21:21:39Z",
"updatedAt": "2023-11-09T21:00:48Z",
"lengthSeconds": 6200,
"owner": {
"id": "545440031",
"displayName": "SoyDairi",
"login": "soydairi"
},
"viewCount": 470,
"game": {
"id": "509658",
"displayName": "Just Chatting",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/509658-{width}x{height}.jpg"
},
"description": "",
"status": "RECORDED"
}
},
"extensions": {
"durationMilliseconds": 43,
"requestID": ""
}
}
[INFO - VOD ADJUSTED CHAPTERS JSON]
{
"data": {
"video": {
"id": "1944355271",
"moments": {
"edges": [
{
"node": {
"id": "",
"type": "GAME_CHANGE",
"positionMilliseconds": 0,
"durationMilliseconds": 6200000,
"description": "Just Chatting",
"subDescription": "",
"thumbnailURL": null,
"moments": null,
"video": null,
"details": {
"game": {
"id": "509658",
"displayName": "Just Chatting",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/509658-40x53.jpg"
}
}
}
}
]
}
}
},
"extensions": {
"durationMilliseconds": 25,
"operationName": "VideoPlayer_ChapterSelectButtonVideo",
"requestID": ""
}
}
[INFO - VOD RAW PLAYLIST]
#EXTM3U
#EXT-X-TWITCH-INFO:ORIGIN="s3",B="false",REGION="",USER-IP="",SERVING-ID="",CLUSTER="cloudfront_vod",USER-COUNTRY="",MANIFEST-CLUSTER="cloudfront_vod"
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="1080p60",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=0,CODECS="avc1.64042A,mp4a.40.2",RESOLUTION=1920x1080,VIDEO="chunked",FRAME-RATE=59.980
https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/chunked/highlight-1944355271.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="Audio Only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=0,CODECS="mp4a.40.2",VIDEO="audio_only"
https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/audio_only/highlight-1944355271.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p60",NAME="720p60",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=0,CODECS="avc1.4D0020,mp4a.40.2",RESOLUTION=1280x720,VIDEO="720p60",FRAME-RATE=59.980
https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/720p60/highlight-1944355271.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="480p30",NAME="480p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=0,CODECS="avc1.4D001F,mp4a.40.2",RESOLUTION=852x480,VIDEO="480p30",FRAME-RATE=29.990
https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/480p30/highlight-1944355271.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="360p30",NAME="360p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=0,CODECS="avc1.4D001E,mp4a.40.2",RESOLUTION=640x360,VIDEO="360p30",FRAME-RATE=29.990
https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/360p30/highlight-1944355271.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="160p30",NAME="160p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=0,CODECS="avc1.4D000C,mp4a.40.2",RESOLUTION=284x160,VIDEO="160p30",FRAME-RATE=29.990
https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/160p30/highlight-1944355271.m3u8
[INFO - VOD EXPANDED PLAYLIST JSON]
[
{
"STREAM-NUMBER": "1",
"TYPE": "VIDEO",
"NAME": "1080p60",
"GROUP-ID": "chunked",
"VIDEO": "chunked",
"CODECS": "avc1.64042A+mp4a.40.2",
"BANDWIDTH": "0",
"RESOLUTION": "1920x1080",
"QUALITY-BY-URL": "chunked",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "59.980",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/chunked/highlight-1944355271.m3u8",
"RES-FPS": "1920x1080p60"
},
{
"STREAM-NUMBER": "2",
"TYPE": "VIDEO",
"NAME": "720p60",
"GROUP-ID": "720p60",
"VIDEO": "720p60",
"CODECS": "avc1.4D0020+mp4a.40.2",
"BANDWIDTH": "0",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "720p60",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "59.980",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/720p60/highlight-1944355271.m3u8",
"RES-FPS": "1280x720p60"
},
{
"STREAM-NUMBER": "3",
"TYPE": "VIDEO",
"NAME": "480p",
"GROUP-ID": "480p30",
"VIDEO": "480p30",
"CODECS": "avc1.4D001F+mp4a.40.2",
"BANDWIDTH": "0",
"RESOLUTION": "852x480",
"QUALITY-BY-URL": "480p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.775",
"FRAME-RATE": "29.990",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/480p30/highlight-1944355271.m3u8",
"RES-FPS": "852x480p30"
},
{
"STREAM-NUMBER": "4",
"TYPE": "VIDEO",
"NAME": "360p",
"GROUP-ID": "360p30",
"VIDEO": "360p30",
"CODECS": "avc1.4D001E+mp4a.40.2",
"BANDWIDTH": "0",
"RESOLUTION": "640x360",
"QUALITY-BY-URL": "360p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "29.990",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/360p30/highlight-1944355271.m3u8",
"RES-FPS": "640x360p30"
},
{
"STREAM-NUMBER": "5",
"TYPE": "VIDEO",
"NAME": "160p",
"GROUP-ID": "160p30",
"VIDEO": "160p30",
"CODECS": "avc1.4D000C+mp4a.40.2",
"BANDWIDTH": "0",
"RESOLUTION": "284x160",
"QUALITY-BY-URL": "160p30",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.775",
"FRAME-RATE": "29.990",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/160p30/highlight-1944355271.m3u8",
"RES-FPS": "284x160p30"
},
{
"STREAM-NUMBER": "6",
"TYPE": "VIDEO",
"NAME": "Audio Only",
"GROUP-ID": "audio_only",
"VIDEO": "audio_only",
"CODECS": "mp4a.40.2",
"BANDWIDTH": "0",
"RESOLUTION": "",
"QUALITY-BY-URL": "audio_only",
"NUMBER-OF-TRACKS": "1",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d1m7jfoe9zdc1j.cloudfront.net/7c197635ba7ade27a084_soydairi_51493578584_3425947261/audio_only/highlight-1944355271.m3u8",
"RES-FPS": ""
}
]
```
#### 5.7.4. VODs with placeholders for quality and URL
Some VODs contain only placeholders in the URLs, and even in 'VIDEO' and 'GROUP-ID', but the headers contain resolution and framerate for all streams except audio_only.
Known placeholders are in URL are: chunked, high, medium, low, mobile, audio_only. Other like "Source" or "Audio Only" are used in headers to designate "chunked" and "audio_only" respectively.
Example ID 138749526:
```
[INFO - VOD INFORMATION JSON]
{
"data": {
"video": {
"title": "Ocarina of Time MST Speedrun in 1:57:30",
"broadcastType": "HIGHLIGHT",
"thumbnailURLs": [
"https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/52f9886174_zfg1_25157553392_641017582//thumb/thumb138749526-320x180.jpg"
],
"createdAt": "2017-04-28T23:45:12Z",
"publishedAt": "2017-04-28T23:45:12Z",
"updatedAt": "2017-04-29T23:35:50Z",
"lengthSeconds": 7292,
"owner": {
"id": "8683614",
"displayName": "Zfg1",
"login": "zfg1"
},
"viewCount": 581,
"game": {
"id": "11557",
"displayName": "The Legend of Zelda: Ocarina of Time",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/11557_IGDB-{width}x{height}.jpg"
},
"description": "pretty good run",
"status": "RECORDED"
}
},
"extensions": {
"durationMilliseconds": 44,
"requestID": ""
}
}
[INFO - VOD ADJUSTED CHAPTERS JSON]
{
"data": {
"video": {
"id": "138749526",
"moments": {
"edges": [
{
"node": {
"id": "",
"type": "GAME_CHANGE",
"positionMilliseconds": 0,
"durationMilliseconds": 7292000,
"description": "The Legend of Zelda: Ocarina of Time",
"subDescription": "",
"thumbnailURL": null,
"moments": null,
"video": null,
"details": {
"game": {
"id": "11557",
"displayName": "The Legend of Zelda: Ocarina of Time",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/11557_IGDB-40x53.jpg"
}
}
}
}
]
}
}
},
"extensions": {
"durationMilliseconds": 40,
"operationName": "VideoPlayer_ChapterSelectButtonVideo",
"requestID": ""
}
}
[INFO - VOD RAW PLAYLIST]
#EXTM3U
#EXT-X-TWITCH-INFO:ORIGIN="s3",B="false",REGION="",USER-IP="",SERVING-ID="",CLUSTER="cloudfront_vod",USER-COUNTRY="",MANIFEST-CLUSTER="cloudfront_vod"
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="Source",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=2818872,CODECS="avc1.64001F,mp4a.40.2",RESOLUTION=1280x720,VIDEO="chunked"
https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/chunked/highlight-138749526.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="audio_only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=348364,CODECS="mp4a.40.2",VIDEO="audio_only"
https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/audio_only/highlight-138749526.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="high",NAME="720p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=1325212,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=1280x720,VIDEO="high",FRAME-RATE=26.250
https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/high/highlight-138749526.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="low",NAME="360p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=661478,CODECS="avc1.42C01E,mp4a.40.2",RESOLUTION=640x360,VIDEO="low",FRAME-RATE=26.250
https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/low/highlight-138749526.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="medium",NAME="480p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=773526,CODECS="avc1.4D401E,mp4a.40.2",RESOLUTION=852x480,VIDEO="medium",FRAME-RATE=26.250
https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/medium/highlight-138749526.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="mobile",NAME="226p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=365754,CODECS="avc1.42C00D,mp4a.40.2",RESOLUTION=400x226,VIDEO="mobile",FRAME-RATE=26.250
https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/mobile/highlight-138749526.m3u8
[INFO - VOD EXPANDED PLAYLIST JSON]
[
{
"STREAM-NUMBER": "1",
"TYPE": "VIDEO",
"NAME": "Source",
"GROUP-ID": "chunked",
"VIDEO": "chunked",
"CODECS": "avc1.64001F+mp4a.40.2",
"BANDWIDTH": "2818872",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "chunked",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/chunked/highlight-138749526.m3u8",
"RES-FPS": "1280x720p"
},
{
"STREAM-NUMBER": "2",
"TYPE": "VIDEO",
"NAME": "720p",
"GROUP-ID": "high",
"VIDEO": "high",
"CODECS": "avc1.4D401F+mp4a.40.2",
"BANDWIDTH": "1325212",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "high",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "26.250",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/high/highlight-138749526.m3u8",
"RES-FPS": "1280x720p26"
},
{
"STREAM-NUMBER": "3",
"TYPE": "VIDEO",
"NAME": "480p",
"GROUP-ID": "medium",
"VIDEO": "medium",
"CODECS": "avc1.4D401E+mp4a.40.2",
"BANDWIDTH": "773526",
"RESOLUTION": "852x480",
"QUALITY-BY-URL": "medium",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.775",
"FRAME-RATE": "26.250",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/medium/highlight-138749526.m3u8",
"RES-FPS": "852x480p26"
},
{
"STREAM-NUMBER": "4",
"TYPE": "VIDEO",
"NAME": "360p",
"GROUP-ID": "low",
"VIDEO": "low",
"CODECS": "avc1.42C01E+mp4a.40.2",
"BANDWIDTH": "661478",
"RESOLUTION": "640x360",
"QUALITY-BY-URL": "low",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "26.250",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/low/highlight-138749526.m3u8",
"RES-FPS": "640x360p26"
},
{
"STREAM-NUMBER": "5",
"TYPE": "VIDEO",
"NAME": "226p",
"GROUP-ID": "mobile",
"VIDEO": "mobile",
"CODECS": "avc1.42C00D+mp4a.40.2",
"BANDWIDTH": "365754",
"RESOLUTION": "400x226",
"QUALITY-BY-URL": "mobile",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.770",
"FRAME-RATE": "26.250",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/mobile/highlight-138749526.m3u8",
"RES-FPS": "400x226p26"
},
{
"STREAM-NUMBER": "6",
"TYPE": "VIDEO",
"NAME": "audio_only",
"GROUP-ID": "audio_only",
"VIDEO": "audio_only",
"CODECS": "mp4a.40.2",
"BANDWIDTH": "348364",
"RESOLUTION": "",
"QUALITY-BY-URL": "audio_only",
"NUMBER-OF-TRACKS": "1",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/52f9886174_zfg1_25157553392_641017582/audio_only/highlight-138749526.m3u8",
"RES-FPS": ""
}
]
```
#### 5.7.5. VODs with resolution and framerate missing for most streams
Some VODs don't contains almost any resolution and/or framerate. ID 134180904:
Only the best quality (chunked) contains resolution in the headers, but not framerate. The rest of the streams contain nothing, since the "QUALITY-BY-URL" are placeholders for parameters that are not fixed, like: high, medium, low, mobile.
```
[INFO - VOD INFORMATION JSON]
{
"data": {
"video": {
"title": "Reviewing very Old Ocarina of Time runs",
"broadcastType": "HIGHLIGHT",
"thumbnailURLs": [
"https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/0df5140b13_zfg1_24990621648_630584348//thumb/thumb134180904-320x180.jpg"
],
"createdAt": "2017-04-08T04:59:59Z",
"publishedAt": "2017-04-08T04:59:59Z",
"updatedAt": "2017-04-10T17:48:29Z",
"lengthSeconds": 30886,
"owner": {
"id": "8683614",
"displayName": "Zfg1",
"login": "zfg1"
},
"viewCount": 2008,
"game": {
"id": "11557",
"displayName": "The Legend of Zelda: Ocarina of Time",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/11557_IGDB-{width}x{height}.jpg"
},
"description": "Watching and reviewing very old and historic Ocarina of Time Speedruns\n\n1st: No Major Skips in 4:46 by Kazooie (2006) \n\n2nd (Starts at 5:10:25): The first OoT any% TAS in 2:33 (2006) (with commentary also from 2006) \n\n3rd (bonus, starts at 7:55:00): The first any% run using wrong warp to Ganon's Castle (2012)",
"status": "RECORDED"
}
},
"extensions": {
"durationMilliseconds": 44,
"requestID": ""
}
}
[INFO - VOD ADJUSTED CHAPTERS JSON]
{
"data": {
"video": {
"id": "134180904",
"moments": {
"edges": [
{
"node": {
"id": "",
"type": "GAME_CHANGE",
"positionMilliseconds": 0,
"durationMilliseconds": 30886000,
"description": "The Legend of Zelda: Ocarina of Time",
"subDescription": "",
"thumbnailURL": null,
"moments": null,
"video": null,
"details": {
"game": {
"id": "11557",
"displayName": "The Legend of Zelda: Ocarina of Time",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/11557_IGDB-40x53.jpg"
}
}
}
}
]
}
}
},
"extensions": {
"durationMilliseconds": 44,
"operationName": "VideoPlayer_ChapterSelectButtonVideo",
"requestID": ""
}
}
[INFO - VOD RAW PLAYLIST]
#EXTM3U
#EXT-X-TWITCH-INFO:ORIGIN="s3",B="false",REGION="",USER-IP="",SERVING-ID="",CLUSTER="cloudfront_vod",USER-COUNTRY="",MANIFEST-CLUSTER="cloudfront_vod"
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="Source",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=2839740,CODECS="avc1.64001F,mp4a.40.2",RESOLUTION=1280x720,VIDEO="chunked"
https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/chunked/highlight-134180904.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="audio_only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=345638,CODECS="mp4a.40.2",VIDEO="audio_only"
https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/audio_only/highlight-134180904.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="high",NAME="high",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=1951534,CODECS="avc1.000000,mp4a.40.2",VIDEO="high"
https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/high/highlight-134180904.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="low",NAME="low",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=838386,CODECS="avc1.000000,mp4a.40.2",VIDEO="low"
https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/low/highlight-134180904.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="medium",NAME="medium",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=1143886,CODECS="avc1.000000,mp4a.40.2",VIDEO="medium"
https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/medium/highlight-134180904.m3u8
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="mobile",NAME="mobile",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=429580,CODECS="avc1.000000,mp4a.40.2",VIDEO="mobile"
https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/mobile/highlight-134180904.m3u8
[INFO - VOD EXPANDED PLAYLIST JSON]
[
{
"STREAM-NUMBER": "1",
"TYPE": "VIDEO",
"NAME": "Source",
"GROUP-ID": "chunked",
"VIDEO": "chunked",
"CODECS": "avc1.64001F+mp4a.40.2",
"BANDWIDTH": "2839740",
"RESOLUTION": "1280x720",
"QUALITY-BY-URL": "chunked",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "1.778",
"FRAME-RATE": "",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/chunked/highlight-134180904.m3u8",
"RES-FPS": "1280x720p"
},
{
"STREAM-NUMBER": "2",
"TYPE": "VIDEO",
"NAME": "high",
"GROUP-ID": "high",
"VIDEO": "high",
"CODECS": "avc1.000000+mp4a.40.2",
"BANDWIDTH": "1951534",
"RESOLUTION": "",
"QUALITY-BY-URL": "high",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/high/highlight-134180904.m3u8",
"RES-FPS": ""
},
{
"STREAM-NUMBER": "3",
"TYPE": "VIDEO",
"NAME": "medium",
"GROUP-ID": "medium",
"VIDEO": "medium",
"CODECS": "avc1.000000+mp4a.40.2",
"BANDWIDTH": "1143886",
"RESOLUTION": "",
"QUALITY-BY-URL": "medium",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/medium/highlight-134180904.m3u8",
"RES-FPS": ""
},
{
"STREAM-NUMBER": "4",
"TYPE": "VIDEO",
"NAME": "low",
"GROUP-ID": "low",
"VIDEO": "low",
"CODECS": "avc1.000000+mp4a.40.2",
"BANDWIDTH": "838386",
"RESOLUTION": "",
"QUALITY-BY-URL": "low",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/low/highlight-134180904.m3u8",
"RES-FPS": ""
},
{
"STREAM-NUMBER": "5",
"TYPE": "VIDEO",
"NAME": "mobile",
"GROUP-ID": "mobile",
"VIDEO": "mobile",
"CODECS": "avc1.000000+mp4a.40.2",
"BANDWIDTH": "429580",
"RESOLUTION": "",
"QUALITY-BY-URL": "mobile",
"NUMBER-OF-TRACKS": "2",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "avc1",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "YES",
"DEFAULT": "YES",
"URL": "https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/mobile/highlight-134180904.m3u8",
"RES-FPS": ""
},
{
"STREAM-NUMBER": "6",
"TYPE": "VIDEO",
"NAME": "audio_only",
"GROUP-ID": "audio_only",
"VIDEO": "audio_only",
"CODECS": "mp4a.40.2",
"BANDWIDTH": "345638",
"RESOLUTION": "",
"QUALITY-BY-URL": "audio_only",
"NUMBER-OF-TRACKS": "1",
"AUDIO-CODEC": "mp4a",
"VIDEO-CODEC": "",
"ASPECT-RATIO": "",
"FRAME-RATE": "",
"AUTOSELECT": "NO",
"DEFAULT": "NO",
"URL": "https://d2nvs31859zcd8.cloudfront.net/0df5140b13_zfg1_24990621648_630584348/audio_only/highlight-134180904.m3u8",
"RES-FPS": ""
}
]
```
#### 5.7.6. VODs with and without html encoded entities in the jsons
NOTE: these JSONs are exactly as returned by the API.
Some VODs do not html encode certain characters/entities, like the single quote `'`, in neither the info json nor the chapters json, the server returns them as-is.
Example ID 2442178271:
```
VODJSONINFO_ORIGINAL:
{"data":{"video":{"title":"Old Gaming Glitched logic actorizer insanty (when is majoras mask 2 #nintendo ?)","broadcastType":"ARCHIVE","thumbnailURLs":["https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/fb4c6e73d3b4c65cebcb_evolvingfetus1_319261514105_1745582733//thumb/thumb0-320x180.jpg","https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/fb4c6e73d3b4c65cebcb_evolvingfetus1_319261514105_1745582733//thumb/thumb1-320x180.jpg","https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/fb4c6e73d3b4c65cebcb_evolvingfetus1_319261514105_1745582733//thumb/thumb2-320x180.jpg","https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/fb4c6e73d3b4c65cebcb_evolvingfetus1_319261514105_1745582733//thumb/thumb3-320x180.jpg"],"createdAt":"2025-04-25T12:05:39Z","publishedAt":"2025-04-25T12:05:39Z","updatedAt":"2025-04-27T05:06:24Z","lengthSeconds":63130,"owner":{"id":"765638629","displayName":"EvolvingFetus1","login":"evolvingfetus1"},"viewCount":98,"game":{"id":"12482","displayName":"The Legend of Zelda: Majora's Mask","boxArtURL":"https://static-cdn.jtvnw.net/ttv-boxart/12482_IGDB-{width}x{height}.jpg"},"description":null,"status":"RECORDED"}},"extensions":{"durationMilliseconds":71,"requestID":"REDACTED"}}
VODJSONCHAPTERS_ORIGINAL:
{"data":{"video":{"id":"2442178271","moments":{"edges":[{"node":{"moments":{"edges":[],"__typename":"VideoMomentConnection"},"id":"cb1acf617c839f57d6d1ddb087c5f481","durationMilliseconds":23871000,"positionMilliseconds":0,"type":"GAME_CHANGE","description":"The Legend of Zelda: Majora's Mask","subDescription":"","thumbnailURL":"","details":{"game":{"id":"12482","displayName":"The Legend of Zelda: Majora's Mask","boxArtURL":"https://static-cdn.jtvnw.net/ttv-boxart/12482_IGDB-40x53.jpg","__typename":"Game"},"__typename":"GameChangeMomentDetails"},"video":{"id":"2442178271","lengthSeconds":63130,"__typename":"Video"},"__typename":"VideoMoment"},"__typename":"VideoMomentEdge"},{"node":{"moments":{"edges":[],"__typename":"VideoMomentConnection"},"id":"5f05297a2844173fdd36172e0467550c","durationMilliseconds":1240000,"positionMilliseconds":23871000,"type":"GAME_CHANGE","description":"Prince of Persia: Warrior Within","subDescription":"","thumbnailURL":"","details":{"game":{"id":"15921","displayName":"Prince of Persia: Warrior Within","boxArtURL":"https://static-cdn.jtvnw.net/ttv-boxart/15921_IGDB-40x53.jpg","__typename":"Game"},"__typename":"GameChangeMomentDetails"},"video":{"id":"2442178271","lengthSeconds":63130,"__typename":"Video"},"__typename":"VideoMoment"},"__typename":"VideoMomentEdge"},{"node":{"moments":{"edges":[],"__typename":"VideoMomentConnection"},"id":"e5387324b64920503f7ac8b508786597","durationMilliseconds":38019000,"positionMilliseconds":25111000,"type":"GAME_CHANGE","description":"Retro","subDescription":"","thumbnailURL":"","details":{"game":{"id":"27284","displayName":"Retro","boxArtURL":"https://static-cdn.jtvnw.net/ttv-boxart/27284-40x53.jpg","__typename":"Game"},"__typename":"GameChangeMomentDetails"},"video":{"id":"2442178271","lengthSeconds":63130,"__typename":"Video"},"__typename":"VideoMoment"},"__typename":"VideoMomentEdge"}],"__typename":"VideoMomentConnection"},"__typename":"Video"}},"extensions":{"durationMilliseconds":27,"operationName":"VideoPlayer_ChapterSelectButtonVideo","requestID":"REDACTED"}}
```
However, in other VODs like 47407552, the single quote ' is html encoded to ' in the info json:
```
VODJSONINFO_ORIGINAL:
{"data":{"video":{"title":"Ocarina of Time Any speedrun (WR?) 57:17 (JP)","broadcastType":"HIGHLIGHT","thumbnailURLs":["https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/64517ac3a1/zfg1_1970006288_1970006288/thumb/thumb0-320x180.jpg"],"createdAt":"2011-10-27T00:43:05Z","publishedAt":"2011-10-27T00:43:04.829599Z","updatedAt":"2016-08-26T23:59:17Z","lengthSeconds":3500,"owner":{"id":"8683614","displayName":"Zfg1","login":"zfg1"},"viewCount":98,"game":null,"description":"bad run, just my first completed with the new route. ? next to WR because zaccio\u0026#39;s 55:52 is suspicious.","status":"RECORDED"}},"extensions":{"durationMilliseconds":43,"requestID":"REDACTED"}}
```
This could happen also in the chapters json of other VODs, but a real example would be needed to confirm.
### 5.8. Geoblocked and OAuth-gated content by Twitch and not the Streamer
Streamers (or Creators), that is, the owner of a Channel, can restrict certain type of content behind a subscription (paywall). Then OAuth/login is required to access it.
Types of content that can be restricted by the streamer:
- Live Stream: yes
- Past Broadcasts (Archives): yes
- Highlights: yes
- Uploads: yes
- Clips : no (always public)
When this happens, it affects the entire VOD / Live Stream, can't be done only for certain renditions (qualities / streams).
Streamers can't geoblock (geofence) any type of content, that is done exclusively by Twitch.
#### 5.8.1. Entire geoblocked VODs
Twitch can geoblock any VOD, Channel or even the entire Twitch, so content cannot be viewed in certain geographical areas. When this happens, looks like the VOD has been deleted or never existed, because no information is provided about any geoblocking.
The biggest example, is that Twitch has shut down operations completely in South Korea, on February 27, 2024 KST. The live streams can still be viewed from South Korea, but VODs can't from any channel.
For example, the channel [LCK](https://www.twitch.tv/lck), which stands for `League of Legends Champions Korea`, they are from South Korea themselves, yet they can't see their own VODs (nor anybody with a Korean IP can). When browsing their Twitch content page from outisde Korea, all content is listed: countless clips (https://www.twitch.tv/lck/clips?filter=clips&range=all) and 751 other vods (https://www.twitch.tv/lck/videos?filter=all&sort=time).
However browsing the channel from Korean IP does not render any VOD content: `No videos found.`, `No Clips Found`.
The most popular VOD by LCK is the ID 1443936760, here is a comparison:
1) Checking its availability from a Canadian IP:
```
$ zsh twitchdownloader-shell.sh info -C clear -R clear -M clear -a true --debug false -u 1443936760
TwitchDownloaderCLI (Shell Edition) 20250621.7; Copyright (C) 2025 dragomerlin (GPL-2.0-only)
[INFO - VOD INFORMATION JSON]
{
"data": {
"video": {
"title": "T1 vs GEN | 2022 LCK Spring FINALS",
"broadcastType": "ARCHIVE",
"thumbnailURLs": [
"https://static-cdn.jtvnw.net/cf_vods/d1m7jfoe9zdc1j/3f06bc50937652d5c85f_lck_46045972141_1648883411//thumb/thumb0-320x180.jpg",
"https://static-cdn.jtvnw.net/cf_vods/d1m7jfoe9zdc1j/3f06bc50937652d5c85f_lck_46045972141_1648883411//thumb/thumb1-320x180.jpg",
"https://static-cdn.jtvnw.net/cf_vods/d1m7jfoe9zdc1j/3f06bc50937652d5c85f_lck_46045972141_1648883411//thumb/thumb2-320x180.jpg",
"https://static-cdn.jtvnw.net/cf_vods/d1m7jfoe9zdc1j/3f06bc50937652d5c85f_lck_46045972141_1648883411//thumb/thumb3-320x180.jpg"
],
"createdAt": "2022-04-02T07:10:16Z",
"publishedAt": "2022-04-02T07:10:16Z",
"updatedAt": "2022-04-02T12:57:21Z",
"lengthSeconds": 20818,
"owner": {
"id": "124425501",
"displayName": "LCK",
"login": "lck"
},
"viewCount": 2800654,
"game": {
"id": "21779",
"displayName": "League of Legends",
"boxArtURL": "https://static-cdn.jtvnw.net/ttv-boxart/21779-{width}x{height}.jpg"
},
"description": "",
"status": "RECORDED"
}
},
"extensions": {
"durationMilliseconds": 41,
"requestID": "REDACTED"
}
}
[INFO - VOD ADJUSTED CHAPTERS JSON]
…
[INFO - VOD RAW PLAYLIST]
…
[INFO - VOD EXPANDED PLAYLIST JSON]
…
```
2) Checking its availability from a South Korean IP:
```
$ zsh twitchdownloader-shell.sh info -C clear -R clear -M clear -a true --debug true -u 1443936760
TwitchDownloaderCLI (Shell Edition) 20250621.7; Copyright (C) 2025 dragomerlin (GPL-2.0-only)
ERROR: Invalid VOD: incorrect | deleted | expired | unpublished | ID for Live stream instead of VOD | CLIP instead of ARCHIVE/UPLOAD/HIGHLIGHT ?
Failed to fetch video info, with return code 2. Exiting.
```
In this case, the json response from `https://gql.twitch.tv/gql` is: `{"data":{"video":null},"extensions":{"durationMilliseconds":2,"requestID":"REDACTED"}}`.
It's not that the data returned from any endpoint does not give any details but informs that somehow the VOD is available; is that the response altogether mimics exactly the behavior of a VOD that is gone for good.
That proves, that some VODs that are declared as deleted or not available, may still exist and be online for certain people.
When browsing from South Korea, not only the VOD playlist can't be retrieved, but the segments themselves are not downloadable either:
```
# Canadian IP
$ curl -sI -o /dev/null -w "%{http_code}\n" https://d3vd9lfkzbru3h.cloudfront.net/44620f8468bb248fd77e_lck_40051862935_1675575010/chunked/2201.ts
200
```
```
# South Korean IP
$ curl -sI -o /dev/null -w "%{http_code}\n" https://d3vd9lfkzbru3h.cloudfront.net/44620f8468bb248fd77e_lck_40051862935_1675575010/chunked/2201.ts
403
```
#### 5.8.2. Renditions being geo-blocked and auth-walled
Starting on May 2025, Twitch started geofencing renditions (not VODs) of quality higher than 1080p. They call it [Enhanced Broadcasting with Multiple Encodes](https://help.twitch.tv/s/article/multiple-encodes?language=en_US). In order to view higher resolutions that that, the user must be logged in (or use OAuth), and be in one of the following countries: Argentina, Australia, Austria, Belgium, Brazil, Canada, Chile, Colombia, Denmark, Finland, France, Germany, Ireland, Italy, Japan, Kuwait, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Qatar, Spain, Sweden, Switzerland, UAE, UK, USA. Other countries and logged-out users are limited to 1080p.
For example, the ID [2491861049](https://www.twitch.tv/liebeskrieger/video/2491861049) has 2 renditions that are geofenced this way: 2160p60 and 1440p60.
The following command captures the part that tells about any geoblocking, according the country of origin and if OAuth is used or not:
```
$ bash twitchdownloader-shell.sh info --debug true -u 2491861049 [--oauth $OAUTH] | grep -F -A1 '(DEBUG): GENERAL_VODJSONVIDEOTOKEN_ORIGINAL:'
```
USA + OAuth:
```
(DEBUG): GENERAL_VODJSONVIDEOTOKEN_ORIGINAL:
{"data":{"videoPlaybackAccessToken":{"value":"{\"authorization\":{\"forbidden\":false,\"reason\":\"\"},\"chansub\":{\"restricted_bitrates\":[]},\"device_id\":null,\"expires\":1750653570,\"https_required\":true,\"privileged\":false,\"user_id\":REDACTED,\"version\":3,\"vod_id\":2491861049,\"maximum_resolution\":\"ULTRA_HD\",\"maximum_video_bitrate_kbps\":12500,\"maximum_resolution_reasons\":{},\"maximum_video_bitrate_kbps_reasons\":[\"AUTHZ_DISALLOWED_BITRATE\"]}","signature":"REDACTED","__typename":"PlaybackAccessToken"}},"extensions":{"durationMilliseconds":115,"operationName":"PlaybackAccessToken_Template","requestID":"REDACTED"}}
```
USA + without OAuth:
```
(DEBUG): GENERAL_VODJSONVIDEOTOKEN_ORIGINAL:
{"data":{"videoPlaybackAccessToken":{"value":"{\"authorization\":{\"forbidden\":false,\"reason\":\"\"},\"chansub\":{\"restricted_bitrates\":[]},\"device_id\":null,\"expires\":1750653623,\"https_required\":true,\"privileged\":false,\"user_id\":null,\"version\":3,\"vod_id\":2491861049,\"maximum_resolution\":\"FULL_HD\",\"maximum_video_bitrate_kbps\":12500,\"maximum_resolution_reasons\":{\"QUAD_HD\":[\"AUTHZ_NOT_LOGGED_IN\"],\"ULTRA_HD\":[\"AUTHZ_NOT_LOGGED_IN\"]},\"maximum_video_bitrate_kbps_reasons\":[\"AUTHZ_DISALLOWED_BITRATE\"]}","signature":"REDACTED","__typename":"PlaybackAccessToken"}},"extensions":{"durationMilliseconds":62,"operationName":"PlaybackAccessToken_Template","requestID":"REDACTED"}}
```
Isle of Man + OAuth:
```
(DEBUG): GENERAL_VODJSONVIDEOTOKEN_ORIGINAL:
{"data":{"videoPlaybackAccessToken":{"value":"{\"authorization\":{\"forbidden\":false,\"reason\":\"\"},\"chansub\":{\"restricted_bitrates\":[]},\"device_id\":null,\"expires\":1750653317,\"https_required\":true,\"privileged\":false,\"user_id\":REDACTED,\"version\":3,\"vod_id\":2491861049,\"maximum_resolution\":\"FULL_HD\",\"maximum_video_bitrate_kbps\":12500,\"maximum_resolution_reasons\":{\"QUAD_HD\":[\"AUTHZ_GEO\"],\"ULTRA_HD\":[\"AUTHZ_GEO\"]},\"maximum_video_bitrate_kbps_reasons\":[\"AUTHZ_DISALLOWED_BITRATE\"]}","signature":"REDACTED","__typename":"PlaybackAccessToken"}},"extensions":{"durationMilliseconds":110,"operationName":"PlaybackAccessToken_Template","requestID":"REDACTED"}}
```
Isle of Man + without OAuth:
```
(DEBUG): GENERAL_VODJSONVIDEOTOKEN_ORIGINAL:
{"data":{"videoPlaybackAccessToken":{"value":"{\"authorization\":{\"forbidden\":false,\"reason\":\"\"},\"chansub\":{\"restricted_bitrates\":[]},\"device_id\":null,\"expires\":1750653341,\"https_required\":true,\"privileged\":false,\"user_id\":null,\"version\":3,\"vod_id\":2491861049,\"maximum_resolution\":\"FULL_HD\",\"maximum_video_bitrate_kbps\":12500,\"maximum_resolution_reasons\":{\"QUAD_HD\":[\"AUTHZ_GEO\",\"AUTHZ_NOT_LOGGED_IN\"],\"ULTRA_HD\":[\"AUTHZ_GEO\",\"AUTHZ_NOT_LOGGED_IN\"]},\"maximum_video_bitrate_kbps_reasons\":[\"AUTHZ_DISALLOWED_BITRATE\"]}","signature":"REDACTED","__typename":"PlaybackAccessToken"}},"extensions":{"durationMilliseconds":62,"operationName":"PlaybackAccessToken_Template","requestID":"REDACTED"}}
```
In some of the responses, there can be seen the values `AUTHZ_NOT_LOGGED_IN` and/or `AUTHZ_GEO`, when higher than 1080p renditions exists in that VOD, but are not being offered for any or both reasons.
The unavailable renditions can still be requested by a proper query, even when the requirements are not met: the results are provided in a tag inside the VOD playlist `GENERAL_VODPLAYLIST` like this:
```
#EXT-X-SESSION-DATA:DATA-ID="com.amazon.ivs.unavailable-media",VALUE="base64_encoded_json"
```
There the field `VALUE` is the base64 encoded json, with all the unavailable renditions. The script `twitchdownloader-shell.sh` does this by default but can be changed with a flag.
To view examples of currently active streams, with higher resolutions than 1080p (Enhanced Broadcasting), there's a webpage set up for that: [Twitch QHD/UHD EB Streams](https://eb.rodney.io/), which updates every 10 minutes.
#### 5.8.3. Summary table about geoblocking and content fencing
| Restriction | Agent | Reason |
| :--- | :--- | :--- |
| Geo-Blocking | Twitch (Platform) | Legal compliance (South Korea), copyright (Esports), licensing (Movie premieres) |
| Subscriber-Only Access | Streamer | Reward paying supporters and increase channel income |
| Follower-Only Chat | Streamer | Reduce spam or chat speed, and encourage viewers to join the community |
| VOD Muting | Twitch (Automated) | An automated system (like Audible Magic) mutes sections of VODs that contain copyrighted music to comply with DMCA |
