Documentation for ts_extract

What is this?

ts_extract is a tool that helps you to recover lost .ts (MPEG2 transport stream)
files. It does so by scanning the files/partitions/devices you specify at the
command line and detecting valid .ts file fragments.

It does then analyze these and see whether they can be recombined into complete
files by looking at stream PIDs (and the PAT/PMT directories) and the continuity
fields in them as well as the PCR.

This can be useful to recover lost files from a hard disk after a disk crash or
after deleting files by mistake. (For a crashed disk, first create an image
using a tool like e.g. dd_rescue.)

Approach

The program does work in three steps

1. It reads all the input files/partitions/disks/images/... and creates a list
   of extents that represent TS file fragments. (This step typically takes the
   longest; the results of it can be stored in a log file that can be reread
   later, skipping over this step.)

2. It then uses the analyzed metadata to reconnect the fragments into (hopefully)
   complete TS streams. (This does take some computation time, tens of minutes
   for tens of thousands of fragments.)

3. The resulting streams can then optionally be copied into files.

TS analysis

The tool looks for a number of features in TS files to detect whether they
belong together:

• The TS packet of 188 does NOT nicely fit into a 4k filesystem block; this
  actually is an advantage for us, as it gives us a test for consecutive
  blocks with only 1/(188/4) error probability.

• The TS packets have a per PID cont counter (4bits) that's increased per
  segment; this gives us a test with a 1/16 error rate; given the fact that
  you find several PIDs in a block typically, this becomes even more useful.

• The TS packets belong to a program that's described by a PAT and PMT;
  we assume that only ONE program is recorded in a .ts file (which is the
  case for e.g. E2 based VDRs); a change in the PID set or the PAT or
  PMT indicates that we have a new stream. (This also applies to the the
  TSID.)

• One of the PIDs of a program typically carries a timestamp (PCR). This
  needs to be continuous and allows us to determine how to reconnect the
  fragments.

Features

• You can filter for transport stream IDs (TSID) and program IDs. If you look
  for a few files only or for a series of files, this is an effective way
  to limit the amount of (CPU) time needed to reconnect the fragments and
  to limit the amount of files and storage required to output the results.

• The state can be saved and reloaded again; this is a good time saver;
  the recommended approach to rescue files is to NOT filter in the first
  step, but save the state (option -l). You can then use that as input
  (option -L) and skip over the first step.

• Options -v (verbose) and -d allow you to get more information and
  observe the inner workings of the program. You can sepcify these options
  multiple time to increase the level of verbosity.

• There's no man page (yet), only the -h (help) summary and this little
  document.

Hints

• If you have deleted files by mistake, make sure you cut off write operations
  to that filesystem as soon (or as much) as possible.

• If you have a crashed disk, do the analysis/extraction on an image (created
  by e.g. dd_rescue.)

• Use options -l and -L to save time. Use -l without filtering (options
  -t and -g) initially and then use -L together with filtering.

Limitations

• The program does not parse tables (PAT, PMT) larger than one packet currently.

• The reconnection routine does only work reliably up to files with up to
  13hrs of stream data (this is due to an optimization).

• The program only deals with stream that contain ONE program. (The PAT
  may reference multiple programs, that's fine.)

License

I release the under the terms of the GNU General Public License (GPL) version 2
or version 3 (at your option).

More information

• See wikipedia page
  for MPEG2 TS description.