[Freevo-devel] Freevo with parallel recording and DVB EPG

[Hartmut S. <har...@su...>]

Hi all,

before I begin, let me thank Dischi and all the people who else worked 
on this project for their great work. I am using Freevo since 2004 
(starting with version 1.5.0, later switching to 1.5.3 and not upgrading 
since then), and I am a big fan of it (and of Python). For me, there's 
no alternative in simplicity, functionality and pragmatism.

In the last three years more and more analog channels were removed from 
my cable in favor of digital TV, but when in late 2013 the analog EPG on 
my cable finally vanished, it was time to make a big switch. I had been 
short on disk space for years, and so it was time to get a bigger hard 
disk and install the newest Freevo on it. While doing this, I also 
intended to realize some wishes that had emerged when using Freevo.

This work turned into a far bigger project than I initially intended, 
keeping me busy (with some interruptions) for almost two years now. I 
was lucky that there were no commits during this time, so I wouldn't 
have to address a moving target (although it's a pity that the community 
seems to have moved away from Freevo; for me it's NOT dead as the patch 
shows: it's ready now for another 10 years).

The result of my work are the attached huge patch for Freevo and the one 
for kaa (I'm not entirely sure if I crafted the kaa XML correctly 
because I wrote my config_dvbstreamer.py by hand). I initially intended 
to dissect the result of my work into separate patches for one issue 
each before submitting, but as work progressed (you will note that I 
have completely rewritten core parts of the recordserver), this became 
increasingly difficult and finally impossible. I'm sorry for that, but 
I'm sure that I have done a piece of solid work that can be taken as a 
whole and will turn Freevo into a new release everyone will embrace. The 
long development phase during which I increasingly used this 
installation for productive recording caused a fair amount of testing to 
be accomplished. I thus consider the result to be pretty stable now. 
Although, nobody knows really...

The following are the most notable changes I made besides, of course, 
making many other small enhancements and fixing bugs:

1) Grabbing EPG from DVB tuner card (dvbstreamer). (An optional text 
dump allows to feed external mechanisms presenting it to other devices.)

2) Parallel use of recording devices of different types (e.g. analog + DVB).

3) Parallel recording on multiple devices of same type.

4) Parallel recording of multiple programs on same DVB device as long as 
they are transmitted on the same transponder channel (including parallel 
recordings of the same channel - needed when recording adjacent shows 
due to them overlapping each other with their record paddings).

5) Different video groups can use the same device, allowing to choose 
between different sets of recording and postprocessing parameters by 
defining multiple video groups ('recorders' in the new terminology) per 
device.

6) Improved manual conflict handling (recursively stacked).

7) Best-effort clean recording in presence of unresolved conflicts.

8) Improved manual scheduling, editing of scheduled recordings 
(accessible also from program guide) and favorites, manual creation of 
favorites, entering of descriptions for existing videos.

9) Favorites with keyword search alternatively to title search.

10) Allowing multiple favorites with same title; a program will always 
be scheduled by the strongest applicable favorite.

11) Favorite reordering by button press (VOL+/VOL-).

12) Automatic daily or weekly repeat of recordings.

13) Direct recording into subfolders of TV_RECORD_DIR.

14) Text input using keyboard, improved SMS-style input, input of 
international charachters.

15) Horizontal scrolling in program guide (and better vertical scrolling).

16) Improved autoshutdown plugin (autologin + quick logoff, anticipatory 
fsck feature).

17) Idlebar Alert plugin as notification channel for recording plugins. 
(The icon is a bit too prominent, maybe someone can create a more decent 
one.)

18) Command plugin again working, can be invoked now also by hotkey.

19) Avidemux plugin for direct video editing (implemented as alternative 
video player).

20) Mplayer ZOOM button, single frame step and improved manual seek.

21) Autoresume feature for playing directories.

22) New password protection mechanism, old one extended for multiple 
passwords per protected directory.

I strived to my best to ensure that existing schedules and favorites 
(missing the new attributes I introduced) will not break with my 
version, but I cannot guarantee that.

I also did my best to not break TV watching (livepause, mplayer), but I 
don't use that (I watch only recordings), so I cannot guarantee. I also 
did not test the new ivtv and vbi2srt recording (I don't have these cards).

I extended the default TV_RECORD_FILE_MASK by adding the channel id, so 
parallel recordings from different channels won't overwrite each other 
even in the case of identical titles (like 'Manual Record') and start times.

I added some comments about things I didn't understand, maybe someone 
else does and can cleanup any issues possibly present. The upsoon plugin 
may have an issue now, I don't use it and did not test.

I am using the Panorama skin. Maybe other skins could also benefit from 
similar modifications like those I did on the Panorama skin.

There are some new configuration settings, so I defined a new config 
version 5.31. In addition to that, the alert, autoshutdown and command 
plugins define the following new configuration settings:

ALERTVIEW_EVENT
AUTOSHUTDOWN_FSCK_TEST
AUTOSHUTDOWN_FSCK_LILO_CMD
AUTOSHUTDOWN_FSCK_GRUB_CMD
COMMANDS_EVENT
COMMANDS_SORT_MENU

The default values for all new configuration options were choosen so 
that existing behaviour remains unchanged (except postprocessing 
serialization, see below).

What is important when switching to the new code is to review the 
TV_CHANNELS configuration. There are a couple of possibilities for 
configuring channels to allow for backward compatibility, but you will 
probably want to use the new format if you have more than one TV card. 
Please read the explanations in local_conf.py.example. Since the 
time-dependent channels code was useless for recording (it did not work, 
see my comments), I removed this feature from the new config syntax to 
not carry around unnecessary baggage in TV_CHANNELS when the new 
features are used. I kept, however, compatibility with existing 
configurations, so you should be able to continue to use it for watching 
if you want.

Additional notes:

- When using the autoshutdown plugin, setting AUTOSHUTDOWN_TIMER_TIMEOUT 
to a negative value activates a special GUI login feature: The first 
menu navigation will implicitly deactivate the autoshutdown timer, thus 
effectively logging the user in. (So, Freevo wouldn't suddenly shutdown 
simply because you leave it alone for a while.) At the same time, the 
default action of the Shutdown menu will be changed to logging the user 
back out with a 10 seconds grace period to cancel logout, causing a 
basically immediate shutdown, provided there is nothing important 
running and no recording due shortly. (Due to the three tests performed 
to detect important processes, the grace period will actually extend up 
to a minute or so.)

- To prevent missing the start of a recording after wakeup due to a 
filesystem check done on boot, the autoshutdown plugin executes the 
command specified in AUTOSHUTDOWN_FSCK_TEST to determine if a filesystem 
check is due. If so, instead of a normal shutdown (or reboot, if nvram 
demands it), it will perform a reboot using the AUTOSHUTDOWN_FSCK_*_CMD 
to reboot into a special boot configuration you provide that has the 
parameter forcefsck added to the kernel command line and boots into 
runlevel 0 (shutdown) like so:

# normal boot configuration
label Freevo
linux /boot/vmlinuz-3.2.0-4-686-pae
append initrd=/boot/initrd.img-3.2.0-4-686-pae root=/dev/sda1 ro quiet 3

# boot into file system check followed by shutdown
label FsCheck
linux /boot/vmlinuz-3.2.0-4-686-pae
append initrd=/boot/initrd.img-3.2.0-4-686-pae root=/dev/sda1 ro quiet 
forcefsck 0

- With Freevo 1.5 I had problems with excess events when repeatedly 
pressing navigation buttons, and with the new Freevo version 
autorepeating keys (FFWD/REW) ceased to work at all (due to the repeats 
being suppressed by Freevo). Upon investigation the implementation 
appeared broken to me, so I fixed it (I think). More importantly, since 
inside Freevo the information necessary to distinguish key repeats from 
repeated key presses is not available, it's the wrong place to do this. 
The information is, however, available to LIRC, so by using the correct 
'repeat' and 'delay' settings you can accomplish the task in the correct 
place:

begin
prog = freevo
button = KEY_DOWN
config = DOWN
repeat = 1
delay = 4
end

The code built into Freevo can then be deactivated by setting in 
local_conf.py:

LIRC_KEY_REPEAT = (0.0, 0.0)

- When entering text using the great TextEntryScreen (I planted it in 
more places now), a keyboard can now be used for text input (its 
presence will be detected by the first letter entered, turning off the 
SMS-style interpretation of number key presses). SMS style input now 
provides better user feedback (cursor movement on commit as seen on cell 
phones) and national characters. The DELETE and REPEAT buttons allow 
deleting characters and cycling through the different character panels 
(first to upper case with automatic switchbackon successful commit). 
Note that for manually committing a letter RIGHT or OK has to be used 
now (the asterisk feature has been disabled for regularity, to get the 
asterisk as text).

- The new password protection feature using DIRECTORY_AUTHORIZATION 
supersedes the existing protection mechanism based on .password (the 
latter was improved too). It allows to have all password protection 
information in one central place, and especially not on the media volume 
itself. Even more important is that protected directories will be hidden 
from the user until he enters the correct password, that is he will have 
no idea what can be revealed by entering a password. (The only hint 
Freevo gives is that there _is_ something to be revealed, since 
otherwise it won't prompt for a password.) There is also no other 
feedback for the password entered than revealing the folders protected 
by it (in fact, any password will be silently accepted and will sit 
there revealing whatever is protected by it, so entering a password is 
simply like pinning an authorization token, a kind of an additional 
non-mandatory login that gives access to additional content throughout 
the menu system). Particularly, Freevo will give no hint about if there 
are more valid passwords to enter to reveal different items.

The password prompt is invoked by pressing the DISPLAY button (the 
MENU_CHANGE_STYLE event is reused when DIRECTORY_AUTHORIZATION is set), 
and the password is cleared by another press of the same button (you can 
do this already while inside the protected directory, so it will be 
invisible after leaving).

It is possible to register different passwords for the same content, so 
if several people are using Freevo, they may all have their private as 
well as shared protected content.

Passwords are no more stored in cleartext but as MD5 hashes, giving a 
minimal level of protection (basically, against accidental eavesdropping 
only - note that a brute force attack against a 6-digit PIN code will 
take only seconds to succed on today's computers).

If you continue to use .password, all your passwords stored in these 
files must be converted to MD5 hashes. As a reward, there can now be 
multiple of them in every .password file if desired (one per line).

- The FULLSCREEN_TOGGLE event in video/mplayer will cycle through 
different zoom configurations as does the corresponding button on my 
Sony TV for SCART input. The autocrop feature didn't work for me since 
the first seconds of a video are often special, causing a wrong crop, so 
I wanted to control this manually. Besides cropping vertically, a 4:3 
video on widescreen can also be stretched horizontally (some Youtube 
videos need that).

- When scheduling a recording from the program guide, there is now an 
additional context menu option "New Recording" that will present you 
with a prefilled edit screen. You will need this if you want to choose a 
different (from the default) recording device, record into a subfolder 
or change other information. A special use for this is recording two 
shows adjacent to each other on a device other than DVB: Simply set the 
start and/or end time so that the recording spans both shows. (They will 
then also both be shown in green in the program guide, and the scheduled 
recording can be accessed from both via 'Edit Recording'.)

It is also possible to create a new scheduled recording from an existing 
one by the same context menu item. So, if you know that part 2 of the 
movie you just scheduled will come next week (it's probably not on the 
EPG yet), you may just clone the recording and adapt the start date.

- When manually scheduling a recording, it is generally unnecessary to 
enter a day or month or year if the show is tonight. Freevo will 
conclude from the start and stop time when tomorrow is meant. I 
recommend you also set the new config setting TV_RECORD_MAXDAYS to 1 to 
remove the item for setting the stop day from the menu.

- When editing the title of a recording having a subtitle, the latter 
will be concatenated to the title and cleared upon save. You may want to 
use this to get more informative listings.

- For recording repeating shows there are now two options: The classic 
one of using a favorite that will schedule the recordings based on the 
EPG, and a new one of scheduling a recording with automatic (dumb) 
repeat (which does not need the EPG).

- The manual_record plugin is now obsolete. Its code has been moved to 
programitem.py, to allow existing scheduled recordings to be edited, and 
a 'Manual Record' menu item has been added the context menu of the 
'Scheduled Recordings' TV menu item.

- From the context menu of the View Favorites TV menu item it is now 
possible to manually create a new favorite. This allows to easily put 
Freevo hunting for a missed show or one read about somewhere else that 
is not present in the program guide.

- For this to work better it is now possible to record directly into 
subfolders (of course, there are more uses for this). So you might want 
to request the hunting favorites to record into a subfolder 'Inbox' 
where you regularly can check for their success and remove the 
corresponding favorites after they did their work.

- If the gap between two programs is shorter than TV_RECORD_PADDING_PRE 
+ TV_RECORD_PADDING_POST, it will be proportionally split between the 
two programs. In TV_RECORD_PADDING_CONFLICT a minimum gap greater than 
zero can be specified to cause programs that are closer to one another 
than this have a conflict.

- Conflicts between programs are now distinguished by the side of the 
conflict: The program with the higher priority (lower priority value) 
has a 'suppress' (because it prevents _another_ program from recording), 
the other program has an 'overlap' (a conflict that prevents _it_ from 
recording). A scheduled program without overlaps will always be shown in 
the EPG as green even when it has suppresses (because it will record 
cleanly). (Programs that are currently recording are shown in red, 
however.) A program with conflicts will be shown in orange. But there 
are more subtleties:

All programs enlisted in conflicts are internally stacked according to 
their priorities. The ones at the bottom are green, as already said. The 
ones one level above will never record cleanly, so they are shown in 
dark orange. Programs at the levels above might be suppressed only by 
programs that are already dark orange, but not by green ones. Since 
there is no point in preventing a (lower prioritized) program from 
recording if it could do so cleanly only to record part of an albeit 
higher prioritized program that is broken anyway, upon recording Freevo 
will ignore programs with suppresses that cannot record cleanly. The 
result is that lower prioritized programs may be selected for recording 
while higher prioritized programs are not. To indicate this fact, the 
programs that will record despite their overlaps to higher prioritized 
programs (that will not record) are shown in the Program Guide in light 
orange.

By using this strategy, Freevo maximizes the number of accomplishable 
clean recordings in the presence of unresolved conflicts at recording 
time. A program that cannot record cleanly will start recording (for the 
rest of its duration) only when it does not (that is, no more) suppress 
another program. (Programs that ARE recording will never be stopped by 
Freevo to free the recorder before their regular end, no matter how high 
prioritized another program due for recording is. But this concerns only 
favorite scheduling and automatic repeat since scheduling by the user 
will remove conflicting programs from the schedule if he chooses to 
resolve the conflict this way, thereby also stopping the recording.)

Programs scheduled by the user will always have highest priority (zero) 
as long as they don't have an automatic repeat. Following are the 
programs scheduled by favorites in their priority order. The lowest 
priority have programs with automatic (dumb) repeat, internally stacked 
by 'weekly' (highest) to 'weekdays'/'weekends' to 'daily' (lowest). On 
equal priority the program starting earlier or the longer one wins, 
further maximizing the expected clean recording time.

As it is implemented now, it is impossible to introduce conflicts into 
the schedule by the user scheduling programs. He will be immediately 
presented with a conflict resolution form presenting the programs in 
conflict. From there he can edit or delete them, search for more 
occurrences and so on, possibly stacking more conflict resolution forms 
if more conflicts are introduced. Consequently, the only way to 
introduce conflicts into the schedule are favorites and automatic repeats.

I must admit that I didn't understand the automatic conflict handling 
code I found in the record server. I don't use automatic conflict 
handling myself, especially because I'm unclear about how and if at all 
an automatism can be made intelligent enough to be helpful in situalions 
too complex to be easily resolved manually. (To be honest, it would also 
have to be extended to handle multiple recording devices now.) For this 
reason I require the user to resolve any conflicts introduced by himself 
manually and replanted this code from where it was to where favorites 
are scheduled. I'm not sure, however, how helpful it will be in this 
place. The original author will probably know better. I have left some 
comments in the code up to this topic.

- For saving space and because my ten years old Pentium 4 is too slow to 
decode HDTV in real time I have to transcode DVB recordings. (Even 
without this need, a remultiplexing step is necessary in order to 
correct the timestamps.) As I was unsuccessful to get the encodingserver 
to work for H264, I implemented a poor man's postprocessing feature 
controlled by a new VideoGroup attribute postprocess_cmd to invoke 
avconv/ffmpeg or whatever you want (you can code any shell command 
sequences here). For an example see the documentation in config.py.

- The record postprocessing threads invoking this are now serialized to 
avoid parallel running transcoding jobs to unnecessarily fragment the 
filesystem when this doesn't speed up things (by using multiple CPU 
cores for instance). (Note that there is no correct queueing, only 
serialization, so the postprocessings will not necessarily get executed 
in launch order.) A new config setting 
RECORDSERVER_PARALLEL_POSTPROCESS_TASKS can be used to control the 
number of parallel running postprocessing tasks (default: 1).

I did not entirely understand if VCR_POST_REC is (analogously to 
VCR_PRE_REC) meant for short running commands after a recording has 
finished, only changing settings or the like. If this is the case, it 
should be probably moved before the serializing 'with' block to not 
break this behaviour. If it is indeed meant for transcoding, it should 
be probably moved up in the code close to my new vg.postprocess_cmd 
stuff and before generating the snapshot. However, I didn't understand 
if the thread will at all wait for the command to finish (it should, 
because of the following TV_RECORD_REMOVE_COMMERCIALS, but I can't tell 
because I didn't understand the Popen* stuff). So I can't tell if the 
serialization will indeed work for VCR_POST_REC. I'd recommend to use 
VCR_POST_REC for settings only stuff and move it out of the 'with'.

- As I understand, the lock files the record server maintains when using 
a device are intended for the TV watching plugins (which themselves will 
NOT lock the device). I think, with my new code I maintained 
compatibility. A more elegant way for handling the dual use of capture 
devices could be to have the view plugins ask the record server if 
accessing some service on some device (video group) is possible without 
conflict. The record server would then communicate back the answer of 
the associated recording plugin. (This way a DVB device could be used 
for simultaneously recording and watching programs that are on the same 
transponder channel, which is not possible using lock files.)

- The dvbstreamer version I'm using (2.1.0-2.3 from Debian Wheezy) 
commonly experiences problems when fetching the EPG, so I restart it 
after every EPG fetch. I use an endless restart loop in conjunction with 
a named pipe /var/cache/freevo/dvbstreamer.input to feed dvbstreamer the 
'quit' command (which is often not enough, though). This code is from my 
.rc script:

case "$1" in
start)
if ! netcat -z localhost 54197; then
log_daemon_msg "Starting dvbstreamer 0"
(while true; do $DVBSTREAMER -r -u dvbstreamer -p control; sleep 2; done 
  >/tmp/dvbstreamer0.out 2>&1) </var/cache/freevo/dvbstreamer.input &
log_end_msg $?
fi
if [ -d /dev/dvb/adapter1 ] && ! netcat -z localhost 54198; then
log_daemon_msg "Starting dvbstreamer 1"
$DVBSTREAMER -dr -a 1 -u dvbstreamer -p control -L /tmp/dvbstreamer1.out
log_end_msg $?
fi
log_daemon_msg "Starting EPG Capture"
(sleep 20; $FREEVO tv_grab >/tmp/tv_grab.out 2>&1; if [ $($DVBCTRL -u 
dvbstreamer -p control lssfs | wc -l) -eq 1 ]; then pid="$(ps -C 
dvbstreamer -o 'pid=,cmd=' | grep -v -- -a)"; echo "restarting $pid" 
 >>/tmp/tv_grab.out; echo "quit"; sleep 10; if [ "$pid" = "$(ps -C 
dvbstreamer -o 'pid=,cmd=' | grep -v -- -a)" ]; then echo "KILLING $pid" 
  >>/tmp/tv_grab.out; kill ${pid%% *} >/dev/null 2>&1; sleep 10; fi; if 
[ "$pid" = "$(ps -C dvbstreamer -o 'pid=,cmd=' | grep -v -- -a)" ]; then 
echo "FORCED KILLING $pid" >>/tmp/tv_grab.out; kill -9 ${pid%% *} 
 >/dev/null 2>&1; fi; fi; while true; do sleep 10d; done) 
 >/var/cache/freevo/dvbstreamer.input &
log_end_msg $?
log_daemon_msg "Starting Freevo Record Server"
$FREEVO --daemon recordserver >/tmp/recordserver.out 2>&1
log_end_msg $?
log_daemon_msg "Starting Freevo Frontend"
/usr/bin/X11/startx $FREEVO >/tmp/freevo.out 2>&1 &
log_end_msg $?
;;
stop)
log_daemon_msg "Stopping Freevo Record Server"
$FREEVO --stop recordserver
log_end_msg $?
log_daemon_msg "Stopping Freevo Frontend"
$FREEVO --stop
log_end_msg $?
log_daemon_msg "Performing Freevo Cache Maintenance"
$FREEVO cache >/dev/null 2>&1
log_end_msg $?
;;

I have a similarly defined EPG refresh command for the command plugin:

<?xml version="1.0" ?>
<freevo>
<command title="EPG aktualisieren">
<cmd>/home/hartmut/freevo/freevo tv_grab >>/tmp/tv_grab.out 2>&amp;1; if 
[ $(/usr/bin/dvbctrl -u dvbstreamer -p control lssfs | wc -l) -eq 1 ]; 
then pid="$(ps -C dvbstreamer -o 'pid=,cmd=' | grep -v -- -a)"; echo 
"restarting $pid"; echo "quit" > /var/cache/freevo/dvbstreamer.input; 
sleep 10; if [ "$pid" = "$(ps -C dvbstreamer -o 'pid=,cmd=' | grep -v -- 
-a)" ]; then echo "KILLING $pid"; kill ${pid%% *}; sleep 10; fi; fi; 
while ! /usr/bin/dvbctrl -u dvbstreamer -p control lssfs; do if [ -n 
"$pid" ]; then echo "FORCED KILLING $pid"; kill -9 ${pid%% *}; pid=""; 
fi; sleep 10; done</cmd>
<info>
<description>Programmführer aktualisieren</description>
</info>
</command>
</freevo>

For security reasons, the EPG grabber refuses to touch dvbstreamer when 
a recording is in progress.

- When doing DVB recordings, I experienced sudden changes of service 
names (for instance adding a forgotten 'HD'). Since dvbstreamer selects 
services by name, this is fatal because the service will not be found, 
causing the recording to fail. So I let the dvbstreamer record plugin 
check on every start if the services defined in TV_CHANNELS exist on the 
cable. To salvage as many recordings as possible, I further implemented 
some fuzzy logic I am very proud of, to select for every missing service 
the most similar by name service out of the ones on the cable that are 
not used in TV_CHANNELS. (Basically, I compute a normalized vector from 
every name in the space of counts of all possible pairs of consecutive 
characters, and then select the one with the maximum dot product against 
the vector computed from the missing name.) The result of this 
substitution is then used for recording and is also communicated to the 
user via the idlebar alert plugin (so he can review his configuration 
and even do a channel scan if indicated).

- The recordserver can persist the alerts communicated to the UI to file 
if RECORDSERVER_ALERTS_LOG is set accordingly. This allows examining 
them even after a shutdown (using, e.g., the commands plugin). You may 
also want to have your transcoding commands log problems to that file 
too then.

- I have introduced some internationalized texts and obsoleted or 
corrected some others. While I strived to not break existing 
translations, there will be a need to adapt the other freevo.po files (I 
did german only). To do this, it would be probably a good idea to 
examine the modifications of i18n/de/LC_MESSAGES/freevo.po.

- I removed the EPG fetch from the cache helper because it does not 
belong there (it has its own helper). If you depend on this, please add 
another call to your shell scripts. For a possible usage see the above 
.rc script.

- The oneclick plugin contains some translations that have nothing to do 
with weather. I don't understand why, and if there should be even more, 
or less, of them. If I broke something, someone understanding this 
plugin might have a look at this.

I would be glad if my patches were integrated into the freevo1 and kaa 
projects. Thank you again for all your great work, and I hope you will 
enjoy mine too.

Hartmut