Here's a patch file that adds the ability to create a .TiVo header file, given chunk0 chunk1 and chunk2. This is a quick and dirty c implementation of Jeremy's perl script that performs the same task.
First off, after reading your patch, I remembered why I never got around to doing this myself: I never really figured out a good interface for this, either API or command line. So it is fairly likely that I will hack this around quite a bit before I settle on something I like. Hope you don't mind :)
I noticed you changed the minor format version from 5 to 0xF, which corresponds to the Fall 2007 TiVo update. I'm not sure what impact this would have on a TiVo that was still running older software, but it may be a good idea to leave the version at 5 since TiVo is pretty likely to preserve backwards compatibility for the foreseeable future (making old files incompatible would anger customers who have collected recordings).
Why did you change the algorithm for calculating the chunk padding? Were you observing different values in the new file format?
I see you used a packed struct for the header. You may have wondered why the version in tivo-parse.h is not packed. This is because packed structs are not portable. I tried this originally, until I discovered that it caused problems on architectures with
strict alignment rules. On x86, you merely incur a performance penalty for unaligned access, but on SPARC it generates a hardware trap, which could be handled to emulate the access in software (at a HUGE performance penalty), or could be passed on to kill the program (which is the behavior I observed on Solaris 9 on SPARC). Take a look at tivo-parse.c to see how I read the header in, and just do it the other way round.
You should avoid seeking as much as possible. It precludes the use of pipes in the future, and messes with caching. You should probably stat() (or fstat()) the files upfront to get their sizes so you can avoid seeking to the end and doing an ftell(). The functions are called _stat and _fstat on windows. These are not necessarily large file safe, but the chunk files should never be anywhere near 2GB anyway ;)
You don't necessarily need to do anything about any of this, since I need to come up with a nice API before I do anything with this anyway I can make these changes then. This is just so you know why I did some of the odd things I did elsewhere in the code...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for taking the time to look at this patch and thanks for the explanations, especially the packed struct info. I found your code well structured and easy to follow and I didn't want to muck around outside of tdmake. My patch on the otherhand, well like I said; quick and dirty. Feel free to use anything from the patch that you like. Personally I find design much harder than implementation, so once you've settled on the calling convention, this patch will be of little use.
I don't see a problem setting the minor version back to 5 just to play it safe. I haven't compared the file structures to see if there is additional benefits with the new version, but I would think it might allow for additional attributes in chunk1 and chunk2. The only reason I can think of not to go to version 0xf would be if someone was running an old version of TiVoDesktop or other app, that expected an older version of TiVo files and wouldn't serve them otherwise. I would think that the TiVo units themselves would handle either format since they *should* be at the latest SW release, unless they're hacked of course:)
My original chunk struct did not include the data. I was writing out the chunk, data, and filler separately. Then I found your perl script and it was much more strait forward. I just never got around to changing my code to match your script for padding the chunk, although I did use it for the mpeg offset.
I used the packed struct as a convienence. Thanks for the information, I hadn't really considered anything except x86 and MIPS. I will definately have a look at how you handled this issue.
As for the pipes, I didn't really see a need for stdin since there were 3 chunks to handle and stdin is not working on Windows. Stdout might be useful, if a caller wanted to write out the header and then append an mpeg file to the end. I guess at this point, tdmake isn't really that useful as a standalone application. A few more options might make it more appealing:
tdmake -m MAK [-c] [-f mpegfile] <chunk0> <chunk1> <chunk2> <outputfile or stdout>
where -f would append an mpeg file to the header to make a complete .TiVo file and -c would optionally cleanup the input files when completed.
On the API side, my first thought is one call to build the complete header and one call to encrypt the chunks. It doesn't look like the chunks are very large, so I could pass all three at once. I could then read in all three chunks from the files and not worry about the file length before hand. Getting the complete header at once would also allow for stdout.
Ok, I think I'll have another go at this. I'll avoid packed structs and cleanup up the padding routine. I'll add some additioanl options to tdmake, and see if I can come up with any ideas on the api side.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Patch File
Logged In: YES
user_id=749423
Originator: NO
I have a couple questions/comments for you.
First off, after reading your patch, I remembered why I never got around to doing this myself: I never really figured out a good interface for this, either API or command line. So it is fairly likely that I will hack this around quite a bit before I settle on something I like. Hope you don't mind :)
I noticed you changed the minor format version from 5 to 0xF, which corresponds to the Fall 2007 TiVo update. I'm not sure what impact this would have on a TiVo that was still running older software, but it may be a good idea to leave the version at 5 since TiVo is pretty likely to preserve backwards compatibility for the foreseeable future (making old files incompatible would anger customers who have collected recordings).
Why did you change the algorithm for calculating the chunk padding? Were you observing different values in the new file format?
I see you used a packed struct for the header. You may have wondered why the version in tivo-parse.h is not packed. This is because packed structs are not portable. I tried this originally, until I discovered that it caused problems on architectures with
strict alignment rules. On x86, you merely incur a performance penalty for unaligned access, but on SPARC it generates a hardware trap, which could be handled to emulate the access in software (at a HUGE performance penalty), or could be passed on to kill the program (which is the behavior I observed on Solaris 9 on SPARC). Take a look at tivo-parse.c to see how I read the header in, and just do it the other way round.
You should avoid seeking as much as possible. It precludes the use of pipes in the future, and messes with caching. You should probably stat() (or fstat()) the files upfront to get their sizes so you can avoid seeking to the end and doing an ftell(). The functions are called _stat and _fstat on windows. These are not necessarily large file safe, but the chunk files should never be anywhere near 2GB anyway ;)
You don't necessarily need to do anything about any of this, since I need to come up with a nice API before I do anything with this anyway I can make these changes then. This is just so you know why I did some of the odd things I did elsewhere in the code...
Logged In: NO
Thanks for taking the time to look at this patch and thanks for the explanations, especially the packed struct info. I found your code well structured and easy to follow and I didn't want to muck around outside of tdmake. My patch on the otherhand, well like I said; quick and dirty. Feel free to use anything from the patch that you like. Personally I find design much harder than implementation, so once you've settled on the calling convention, this patch will be of little use.
I don't see a problem setting the minor version back to 5 just to play it safe. I haven't compared the file structures to see if there is additional benefits with the new version, but I would think it might allow for additional attributes in chunk1 and chunk2. The only reason I can think of not to go to version 0xf would be if someone was running an old version of TiVoDesktop or other app, that expected an older version of TiVo files and wouldn't serve them otherwise. I would think that the TiVo units themselves would handle either format since they *should* be at the latest SW release, unless they're hacked of course:)
My original chunk struct did not include the data. I was writing out the chunk, data, and filler separately. Then I found your perl script and it was much more strait forward. I just never got around to changing my code to match your script for padding the chunk, although I did use it for the mpeg offset.
I used the packed struct as a convienence. Thanks for the information, I hadn't really considered anything except x86 and MIPS. I will definately have a look at how you handled this issue.
As for the pipes, I didn't really see a need for stdin since there were 3 chunks to handle and stdin is not working on Windows. Stdout might be useful, if a caller wanted to write out the header and then append an mpeg file to the end. I guess at this point, tdmake isn't really that useful as a standalone application. A few more options might make it more appealing:
tdmake -m MAK [-c] [-f mpegfile] <chunk0> <chunk1> <chunk2> <outputfile or stdout>
where -f would append an mpeg file to the header to make a complete .TiVo file and -c would optionally cleanup the input files when completed.
On the API side, my first thought is one call to build the complete header and one call to encrypt the chunks. It doesn't look like the chunks are very large, so I could pass all three at once. I could then read in all three chunks from the files and not worry about the file length before hand. Getting the complete header at once would also allow for stdout.
Ok, I think I'll have another go at this. I'll avoid packed structs and cleanup up the padding routine. I'll add some additioanl options to tdmake, and see if I can come up with any ideas on the api side.
Logged In: YES
user_id=749423
Originator: NO
This patch is being superceded by #1851705 (tdmake2).