Thread: [Ipodlinux-devel] iTunesDB import

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

is there a way to import the iTunes database into the file browser instead of all the crazy f0* directories?

Currently, I don't think there is.

I am working on porting a library for reading the itunesDB to the ipod,=20
but it's going slowly... Don't know if anyone else is doing the same?

Terje

On Tue, 06 Apr 2004 11:03:17 +0000, <ic...@co...> wrote:

> is there a way to import the iTunes database into the file browser=20
> instead of all the crazy f0* directories?
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id=3D1470&alloc_id=3D3638&op=3Dc=
lick
> _______________________________________________
> iPodlinux-devel mailing list
> iPo...@li...
> https://lists.sourceforge.net/lists/listinfo/ipodlinux-devel

--=20
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

On Tue, Apr 06, 2004 at 02:11:16PM +0200, Terje Wiesener wrote:
> Currently, I don't think there is.
> 
> I am working on porting a library for reading the itunesDB to the ipod, 
> but it's going slowly... Don't know if anyone else is doing the same?

I am planing on rewriting the gtkpod itunesDB code so that it does not
depend on glib as soon as I find the time. Which might be in around two
weeks. 

But you are od course welcom to beat me to it. 
The gtkpod code should be a very good basis. It is fairly well tested
and supports most features (rating, playcount, volume adj. jsut to name
a few) and under active development.

Jens

> 
> Terje
> 
> On Tue, 06 Apr 2004 11:03:17 +0000, <ic...@co...> wrote:
> 
> >is there a way to import the iTunes database into the file browser 
> >instead of all the crazy f0* directories?

-- 
Jens Taprogge

> I am planing on rewriting the gtkpod itunesDB code so that it does not
> depend on glib as soon as I find the time. Which might be in around two
> weeks.

That is the code I'm working with also. I'm trying to make a wrapper for=20
all the glib types and functions needed, so the code itself doesn't have=20
to be rewritten so much. However, I'm not sure what to do with those pesk=
y=20
UTF strings...

Work has been very slow though, especially after some unfortunate disk=20
crashes courtesy of Partition Magic... Things are finally up and running=20
again now, so I'll see if I get time to look at it one of these days.

cheers
Terje

Hi all

I've managed to port some of the itunesdb functions from the gtkpod=20
project.

It runs under ipodlinux, and parsing my database (around 1600 songs) take=
s=20
~30 sec on the iPod.

So far, it doesn't do anything fancy, other than print the info about eac=
h=20
track, but I suspect it should be quite easy to integrate with for=20
instance podzilla.

The source can be downloaded from=20
http://orz.dyndns.org/stash/parsedb.tar.gz (When looking at the code,=20
please remember I haven't written a line C in about 3 years...)

Happy easter =3D)

Terje

On Apr 6, 2004, at 3:04 PM, Jens Taprogge wrote:

> On Tue, Apr 06, 2004 at 02:11:16PM +0200, Terje Wiesener wrote:
>> Currently, I don't think there is.
>>
>> I am working on porting a library for reading the itunesDB to the 
>> ipod,
>> but it's going slowly... Don't know if anyone else is doing the same?
>
> I am planing on rewriting the gtkpod itunesDB code so that it does not
> depend on glib as soon as I find the time. Which might be in around two
> weeks.
>
> But you are od course welcom to beat me to it.
> The gtkpod code should be a very good basis. It is fairly well tested
> and supports most features (rating, playcount, volume adj. jsut to name
> a few) and under active development.
>

Hi Jens,

last time I looked, the gtkpod iTunesDB code was really slow... on my 
dual 2.4 GHz Xeon that is.  It took about five minutes to parse my 4500 
song db (perhaps the GUI code was slowing things down a bit too).  I 
shiver at the thought of running it on my iPod.  If we ever wanted to 
add playcount updating to podzilla, it would not be too wise either, as 
gtkpod actually re-writes the entire database upon export.  I think the 
way to go is actually a proper db driver implementation - updating 
records in place (or, at least, with a cache mechanism for updates, so 
as to minimise disk access).

I have started work on a playqueue in podzilla - for now I'm going to 
use my own simple d.b. files for browsing songs (to be generated with 
libid3tag on the host).  I am working on the principle of caching up to 
16 MB of the next songs in the playqueue, and actually playing them 
with libmad (rather than forking a sub-process).  (Intel's code is my 
second candidate, but its not GPLed).

When I've gotten that working, I would port the code to the binary 
iTunesDB format, and add play-count updating.
For the time-being, I think Bernards idea of a link farm is the 
quickest way we'll get user-friendly song browsing in podzilla :-).

Cheers,
	Jonathan.

Hi Jonathan,

On Wed, Apr 07, 2004 at 09:22:13AM +0200, Jonathan C. Ross wrote:
> Hi Jens,
> 
> last time I looked, the gtkpod iTunesDB code was really slow... on my 
> dual 2.4 GHz Xeon that is.  It took about five minutes to parse my 4500 
> song db (perhaps the GUI code was slowing things down a bit too).  I 
> shiver at the thought of running it on my iPod.  If we ever wanted to 
> add playcount updating to podzilla, it would not be too wise either, as 
> gtkpod actually re-writes the entire database upon export.  I think the 
> way to go is actually a proper db driver implementation - updating 
> records in place (or, at least, with a cache mechanism for updates, so 
> as to minimise disk access).
The slowless is almost certainly caused by the gui. Depending on your
settings it is adding the entries in the gui as soon as it is reading
them (IIRC at least). 

Still I fully agree on the in place editing.

> I have started work on a playqueue in podzilla - for now I'm going to 
> use my own simple d.b. files for browsing songs (to be generated with 
> libid3tag on the host).  I am working on the principle of caching up to 
> 16 MB of the next songs in the playqueue, and actually playing them 
> with libmad (rather than forking a sub-process).  (Intel's code is my 
> second candidate, but its not GPLed).
> 
> When I've gotten that working, I would port the code to the binary 
> iTunesDB format, and add play-count updating.
> For the time-being, I think Bernards idea of a link farm is the 
> quickest way we'll get user-friendly song browsing in podzilla :-).

I will try to give the iTunesDB a shot tomorrow night. So if it is not
to complicated perhaps we have s/th by the day after tomorrow.

Cheers
Jens

-- 
Jens Taprogge

Hi all,

Sorry I haven't been following this thread too closely but hopefully you
guys make some progress on this ;)

I dug up some old code I was working on for reading the itunes library. 
The simple example just reads the iTunes db (in current directory) and
prints out some stuff like file name and track name.  Maybe this would
be easy to hack together to create a simple sim-linker...

Anyhow the code is at
http://ipodlinux.sourceforge.net/download/beta/it-src.tgz.

Extract that, run ./configure && make and it should create a binary
'src/it'.  Copy/link your iTunesDB to the current directory and run that
and it should print out stuff.

It uses the iTunesDB2 library from qahog...

cheers,
bern.

On Wed, 2004-04-07 at 23:05, Jens Taprogge wrote:
> Hi Jonathan,
> 
> On Wed, Apr 07, 2004 at 09:22:13AM +0200, Jonathan C. Ross wrote:
> > Hi Jens,
> > 
> > last time I looked, the gtkpod iTunesDB code was really slow... on my 
> > dual 2.4 GHz Xeon that is.  It took about five minutes to parse my 4500 
> > song db (perhaps the GUI code was slowing things down a bit too).  I 
> > shiver at the thought of running it on my iPod.  If we ever wanted to 
> > add playcount updating to podzilla, it would not be too wise either, as 
> > gtkpod actually re-writes the entire database upon export.  I think the 
> > way to go is actually a proper db driver implementation - updating 
> > records in place (or, at least, with a cache mechanism for updates, so 
> > as to minimise disk access).
> The slowless is almost certainly caused by the gui. Depending on your
> settings it is adding the entries in the gui as soon as it is reading
> them (IIRC at least). 
> 
> Still I fully agree on the in place editing.
> 
> > I have started work on a playqueue in podzilla - for now I'm going to 
> > use my own simple d.b. files for browsing songs (to be generated with 
> > libid3tag on the host).  I am working on the principle of caching up to 
> > 16 MB of the next songs in the playqueue, and actually playing them 
> > with libmad (rather than forking a sub-process).  (Intel's code is my 
> > second candidate, but its not GPLed).
> > 
> > When I've gotten that working, I would port the code to the binary 
> > iTunesDB format, and add play-count updating.
> > For the time-being, I think Bernards idea of a link farm is the 
> > quickest way we'll get user-friendly song browsing in podzilla :-).
> 
> I will try to give the iTunesDB a shot tomorrow night. So if it is not
> to complicated perhaps we have s/th by the day after tomorrow.
> 
> Cheers
> Jens

On Wed, Apr 07, 2004 at 11:05:02PM +0200, Jens Taprogge wrote:
> I will try to give the iTunesDB a shot tomorrow night. So if it is not
> to complicated perhaps we have s/th by the day after tomorrow.

Well I guess better late than never...

I have finally managed to take a closer look at the problem. I ahve
decided to implement a library from scratch. 

So far it can read most of the information on tracks (as opposed to
playlists). Writing is not yet supported. But the library is designed in
a way that makes it possible to update single entries without rewriting
all the whole file. 

I think it might also be a bit faster than the gtkpod implementation.
But I have not measured that. However the test program takes about
127 milliseconds to read a 1779 songs iTunesDB on a 2.4 GHz Pentium 4M
(output piped to /dev/null).

So far the library is only 6480 bytes large (stripped).

So far the lib does not handle converison of the iTunesDB strings
(UTF-16). And I am not sure it should. Basically I am undecided between
not handling conversion (and leaving that part to the app) or converting
everything to UTF-8. The latter has some performance and/or memory usage
advantages. What does everyone else think?

Also please comment on the api. It is not documented to well (a bit of
documentation is in itunesdb.h) but I think it should become pretty
clear by looking at the test program.

You can find the tarball at 
http://www-users.rwth-aachen.de/jens.taprogge/files/libitunesdb_0.1.tar.gz

Best Regards
Jens

PS: Btw. it should be really easy to change the test program to output a
shellscript to create a link-farm.

-- 
Jens Taprogge

On Sun, Apr 11, 2004 at 01:04:33AM +0200, Jens Taprogge wrote:
> So far the lib does not handle converison of the iTunesDB strings
> (UTF-16). And I am not sure it should. Basically I am undecided between
> not handling conversion (and leaving that part to the app) or converting
> everything to UTF-8. The latter has some performance and/or memory usage
> advantages. What does everyone else think?

Verision 0.2 of the library converts all strings to UTF8. It seems to be
much more convinient that way, saves nearly 50% memory on the strings
and does not add too much overhead. (Since the filenames are within
ASCII range the pathnames can be used 1:1 now).

The tarball is at:
http://www-users.rwth-aachen.de/jens.taprogge/files/libitunesdb_0.2.tar.gz

Best Regards
Jens
-- 
Jens Taprogge

Hi

Seems we are working on the same thing, basically =3D)

I just finished my version 0.11 yesterday, downloadable from=20
http://orz.dyndns.org/stash/parsedb0.11.tar.gz ,
which parses the playlists as well as the tracks.

I would be very interested in the performance of your program, when run o=
n=20
the ipod. My port parses my 1600 track db with output to /dev/null in 85=20
milliseconds on my 1800MHz Athlon, but uses a whopping 30 seconds on the=20
ipod.

I have tried to build your program, but the ./configure script fails with=
:
	checking how to run the C++ preprocessor... /lib/cpp
	configure: error: C++ preprocessor "/lib/cpp" fails sanity check
I guess I'm missing some package.

Anyway, good work, I'll probably discontinue work on the port if your=20
library turns out to be more efficient =3D)

Terje

PS: Funny that we both ended up using the same UTF conversion functions =3D=
)

> Verision 0.2 of the library converts all strings to UTF8. It seems to b=
e
> much more convinient that way, saves nearly 50% memory on the strings
> and does not add too much overhead. (Since the filenames are within
> ASCII range the pathnames can be used 1:1 now).
>
> The tarball is at:
> http://www-users.rwth-aachen.de/jens.taprogge/files/libitunesdb_0.2.tar=
.gz

Hi Terje,

On Sun, Apr 11, 2004 at 03:43:31PM +0200, Terje Wiesener wrote:
> Hi
> 
> Seems we are working on the same thing, basically =)
> 
> I just finished my version 0.11 yesterday, downloadable from 
> http://orz.dyndns.org/stash/parsedb0.11.tar.gz ,
> which parses the playlists as well as the tracks.
> 
> I would be very interested in the performance of your program, when run on 
> the ipod. My port parses my 1600 track db with output to /dev/null in 85 
> milliseconds on my 1800MHz Athlon, but uses a whopping 30 seconds on the 
> ipod.
I have not yet put linux on the ipod. And I will likely not find the
time until the week after next week. I would certainly like to know
if/how well it runs on the ipod. There might be endianess problems in
two places. But that should be fixable easyly.

One way to optimize for ipod use it to only parse the information the
ipod uses or is going to use (most likely Artist, Album, Track Name,
Track Number, Volume Adjust, Playcount, Rating, Genre). That would save
us the UTF conversion etc. I have been thinking about adding that. A
rough estimate is that it would increase performance by about 

Bernard: I guess using mmap is not an option on the ipod since it does
not have a full scale MMU?

> I have tried to build your program, but the ./configure script fails with:
> 	checking how to run the C++ preprocessor... /lib/cpp
> 	configure: error: C++ preprocessor "/lib/cpp" fails sanity check
> I guess I'm missing some package.
This is the first time I have done an autotools package. So it is not
unlikely something is broken in that regard. What version of the
autotools programs are you using? (I am using autoconf 2.59, automake
1.7.9)
Here it says:
	checking how to run the C++ preprocessor... g++ -E
You might also want to try setting the C++ preprocessor manually by
specifying CXXCPP (see configure --help).

> Anyway, good work, I'll probably discontinue work on the port if your 
> library turns out to be more efficient =)
Thanks.
I think a native implementation is preferable. (That's why I ended up
coding it.) It should be quite fast. There also is still room for
improvement (by inlining certain functions that are only used once
etc.). But for now I want the implementation to be correct and clean in
the first place.

Adding playlist support should be easy.

Perhaps we can join our efforts.

If the project is appreciated by the other developers I will put it in
CVS on sf.org.

> Terje
> 
> PS: Funny that we both ended up using the same UTF conversion functions =)
Well, I have actually taken a look on your port. And than figures that
the implementation by unicode.org should be the most reliable. Also the
api did fit perfectly. 

Thank you very much for your feedback!

Jens

-- 
Jens Taprogge

> I have not yet put linux on the ipod. And I will likely not find the
> time until the week after next week. I would certainly like to know
> if/how well it runs on the ipod. There might be endianess problems in
> two places. But that should be fixable easyly.
[snip]
> This is the first time I have done an autotools package. So it is not
> unlikely something is broken in that regard. What version of the
> autotools programs are you using? (I am using autoconf 2.59, automake
> 1.7.9)
> Here it says:
> 	checking how to run the C++ preprocessor... g++ -E
> You might also want to try setting the C++ preprocessor manually by
> specifying CXXCPP (see configure --help).

I don't remember the version of autoconf, but I am using automake 1.7.=20
Using the newest Debian unstable packages for both. I'll check it next=20
time I boot into linux.

If you have the build environment set up for compiling arm-binaries,=20
please feel free to send me a binary and I'll see how it performs on the=20
pod.

Terje

Version 0.3 adds functionality to _update_ certain entries. In other
words not the whole db is rewritten but only the parts that actually
changed. 

Currently functions to change the playcount, last_played and rating
entries are implemented.

Apart from that the callback API chaged slightly. The user can now pass
data to the callbacks.

The tarball can be found at:
http://www-users.rwth-aachen.de/jens.taprogge/files/libitunesdb_0.3.tar.gz

Best Regards
Jens

-- 
Jens Taprogge

On Tue, 2004-04-06 at 21:03, ic...@co... wrote:
> is there a way to import the iTunes database into the file browser instead of all the crazy f0* directories?
> 

Not yet afaik. However, the iTunes DB file format has been reverse
engineered, so it's possible to implement it. Your contribution
is welcome :)

Ben.

There is VB source code for doing this that I wrote for a windows program
called SharePod.

The website is at http://www25.brinkster.com/carmagt/sharepod/, you can
download the source from there.

It reads all song information, playlists and their contents and the
On-The-Go playlist ;)

Shouldnt be much of a mission to convert the useful bits to C...

Jeff

----- Original Message ----- 
From: "Terje Wiesener" <wie...@sa...>
To: <ic...@co...>; <ipo...@li...>
Sent: Wednesday, April 07, 2004 12:11 AM
Subject: Re: [Ipodlinux-devel] iTunesDB import

Currently, I don't think there is.

I am working on porting a library for reading the itunesDB to the ipod,
but it's going slowly... Don't know if anyone else is doing the same?

Terje

On Tue, 06 Apr 2004 11:03:17 +0000, <ic...@co...> wrote:

> is there a way to import the iTunes database into the file browser
> instead of all the crazy f0* directories?
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> _______________________________________________
> iPodlinux-devel mailing list
> iPo...@li...
> https://lists.sourceforge.net/lists/listinfo/ipodlinux-devel

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=ick
_______________________________________________
iPodlinux-devel mailing list
iPo...@li...
https://lists.sourceforge.net/lists/listinfo/ipodlinux-devel

There are also some good(?) libraries already available in C but they are glib
based so need quite a few support libraries that we don't currently use on the
iPod.  A good simple C library would be great!

A simple short term fix would be to use one of these libraries on the host to
create symbolic links between the /iPod_Control/... files and
/music/Artist/Album/Song or something similar.

There is also a perl package that can be used to read the .db files which
might be well-suited to the symbolic link creation task.

Jeffrey Harris <jef...@cl...> said:

> There is VB source code for doing this that I wrote for a windows program
> called SharePod.
> 
> The website is at http://www25.brinkster.com/carmagt/sharepod/, you can
> download the source from there.
> 
> It reads all song information, playlists and their contents and the
> On-The-Go playlist ;)
> 
> Shouldnt be much of a mission to convert the useful bits to C...
> 
> Jeff
> 
> ----- Original Message ----- 
> From: "Terje Wiesener" <wie...@sa...>
> To: <ic...@co...>; <ipo...@li...>
> Sent: Wednesday, April 07, 2004 12:11 AM
> Subject: Re: [Ipodlinux-devel] iTunesDB import
> 
> 
> Currently, I don't think there is.
> 
> I am working on porting a library for reading the itunesDB to the ipod,
> but it's going slowly... Don't know if anyone else is doing the same?
> 
> Terje
> 
> On Tue, 06 Apr 2004 11:03:17 +0000, <ic...@co...> wrote:
> 
> > is there a way to import the iTunes database into the file browser
> > instead of all the crazy f0* directories?
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: IBM Linux Tutorials
> > Free Linux tutorial presented by Daniel Robbins, President and CEO of
> > GenToo technologies. Learn everything from fundamentals to system
> > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> > _______________________________________________
> > iPodlinux-devel mailing list
> > iPo...@li...
> > https://lists.sourceforge.net/lists/listinfo/ipodlinux-devel
> 
> 
> 
> -- 
> Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=ick
> _______________________________________________
> iPodlinux-devel mailing list
> iPo...@li...
> https://lists.sourceforge.net/lists/listinfo/ipodlinux-devel
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> _______________________________________________
> iPodlinux-devel mailing list
> iPo...@li...
> https://lists.sourceforge.net/lists/listinfo/ipodlinux-devel
> 

-- 