"failed to generate collection"

[Anonymous]

Hi all,
ive been playing with rompr for few days,
when my mopidy db was small everything worked ok but when i scanned all my files rompr started to give me "failed to generate collection"
rompr error log doesnt show any error,
apache error log show:

[:error] [pid 13085] [client 127.0.0.1:54654] PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/rompr/player/mpd/sockets.php on line 47, referer: http://localhost/rompr/

i tried any solution i could find,
i verified that the rompr conf is working from here: https://sourceforge.net/p/rompr/discussion/general/thread/a4ab9cea/?limit=25&page=1#ec61

(btw i couldnt run "sudo a2enconf rompr" until i changed the filename to "rompr.conf")

i changed these value:
max_execution_time = 300

i added those to rompr:
php_admin_value post_max_size 32M
php_admin_value memory_limit 256M
php_admin_value upload_max_filesize 32M

but still getting those errors....

i cannot find anymore solutions online....

please help

[Fat German Productions]

php_admin_value memory_limit 256M

You've gone over the memory limit Allowed memory size of 268435456 bytes exhausted

You'll need to increase the memory_limit value.

I'm surprised you need that much memory though, how many tracks are in your collection?

[Anonymous]

350K on sqlite
ill try with 512M
thanks

[Fat German Productions]

350,000 tracks? Holy moly, I think you're going to struggle.

Rompr uses significantly more memory when files are not fully tagged. As a minimum you should have Title, Artist, Album, Album Artist, Track Number, and Disc Number tages set correctly. for as many of your tracks as you can manage. Obviously if your tracks are coming from Spotify then there's nothing you can do about that and you may find you need vast amounts of memory for that many tracks.

[Anonymous]

most of my files are fully tagged...
i'll try reducing directories.

thanks :(

[Anonymous]

512M didnt help also:

[Mon Mar 19 17:44:41.288999 2018] [:error] [pid 19120] [client 127.0.0.1:58714] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/rompr/player/mpd/sockets.php on line 47, referer: http://localhost/rompr/

[Fat German Productions]

I'll look into it further, if everything is tagged then you may have found a a memory leak that only shows up when you have huge numbers of tracks.

[Anonymous]

i removed one big music directory, now mopidy report 158M tracks,
the memory limit is 512M and i got the error again:

[Mon Mar 19 17:44:41.288999 2018] [:error] [pid 19120] [client 127.0.0.1:58714] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/rompr/player/mpd/sockets.php on line 47, referer: http://localhost/rompr/

one other question, maybe or not related:
in rompr error log, i see jumps in timestamp, like time:

20:02:46 :    19119 : INIT                Using MPD Host Default
18:02:46 :    19119 : MPD                 MPD Command status
20:02:56 :    19117 : COOKIE              Pref currenthost overridden by Cookie  - Value : Default
20:02:56 :    19117 : COOKIE              Pref player_backend overridden by Cookie  - Value : mopidy
20:02:56 :    19117 : INIT                Using MPD Host Default
18:02:56 :    19117 : MPD                 MPD Command status
20:03:06 :    19121 : COOKIE              Pref currenthost overridden by Cookie  - Value : Default
20:03:06 :    19121 : COOKIE              Pref player_backend overridden by Cookie  - Value : mopidy
20:03:06 :    19121 : INIT                Using MPD Host Default
18:03:06 :    19121 : MPD                 MPD Command status
20:03:16 :    19120 : COOKIE              Pref currenthost overridden by Cookie  - Value : Default
20:03:16 :    19120 : COOKIE              Pref player_backend overridden by Cookie  - Value : mopidy
20:03:16 :    19120 : INIT                Using MPD Host Default
18:03:16 :    19120 : MPD                 MPD Command status

why is that? (my timezone is +2)

[Fat German Productions]

The time thing is probably because your php.ini doesn't have a valid setting for

date.timezone

and rompr is defaulting to UTC. This happens after the INIT and COOKIE messages. It's not related to the memory problem.

The largest collection I'm ever aware of anybody using was 30K tracks. I would be interested in seeing a full debug log of what happens when you try to update your collection, it might give me something to go on.

[Anonymous]

now mpd is with 133400 tracks,
rompr max memory is at 1024M
i rescan the DB and after a long time rompr gives a "timeout" error, nothing in apache log.

here is the full rompr log: (30M so i zipped it) https://expirebox.com/download/312c89c17a649b7440e7513edd22d6ea.html

[Fat German Productions]

Hmm. Thanks. What I can see from that is that after a certain point you start to get a load of SQL errors, like the database engine is not working. I wonder if this might be an sqlite issue. Can you try with MySQL?

[Anonymous]

im already on mysql with rompr!
![](https://imgur.com/wgdch1v

[Fat German Productions]

Well I've had a look and the only place rompr uses significant memory is in sorting albums with incomplete tags. You should check whether all your tracks have AlbumArtist and Disc set correctly. I can tell from the log that a lot of them do but it doesn't show me the ones that don't (if there are any).
You also seem to have a lot of duplicate albums. I don't know if this is part of the problem but rompr won't display duplicates.

[Fat German Productions]

I think I've found the problem. There looks to be a bug in mopidy-local-sqlite.

If you have multiple copies of the same track by the same artist on multiple different albums (which you do, I can tell from the debug output), local-sqlite sometimes puts them all on the same album. There are other issues too with it splitting single albums up into multiples. I think this might explain why rompr is getting confused when parsing the output.

I fixed this on my setup by changing the configuration in mopidy.conf

[local-sqlite]
enabled = true
use_album_mbid_uri = false
use_artist_sortname = false

Then I deleted the mopidy database at ~/.local/share/mopidy/local-sqlite/library.db, and re-ran mopidy local scan. It's probably wise in your case to delete the rompr database too before creating the collection.

I have also found a way to reduce rompr's memory usage still further but I don't think it will affect you. However it'll be in the next release.

[Fat German Productions]

Timeout issues are a web server problem. I've added a section on dealing with this on the installation page of the wiki.