I'm trying to import 10M games from FICS but it's very slow and keeps crashing (even though my computer has 16GB of ram). Maybe input buffering and multithreading could help on newer systems?
I am not sure what limit you are hitting, if any. On my system, i have sg4 files > 2^31 = 2,147,483,648 bytes, so i am not sure your ticket is valid.
If you can pin point the cause of your problem , please be more specific.
There are limits on the number of games, Rounds, Names, etc.
But - if you are hitting some limit - raising the limits of si4 is not a priority for ScidvsPC.
It is just not productive on any level imho.
Cheers.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I created a new database and tried to to import this FICS file (unzipped of course). As soon as the corresponding sg4 file hits 2,147,483,501 bytes the importing window shows this error:
Importing file C:/Users/Michael/Desktop/tf/ficsall/ficsgamesdb_2006_chess_nomovetimes_1519430.pgn
Error writing game file.Error saving game in database.
I can then see a little over 10M games in the game list window but the database file itself is corrupted (can't reopen after close). I installed Scid vs PC-4.18.1.x64.exe on Windows 10. Scid 4.6.4 (non vs-pc) also shows errors and stops importing at the same point but doesn't corrupt the file.
Maybe a single write handle is only good for 2 gb?
Last edit: Michael Pang 2017-12-28
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
it seems that this problem still persists. One thing though, scid (not scid vs pc) can import all the database , but if its opened by scid vs pc, weirdly enough gives error "error reading game"
Ok, i have verified this is definitely a reproducable bug , and only on windows.
We can't load any games in an sg4 file past the 2gb limit. Of course this is kindof huge, and getting close to the 16 million game limit anyhow, but hopefully it'll get fixed before too long.... edit ... maybe it'll get fixed one day :(
Last edit: Steven 2022-01-20
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Michael also wrote
I am not sure what limit you are hitting, if any. On my system, i have sg4 files > 2^31 = 2,147,483,648 bytes, so i am not sure your ticket is valid.
If you can pin point the cause of your problem , please be more specific.
There are limits on the number of games, Rounds, Names, etc.
But - if you are hitting some limit - raising the limits of si4 is not a priority for ScidvsPC.
It is just not productive on any level imho.
Cheers.
I created a new database and tried to to import this FICS file (unzipped of course). As soon as the corresponding sg4 file hits 2,147,483,501 bytes the importing window shows this error:
I can then see a little over 10M games in the game list window but the database file itself is corrupted (can't reopen after close). I installed Scid vs PC-4.18.1.x64.exe on Windows 10. Scid 4.6.4 (non vs-pc) also shows errors and stops importing at the same point but doesn't corrupt the file.
Maybe a single write handle is only good for 2 gb?
Last edit: Michael Pang 2017-12-28
Thanks Michael. I will try to look at this when i get the chance, but may not be too soon.
It is vaguely possible you are hitting a file system limit i guess, but seems unlikely. Do you have any other files largere than 2gb ?
Yes, for example the pgns I downloaded. And thank you for developing this software :)
it seems that this problem still persists. One thing though, scid (not scid vs pc) can import all the database , but if its opened by scid vs pc, weirdly enough gives error "error reading game"
Last edit: Ajayasub 2021-07-08
Ok, i have verified this is definitely a reproducable bug , and only on windows.
We can't load any games in an sg4 file past the 2gb limit. Of course this is kindof huge, and getting close to the 16 million game limit anyhow, but hopefully it'll get fixed before too long.... edit ... maybe it'll get fixed one day :(
Last edit: Steven 2022-01-20
Ok, this is fixed in ScidvsPC 4.23 (Betas out now).