Menu
▾
▴
Re: [scipion-users] CTFFind4 failing on large datasets
|
From: Pablo C. <pc...@cn...> - 2022-08-29 10:17:17
|
Thank you Maria! The error you are reporting may be unrelated with "CTFFind4 in Scipion". “sqlite3.OperationalError: database is locked" Is related, probably to concurrency. There are several factors that may cause this. *threads*: If you specify a high number of threads, many of them will process at the same time CTFs. This is desirable and we have made the effort to enable as many threads as available, but if this happens reducing the threads may help. *streaming*: Streaming processing implied many active protocols that favors concurrency. *filesystem*: A slow filesystem (e.g.: network filesystem mounted over a a slow ethernet connection) may provide locks. In theory, CTFfind and many others, should cope nicely with large datasets, or at least is being doing this for a long time without any issue. This may be a specific issue. On 29/8/22 9:07, Anastasina, Maria wrote: > Dear Scipion users, > > I’m get an unexpected error when running CTFFind4 in Scipion 3.0.8 on a large (50K) set of micrographs. The protocol fails after processing about 70 % of micrographs with the error: “sqlite3.OperationalError: database is locked”. > > Sometimes re-launching in continuation mode helps to process few more micrographs, but the protocol will eventually fail again with the same error. Breaking the data set in smaller chunks (about 10K micrographs each) helps and the protocol ran in the same conditions succeeds for all the micrographs. > > I wonder does this indicate that CTFFind4 has problems handling large datasets and should be used on smaller datasets? Or is there possibly another problem that I am not noticing? I’m not attaching run.stdout as it is pretty large (31MB), if need I can upload it to eg grdive. > > Thank you, > Maria > > > > > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users -- Pablo Conesa - *Madrid Scipion <http://scipion.i2pc.es> team* |
