wekan / Tickets / #1096 Maximum call stack size exceeded

Anonymous - 2017-06-29

Originally posted by: tlmorgen

We're having this problem as well. At approximately 60+ presences and 6000+ cards we were getting this crash every few hours. Mongod is not shockingly busy during these periods, but node was pegging its core. We have installed an HTTP load balancer and are running 6 node instances now. It would be great if this was resolved as I expect one node instance is enough for such small figures.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-06-29

Originally posted by: GavinLilly

Those usage figures sound similar or higher to ours so it's good (?) to hear that we're not the only ones having this problem. Today we were experiencing this issue every few hours. We didn't experience it on our old LibreBoard instance (don't know the version but I know it was on CentOS v6).
Out of interest are you running all 6 node instances on different machines and have you experienced any concurrency issues? I didn't realise Wekan could even handle running in a load balanced environment

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-06-29

Originally posted by: tlmorgen

All the node instances and the single mongod instance are on the same host (8 core). I expect that mongo takes care of most of the concurrency problems. Also I'm just hoping that my users aren't creating in-flight conflicts :/

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-06-29

Originally posted by: xet7

Can you try to debug more with this?
https://stackoverflow.com/questions/6095530/maximum-call-stack-size-exceeded-error

Are you running latest Wekan v0.27 ? There was fix for duplicate id generation:
https://github.com/wekan/wekan/blob/devel/CHANGELOG.md

You could also try cleanup scripts:
https://github.com/wekan/wekan-cleanup

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-06-29

Originally posted by: xet7

Are all those many Wekan instances using same MongoDB database with same boards etc? I don't know how well MongoDB handles concurrency or will there be locking issues, Wekan instances overwriting each others data.

I could sometime try to get Wekan working with newest ToroDB, to see if ToroDB can replace MongoDB fully. Related project:
https://github.com/wekan/wekan-postgresql

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-06-30

Originally posted by: GavinLilly

We're on v0.26 currently so will try the 0.27 version and will try the cleanup script as well.

I'm not sure how I'm going to be able to debug it more though because this isn't something that is easily reproducbible. We have over 1000 unique users of the tool and this error seems to happen randomly within that user base.

Looking around at similar issues with other users of MeteorJS it seems to be an issue with recursion; perhaps part of the code which removes documents (I don't know if this is attachments or JS documents)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-07-03

Originally posted by: GavinLilly

I thought that I had created some stability by upgrading and cleaning the DB but then this morning I had this error.

Jul 03 09:27:05 Wekan[28778]: Internal exception while processing message { msg: 'sub',
Jul 03 09:27:05 Wekan[28778]: id: 'r6JkooY5kTBEKRWeB',
Jul 03 09:27:05 Wekan[28778]: name: 'meteor_autoupdate_clientVersions',
Jul 03 09:27:05 Wekan[28778]: params: [] } Maximum call stack size exceeded RangeError: Maximum call stack size exceeded

Everything seems to be linked to exceeding the call stack size. I can increase it but I feel like that's just a workaround to a more inherent problem. Are there some recommendations for higher call stack settings or some other fixes?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-07-04

Originally posted by: alexandrevl

We are here are facing the same problem in version 0.27. We have more de 1000 unique user as well and 1000+ boards in a docker. We cleaned Mongo and the issue persists.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-07-05

Originally posted by: GavinLilly

Is this an indicator that Wekan (and/or Meteor) does not scale to 1000+ userbases?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-07-05

Originally posted by: GavinLilly

Another failure today:

Jul 05 13:37:50 Wekan[27055]: Internal exception while processing message { msg: 'connect',
Jul 05 13:37:50 Wekan[27055]: session: 'a9giZbmfumPZoEcG7',
Jul 05 13:37:50 Wekan[27055]: version: '1',
Jul 05 13:37:50 Wekan[27055]: support: [ '1', 'pre2', 'pre1' ] } Maximum call stack size exceeded RangeError: Maximum call stack size exceeded
Jul 05 13:41:11 Wekan[27055]: packages/meteor.js:786
Jul 05 13:41:11 Wekan[27055]: }).run();

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-07-20

Originally posted by: GavinLilly

Is there something I can do to help debug this issue? Any particular way of launching Wekan to provide a more in depth log of errors?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-07-20

Originally posted by: xet7

@GavinLilly

Yes. Do Google searches with that error message to find if somebody has fix for this, and add links to this issue.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-07-24

Originally posted by: GavinLilly

After taking a deeper look at the system log I can see that the previously reported errors do not appear to be the cause of the crashing. In fact it seems pretty impossible to see what is causing it because of the lack of verbosity in the failure message. When I see that Systemd has observed the Wekan process failing I get one of the following errors:

Jul 21 11:30:07 Wekan[5229]: packages/meteor.js:786 Jul 21 11:30:07 Wekan[5229]: }).run(); Jul 21 11:30:07 Wekan[5229]: ^ Jul 21 11:30:07 Wekan[5229]: RangeError: Maximum call stack size exceeded Jul 20 17:03:57 Wekan[9145]: packages/meteor.js:786 Jul 20 17:03:57 Wekan[9145]: }).run(); Jul 19 17:12:05 Wekan[4494]: packages/meteor.js:786

This points to the "_scheduleRun" function in Meteor. There's very little online about this error except for the following GitHub issue which was not resolved
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-08-21

Originally posted by: GavinLilly

I wasn't able to fix the issues we were having but I have implemented a workaround through load balancing our Wekan instances. We now have 3x Wekan instances on separate virtual machines and an Nginx instance doing hash load balancing. We had no issues for ~7 days and then all 3 started to fail with high memory/CPU usage. For the time being I'm restarting each Wekan instance nightly to keep them going during operating hours

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-08-21

Ticket changed by: GavinLilly

status: open --> closed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-08-21

Ticket changed by: GavinLilly

status: closed --> open
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-08-21

Originally posted by: xet7

I could try updating meteor and every package like in that Github issue mentioned by @GavinLilly

@ocdtrekkie @kentonv

Do you know is there any issues with using newer version of Meteor with Sandstorm?
Does capnp only work with Meteor 1.4 ?
Currently Wekan uses Meteor 1.4.4.1 as can be seen in Dockerfile.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2017-08-21

Originally posted by: ocdtrekkie

@xet7 meteor-spk 0.3.2 (current) supports Meteor 1.4. Shouldn't be any concerns for Sandstorm.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Originally posted by: pierreozoux

I see the same error, but without the following about the Maximum call stack size exceeded:

app_1  | Exception in queued task: Error: Removed nonexistent document J7EHNvXkG86gaGjsD
app_1  |     at [object Object]._.extend.removed (packages/ddp-server/livedata_server.js:202:17)
app_1  |     at [object Object]._.extend.removed (packages/ddp-server/livedata_server.js:394:10)
app_1  |     at [object Object]._.extend.removed (packages/ddp-server/livedata_server.js:1281:19)
app_1  |     at _cursor.observeChanges.removed (packages/cottz:publish-relations/lib/server/cursor/cursor.js:52:13)
app_1  |     at packages/mongo/observe_multiplex.js:182:30
app_1  |     at Array.forEach (native)
app_1  |     at Function._.each._.forEach (packages/underscore.js:139:11)
app_1  |     at Object.task (packages/mongo/observe_multiplex.js:176:9)
app_1  |     at [object Object]._.extend._run (packages/meteor.js:807:18)
app_1  |     at packages/meteor.js:785:14

I had some instabilities lately, and it was maybe due to that, it is the only error I see in the logs.

(Tell me if you prefer me to open a new issue)

Anonymous - 2017-11-29

Ticket changed by: xet7

status: open --> closed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Maximum call stack size exceeded

Milestone

Searches

Help

#1096 Maximum call stack size exceeded

Discussion