#101 FIXED IN SVN: Hang on big names arrays

[Milan Bouchet-Valat]

Today I added a numeric variable to a big data frame, and rkward.frontend started eating GBs of memory and using all CPU, never finishing. I'm able to reproduce the hang with the attached .RData file, which contains that variable as an array. Loading this file always makes my RKWard hang. This is quite weird, since either converting the array to a vector, or setting dimnames to NULL is enough to fix the problem. But changing the names to 1:length(a) for example has no effect, nor changing the values of a.

You can download the .RData file from: http://ubuntuone.com/24J68sdk9kd6PUCtO6Hd1f

No need to say I'm very interested in debugging this, since I need to work with this data. ;-)

I'm using RKWard 0.5.6z+0.5.7+devel1 on Fedora 15 64-bits.

> str(a)
num [1:1985324(1d)] 0.2619 0.7744 -0.0511 -0.0113 0.256 ...
- attr(*, "dimnames")=List of 1
..$ : chr [1:1985324] "1969" "1969" "1969" "1969" ...

Backtrace of rkward while hanging on load:
(gdb) ba
#0 0x000000000048e990 in QList<RObject*>::indexOf (this=<optimized out>, t=
@0x7fff99b712a8, from=<optimized out>) at /usr/include/QtCore/qlist.h:833
#1 0x00000000004e84a2 in RContainerObject::updateChildren (this=0x2b22d90,
new_children=<optimized out>)
at /home/milan/Dev/rkward/rkward/rkward/core/rcontainerobject.cpp:180
#2 0x00000000004e8911 in RContainerObject::updateStructure (this=0x2b22d90,
new_data=<optimized out>)
at /home/milan/Dev/rkward/rkward/rkward/core/rcontainerobject.cpp:93
#3 0x00000000004e7e11 in RContainerObject::updateChildStructure (this=
0x2565260, child=0x2b22d90, new_data=0x2c83710, just_created=true)
at /home/milan/Dev/rkward/rkward/rkward/core/rcontainerobject.cpp:55
#4 0x00000000004e7b8f in RContainerObject::createChildFromStructure (this=
0x2565260, child_data=0x2c83710, child_name=..., position=0)
at /home/milan/Dev/rkward/rkward/rkward/core/rcontainerobject.cpp:128
#5 0x00000000004e8014 in RContainerObject::updateChildStructure (this=
0x2565260, child=0x2c85970, new_data=0x2c83710,
just_created=<optimized out>)
at /home/milan/Dev/rkward/rkward/rkward/core/rcontainerobject.cpp:72
#6 0x00000000004e0477 in rCommandDone (command=0x2c83710, this=0x2c85970)
at /home/milan/Dev/rkward/rkward/rkward/core/robject.cpp:263
#7 RObject::rCommandDone (this=0x2c85970, command=0x2c83710)
at /home/milan/Dev/rkward/rkward/rkward/core/robject.cpp:253
#8 0x00000000004db7fb in RKVariable::rCommandDone (this=0x2c85970, command=
---Type <return> to continue, or q <return> to quit---
0x2c83710) at /home/milan/Dev/rkward/rkward/rkward/core/rkvariable.cpp:120
#9 0x00000000004fdd81 in RCommand::finished (this=0x2c83710)
at /home/milan/Dev/rkward/rkward/rkward/rbackend/rcommand.cpp:113
#10 0x00000000004f57a0 in RInterface::handleCommandOut (this=<optimized out>,
command=0x2c83710)
at /home/milan/Dev/rkward/rkward/rkward/rbackend/rinterface.cpp:209
#11 0x00000000004fc80d in RInterface::handleRequest (this=0x24c6ea0, request=
0x7fad6c07f2e0)
at /home/milan/Dev/rkward/rkward/rkward/rbackend/rinterface.cpp:324
#12 0x00000035e4f70c0c in QObject::event (this=0x24c5cf0, e=<optimized out>)
at kernel/qobject.cpp:1248
#13 0x00000035f1bb9324 in notify_helper (e=0x7fad6c10a500, receiver=0x24c5cf0,
this=0x21a7df0) at kernel/qapplication.cpp:4481
#14 QApplicationPrivate::notify_helper (this=0x21a7df0, receiver=0x24c5cf0, e=
0x7fad6c10a500) at kernel/qapplication.cpp:4453
#15 0x00000035f1bbe1b1 in QApplication::notify (this=0x7fff99b721c0, receiver=
0x24c5cf0, e=0x7fad6c10a500) at kernel/qapplication.cpp:4360
#16 0x00000035eb841d56 in KApplication::notify(QObject*, QEvent*) ()
from /usr/lib64/libkdeui.so.5
#17 0x00000035e4f5a20c in QCoreApplication::notifyInternal (this=
0x7fff99b721c0, receiver=0x24c5cf0, event=0x7fad6c10a500)
at kernel/qcoreapplication.cpp:787
#18 0x00000035e4f5d7d4 in sendEvent (event=0x7fad6c10a500, receiver=0x24c5cf0)
---Type <return> to continue, or q <return> to quit---
at kernel/qcoreapplication.h:215
#19 QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=0,
data=0x2180070) at kernel/qcoreapplication.cpp:1428
#20 0x00000035e4f84973 in sendPostedEvents () at kernel/qcoreapplication.h:220
#21 postEventSourceDispatch (s=0x21ab8c0)
at kernel/qeventdispatcher_glib.cpp:277
#22 0x00000035df6427ed in g_main_dispatch (context=0x21ab4e0) at gmain.c:2441
#23 g_main_context_dispatch (context=0x21ab4e0) at gmain.c:3014
#24 0x00000035df642fc8 in g_main_context_iterate (context=0x21ab4e0,
block=<optimized out>, dispatch=1, self=<optimized out>) at gmain.c:3092
#25 0x00000035df64325c in g_main_context_iteration (context=0x21ab4e0,
may_block=1) at gmain.c:3155
#26 0x00000035e4f84dcf in QEventDispatcherGlib::processEvents (this=0x21a7490,
flags=<optimized out>) at kernel/qeventdispatcher_glib.cpp:422
#27 0x00000035f1c5c12e in QGuiEventDispatcherGlib::processEvents (
this=<optimized out>, flags=<optimized out>)
at kernel/qguieventdispatcher_glib.cpp:207
#28 0x00000035e4f59722 in QEventLoop::processEvents (this=<optimized out>,
flags=...) at kernel/qeventloop.cpp:149
#29 0x00000035e4f5991f in QEventLoop::exec (this=0x7fff99b72110, flags=...)
at kernel/qeventloop.cpp:201
#30 0x00000035e4f5da67 in QCoreApplication::exec ()
at kernel/qcoreapplication.cpp:1064
---Type <return> to continue, or q <return> to quit---
#31 0x0000000000432634 in main (argc=<optimized out>, argv=<optimized out>)
at /home/milan/Dev/rkward/rkward/rkward/main.cpp:177
(gdb) list
828 Q_OUTOFLINE_TEMPLATE int QList<T>::indexOf(const T &t, int from) const
829 {
830 if (from < 0)
831 from = qMax(from + p.size(), 0);
832 if (from < p.size()) {
833 Node *n = reinterpret_cast<Node *>(p.at(from -1));
834 Node *e = reinterpret_cast<Node *>(p.end());
835 while (++n != e)
836 if (n->t() == t)
837 return int(n - reinterpret_cast<Node *>(p.begin()));

[Thomas Friedrichsmeier]

Hi!

The core problem with that object is that it looked like an object containing 1985324 sub-objects to RKWard. I have addressed this in the development version (http://p.sf.net/rkward/svn) by making sure that RKWard will never try to look at more than 100000 children of any object, and specifically will not treat arrays as hierarchical named objects (which it never did quite right, anyway). That should solve the hang.

That said, that object of yours really is a bit unusual. To me it looks more like you wanted to have two separate variables, as in:
x <- data.frame (year=names (a), as.vector (a))
or, alternatively:
x <- sapply (unique (names (a)), function (x) as.vector (a[x == names (a)]))

While names() can contain duplicates for most R objects, this is rarely ever useful, and typically it indicates a problem. E.g. what's the use of
a["2009"]
?

Regards
Thomas

[Milan Bouchet-Valat]

Ah, cool, I'll try with a more recent SVN then. I would have updated before reporting, but my setup is a little messed up ATM, so that's not as easy as it sounds. ;-)

That 'a' object doesn't make any sense, it's just that I extracted the problematic variable from a big data frame using a<-df$a. Since it was the smallest unit that triggered the bug, I kept it, but it isn't useful by itself. The names probably correspond to the first column of the data frame, with contains the survey year, but is of course used in relation with many more variables.