[Crm114-general] Markovian microgrooming hangs

[Kai R. <Kai...@ar...>]

Every once in a blue moon, I have problems with training hanging indefinitely during a microgroom. The process just sits there eating CPU, but neither strace nor ltrace shows anything. 

I am running version 20100106-BlamceMichelson on Linux and using the Markovian classifier with unique and microgroom flags.

After some debugging there seems to be some very rare occasions when the crm_zapcss finds itself in a situation where it is virtually unable to find any data to zero out, i.e. zcountdown is larger than zero and it just keeps incrementing vcut higher and higher without accomplishing anything. 

I don't know if it just is bad luck with regarding to the data or if the distance calculations overflow the long value being compared against. I do not fully understand the intricacies of the data structures and interpretations of their contents, but the following patch seems to solve the problems I am having:



--- crm114-20100106-BlameMichelson.src/crm_css_maintenance.c.orig       2013-08-26 14:03:58.000000000 +0300
+++ crm114-20100106-BlameMichelson.src/crm_css_maintenance.c    2013-08-26 14:03:39.000000000 +0300
@@ -260,7 +260,7 @@
 #define DWEIGHT 1.0
 #define DWEIGHT2 0.0

-  long vcut;
+  double vcut, dist, nextcut;
   long zcountdown;
   unsigned long packlen;
   unsigned long k;
@@ -271,9 +271,10 @@
   //  fprintf (stderr, " S: %ld, E: %ld, L: %ld ", start, end, packlen );
   zcountdown = packlen / 32.0 ;  //   get rid of about 3% of the data
   actually_zeroed = 0;
-  while (zcountdown > 0)
+  while (zcountdown > 0 && vcut > 0)
     {
-      //  fprintf (stderr, " %ld ", vcut);
+      //  fprintf (stderr, " %.1f ", vcut);
+      nextcut = 0;
       for (k = start; k <= end;  k++)
        {
          if (h[k].key != 0 )       // key == 0 means "special- don't zero!"
@@ -282,21 +283,23 @@
              if (h[k].value > 0)      //  can't zero it if it's already zeroed
              {
                //  fprintf (stderr, "b");
-               if ((VWEIGHT * h[k].value) +
-                   (VWEIGHT2 * h[k].value * h[k].value ) +
-                   (DWEIGHT * (k - h[k].hash % hs)) +
-                   (DWEIGHT2 * (k - h[k].hash % hs) * (k - h[k].hash % hs))
-                   <= vcut)
+               dist = (VWEIGHT * h[k].value) +
+                (VWEIGHT2 * h[k].value * h[k].value ) +
+                      (DWEIGHT * (k - h[k].hash % hs)) +
+                      (DWEIGHT2 * (k - h[k].hash % hs) * (k - h[k].hash % hs));
+               if (dist <= vcut)
                  {
                    //  fprintf (stderr, "*");
                    h[k].value = 0;
                    zcountdown--;
                    actually_zeroed++;
-                 };
+                 }
+               else if( dist < nextcut || nextcut==0 )
+                 nextcut = dist;
              };
            };
        };
-      vcut++;
+      vcut = nextcut;
     };
   return (actually_zeroed);
 }



--
Kai...@ar...     GSM  +358-40-767 8282
Oy Arrak Software Ab   http://www.arrak.fi