Hi,

After analyzing several +30945 seconds leaps, using the coLinux debug version, I managed to solved the mystery.

The root cause of the problem is a timer read hardware glitch.
The glitch occurs when the 24th bit of the counter should overflow, but it doesn't.
For example:
Instead of reading 0x16242A000004, a value of 0x162429000004 was read.
Instead of reading 0x13510E000005, a value of 0x13510D000005 was read.

This causes an error of  -4.68 seconds (for a 3579545 frequency).
The code in callback_return_jiffies(), together with an inaccuracy in co_div64() causes some interesting results:

The first time the glitch occurs the time leaps forward by 24 seconds.
When subsequent glitches occur, time will leap forward by 30945 and 30956 seconds alternatively.

The results can be verified with the simulation below.

Problem resolution:
It seems this clock reading problem is well known and there are several workarounds for it.
For example see the the thread http://www.mail-archive.com/freebsd-current@freebsd.org/msg34826.html and http://support.microsoft.com/kb/274323 .
In any case, I think that a simple check to verify that clock did not shift backward from previous read is a must.
In addition, if the clock changed more than a threshold, then the clock should be read again in loop until below threshold.

Code to simulate the time leap:

void simulate()
{
    unsigned long long correct_cur_timestamp, prev_timestamp;
    unsigned long long timestamp, freq, timestamp_diff, remainder;
    unsigned long jiffies;
    int loops = 0;

    freq = 3579545;

    /*
     * dummy initial value
     */
    correct_cur_timestamp = 0x12345678;
    prev_timestamp = correct_cur_timestamp - (freq/100);

    remainder = 0;

    while(1)
    {
        if ((++loops % 100000) == 0)
        {
            printf("Simulating glitch. loops: %ld\n", loops);
            timestamp = correct_cur_timestamp - 0x01000000;
        }
        else
        {
            timestamp = correct_cur_timestamp;
        }
        timestamp_diff = remainder;
        timestamp_diff += 100 * (timestamp - prev_timestamp);  /* HZ value */
        jiffies = co_div64(timestamp_diff, freq);

        if (jiffies > 1)
        {
            printf("timestamp new=%lu %lx%08lxh old=%lu %lx%08lxh rem=%lu %lx%08lxh\n",
                (unsigned long)timestamp, (unsigned long)(timestamp>>32), (unsigned long)timestamp,
                (unsigned long)prev_timestamp, (unsigned long)(prev_timestamp>>32), (unsigned long)prev_timestamp,
                (unsigned long)remainder, (unsigned long)(remainder>>32), (unsigned long)remainder);

            printf("jiffies=%ld, diff=%ld %lx%08lxh freq=%ld %lx%08lxh\n",
                jiffies,
                (unsigned long)timestamp_diff, (unsigned long)(timestamp_diff>>32), (unsigned long)timestamp_diff,
                (unsigned long)freq, (unsigned long)(freq>>32), (unsigned long)freq);

        }

        remainder = timestamp_diff - (jiffies * freq);
        prev_timestamp = timestamp;

        /*
         * Advance clock
         */
        correct_cur_timestamp += (freq/100);
    }
}


Thanks,
Ron