#1038 Liferea Pulls Continuous Updates from TT-RSS

v1.8
closed-fixed
Networking (27)
7
2013-07-22
2012-11-01
No

When setting up a TT-RSS source, Liferea pulls continuous updates, downloading all of the feed data and then immediately downloading the data again. This cycle will continue until Liferea is closed, causing high load on the TT-RSS instance, and will immediately resume when Liferea is restarted. While this issue causes no noticable problems in Liferea's UI, if running TT-RSS on a small server it can cause enough server-side load to interfere with other webapps.

I see this behavior with both Liferea 1.8.7 and 1.8.10, both using TT-RSS 1.6.0 as a back end. However, this post from the liferea-devel mailing list seems to indicate that the issue has been present since at least Liferea 1.8.0 and TT-RSS 1.5.7: http://sourceforge.net/mailarchive/forum.php?thread_name=4F01473E.8060803%40exyr.org&forum_name=liferea-devel

A lightly sanitized log produced using liferea --debug-all is attached.

Discussion

  • Anonymous

    Anonymous - 2012-11-01
     
  • Walter Bell

    Walter Bell - 2013-02-24

    I'm seeing the same issue in 1.8. Here's a patch to fix the issue-- from looking at other sources, the ttrss source doesn't enforce an update interval.

     
  • mossroy

    mossroy - 2013-03-24

    I also face the same issue.
    My tt-rss 1.7.5 instance is installed on a sheevaplug server (1.2 GHz ARM single core)
    If I start Liferea, the server's CPU reaches 100% and does not go below until Liferea is shut down.

     
  • James Le Cuirot

    James Le Cuirot - 2013-03-27

    I am also seeing this under 1.10-rc1. The patch doesn't apply cleanly so here's a new one that seems to work.

    As well as this, I am seeing a massive memory leak when using tt-rss under both 1.8.12 and 1.10-rc1. At one point, it was using over 2GB. I'm not sure if it's related but I will monitor the RAM usage now that this patch has been applied. However, I suspect the problem will simply take much much longer to appear now that it's no longer checking continuously.

     
  • mase76

    mase76 - 2013-05-07

    The patch did not solve it for me.

     
  • Simon Kågedal Reimer

    James' patch works well for me, although it seems he forgot to include a #define of TTRSS_SOURCE_UPDATE_INTERVAL. Here's a git format-patch formatted patch against git commit 845cbdfd63512b5b2afa0b7551f3ee8981479695 with the define included.

    This code might need a more thorough review, but I'm hoping a working fix for this rather serious bug can be added soon.

     
  • James Le Cuirot

    James Le Cuirot - 2013-05-08

    Actually that line is already present in the 1.10-rc1 tarball but it was removed in commit 868e8187bde1a12f915e70b4f706f6adbcffdc54 because it is apparently useless!

     
  • Simon Kågedal Reimer

    James, aha!

     
  • mase76

    mase76 - 2013-05-08

    I patched against the Debian Package of 1.10rc1.
    It seems to work, but the update message in the status
    bar does not disappear every time. But I did not see
    related traffic with wireshark.

     
  • Lars Windolf

    Lars Windolf - 2013-05-08

    Hi Guys!

    First thing I need to say: I'm really sorry I didn't find time to support you earlier and give proper feedback.

    Actually I believe current git master has it already fixed. Can you please retest?

    The actual bug was in ttrss_source_auto_update () not behaving the same as the Google variant.

    The patch you guys created would also solve it, but I prefer the solution in git as it simplifies code and corrects invalid behaviour. I do not want to introduce a workaround instead. I hope you understand this.

    @Simon: As for the commit you mentioned. It is about simplifying the implementation of the TinyTinyRSS implementation. This commit makes it use feed metadata as all other node types do and removes the unnecessary timestamp handling. This is because each node source type relying on default updating shouldn't implement it's own handling.

     
  • Simon Kågedal Reimer

    Hi Lars, thanks for replying!

    I just did a pull and a recompile and unfortunately, no, I don't think git master has it fixed. If I run it with --debug-update, I see it pulling updates every 10 seconds.

    Using general mechanisms for handling feed metadata sounds great, but I don't quite understand how the ttrss_source_auto_update introduced in that commit, which is the same as in current git master could work:

    static void
    ttrss_source_auto_update (nodePtr node)
    {
            GTimeVal        now;
            ttrssSourcePtr  source = (ttrssSourcePtr) node->data;
    
            if (source->loginState == TTRSS_SOURCE_STATE_IN_PROGRESS) 
                    return; /* the update will start automatically anyway */
    
            g_get_current_time (&now);
    
            ttrss_source_update (node);
    }
    

    As far as my understanding goes, this is called every ten seconds by feedlist_auto_update. The only thing that would stop it from calling ttrss_source_update is if we're currently logging in. And ttrss_source_update calls subscription_update which starts pulling the updates.

    google_source_auto_update, by contrast, does the kind of timestamp checking that James' patch does, and that it seems ttrss_source_auto_update was doing before the above mentioned commit.

    Regards, Simon

     
    Last edit: Simon Kågedal Reimer 2013-05-09
  • Lars Windolf

    Lars Windolf - 2013-05-09

    Hi Simon,

    you are right ttrss_source_auto_update() is called every 10s because it should check every 10s if there is a pending update. If TinyTinyRSS is in state logged in it simply calls ttrss_source_update() which runs subscription_update() for each feed. Only subscription_update() performs the update interval check.

    This way we have the same code active for real feeds and TinyTinyRSS feeds.

    I did test with a 5min update interval so I see dozens of "ttrss_source_auto_update()" calls before the feeds are updated after 5min. Do you really see feed update processing handling after each ttrss_source_auto_update() for reach of the feeds in TinyTinyRSS?

    Cheers
    Lars

     
  • Simon Kågedal Reimer

    Hi again!

    Yes, I really see that. To be really sure it was nothing funny on my side I compiled a fresh git clone:

    git clone git://git.code.sf.net/p/liferea/code liferea
    cd liferea
    ./autogen.sh
    make
    sudo make install
    

    And cleaned up everything on the user side:

    rm -rf ~/.local/share/liferea
    rm -rf ~/.cache/liferea
    rm -rf ~/.config/liferea
    gsettings reset-recursively net.sf.liferea
    

    (I hope that gets everything)

    Then ran the freshly compiled Liferea with --debug-update. I added a ttrss source, a new local install of Tiny Tiny RSS 1.7.8 - and yes, updates every ten seconds.

    I attach the stdout+stderr output of running this for about a minute. You'll see several updates of the same feeds.

    I fail to see where in subscription_update there is timestamp checking. Shouldn't this involve subscription_auto_update? This seems to be where the timestamps are compared, but it is never reached through ttrss_source_auto_update.

    In ttrss_source_auto_update, I tried simply changing:

    ttrss_source_update (node);
    

    To:

    subscription_auto_update (node->subscription);
    

    This instead gives me no updates at all, even after timeout.

    Regards, Simon

     
  • Lars Windolf

    Lars Windolf - 2013-05-09

    Hi Simon,

    thanks for testing. Actually I messed up. I hadn't pushed the fix yet. It is online now (http://sourceforge.net/p/liferea/code/ci/master/tree/src/fl_sources/ttrss_source.c?diff=052111130e66067c29d885379f882cac92b1540a).

    It is as you tried just a replacement of ttrss_source_update() with subscription_auto_update() because this is the central function doing interval checking based on the updateState of a subscription.

    For me this works. I tried multiple times. I consider this to be solving the problem.

     
  • Simon Kågedal Reimer

    Allright! Mystery solved. :)

    I will test it again tonight.

     
  • Simon Kågedal Reimer

    Confirming that things work as expected in current git master!

    (If anyone in this thread has problems with this so that it doesn't run according to preferences, it might be that you've previously run Liferea with James' (or my) patch. This sets the update interval on ttrss subscriptions to once every day, saved in the database. Just remove the ttrss source and re-add it and things will work.)

     
  • Lars Windolf

    Lars Windolf - 2013-05-11
    • status: open --> open-fixed
    • assigned_to: Lars Windolf
    • Priority: 5 --> 7
     
  • Lars Windolf

    Lars Windolf - 2013-07-22

    Closing this

     
  • Lars Windolf

    Lars Windolf - 2013-07-22
    • status: open-fixed --> closed-fixed