java-gnome-developer

[Java-gnome-developer] Pixbuf memory leak

[William T. <wil...@gm...>]

Hi all,

I've come across a memory leak with Pixbuf - whenever I construct one,
lots of memory is leaked. The test below leaks ~200MB when loading a
6kb image 10,000 times. Commenting out the Pixbuf construction leads
to steady memory use.
This memory usage isn't reported by the JVM.  I've come across this
behaviour with 4.0.19 and 4.1.1.

Whilst I'd love a proper solution to this, I wonder if anyone has a
quick and dirty workaround for freeing up this memory? I'm doing a
production run of approximately 6000 maps for a website, which grinds
to a halt after 100 or so maps, using all 8GB of RAM.

Thanks

Will Temperley


  @Test
  public void itLeaks() throws IOException {
    Gtk.init(null);

    for (int i = 0; i < 10000; i++) {
      System.out.println(i);

      BufferedImage b = ImageIO.read(new File("/tmp/x.png"));
      ByteArrayOutputStream bos = new ByteArrayOutputStream();
      ImageIO.write(b, "PNG", bos);
      byte[] arr = bos.toByteArray();

      //comment out and memory usage remains steady
      Pixbuf pb = new Pixbuf(arr);

    }

  }

Re: [Java-gnome-developer] Pixbuf memory leak

[Andrew C. <an...@op...>]

On Wed, 2011-10-19 at 18:01 +0100, William Temperley wrote:
>  I wonder if anyone has a
> quick and dirty workaround for freeing up this memory? I'm doing a
> production run of approximately 6000 maps for a website, which grinds
> to a halt after 100 or so maps, using all 8GB of RAM.

Eek. That's no good.

So after 6+ years at this, we're all starting to hit production usage
where we stress the memory management system. Java's behaviour hasn't
always been pretty. 

The problem, I think is:

a) that GLib allocated memory is outside the visibility of the JVM

 cross product

b) the JVM garbage collector only runs when *it* thinks that it is out
of memory.

This second point is somewhat counter to expectation, but of late I've
noticed that Java doesn't "waste" time with full GC (let alone actual
page release back to host OS) until it thinks it is a last resort.

(in other words, there's sorta an enterprise-y assumption that it's the
only big thing running, so it doesn't need to be in a rush to return
resources. If OOM killer comes along and asks for pages then the JVM has
lots to offer, but until then you just get enormous VSZ sizes. I'm not
über thrilled about this.)

Anyway, the side effect of (a) × (b) is that our GObjects don't get
free'd until Java processes the Proxy object being out of scope and they
are finalized; that doesn't happen until the GC has done its thing. The
objects *will* get freed, but when is open to question.

So as an immediate debug, I would ask what the effect of sticking a
System.gc() in right after your Pixbuf goes out of scope. I know we've
all been trained out of making that call over the years, but it might be
worth offering the hint.

If that doesn't work, we'll look deeper. If it does work, we'll look
deeper.

AfC
Sydney

Re: [Java-gnome-developer] Pixbuf memory leak

[William T. <wil...@gm...>]

On 20 October 2011 01:46, Andrew Cowie <an...@op...> wrote:
> On Wed, 2011-10-19 at 18:01 +0100, William Temperley wrote:
>>  I wonder if anyone has a
>> quick and dirty workaround for freeing up this memory? I'm doing a
>> production run of approximately 6000 maps for a website, which grinds
>> to a halt after 100 or so maps, using all 8GB of RAM.
>
> Eek. That's no good.
>

I've had to resort to scripting the app with Python to do batches -
ugly but works!

>
> The problem, I think is:
>
> a) that GLib allocated memory is outside the visibility of the JVM
>
>  cross product
>
> b) the JVM garbage collector only runs when *it* thinks that it is out
> of memory.
>
> This second point is somewhat counter to expectation, but of late I've
> noticed that Java doesn't "waste" time with full GC (let alone actual
> page release back to host OS) until it thinks it is a last resort.

I've found that Linux has been killing my large process when all
memory and virtual memory has gone. I guess this could be due to
either the JVM not knowing the proxy objects are huge and should be
cleaned up, or the memory has been leaked by the proxies.

> Anyway, the side effect of (a) × (b) is that our GObjects don't get
> free'd until Java processes the Proxy object being out of scope and they
> are finalized; that doesn't happen until the GC has done its thing. The
> objects *will* get freed, but when is open to question.
>
> So as an immediate debug, I would ask what the effect of sticking a
> System.gc() in right after your Pixbuf goes out of scope. I know we've
> all been trained out of making that call over the years, but it might be
> worth offering the hint.

I've just tried that and I'm still seeing the same memory increases
unfortunately.  This didn't work in

>
> If that doesn't work, we'll look deeper. If it does work, we'll look
> deeper.
>

I'm fairly sure it's Pixbuf that's causing this. For a bit of
background, I do a lot of drawing hefty polygons onto Cairo canvases
and draw images using the Pixbuf.  When isolating the leak by
commenting out code (having had no luck with profilers as they don't
see the allocated native memory) I ran the app with Pixbufs being
constructed but not used and got massive memory use. Commenting out
the construction of the Pixbuf - i.e. just one line - lead to steady
memory usage.

OTOH, The good news is I've been doing some fairly hefty work creating
and destroying hundreds of Cairo canvases and drawing massive polygons
on them (some with over 60,000 vertices) and I've not seen leaks with
these.

Will

Re: [Java-gnome-developer] Pixbuf memory leak

[Andrew C. <an...@op...>]

On Fri, 2011-10-21 at 10:09 +0100, William Temperley wrote:
> I'm fairly sure it's Pixbuf that's causing this. For a bit of
> background, I do a lot of drawing hefty polygons onto Cairo canvases
> and draw images using the Pixbuf.

Hm.

Have you considered using a Cairo ImageSurface? You can write images
with that.

AfC
Mount Wilson

Re: [Java-gnome-developer] Pixbuf memory leak

[Vreixo F. L. <met...@ya...>]

Hi all,

please correct me if I'm wrong, I haven't looked at this since one or two years, but I am pretty

sure about the reason for the leak in the test code. As it never enters gtk main loop, our reference
to the underlying GdkPixbuf object is never released. That is because we release it in a g_idle
function that is only executed inside the gtk main loop. So just doing:


          while (Gtk.eventsPending()) {
              Gtk.mainIterationDo(false);
          }

after Pixbuf creation will solve the memory leak. I've tested it and it works.


***

Details hereafter, you can skip this unless you understand our memory management system. Please
note I may be forgetting something, it was a long time ago since the last look I took at this stuff.
****

So, let's start. When the Pixbuf is created, a native gdk function is invoked, which in turn returns a
GObject WE OWN. When the Proxy is created, java side, in org.gnome.glib.Object we invoke
GObject.addToggleRef. This adds a new gtk ref (toggle ref) together with a strong ref in java. At this
moment, we hold two native ref to the GdkPixbuf (the original plus the toggle) and two java refs
(the one corresponding to the pb ref in our code plus the internal one). All this happens in bindings_java_memory_ref.
Of course, our original ref is no longer needed (we already hold the toggle ref) so we dispose it.
However, we do that in a g_idle function, to avoid the toggle ref for being "toggled" unnecessarily
(ok, unnecessarily in most cases, where the object is right next added to a container). In this case, as
the main loop was never executed, the original ref was never removed.

Back in our code, after one loop step, the pb var is out of scope. However, we still keep the additional java
ref, so the Pixbuf object never goes out of scope java side, so the GC never collects it.

Note that this extra ref is released (actually turned weak) after the toggle ref is the last one, which happens
when both the main loop is executed (and thus original ref is removed), and pb goes out of scope.

I hope I have explained it properly, just comment here if you need more details.

Cheers
Vreixo

PS: AfC, I think I need a chat with you this week about builder related stuff. I hope we both have time for that.







----- Mensagem original -----
De: William Temperley <wil...@gm...>
Para: java-gnome-developer <jav...@li...>
Cc: 
Enviadas: Quarta-feira, 19 de Outubro de 2011 19:01
Assunto: [Java-gnome-developer] Pixbuf memory leak

Hi all,

I've come across a memory leak with Pixbuf - whenever I construct one,
lots of memory is leaked. The test below leaks ~200MB when loading a
6kb image 10,000 times. Commenting out the Pixbuf construction leads
to steady memory use.
This memory usage isn't reported by the JVM.  I've come across this
behaviour with 4.0.19 and 4.1.1.

Whilst I'd love a proper solution to this, I wonder if anyone has a
quick and dirty workaround for freeing up this memory? I'm doing a
production run of approximately 6000 maps for a website, which grinds
to a halt after 100 or so maps, using all 8GB of RAM.

Thanks

Will Temperley


  @Test
  public void itLeaks() throws IOException {
    Gtk.init(null);

    for (int i = 0; i < 10000; i++) {
      System.out.println(i);

      BufferedImage b = ImageIO.read(new File("/tmp/x.png"));
      ByteArrayOutputStream bos = new ByteArrayOutputStream();
      ImageIO.write(b, "PNG", bos);
      byte[] arr = bos.toByteArray();

      //comment out and memory usage remains steady
      Pixbuf pb = new Pixbuf(arr);

    }

  }

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
_______________________________________________
java-gnome-developer mailing list
jav...@li...
https://lists.sourceforge.net/lists/listinfo/java-gnome-developer

Re: [Java-gnome-developer] Pixbuf memory leak

[William T. <wil...@gm...>]

On 23 October 2011 15:55, Vreixo Formoso Lopes <met...@ya...> wrote:
> Hi all,
>
> please correct me if I'm wrong, I haven't looked at this since one or two years, but I am pretty
>
> sure about the reason for the leak in the test code. As it never enters gtk main loop, our reference
> to the underlying GdkPixbuf object is never released. That is because we release it in a g_idle
> function that is only executed inside the gtk main loop. So just doing:
>
>
>           while (Gtk.eventsPending()) {
>               Gtk.mainIterationDo(false);
>           }
>
> after Pixbuf creation will solve the memory leak. I've tested it and it works.
>
>

Thanks very much indeed Vrexio, I can confirm running this code after
this fixes the issue in the test and my application. My app is unusual
as it does not have a Gtk.main() loop, presumably I would not have
seen this issue if it did.

Best regards,

Will Temperley