jung-support

[Jung-support] Does PageRank Take Much Memory?

[Bing Li <lb...@gm...>]

Dear all,

It is found that a large amount of memory is taken when ranking with
PageRank, from JUNG 2.0.1. When starting to rank, the memory consumption
goes high and when the ranking is done, it memory usage gets low.

According to jconsole, the memory usage jumps with a large range, such as
200M ~1.2G. Is it normal?

I am still not sure if it is caused by PageRank. However, I notice one
problem that I have to initialize an instance of PageRank and Graph each
time when ranking a group.

I think it must be better if the PageRank or Graph can be reused. When a
new graph is available, PageRank or Graph can be reset or cleared. But no
such methods are available, right? Any other ways to solve the potential
problem?

The code is listed as follows. The method is invoked many times because
there are a lot of graphs in my systems and each of them must be ranked
periodically. So the memory usage goes up and down in a large range
periodically.

        public Map<T, Double> Rank(HashMap<T, Vertex<T>> vertexMap,
Set<Edge<T>> edgeSet)
        {
                HashMap<T, Double> finalRankMap = new HashMap<T, Double>();
                PageRank<Vertex<T>, Edge<T>> ranker;
                Graph<Vertex<T>, Edge<T>> graph = new
DirectedSparseMultigraph<Vertex<T>, Edge<T>>();
                for (Edge<T> edge : edgeSet)
                {
                        graph.addEdge(edge, edge.GetSourceVertex(),
edge.GetDestinationVertex(), EdgeType.DIRECTED);
                }
                ranker = new PageRank<Vertex<T>, Edge<T>>(graph,
this.weightTransformer, RankingConfig.RANDOM_JUMP_PROPABILITY);
                ranker.evaluate();

                for (Vertex<T> vertex : vertexMap.values())
                {
                        try
                        {
                                finalRankMap.put(vertex.GetKey(),
ranker.getVertexScore(vertex));
                        }
                        catch (IllegalArgumentException e)
                        {
                                e.printStackTrace();
                        }
                }
                return finalRankMap;
        }

I appreciates so much for your help!

Best regards,
Bing

Re: [Jung-support] Does PageRank Take Much Memory?

[Joshua O'M. <jos...@gm...>]

Bing:

You're creating a lot of objects, so the memory usage is going to go up.
 Since you're not actually running out of memory, it sounds like the Java
garbage collector is doing its job; as soon as the graph and ranker go out
of scope (i.e., when the rank() method returns) they are eligible for GC
and will be collected when the space is needed.  It's not clear that you
have an actual problem here.

I would point out that there's not necessarily a lot of point to building
an edgeSet, i.e., the Graph encodes this information itself so there's no
need for you to do it again.  That is, just build the Graph instead of the
edge set and you'll save some space (and time) right there.

Joshua

On Wed, Sep 26, 2012 at 8:50 AM, Bing Li <lb...@gm...> wrote:

> Dear all,
>
> It is found that a large amount of memory is taken when ranking with
> PageRank, from JUNG 2.0.1. When starting to rank, the memory consumption
> goes high and when the ranking is done, it memory usage gets low.
>
> According to jconsole, the memory usage jumps with a large range, such as
> 200M ~1.2G. Is it normal?
>
> I am still not sure if it is caused by PageRank. However, I notice one
> problem that I have to initialize an instance of PageRank and Graph each
> time when ranking a group.
>
> I think it must be better if the PageRank or Graph can be reused. When a
> new graph is available, PageRank or Graph can be reset or cleared. But no
> such methods are available, right? Any other ways to solve the potential
> problem?
>
> The code is listed as follows. The method is invoked many times because
> there are a lot of graphs in my systems and each of them must be ranked
> periodically. So the memory usage goes up and down in a large range
> periodically.
>
>         public Map<T, Double> Rank(HashMap<T, Vertex<T>> vertexMap,
> Set<Edge<T>> edgeSet)
>         {
>                 HashMap<T, Double> finalRankMap = new HashMap<T, Double>();
>                 PageRank<Vertex<T>, Edge<T>> ranker;
>                 Graph<Vertex<T>, Edge<T>> graph = new
> DirectedSparseMultigraph<Vertex<T>, Edge<T>>();
>                 for (Edge<T> edge : edgeSet)
>                 {
>                         graph.addEdge(edge, edge.GetSourceVertex(),
> edge.GetDestinationVertex(), EdgeType.DIRECTED);
>                 }
>                 ranker = new PageRank<Vertex<T>, Edge<T>>(graph,
> this.weightTransformer, RankingConfig.RANDOM_JUMP_PROPABILITY);
>                 ranker.evaluate();
>
>                 for (Vertex<T> vertex : vertexMap.values())
>                 {
>                         try
>                         {
>                                 finalRankMap.put(vertex.GetKey(),
> ranker.getVertexScore(vertex));
>                         }
>                         catch (IllegalArgumentException e)
>                         {
>                                 e.printStackTrace();
>                         }
>                 }
>                 return finalRankMap;
>         }
>
> I appreciates so much for your help!
>
> Best regards,
> Bing
>
>
> ------------------------------------------------------------------------------
> How fast is your code?
> 3 out of 4 devs don\\\'t know how their code performs in production.
> Find out how slow your code is with AppDynamics Lite.
> http://ad.doubleclick.net/clk;262219672;13503038;z?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Jung-support mailing list
> Jun...@li...
> https://lists.sourceforge.net/lists/listinfo/jung-support
>
>


-- 
jos...@gm.........sites.google.com/site/joshuaomadadhain/
   Joshua O'Madadhain: Information Scientist, Musician, Philosopher-At-Tall
It's that moment of dawning comprehension that I live for.  -- Bill
Watterson
 My opinions are too rational and insightful to be those of any
organization.

Re: [Jung-support] Does PageRank Take Much Memory?

[Bing Li <lb...@gm...>]

Dear Joshua,

Thanks so much for your reply!

In my system, there are a lot of graphs. To speed the ranking, concurrency
solution is designed. But the memory is costed a lot.

For the edgeSet, the reason is that I need to set weights for each edge.
The weights depend on my system's application tightly. May I have other way
to set the weights and save the memory?

Best regards,
Bing

On Thu, Sep 27, 2012 at 12:48 AM, Joshua O'Madadhain <
jos...@gm...> wrote:

> Bing:
>
> You're creating a lot of objects, so the memory usage is going to go up.
>  Since you're not actually running out of memory, it sounds like the Java
> garbage collector is doing its job; as soon as the graph and ranker go out
> of scope (i.e., when the rank() method returns) they are eligible for GC
> and will be collected when the space is needed.  It's not clear that you
> have an actual problem here.
>
> I would point out that there's not necessarily a lot of point to building
> an edgeSet, i.e., the Graph encodes this information itself so there's no
> need for you to do it again.  That is, just build the Graph instead of the
> edge set and you'll save some space (and time) right there.
>
> Joshua
>
> On Wed, Sep 26, 2012 at 8:50 AM, Bing Li <lb...@gm...> wrote:
>
>> Dear all,
>>
>> It is found that a large amount of memory is taken when ranking with
>> PageRank, from JUNG 2.0.1. When starting to rank, the memory consumption
>> goes high and when the ranking is done, it memory usage gets low.
>>
>> According to jconsole, the memory usage jumps with a large range, such as
>> 200M ~1.2G. Is it normal?
>>
>> I am still not sure if it is caused by PageRank. However, I notice one
>> problem that I have to initialize an instance of PageRank and Graph each
>> time when ranking a group.
>>
>> I think it must be better if the PageRank or Graph can be reused. When a
>> new graph is available, PageRank or Graph can be reset or cleared. But no
>> such methods are available, right? Any other ways to solve the potential
>> problem?
>>
>> The code is listed as follows. The method is invoked many times because
>> there are a lot of graphs in my systems and each of them must be ranked
>> periodically. So the memory usage goes up and down in a large range
>> periodically.
>>
>>         public Map<T, Double> Rank(HashMap<T, Vertex<T>> vertexMap,
>> Set<Edge<T>> edgeSet)
>>         {
>>                 HashMap<T, Double> finalRankMap = new HashMap<T,
>> Double>();
>>                 PageRank<Vertex<T>, Edge<T>> ranker;
>>                 Graph<Vertex<T>, Edge<T>> graph = new
>> DirectedSparseMultigraph<Vertex<T>, Edge<T>>();
>>                 for (Edge<T> edge : edgeSet)
>>                 {
>>                         graph.addEdge(edge, edge.GetSourceVertex(),
>> edge.GetDestinationVertex(), EdgeType.DIRECTED);
>>                 }
>>                 ranker = new PageRank<Vertex<T>, Edge<T>>(graph,
>> this.weightTransformer, RankingConfig.RANDOM_JUMP_PROPABILITY);
>>                 ranker.evaluate();
>>
>>                 for (Vertex<T> vertex : vertexMap.values())
>>                 {
>>                         try
>>                         {
>>                                 finalRankMap.put(vertex.GetKey(),
>> ranker.getVertexScore(vertex));
>>                         }
>>                         catch (IllegalArgumentException e)
>>                         {
>>                                 e.printStackTrace();
>>                         }
>>                 }
>>                 return finalRankMap;
>>         }
>>
>> I appreciates so much for your help!
>>
>> Best regards,
>> Bing
>>
>>
>> ------------------------------------------------------------------------------
>> How fast is your code?
>> 3 out of 4 devs don\\\'t know how their code performs in production.
>> Find out how slow your code is with AppDynamics Lite.
>> http://ad.doubleclick.net/clk;262219672;13503038;z?
>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>> _______________________________________________
>> Jung-support mailing list
>> Jun...@li...
>> https://lists.sourceforge.net/lists/listinfo/jung-support
>>
>>
>
>
> --
> jos...@gm.........sites.google.com/site/joshuaomadadhain/
>    Joshua O'Madadhain: Information Scientist, Musician, Philosopher-At-Tall
> It's that moment of dawning comprehension that I live for.  -- Bill
> Watterson
>  My opinions are too rational and insightful to be those of any
> organization.
>
>

Re: [Jung-support] Does PageRank Take Much Memory?

[Joshua O'M. <jos...@gm...>]

On Wed, Sep 26, 2012 at 10:54 PM, Bing Li <lb...@gm...> wrote:

> Dear Joshua,
>
> Thanks so much for your reply!
>
> In my system, there are a lot of graphs. To speed the ranking, concurrency
> solution is designed. But the memory is costed a lot.
>

So decrease the amount of Java heap available, and GC will happen more
aggressively.  At some point, obviously, you'll run out of memory if you
decrease it enough, and the less memory you have the more time you'll spend
doing garbage collection, so you'll have to decide what tradeoff you're
willing to accept.


>
> For the edgeSet, the reason is that I need to set weights for each edge.
> The weights depend on my system's application tightly. May I have other way
> to set the weights and save the memory?
>

That doesn't really answer the question as to why you need the edge set.
 For PageRank, you need to supply (as you are) a Transformer from edges to
weights.  JUNG does not care how your Transformer works, so there are
several different ways you can implement it, as discussed here:
https://sourceforge.net/apps/trac/jung/wiki/JUNGManual#UserData

Good luck!

Joshua


>
> Best regards,
> Bing
>
> On Thu, Sep 27, 2012 at 12:48 AM, Joshua O'Madadhain <
> jos...@gm...> wrote:
>
>> Bing:
>>
>> You're creating a lot of objects, so the memory usage is going to go up.
>>  Since you're not actually running out of memory, it sounds like the Java
>> garbage collector is doing its job; as soon as the graph and ranker go out
>> of scope (i.e., when the rank() method returns) they are eligible for GC
>> and will be collected when the space is needed.  It's not clear that you
>> have an actual problem here.
>>
>> I would point out that there's not necessarily a lot of point to building
>> an edgeSet, i.e., the Graph encodes this information itself so there's no
>> need for you to do it again.  That is, just build the Graph instead of the
>> edge set and you'll save some space (and time) right there.
>>
>> Joshua
>>
>> On Wed, Sep 26, 2012 at 8:50 AM, Bing Li <lb...@gm...> wrote:
>>
>>> Dear all,
>>>
>>> It is found that a large amount of memory is taken when ranking with
>>> PageRank, from JUNG 2.0.1. When starting to rank, the memory consumption
>>> goes high and when the ranking is done, it memory usage gets low.
>>>
>>> According to jconsole, the memory usage jumps with a large range, such
>>> as 200M ~1.2G. Is it normal?
>>>
>>> I am still not sure if it is caused by PageRank. However, I notice one
>>> problem that I have to initialize an instance of PageRank and Graph each
>>> time when ranking a group.
>>>
>>> I think it must be better if the PageRank or Graph can be reused. When a
>>> new graph is available, PageRank or Graph can be reset or cleared. But no
>>> such methods are available, right? Any other ways to solve the potential
>>> problem?
>>>
>>> The code is listed as follows. The method is invoked many times because
>>> there are a lot of graphs in my systems and each of them must be ranked
>>> periodically. So the memory usage goes up and down in a large range
>>> periodically.
>>>
>>>         public Map<T, Double> Rank(HashMap<T, Vertex<T>> vertexMap,
>>> Set<Edge<T>> edgeSet)
>>>         {
>>>                 HashMap<T, Double> finalRankMap = new HashMap<T,
>>> Double>();
>>>                 PageRank<Vertex<T>, Edge<T>> ranker;
>>>                 Graph<Vertex<T>, Edge<T>> graph = new
>>> DirectedSparseMultigraph<Vertex<T>, Edge<T>>();
>>>                 for (Edge<T> edge : edgeSet)
>>>                 {
>>>                         graph.addEdge(edge, edge.GetSourceVertex(),
>>> edge.GetDestinationVertex(), EdgeType.DIRECTED);
>>>                 }
>>>                 ranker = new PageRank<Vertex<T>, Edge<T>>(graph,
>>> this.weightTransformer, RankingConfig.RANDOM_JUMP_PROPABILITY);
>>>                 ranker.evaluate();
>>>
>>>                 for (Vertex<T> vertex : vertexMap.values())
>>>                 {
>>>                         try
>>>                         {
>>>                                 finalRankMap.put(vertex.GetKey(),
>>> ranker.getVertexScore(vertex));
>>>                         }
>>>                         catch (IllegalArgumentException e)
>>>                         {
>>>                                 e.printStackTrace();
>>>                         }
>>>                 }
>>>                 return finalRankMap;
>>>         }
>>>
>>> I appreciates so much for your help!
>>>
>>> Best regards,
>>> Bing
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> How fast is your code?
>>> 3 out of 4 devs don\\\'t know how their code performs in production.
>>> Find out how slow your code is with AppDynamics Lite.
>>> http://ad.doubleclick.net/clk;262219672;13503038;z?
>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> _______________________________________________
>>> Jung-support mailing list
>>> Jun...@li...
>>> https://lists.sourceforge.net/lists/listinfo/jung-support
>>>
>>>
>>
>>
>> --
>> jos...@gm.........sites.google.com/site/joshuaomadadhain/
>>    Joshua O'Madadhain: Information Scientist, Musician,
>> Philosopher-At-Tall
>> It's that moment of dawning comprehension that I live for.  -- Bill
>> Watterson
>>  My opinions are too rational and insightful to be those of any
>> organization.
>>
>>
>


-- 
jos...@gm.........sites.google.com/site/joshuaomadadhain/
   Joshua O'Madadhain: Information Scientist, Musician, Philosopher-At-Tall
It's that moment of dawning comprehension that I live for.  -- Bill
Watterson
 My opinions are too rational and insightful to be those of any
organization.

Re: [Jung-support] Does PageRank Take Much Memory?

[Bing Li <lb...@gm...>]

Dear Joshua,

Before executing the method, Rank(), listed previously, edgeSet is set as
follows.

       ...
       this.edgeSet.add(new Edge<String>(this.GetVertex(hostKey),
this.GetVertex(neighborKey),
weight));
       ...

The weight is calculated according to the specific scenarios in my system.
All of edges are saved in the set before invoking the method of Rank().

Is it not required? It costs more memory?

Your help is highly appreciated.

Best regards,
Bing

On Fri, Sep 28, 2012 at 12:33 AM, Joshua O'Madadhain <
jos...@gm...> wrote:

> On Wed, Sep 26, 2012 at 10:54 PM, Bing Li <lb...@gm...> wrote:
>
>> Dear Joshua,
>>
>> Thanks so much for your reply!
>>
>> In my system, there are a lot of graphs. To speed the ranking,
>> concurrency solution is designed. But the memory is costed a lot.
>>
>
> So decrease the amount of Java heap available, and GC will happen more
> aggressively.  At some point, obviously, you'll run out of memory if you
> decrease it enough, and the less memory you have the more time you'll spend
> doing garbage collection, so you'll have to decide what tradeoff you're
> willing to accept.
>
>
>>
>> For the edgeSet, the reason is that I need to set weights for each edge.
>> The weights depend on my system's application tightly. May I have other way
>> to set the weights and save the memory?
>>
>
> That doesn't really answer the question as to why you need the edge set.
>  For PageRank, you need to supply (as you are) a Transformer from edges to
> weights.  JUNG does not care how your Transformer works, so there are
> several different ways you can implement it, as discussed here:
> https://sourceforge.net/apps/trac/jung/wiki/JUNGManual#UserData
>
> Good luck!
>
> Joshua
>
>
>>
>> Best regards,
>> Bing
>>
>> On Thu, Sep 27, 2012 at 12:48 AM, Joshua O'Madadhain <
>> jos...@gm...> wrote:
>>
>>> Bing:
>>>
>>> You're creating a lot of objects, so the memory usage is going to go up.
>>>  Since you're not actually running out of memory, it sounds like the Java
>>> garbage collector is doing its job; as soon as the graph and ranker go out
>>> of scope (i.e., when the rank() method returns) they are eligible for GC
>>> and will be collected when the space is needed.  It's not clear that you
>>> have an actual problem here.
>>>
>>> I would point out that there's not necessarily a lot of point to
>>> building an edgeSet, i.e., the Graph encodes this information itself so
>>> there's no need for you to do it again.  That is, just build the Graph
>>> instead of the edge set and you'll save some space (and time) right there.
>>>
>>> Joshua
>>>
>>> On Wed, Sep 26, 2012 at 8:50 AM, Bing Li <lb...@gm...> wrote:
>>>
>>>> Dear all,
>>>>
>>>> It is found that a large amount of memory is taken when ranking with
>>>> PageRank, from JUNG 2.0.1. When starting to rank, the memory consumption
>>>> goes high and when the ranking is done, it memory usage gets low.
>>>>
>>>> According to jconsole, the memory usage jumps with a large range, such
>>>> as 200M ~1.2G. Is it normal?
>>>>
>>>> I am still not sure if it is caused by PageRank. However, I notice one
>>>> problem that I have to initialize an instance of PageRank and Graph each
>>>> time when ranking a group.
>>>>
>>>> I think it must be better if the PageRank or Graph can be reused. When
>>>> a new graph is available, PageRank or Graph can be reset or cleared. But no
>>>> such methods are available, right? Any other ways to solve the potential
>>>> problem?
>>>>
>>>> The code is listed as follows. The method is invoked many times because
>>>> there are a lot of graphs in my systems and each of them must be ranked
>>>> periodically. So the memory usage goes up and down in a large range
>>>> periodically.
>>>>
>>>>         public Map<T, Double> Rank(HashMap<T, Vertex<T>> vertexMap,
>>>> Set<Edge<T>> edgeSet)
>>>>         {
>>>>                 HashMap<T, Double> finalRankMap = new HashMap<T,
>>>> Double>();
>>>>                 PageRank<Vertex<T>, Edge<T>> ranker;
>>>>                 Graph<Vertex<T>, Edge<T>> graph = new
>>>> DirectedSparseMultigraph<Vertex<T>, Edge<T>>();
>>>>                 for (Edge<T> edge : edgeSet)
>>>>                 {
>>>>                         graph.addEdge(edge, edge.GetSourceVertex(),
>>>> edge.GetDestinationVertex(), EdgeType.DIRECTED);
>>>>                 }
>>>>                 ranker = new PageRank<Vertex<T>, Edge<T>>(graph,
>>>> this.weightTransformer, RankingConfig.RANDOM_JUMP_PROPABILITY);
>>>>                 ranker.evaluate();
>>>>
>>>>                 for (Vertex<T> vertex : vertexMap.values())
>>>>                 {
>>>>                         try
>>>>                         {
>>>>                                 finalRankMap.put(vertex.GetKey(),
>>>> ranker.getVertexScore(vertex));
>>>>                         }
>>>>                         catch (IllegalArgumentException e)
>>>>                         {
>>>>                                 e.printStackTrace();
>>>>                         }
>>>>                 }
>>>>                 return finalRankMap;
>>>>         }
>>>>
>>>> I appreciates so much for your help!
>>>>
>>>> Best regards,
>>>> Bing
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> How fast is your code?
>>>> 3 out of 4 devs don\\\'t know how their code performs in production.
>>>> Find out how slow your code is with AppDynamics Lite.
>>>> http://ad.doubleclick.net/clk;262219672;13503038;z?
>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>>> _______________________________________________
>>>> Jung-support mailing list
>>>> Jun...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/jung-support
>>>>
>>>>
>>>
>>>
>>> --
>>> jos...@gm.........sites.google.com/site/joshuaomadadhain/
>>>    Joshua O'Madadhain: Information Scientist, Musician,
>>> Philosopher-At-Tall
>>> It's that moment of dawning comprehension that I live for.  -- Bill
>>> Watterson
>>>  My opinions are too rational and insightful to be those of any
>>> organization.
>>>
>>>
>>
>
>
> --
> jos...@gm.........sites.google.com/site/joshuaomadadhain/
>    Joshua O'Madadhain: Information Scientist, Musician, Philosopher-At-Tall
> It's that moment of dawning comprehension that I live for.  -- Bill
> Watterson
>  My opinions are too rational and insightful to be those of any
> organization.
>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Jung-support mailing list
> Jun...@li...
> https://lists.sourceforge.net/lists/listinfo/jung-support
>
>

Re: [Jung-support] Does PageRank Take Much Memory?

[Joshua O'M. <jos...@gm...>]

On Thu, Sep 27, 2012 at 7:35 PM, Bing Li <lb...@gm...> wrote:

> Dear Joshua,
>
> Before executing the method, Rank(), listed previously, edgeSet is set as
> follows.
>
>        ...
>        this.edgeSet.add(new Edge<String>(this.GetVertex(hostKey), this.GetVertex(neighborKey),
> weight));
>        ...
>
> The weight is calculated according to the specific scenarios in my system.
> All of edges are saved in the set before invoking the method of Rank().
>
> Is it not required? It costs more memory?
>

You don't need this.  Unless there's more to an Edge than you describe
here--it looks like it's defined by its source, destination, and
weight--then you can just do this:

graph.addEdge(new Edge(weight), this.GetVertex(hostkey),
this.GetVertex(neighborKey));

If edge weights are guaranteed unique, then it's even easier, just let the
edge type itself be the weight.

The graph holds the relationship between the vertices, the Edge doesn't
need to do that.
Skipping building the edgeSet will save you both memory and time.

Joshua


>
> Your help is highly appreciated.
>
> Best regards,
> Bing
>
> On Fri, Sep 28, 2012 at 12:33 AM, Joshua O'Madadhain <
> jos...@gm...> wrote:
>
>> On Wed, Sep 26, 2012 at 10:54 PM, Bing Li <lb...@gm...> wrote:
>>
>>> Dear Joshua,
>>>
>>> Thanks so much for your reply!
>>>
>>> In my system, there are a lot of graphs. To speed the ranking,
>>> concurrency solution is designed. But the memory is costed a lot.
>>>
>>
>> So decrease the amount of Java heap available, and GC will happen more
>> aggressively.  At some point, obviously, you'll run out of memory if you
>> decrease it enough, and the less memory you have the more time you'll spend
>> doing garbage collection, so you'll have to decide what tradeoff you're
>> willing to accept.
>>
>>
>>>
>>> For the edgeSet, the reason is that I need to set weights for each edge.
>>> The weights depend on my system's application tightly. May I have other way
>>> to set the weights and save the memory?
>>>
>>
>> That doesn't really answer the question as to why you need the edge set.
>>  For PageRank, you need to supply (as you are) a Transformer from edges to
>> weights.  JUNG does not care how your Transformer works, so there are
>> several different ways you can implement it, as discussed here:
>> https://sourceforge.net/apps/trac/jung/wiki/JUNGManual#UserData
>>
>> Good luck!
>>
>> Joshua
>>
>>
>>>
>>> Best regards,
>>> Bing
>>>
>>> On Thu, Sep 27, 2012 at 12:48 AM, Joshua O'Madadhain <
>>> jos...@gm...> wrote:
>>>
>>>> Bing:
>>>>
>>>> You're creating a lot of objects, so the memory usage is going to go
>>>> up.  Since you're not actually running out of memory, it sounds like the
>>>> Java garbage collector is doing its job; as soon as the graph and ranker go
>>>> out of scope (i.e., when the rank() method returns) they are eligible for
>>>> GC and will be collected when the space is needed.  It's not clear that you
>>>> have an actual problem here.
>>>>
>>>> I would point out that there's not necessarily a lot of point to
>>>> building an edgeSet, i.e., the Graph encodes this information itself so
>>>> there's no need for you to do it again.  That is, just build the Graph
>>>> instead of the edge set and you'll save some space (and time) right there.
>>>>
>>>> Joshua
>>>>
>>>> On Wed, Sep 26, 2012 at 8:50 AM, Bing Li <lb...@gm...> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> It is found that a large amount of memory is taken when ranking with
>>>>> PageRank, from JUNG 2.0.1. When starting to rank, the memory consumption
>>>>> goes high and when the ranking is done, it memory usage gets low.
>>>>>
>>>>> According to jconsole, the memory usage jumps with a large range, such
>>>>> as 200M ~1.2G. Is it normal?
>>>>>
>>>>> I am still not sure if it is caused by PageRank. However, I notice one
>>>>> problem that I have to initialize an instance of PageRank and Graph each
>>>>> time when ranking a group.
>>>>>
>>>>> I think it must be better if the PageRank or Graph can be reused. When
>>>>> a new graph is available, PageRank or Graph can be reset or cleared. But no
>>>>> such methods are available, right? Any other ways to solve the potential
>>>>> problem?
>>>>>
>>>>> The code is listed as follows. The method is invoked many times
>>>>> because there are a lot of graphs in my systems and each of them must be
>>>>> ranked periodically. So the memory usage goes up and down in a large range
>>>>> periodically.
>>>>>
>>>>>         public Map<T, Double> Rank(HashMap<T, Vertex<T>> vertexMap,
>>>>> Set<Edge<T>> edgeSet)
>>>>>         {
>>>>>                 HashMap<T, Double> finalRankMap = new HashMap<T,
>>>>> Double>();
>>>>>                 PageRank<Vertex<T>, Edge<T>> ranker;
>>>>>                 Graph<Vertex<T>, Edge<T>> graph = new
>>>>> DirectedSparseMultigraph<Vertex<T>, Edge<T>>();
>>>>>                 for (Edge<T> edge : edgeSet)
>>>>>                 {
>>>>>                         graph.addEdge(edge, edge.GetSourceVertex(),
>>>>> edge.GetDestinationVertex(), EdgeType.DIRECTED);
>>>>>                 }
>>>>>                 ranker = new PageRank<Vertex<T>, Edge<T>>(graph,
>>>>> this.weightTransformer, RankingConfig.RANDOM_JUMP_PROPABILITY);
>>>>>                 ranker.evaluate();
>>>>>
>>>>>                 for (Vertex<T> vertex : vertexMap.values())
>>>>>                 {
>>>>>                         try
>>>>>                         {
>>>>>                                 finalRankMap.put(vertex.GetKey(),
>>>>> ranker.getVertexScore(vertex));
>>>>>                         }
>>>>>                         catch (IllegalArgumentException e)
>>>>>                         {
>>>>>                                 e.printStackTrace();
>>>>>                         }
>>>>>                 }
>>>>>                 return finalRankMap;
>>>>>         }
>>>>>
>>>>> I appreciates so much for your help!
>>>>>
>>>>> Best regards,
>>>>> Bing
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> How fast is your code?
>>>>> 3 out of 4 devs don\\\'t know how their code performs in production.
>>>>> Find out how slow your code is with AppDynamics Lite.
>>>>> http://ad.doubleclick.net/clk;262219672;13503038;z?
>>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>>>> _______________________________________________
>>>>> Jung-support mailing list
>>>>> Jun...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/jung-support
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> jos...@gm.........
>>>> sites.google.com/site/joshuaomadadhain/
>>>>    Joshua O'Madadhain: Information Scientist, Musician,
>>>> Philosopher-At-Tall
>>>> It's that moment of dawning comprehension that I live for.  -- Bill
>>>> Watterson
>>>>  My opinions are too rational and insightful to be those of any
>>>> organization.
>>>>
>>>>
>>>
>>
>>
>> --
>> jos...@gm.........sites.google.com/site/joshuaomadadhain/
>>    Joshua O'Madadhain: Information Scientist, Musician,
>> Philosopher-At-Tall
>> It's that moment of dawning comprehension that I live for.  -- Bill
>> Watterson
>>  My opinions are too rational and insightful to be those of any
>> organization.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://ad.doubleclick.net/clk;258768047;13503038;j?
>>
>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>> _______________________________________________
>> Jung-support mailing list
>> Jun...@li...
>> https://lists.sourceforge.net/lists/listinfo/jung-support
>>
>>
>


-- 
jos...@gm.........sites.google.com/site/joshuaomadadhain/
   Joshua O'Madadhain: Information Scientist, Musician, Philosopher-At-Tall
It's that moment of dawning comprehension that I live for.  -- Bill
Watterson
 My opinions are too rational and insightful to be those of any
organization.

Re: [Jung-support] Does PageRank Take Much Memory?

[Bing Li <lb...@gm...>]

Joshua,

I see. I appreciate so much for your help!

Best regards,
Bing

On Fri, Sep 28, 2012 at 2:07 PM, Joshua O'Madadhain <
jos...@gm...> wrote:

> On Thu, Sep 27, 2012 at 7:35 PM, Bing Li <lb...@gm...> wrote:
>
>> Dear Joshua,
>>
>> Before executing the method, Rank(), listed previously, edgeSet is set as
>> follows.
>>
>>        ...
>>        this.edgeSet.add(new Edge<String>(this.GetVertex(hostKey), this.GetVertex(neighborKey),
>> weight));
>>        ...
>>
>> The weight is calculated according to the specific scenarios in my
>> system. All of edges are saved in the set before invoking the method of
>> Rank().
>>
>> Is it not required? It costs more memory?
>>
>
> You don't need this.  Unless there's more to an Edge than you describe
> here--it looks like it's defined by its source, destination, and
> weight--then you can just do this:
>
> graph.addEdge(new Edge(weight), this.GetVertex(hostkey),
> this.GetVertex(neighborKey));
>
> If edge weights are guaranteed unique, then it's even easier, just let the
> edge type itself be the weight.
>
> The graph holds the relationship between the vertices, the Edge doesn't
> need to do that.
> Skipping building the edgeSet will save you both memory and time.
>
> Joshua
>
>
>>
>> Your help is highly appreciated.
>>
>> Best regards,
>> Bing
>>
>> On Fri, Sep 28, 2012 at 12:33 AM, Joshua O'Madadhain <
>> jos...@gm...> wrote:
>>
>>> On Wed, Sep 26, 2012 at 10:54 PM, Bing Li <lb...@gm...> wrote:
>>>
>>>> Dear Joshua,
>>>>
>>>> Thanks so much for your reply!
>>>>
>>>> In my system, there are a lot of graphs. To speed the ranking,
>>>> concurrency solution is designed. But the memory is costed a lot.
>>>>
>>>
>>> So decrease the amount of Java heap available, and GC will happen more
>>> aggressively.  At some point, obviously, you'll run out of memory if you
>>> decrease it enough, and the less memory you have the more time you'll spend
>>> doing garbage collection, so you'll have to decide what tradeoff you're
>>> willing to accept.
>>>
>>>
>>>>
>>>> For the edgeSet, the reason is that I need to set weights for each
>>>> edge. The weights depend on my system's application tightly. May I have
>>>> other way to set the weights and save the memory?
>>>>
>>>
>>> That doesn't really answer the question as to why you need the edge set.
>>>  For PageRank, you need to supply (as you are) a Transformer from edges to
>>> weights.  JUNG does not care how your Transformer works, so there are
>>> several different ways you can implement it, as discussed here:
>>> https://sourceforge.net/apps/trac/jung/wiki/JUNGManual#UserData
>>>
>>> Good luck!
>>>
>>> Joshua
>>>
>>>
>>>>
>>>> Best regards,
>>>> Bing
>>>>
>>>> On Thu, Sep 27, 2012 at 12:48 AM, Joshua O'Madadhain <
>>>> jos...@gm...> wrote:
>>>>
>>>>> Bing:
>>>>>
>>>>> You're creating a lot of objects, so the memory usage is going to go
>>>>> up.  Since you're not actually running out of memory, it sounds like the
>>>>> Java garbage collector is doing its job; as soon as the graph and ranker go
>>>>> out of scope (i.e., when the rank() method returns) they are eligible for
>>>>> GC and will be collected when the space is needed.  It's not clear that you
>>>>> have an actual problem here.
>>>>>
>>>>> I would point out that there's not necessarily a lot of point to
>>>>> building an edgeSet, i.e., the Graph encodes this information itself so
>>>>> there's no need for you to do it again.  That is, just build the Graph
>>>>> instead of the edge set and you'll save some space (and time) right there.
>>>>>
>>>>> Joshua
>>>>>
>>>>> On Wed, Sep 26, 2012 at 8:50 AM, Bing Li <lb...@gm...> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> It is found that a large amount of memory is taken when ranking with
>>>>>> PageRank, from JUNG 2.0.1. When starting to rank, the memory consumption
>>>>>> goes high and when the ranking is done, it memory usage gets low.
>>>>>>
>>>>>> According to jconsole, the memory usage jumps with a large range,
>>>>>> such as 200M ~1.2G. Is it normal?
>>>>>>
>>>>>> I am still not sure if it is caused by PageRank. However, I notice
>>>>>> one problem that I have to initialize an instance of PageRank and Graph
>>>>>> each time when ranking a group.
>>>>>>
>>>>>> I think it must be better if the PageRank or Graph can be reused.
>>>>>> When a new graph is available, PageRank or Graph can be reset or cleared.
>>>>>> But no such methods are available, right? Any other ways to solve the
>>>>>> potential problem?
>>>>>>
>>>>>> The code is listed as follows. The method is invoked many times
>>>>>> because there are a lot of graphs in my systems and each of them must be
>>>>>> ranked periodically. So the memory usage goes up and down in a large range
>>>>>> periodically.
>>>>>>
>>>>>>         public Map<T, Double> Rank(HashMap<T, Vertex<T>> vertexMap,
>>>>>> Set<Edge<T>> edgeSet)
>>>>>>         {
>>>>>>                 HashMap<T, Double> finalRankMap = new HashMap<T,
>>>>>> Double>();
>>>>>>                 PageRank<Vertex<T>, Edge<T>> ranker;
>>>>>>                 Graph<Vertex<T>, Edge<T>> graph = new
>>>>>> DirectedSparseMultigraph<Vertex<T>, Edge<T>>();
>>>>>>                 for (Edge<T> edge : edgeSet)
>>>>>>                 {
>>>>>>                         graph.addEdge(edge, edge.GetSourceVertex(),
>>>>>> edge.GetDestinationVertex(), EdgeType.DIRECTED);
>>>>>>                 }
>>>>>>                 ranker = new PageRank<Vertex<T>, Edge<T>>(graph,
>>>>>> this.weightTransformer, RankingConfig.RANDOM_JUMP_PROPABILITY);
>>>>>>                 ranker.evaluate();
>>>>>>
>>>>>>                 for (Vertex<T> vertex : vertexMap.values())
>>>>>>                 {
>>>>>>                         try
>>>>>>                         {
>>>>>>                                 finalRankMap.put(vertex.GetKey(),
>>>>>> ranker.getVertexScore(vertex));
>>>>>>                         }
>>>>>>                         catch (IllegalArgumentException e)
>>>>>>                         {
>>>>>>                                 e.printStackTrace();
>>>>>>                         }
>>>>>>                 }
>>>>>>                 return finalRankMap;
>>>>>>         }
>>>>>>
>>>>>> I appreciates so much for your help!
>>>>>>
>>>>>> Best regards,
>>>>>> Bing
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> How fast is your code?
>>>>>> 3 out of 4 devs don\\\'t know how their code performs in production.
>>>>>> Find out how slow your code is with AppDynamics Lite.
>>>>>> http://ad.doubleclick.net/clk;262219672;13503038;z?
>>>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>>>>> _______________________________________________
>>>>>> Jung-support mailing list
>>>>>> Jun...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/jung-support
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> jos...@gm.........
>>>>> sites.google.com/site/joshuaomadadhain/
>>>>>    Joshua O'Madadhain: Information Scientist, Musician,
>>>>> Philosopher-At-Tall
>>>>> It's that moment of dawning comprehension that I live for.  -- Bill
>>>>> Watterson
>>>>>  My opinions are too rational and insightful to be those of any
>>>>> organization.
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> jos...@gm.........sites.google.com/site/joshuaomadadhain/
>>>    Joshua O'Madadhain: Information Scientist, Musician,
>>> Philosopher-At-Tall
>>> It's that moment of dawning comprehension that I live for.  -- Bill
>>> Watterson
>>>  My opinions are too rational and insightful to be those of any
>>> organization.
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://ad.doubleclick.net/clk;258768047;13503038;j?
>>>
>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> _______________________________________________
>>> Jung-support mailing list
>>> Jun...@li...
>>> https://lists.sourceforge.net/lists/listinfo/jung-support
>>>
>>>
>>
>
>
> --
> jos...@gm.........sites.google.com/site/joshuaomadadhain/
>    Joshua O'Madadhain: Information Scientist, Musician, Philosopher-At-Tall
> It's that moment of dawning comprehension that I live for.  -- Bill
> Watterson
>  My opinions are too rational and insightful to be those of any
> organization.
>
>
>
> ------------------------------------------------------------------------------
> Got visibility?
> Most devs has no idea what their production app looks like.
> Find out how fast your code is with AppDynamics Lite.
> http://ad.doubleclick.net/clk;262219671;13503038;y?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Jung-support mailing list
> Jun...@li...
> https://lists.sourceforge.net/lists/listinfo/jung-support
>
>