Menu

#16 Island filtering should happen in graph compression stage

open
nobody
None
5
2012-10-05
2012-10-05
Anonymous
No

Islands are nodes which are unconnected to other nodes. Islands are currently filtered out in RemoveTipsAvro.

We should be trying to filter out islands (and other graph structures) as close as possible to the location in the processing where they are created. For an island, this means right after graph compression. The downside of moving island filtering into the compression stage is that we have two different MR jobs QuickMerge/PairMergAvro where these islands might be produced. So there two places we need to add the filtering. However, I think we could create a common class NodeFilter which gets applied in both places.

Discussion

MongoDB Logo MongoDB