This problem is identical to bug ID 1525332 and has resurface. I added a comment to that bug and tried to attach xpart.g but could not. The problem:
When I programmatically use rtedge, the object is not filling the view so I get very few pixels compared to doing rtedge in mged. I've attached xpart.g. Try rtedge on object "curved" inside and outside mged to observe the difference. See that rtedge works when called from inside mged but not as an executable.