There are two options i can suggest you:
1) Requires less work. Use a subclass of CompositeLayoutProvider.
2) Requires more work, but in general case better results. Use Draw2D
CompoundDirectedGraphLayout. You'll have to write a GMF layout provider
for this one... it's not adopted by GMF yet.
Hope this helps.