On a previous project, I came up with a UI pattern that might be useful for this effort. I called them "Edilogs" -- basically they were editors that had the look, feel, keybindings, and content of a dialog box. They'd often have "OK" and "Cancel" buttons at the bottom which could be activated via the "esc" and "enter" keys, and for edilogs that supported an "apply" binding, the save action was redirected to be the same as apply.
You would use them in situations where you'd want to ask the user a quick question, present a wizard, etc., and could be used as a fairly direct replacement for most dialog boxes. The big difference between edilogs and dialogs is that edilogs are always nonmodal and they are docked in the editor area so can be rearranged using the usual workbench layout system, and navigated using the back stack and other useful tools for editor navigation.
Edilogs are actually already familiar to most users since they look and feel a lot like web pages. When a website needs to ask you something, it normally directs you to another page rather than opening a popup window. In our UX study, users were not suprised by the change and 100% of the user feedback we received indicated a preference for the edilogs over the dialogs they replaced.
At the time, I'd written a bunch of utilities with API very similar to various JFace dialog box utilities, so switching from a dialog to an edilog was usually just a matter of switching base classes and changing a few lines of code.
You wouldn't want to use edilogs in all circumstances -- they would steal focus if ever opened by a background task so should only ever be opened in response to an explicit user gesture, but the metaphor works well for thiings like wizards, preference pages, confirmation dialogs, and other UI that gathers more information from the user in response to a user-initiated command.