First, the AgilPro Simulator is a desktop tool that actually executes the
process (and not only simulates it), but without the use of a process
engine below. This means it can't handle multiple instances of the same
process. I agree that using existing process engines sometimes is a lot of
overhead, but for bigger processes with thousands of instances it is alas
not possible otherwise.
My first thought concerning your question has been the Process Virtual
Machine (PVM) [1] that is the basis for JBPM as well as Bull Nova Bonita.
If you are searching for an API then this might be the place.
On the other side, Mickael is also working on the tooling side (as he
already mentioned) and thereby concerned about a minimal set of API (an
image can be seen at [2]). Maybe that might be something that is
interesting for you?
But both are not concerned about a presentation layer for asking
questions, etc. Maybe imixs [3] might be interesting for you, I believe to
remember that they got something like that in their tooling.
That's all for the moment I can think of, so I'm sorry if I was not able
to help you at the current state.