RE: [hyades-dev] XML-based command data, binary data, pipelines

[Allan K Pratt <apratt@xxxxxxxxxx>]

Harm wrote: "I would vote yes to having XML and the prime vehicle and 
leaving the binary door open but not used by any Hyades tools."

Let's talk about the "binary door" for a second.

I think the "binary door" that Andy sees is that the whole communications 
pipeline can be replaced. Victor and I have talked this over, and that's 
not what Victor means by it. Maybe not Harm either.

First, why talk about a binary door at all? Answer: to satisfy the 
requirement (which I now see isn't captured) that a low-overhead path 
exist from client to agent for bulk binary data. This data is expected to 
include Java classes, test drivers, data pools, and even whole 
applications.

This data might never reside in a file, so the file transfer requirement 
doesn't already satisfy this. We can back our way in by telling users, 
"Put it in a file and use file transfer," but this could be inconvenient. 
It's all about tradeoffs.

One way to get binary data through the XML command path is to use base64 
encoding or similar. This has space (1.4x) and encode/decode memory and 
time costs.

Another way is to split the payload into two parts: an XML part and a 
(possibly empty) binary part. Most commands wouldn't use the binary part, 
but it would be there for those that wanted it. We still get the benefits 
of the XML part, but we also get the performance (in time, space, and 
bandwidth) of un-encoded binary data for those situations where it's 
desired. (Note that an HTTP transport system will have to base64 encode 
the binary part after all, but at least the TCP/IP transport won't have 
to.)

As for "replacing the whole communications pipeline," I'm not sure quite 
what that means any more. We drew boxes with a "command extractor" on the 
right and an arrow down from there into the HCE. What exactly is that 
interface? At one time we said "it's the binary data represented as 
structs" but I don't know if that's true any more. Does the HCE have a big 
"union" somewhere that knows the internal parameter data for every 
command? I think not. Does the command extractor use DOM parsing to create 
a structured representation of the XML? I didn't think we wanted that. 
Does it pass the raw XML without processing it? Maybe. Now I don't know 
what that looks like any more. And remember, that's one end of the 
pipeline we're supposed to be able to replace.

So: if the mechanism for binary data is to use a whole different pipeline, 
what does the end of that pipeline look like? And if not, what about an 
optional binary blob following the XML?

-- Allan Pratt, apratt@xxxxxxxxxx
Rational software division of IBM