[hyades-dev] The state of the HCE project

[Allan K Pratt <apratt@xxxxxxxxxx>]

I want to start this conversation in advance of our face-to-face meeting 
in two weeks. I will be on vacation next week so I only have a few days to 
talk it over by mail. Here goes.

Does anybody else feel like this is a house of cards? Each time we come up 
with a new requirement we implement a new subsystem or command to deal 
with it, but there isn't much symmetry or logical structure (you don't 
have to call it "architecture") in evidence. Maybe it's just me.

Here are some examples of what I'm talking about: 

I'm very worried that we're inventing a wire protocol and dispatch system 
when we should be adopting an existing standard. We've got layers, payload 
normalizers, headers, and envelopes. magic numbers and MD5 codes, but we 
don't talk about whether you could really recover from data errors and 
lost bytes on the data stream. We have source and destination addresses, 
multiple interconnections, brokers and controllers. Hasn't this been done 
before?

We've got structures that must be serialized and deserialized. We have 
string encodings and byte orders. In the current proposals the structures 
have fixed form: each command in the protocol has arguments of known types 
in a known order. History is littered with such protocols, all thrown away 
when they proved too resistant to changes. It has been proven more than 
once that no matter how many predefined data structures you put into a 
design there is always going to be another one tomorrow. Are we going to 
figure it out the hard way? Again? We ourselves are throwing away the 
RAC's messaging system and inventing another. Formats like XML that are 
more tolerant of loosely-coupled endpoints emerged for a reason. 

We have created a variable getting/setting system for agents and used it 
to perform the task of a property/attribute query system also, but then 
when the question of versioning comes up we invent another command, a 
whole new interface, that all components must implement. Why not have them 
all implement the variable system, and have the version number be one of 
the variables? This would be more consistent and more streamlined. 

Moreover, the ability to query a component's version number is only a 
partial fix to a larger problem: which versions am I compatible with? 
Should I force the loading of an old version? Elsewhere this is called 
"DLL Hell" and requires very large and complex scaffolding to address. 
(Microsoft uses "side-by-side execution" and Manifests.) I can't tell if 
the query solution is adequate because I can't tell what problems we're 
trying to solve, and what problems we've decided to shrug off.

There's another danger signal here: When a software group says they should 
scrap an existing, working system and start over, one should be very 
skeptical. Will it really be better? Will it even work as well? Will the 
group finish reimplementing in finite time? The existing system represents 
many man-years of discovering and solving non-obvious problems with 
specialized, sometimes arcane and platform-specific solutions. Is it 
really better to throw that accumulated knowledge away and start over?

I'm not picking on any of these single ideas themselves - this whole 
system feels like it's getting built in the dark, a piece at a time. 
Partly, as I've harped on before, it's because we don't really know what 
we're building. What motivated this in the first place? We can't know if 
we're on a path to a solution, let alone a good path, if we don't know 
what the problem is. What's the scope for this effort? What's our charter? 
How do we know when we're finished? How can we tell when we're 
overreaching if we don't know the boundaries?

I think we should step back from byte-by-byte protocols and address these 
questions. I welcome discussion from both assenters and dissenters.

-- Allan Pratt, apratt@xxxxxxxxxx
Rational software division of IBM