Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [vtp-dev] Questions, directions, techie stuff

Title: Re: [vtp-dev] Questions, directions, techie stuff
Nino,

    You are absolutely right that there are drawbacks with the single file storage approach.  I'll try to outline why we decided to go that direction, as our original design actually used multiple files.  Most of the issues we’ve seen when multiple developers make changes to an application can be attributed to two primary causes: the information is stored in XML format and there is a separation between the storage format and the medium familiar to the developer.  Let’s take a look a closer look at these problems.

The XML format is fabulous for certain use cases.  Storing highly structured data that contains large amounts of text that needs to be accessible on multiple platforms and manageable by humans is one of them.  And this is exactly the type of data that makes up our application definitions.  The problem is that is gives the diff engine of most source control systems fits.  Nearly by definition, all data markers in XML are variable length in both individual composition (attributes) or number and order (sub elements).  In my experience the most usual culprits of incorrect difference calculation are elements with multiple attributes, especially when the number or which attributes are dynamic, and the end tags of elements with children.  To help avoid the havoc caused by these instances, the application document generated by the VTP follows some strict rules:

  • XML indentation is fully enforced on all elements and levels of nesting
  • Start tag pre-ambles are always placed on their own line
  • Element attributes are always placed on their own line
  • Element attributes always occur in the same order regardless of presence
  • Element end tags are always placed on their own line
  • Element collections are always written in the same order they were read or created unless there is an overriding ordering mechanism

However, even with all these rules in place, there are still instances of conflict when merging changes from independent sources.  This brings us to the other basic driver; the developer is not familiar with the XML format.  Our development environment presents the developer with a 100% graphical representation of all information present in the XML document.  This goes so far as the editor doesn’t even bother with the underlying file until a change needs to be committed to storage.  Often times, the developer doesn’t even have experience with XML or other central technologies employed by the VTP let alone a working knowledge of the schema and nuances of our specific XML language.  Eclipse is great in that it happily displays the head to head conflict resolution UI.  Given the target audience, this can result in even further data loss depending on their tendency to click buttons.

I’ve now done something I hate to do, describe an array of problems without really offering any solutions.  I’m not sure if there are any “good and easy” ones.  OK, back to why we migrated to single file over multiple.  The main reason was that using multiple files to store the data opened our project structure up to a possible inconsistent state, depending on user behavior.  In a traditional java project, having many different files with changes in them and various states of committal to the source repository is normal and often doesn’t even result in an inconsistent state.  However, our applications are very integrated and represent a continuous flow of logic from the single entry point to its final states.  There are very few edit oriented operations that could be performed that wouldn’t leave the entire project in an inconsistent state if all documents weren’t checked in.  Although this type of situation is fairly common for those of us who are familiar with the standard software development lifecycle, it can be quite a learning curve for much of the target audience.

Another reason we decided to perform the migration was all the other issues with XML were still present in the multiple file approach, so we decided to merge all the files into a single source.  To be honest, I’m not entirely sold on this approach, but it solved more issues than it presented at the time.  Choices like these really resonate with what Kent Beck discussed in his Design is an Island piece on Planet Eclipse.  I would love to enable a greater liberty in changes being made by multiple developers.  I would like to mention that with a moderate amount of discipline, multiple developers can still edit the same project.  If the developers remain within say a single canvas or don’t modify the same canvas, the likelihood of a conflict is greatly reduced thanks to the efforts I mentioned above.  Also, there is the ability to separate your application into several independent projects as application fragments.  These can obviously be edited without fear of collision.

Ultimately, I think the only real solution is to provide a completely custom conflict resolution UI that allows the application designer to merge changes using the medium they are used to.  Showing the changes in a head-to-head visual display and simple point and click change acceptance.  This would almost be a project in and of itself and consequently has been a lower priority than other more easily reachable fruit.  I’ve been recently playing around with a couple of ideas on how to make the XML rules we follow more effective.  My favorite one so far is to sort of watermark the end element tag so as to identify it uniquely even within a list of similar elements.  For example, most of our elements represent objects that have an identifying attribute like ‘id’.  The line containing the end tag would have an XML comment added just after the tag content that contained the identifier.  This would allow the line based diff engines to more easily match groups of lines together in cohesive segments.  I haven’t fully worked out the details yet.

Trip Gilman


On 4/14/09 10:57 AM, "Nino Martinez" <nino.martinez.wael@xxxxxxxxx> wrote:

> Hi Trip
>
> Now that you mention a thing about the runtime platform, as I see it the
> interactions you describe in the visual layout all goes into one xml
> container? Im wondering why it's done this way? Nortel's SCE, which im
> heading into using everyday, also has this approach.
>
> Making it impossible to be more than one dev on projects. Since you
> overlap changes in the xml file.. Or am I wrong about VTP (I hope so)..
>
> If it instead could be contained in some code generation or something
> like it, maybe taking advantage of java7's scripting possibilities (when
> it comes) or something else to leverage this problem.. However this
> would probably impose a big code change..
>
> regards Nino
>
>
> On Sun, 2009-04-12 at 19:27 -0500, Trip Gilman wrote:
>> David
>>
>> First I'd like to thank you for your interest in the VTP and welcome you
>> back to the industry.  I've provided a broad overview of where we are now as
>> a project and also weaved some of my detailed thoughts on your questions
>> into your original text below.
>>
>> You are correct in that the current VTP is a complete departure from the
>> codebase originally contributed by IBM.  I would like to take a moment to
>> point out some of the fundamental changes between the original vision and
>> our current direction and then discuss the reasoning for our approach.
>>
>> First, the xml editors for GRXML grammars have been replaced with the
>> generic xml editor provided by the basic eclipse development framework.  The
>> original VTP utilized a customized xml editor based on those provided by the
>> Web Tools Project.  The removal of this dependency reduced the overall size
>> of our software by nearly 100M and eliminated several layers of complexity
>> for no real loss of functionality.
>>
>> Second, and probably the most centrally important, the VTP employs a common
>> runtime architecture as opposed to code generation.  I don't want to focus
>> on the choice between the two, but I would like to illustrate how we
>> leverage our choice of a runtime architecture.
>>
>> The voice tools project has almost become a misnomer.  Our development and
>> runtime frameworks are no longer just for VXML, but have evolved into a
>> generic platform for describing interactions.  The core of the development
>> environment is blissfully unaware that the user intends to use the
>> application they are creating as the source of a VXML interaction.  The
>> runtime environment executes the interaction within a process engine that
>> emits basic interactive commands that are interpreted into VXML at the final
>> moment.  This is important to understand as this philosophy is pervasive
>> throughout the codebase.
>>
>> The user creates applications by adding Actions to a canvas and then
>> defining the transitions between those actions.  The most common Actions,
>> such as a Menu or a Prompt(Output), are standard components provided by the
>> framework.  However, third-party developers can easily and quickly create
>> their own action libraries for users to use.  I want to reiterate that
>> although these actions may share the names of the elements that make up
>> VXML, they are not the same.
>>
>> The designer has an abstraction framework that models the nuances of
>> interaction types on these actions, allowing interaction specific
>> configuration.  For example, the VTP comes with an implementation of the
>> Voice (VXML) interaction type.  This provides interaction specific UIs to
>> manage VXML settings such as barge-in and dtmf/speech entry.
>>
>> This framework has been used to create other interaction types such as
>> Instant Messaging.  An application can support any combination of
>> interaction types the user has available to them.  This is very powerful in
>> that the same application deployed once, could drive the interactions of
>> both telephone and IM based customers.
>>
>> On the runtime side, things are equally abstracted.  As I mentioned earlier,
>> the artifacts produced by our designer (upon export) are actually process
>> meta-language documents.  These documents are executed by the process engine
>> at the core of the runtime.  As the process is executed, interactive
>> commands are generated.  These commands are then filtered through a platform
>> abstraction layer and evaluated.  The VTP includes a platform layer designed
>> for VXML interactions that is capable of transforming the interactive
>> commands into their representative VXML documents.
>>
>> The platform abstraction layer is powerful on two fronts.  First, it allows
>> the VXML generation process to be customized to the specific VXML Browser
>> platform requesting the document.  This means that a single application
>> instance can service multiple VXML platforms simultaneously without
>> re-export or compilation.  The VXML session could even bounce back and forth
>> between the platforms without issue.
>>
>> Second, the platform abstraction is open to third-parties to add
>> implementations for additional platform types and output mechanisms.  The
>> VXML platform generates XML documents and processes requests and responses
>> over HTTP.  Other implementations could simply execute direct API calls to
>> other software or hardware, never producing a document artifact.  They could
>> also support transports other than HTTP.
>>
>> I hope this helps illuminate the project for you.
>>
>> Trip Gilman
>>
>>
>> On 4/10/09 8:33 PM, "David Reich" <David.Reich@xxxxxxxxxxx> wrote:
>>
>>> Hi,
>>>
>>> First, let me (re)introduce myself.  David Reich, formerly of IBM.  I was
>>> involved in the creation of VoiceXML and was the guy who was the initial
>>> driver behind all of the Eclipse-based voice tooling, first from IBM's
>>> toolkit, and it's subsequent donation to Eclipse and VTP.  I've since left
>>> IBM
>>> and as you know, IBM's involvement in voice has waned.  But, I am now with a
>>> new company, happily back in VoiceXML, CCXML and SRGS, and we are doing a
>>> lot
>>> with VoiceXML and CCXML and I was brought in to drive our development team,
>>> and one of the areas I am looking at is the development of our application
>>> artifacts, and VoiceXML application generation.
>>>
>>> With that, I've spen the last couple of days figuring out where things are
>>> with this project, how we can use it, how we might give back to it and so
>>> on.
>>> From what I've been able to see thus far, it looks like the current code
>>> base
>>> bears little to no resemblance to the VTP 1.0 code that came in from my
>>> former
>>> team, but, on the other hand, there is a nicely-coming-along visual editor
>>> without which all VTP would be is the XML editor with DTDs plugged in and
>>> while nice, not really compelling.
>>>
>>> I was hoping to provoke some discussion and see about some combination of
>>> code
>>> to create an even more gener(ic)ally useful environment.  With that, some
>>> thoughts and the hope we can put some other things together.  In no specific
>>> order:
>>>
>>>
>>>  *   The original IBM code seems to be gone.  When things moved to SVN, the
>>> CVS repository went away, so unless someone has that source code, it's gone
>>> forever (I've checked with the (remnants of) the IBM team, and without some
>>> serious effort, it's not readily available.  So, how can we get that back,
>>> even if we only want to take small bits (or even none?) of it?  There's got
>>> to
>>> be useful things there, and we should still look at that.
>>
>> The original IBM source still exists, however it is not readily available as
>> it has been archived from CVS by the webmasters to reclaim space and keep
>> the repositories current.  If you would like a copy of the archive, I would
>> be more than happy to make the request for you.
>>
>>>  *   In the original tools, of which the VTP 1.0 code was a part, the vision
>>> was a call flow builder similar to what WebMethods has been working on, but
>>> rather than generate a metalanguage, have it generate VoiceXML and have a
>>> tabbed view (kinda like FrontPage) where you see the call flow view, and the
>>> VoiceXML source view, and the editor would allow you to tweak the VoiceXML
>>> in
>>> source edit mode, affording finer-grained control of the VoiceXML.
>>
>> Very early on we abandoned code generation in favor of a runtime
>> architecture to overcome several challenges we were faced with when
>> developing applications in the field.  Each time we encountered a new VXML
>> platform, there were always a myriad of minute differences or interesting
>> interpretations of the specification.  We considered just coding these
>> changes into the code generation piece until we ran into a client that had
>> multiple platforms in service.  The solution was to export the application
>> twice, once for each platform.  This instance was one of the driving forces
>> in the transition to a runtime architecture.
>>
>> Another driving factor of the move, was the realization that we were already
>> building a runtime, just not officially and not effectively.  We had
>> developed a loose library of utility functions and were using them over and
>> over again in all our applications.  More and more functionality was being
>> encoded into this framework in an effort to hasten development.  Making the
>> runtime a first class citizen in the architecture allowed us to leverage the
>> advances we had made while removing most of the hurdled we were facing.
>>
>>>  *   Linkages to the Eclipse framework for task-list items in validation,
>>> warnings, errors, etc.
>>
>> Absolutely, integration to the tagging systems in Eclipse is currently on
>> the wish list.
>>
>>>  *   Enabling (as Randy has told me is the direction) a derivative or
>>> extension plugin to generate code that spews forth VoiceXML.  This could
>>> take
>>> the form of the OpenMethods servlet, or even perhaps an extension one could
>>> write
>>>
>>> to do JSPs, or other types of markup/tag/metalanguage since not everyone
>>> will
>>> want just the VoiceXML from the OpenMethods servlet, or wants to make
>>> further
>>> tweaks to the VoiceXML.
>>
>> I won't say that it would be impossible to generate the resulting VXML from
>> an application statically, but at some point you'd end up recreating the
>> logic inherent in the runtime in JSP or some other language.  In the case of
>> multi-modal applications, simply duplicate the application several times
>> over.
>>
>> On a side note, the modification of generated code after export can be a
>> very tempting practice as many problems can be solved quickly this way.
>> However, I never recommend the use of this.  It places a process requirement
>> on the development and deployment staff to always re-apply the changes to
>> the code each time there is an update.  What happens if changes made to the
>> code require the workaround to be updated?  This can easily balloon into a
>> maintenance nightmare.  I would say that it should be the goal of any
>> system, whether code generation or runtime, to avoid the need of this at all
>> costs.
>>
>>>  *   A broader property sheet for each call element in the palette on the
>>> canvas to specify items such as barge-in, SSML tags or information, prompt
>>> audio files to be played and so on. Specifically, the block would play one
>>> of
>>> a set of prompts based on current state (such as "Welcome to..."  or
>>> "Welcome
>>> back to..." based on some global variables in the ECMAscript for example.
>>> This tool should should hopefully allow for the export of this property
>>> sheet
>>> data in different formats, but then again, as open source, we can work
>>> together on extensions for our specific needs and write the hooks and
>>> extension code for different needs.
>>
>> The designer already supports this type of configuration in several ways.
>> At a base level, the developer can use _javascript_ variables to indicate
>> prompt content.  SSML can also be included in prompt content simply as text.
>> At a higher level, the designer employs a system called branding to allow
>> very detailed configuration for different user segments.  Branding can be
>> used for personalization, line of business differentiation, or even
>> multi-tenanting of an application.
>>
>> On the export aspect of this, I assume you intend to use the artifacts
>> within a static JSP/VXML document.
>>
>>>  *   Have the palette be extensible where one can take prebuilt modules (say
>>> an authentication subflow) add them to the palette, and drop it into a new
>>> or
>>> existing application.
>>
>> The designer is fully extensible.  Third-parties can easily publish
>> additional modules for use.  Developers are also able to package up common
>> call flow elements into reusable components called dialogs or even
>> significant portions of a call flow into a fragment.
>>
>>> That's enough or now, but you get the general idea.   I'm hoping we can make
>>> this a bit more granular and extensible, and some of that is in the earlier
>>> code (yes, I also have a fondness for my baby ;-) and marrying that with the
>>> newer code, I think we can do a lot here.
>>>
>>> Comments?
>>>
>>> Thanks....
>>>
>>> David Reich
>>> AIT Architect,  Adeptra, Inc.
>>>
>>>
>>>
>>> _______________________________________________
>>> vtp-dev mailing list
>>> vtp-dev@xxxxxxxxxxx
>>> https://dev.eclipse.org/mailman/listinfo/vtp-dev
>>
>> _______________________________________________
>> vtp-dev mailing list
>> vtp-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/vtp-dev
>
> _______________________________________________
> vtp-dev mailing list
> vtp-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/vtp-dev

Back to the top