Re: [stem-dev] STEM Feature for single KML file animation -- Dev Questio

[James Kaufman <jhkauf@xxxxxxxxxx>]

Thank you Alex!!
We really appreciate your efforts here!
Would you be willing to create a stem
wiki page on kml/google earth integration documenting what you have been
able to do for others?

I'm not sure about your application
but another short term fix might be to run STEM and generate the kml files
as step 1 and play back on google earth as step 2.

or,... maybe get a massive server with
lots of cores so you can up the number of threads.

Anyway please let us know. Thank you
for looking at the KML display classes....


Best Regards,
Jamie

IBM Almaden Research Center, 650 Harry Rd.
San Jose, CA 95120-6099
email: jhkauf@xxxxxxxxxx
phone: (408) 927-2477  (tie 457-2477)

From:	alex@xxxxxxxxxxxxxx
To:	stem-dev@xxxxxxxxxxx,
Cc:	matt.davis@xxxxxxxxxxx, James Kaufman/Almaden/IBM@IBMUS
Date:	10/08/2014 08:18 AM
Subject:	re: STEM Feature for single KML file animation -- Dev Questions

---

Hello James,

No worries... I'm sure you're very busy.  I appreciate the time you
have 
taken to explain the parameters.

A.  Thanks.  I will see if I can extract the population data
using the 
Java code you've included.

B.  Also great info here.  I'll hunt down the Sequencer object
to get 
the start and end dates.

C. I wrote: "Why is my simulation's 'cycle' counter jumping around..."

You were absolutely correct... it turned out to be a threading problem.
 
I resolved it by doing the following:

   -Increased the STEM simulation step time from 250ms to 750ms
   -Set the STEM Process Priority to High in the OS
   -Allocated STEM 2GB/2GB of memory on launch
   -Slowed the new KML writer so it writes only once per 5 cycles
(rather 
than each cycle).
   -Reduced the complexity of the simulation... the max output of
the KML 
file can only be about 400MB.

I learned that if the single KML file is greater than about 400 MB, 
Google Earth will crash when loading it.

So unfortunately, this limits the 'granularity' of KML file animations.
 
There is probably a way around this... at present, I write the Polygons
to represent a county repeatedly (at each simulation step).  This
makes 
county-level data cause the KML animation file to be very large 
(hundreds of megs) .. There is lots of redundant data since the same 
polygon coordinates gets written over and over, repeated on each 
TimeSpan.

I see three workarounds to the large KML output file sizes...  The
key 
will be to eliminate the KML polygon data redundancy over TimeSpans...

C.1. Utilize image overlays (jpegs) for countries, regions and counties
etc... rather than having Google Earth draw  STEM's region polygons
over 
and over, use colored JPG overlays, shape files, something like that.

C.2. OR: Have GE import STEM county-level coordinates as a resource 
'as-needed', and use Google Earth's 'Feature' loading capability to 
cache only the data in view.  This way it doesn't all get loaded into
memory at once, and might eliminate redundancy.

C.3. OR: Quick and Dirty solution:  Reduce the number of points 
(resolution) that is used to represent county data when writing to the
KML file by Ramer–Douglas–Peucker algorithm.  So the county polygons
will not be so complex in the KML, and will take up less space.  ie.
Reduce a county polygon from ~50 coordinate pairs to 10 coordinate 
pairs.

http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm

D. Thanks.  I think I will try to use your standard methods (the Aspect
object, scaling, etc) for coloring the KML output polygons.  That
way 
everything is consistent across the STEM platform.  I want the KML
animation to support log scaling , just like the GUI does.

E. Okay. I'm pretty sure the STEM GE Servlet stuff is broken on my 
Windows 7 x64 environment.  For some reason, the web server has not
ever 
run locally (nothing is bound on port 8080 or whatever it is).  I
think 
this functionality might be broken or deprecated in the current versions
of Java and Google Earth.

The existing GE functionality in the latest STEM release was able to do
two major tasks (1) write a directory with one kml file per iteration --
numbered, or (2) launch Google Earth to load a control.kml file once per
iteration.   I don't think I was able to get the GE Servlet stuff
working at all, but I will test again.

F. > What is the difference between KmlDisplay,  KmlDisplayCustom,
and 
KmlDisplaySelection? Is any of this functionality deprecated?   "Some
of 
these are really old and may go back to older versions of google earth."

Okay, I may try to re-factor some of this code.  I think 
KmlDisplaySelection was meant to display a sub-set of a geographic area
to KML, but I don't know if it's working.  I think some of this code
can 
be pulled out, fixed, or refactored, especially if the Servlet stuff is
actually broken for everyone.

Anyway, I will keep digging and clean up the code as I go.  I'd like
to 
pull out or fix any deprecated features that are broken.

Also, I'm tempted to convert the KML write functionality to a Logger 
object, unless there is a reason not to do that...

Thanks,
Alex (aka Vincent R)


On 2014-10-07 16:38, James Kaufman wrote:
> Vincent,
> Sorry for the slow response.
> Re your questions
> 
>  > A. Where is the 'county' population data stored? For example,
how
> do I go from a key like 'US-NY-36039' (county identifier) to retrieve
> the original population in the county and the current population in
> the county at a time-step? (to determine number of fatalities , or
> total number of Infections as an absolute value rather than percent).
> 
> if you have the node you can use something like the following
>  EList<NodeLabel> labelList = node.getLabels();
>  FOR(NodeLabel nl:labelList) {
>  IF(nl INSTANCEOF PopulationLabel) {
>  PopulationLabel pl = (PopulationLabel) nl;
>  DOUBLE population = pl.getCurrentPopulationValue().getCount();
>  }
>  }
> You should be able to get the node from the graph. (ie
> graph.getNode(nodeURI))
> 
>  > B. Where can I find a simulation's actual 'start' and 'end'
dates
> in String or DateTime format? What object and where is it?
>  Get the Sequencer from the scenerio (from the simulation). The
> sequencer has the start and end dates.
> the sequencer has method getStartTime and getEndTime
> they return "STEMtime" objects but STEMtime.getDate() gives
you a
> Date.
> 
>  > C. Why is my simulation's 'cycle' counter jumping around
at the
> beginning? It starts at cycle=0, then goes to cycle=2, then to
> cycle=7,8,9,10 etc?
>  The gui runs in it's own thread so you are running either a
big or
> small simulation. On the menu bar you can go to >window>preferences.
> then >STEM>Simulation Managment and either increase the simulation
> delay so the gui keeps up. Or decrease it if you need.
> 
>  > D. Is there a 'proper' way to go about scaling an 'I' value
to a
> polygon color? Should I just use the Aspect object? Any tips on this?
> 
> I think it just depends what you are trying to show. We usually
> normalize I by the population but you might be interested in small
> values at the start of an epidemic so you might need a scale factor
or
> log scaling.
> 
>  > E. Are there any known bugs with the GE Servlet stuff?
I'm getting
> some Servlet exceptions (can't connect to local web server). Is this
> feature broken (the network pipe)? It's not
> important -- just curious.
> 
>  We have not run this in a very long time. Matt, any ideas on
this?
> 
>  > F. Why are there multiple classes which seem to be involved
in
> writing KML files... What is the difference between KmlDisplay,
> KmlDisplayCustom, and KmlDisplaySelection? Is any of this
> functionality deprecated? Some of these are really old and may go
back
> to older versions of google earth.
> 
> I'm sorry not to be more helpful....
> 
> Best Regards,
>  Jamie
> 
>  IBM Almaden Research Center, 650 Harry Rd.
>  San Jose, CA 95120-6099
>  email: jhkauf@xxxxxxxxxx
>  phone: (408) 927-2477 (tie 457-2477)