Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [stem-dev] STEM Feature for single KML file animation -- Dev Questions

I've fix most of the issues in the second recent reply with work-arounds. The original questions I asked in the first post I'm still unclear on.

To get the KML simulation to work without skipping too much, I did the following:

(1) Increased STEM memory to 2048MB / 2048MB
(2) Increase Win7 Process Priority on STEM Java.exe to High from Normal.
(3) Decreased iteration speed from 250ms / cycle to 1000ms / cycle in config.
(4) Set KML to only log once per 2nd cycle.
(5) Additional workaround might involve calling work.join() in GEJob to wait until IO returns

With the above first 4 changes, I'm getting very consistent results with 50% smaller kml files.

-vr

On 2014-10-04 14:40, vruslan@xxxxxxxxxxxxxx wrote:
On a related note... I have noticed my neat STEM KML animation file is
skipping cycles, and not just at the beginning.

Some of the cycle skipping seems related to Thread IO Bottleneck (we
ARE writing up to many megs of KML data per simulation 'day'). . . And
yes, Some cycle skipping seems related to low memory. . .

But a decent amount of the cycle skipping seems to be somewhat random...

For example, here is some of my output 'operon.kml' (a 250MB
uncompressed KML animation of Ebola in the U.S.)...  We skip from Day
293 to Day 300...  Even excluding IO or memory issues, my KML skips
from Day 293 to 296.  Any ideas?

Operon.kml:

<Placemark id="pm0">
<name>Clay County, 46027, SD US-SD-46027 I: 8.886536660398633E-5</name>
<snippet>empty</snippet>
<description><![CDATA[test description here]]></description>
<TimeSpan>
<!-- KML Timespan supposed to be start day and end day seperated by a
one-day interval.  Paramter comes from the STEM cycle variable . . .
For some reason we skip over 6 days here. -->
<begin>293</begin>
<end>300</end>
</TimeSpan>
<styleUrl>#Style1</styleUrl>
<MultiGeometry>
<Polygon id="polygon1">
<altitudeMode>clampToGround</altitudeMode>
<outerBoundaryIs>
<LinearRing>

Notice it jumps from Day 293 to 300 in my KML output . . . I'm
expecting to see the STEM cycle go up in intervals of one, not
randomly six at a time. . . Here is the associated console debug
output . .

STEM Debug Console Output:

StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEInterface: Done. cycle=291
GEJob: c:\kml2/operon.kml I 291 false false
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=292
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   932096K
GE: Memory Used:  548124K
GE: Percent Used: 0.5880561290092436
GEInterface.simulationChanged: Operon KML Option
GEInterface.simulationChanged: entry folder != null
StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEJob: Skip KML generation for cycle 292          *** WHY does Day 292
fail?  Thread bottleneck, right?
GEInterface: Done. cycle=292
KmlDisplay: write c:\kml2/operon.kml with 2084 entries.
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=293
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   932096K
GE: Memory Used:  794903K
GE: Percent Used: 0.8528129579732131
GE: Memory used after GC:  762479K
GE: Percent Used: 0.7508972083336616
GEInterface.simulationChanged: Operon KML Option
GEInterface.simulationChanged: entry folder != null
StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEInterface: Done. cycle=293
GEJob: c:\kml2/operon.kml I 293 false false      *** Here, Day 293
succeeds.  No Thread IO Bottleneck.
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=296  *** Now why are
we skipping to Day 296?  Is this my code or STEM ?
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   997888K
GE: Memory Used:  660283K
GE: Percent Used: 0.6616812442253038
GEInterface.simulationChanged: Operon KML Option
GEInterface.simulationChanged: entry folder != null
StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEJob: Skip KML generation for cycle 296         *** But Writing Day
296 KML fails.  Thread bottleneck?
GEInterface: Done. cycle=296
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=297
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   1012992K
GE: Memory Used:  659538K
GE: Percent Used: 0.6510798579850581
GEInterface.simulationChanged: Operon KML Option
GEInterface.simulationChanged: entry folder != null
StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEJob: Skip KML generation for cycle 297       *** Writing Day 297 KML
fails.  Thread bottleneck?
GEInterface: Done. cycle=297
KmlDisplay: write c:\kml2/operon.kml with 2208 entries.
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=298
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   1014272K
GE: Memory Used:  594078K
GE: Percent Used: 0.5857188933910233
GEInterface.simulationChanged: Operon KML Option
GEInterface.simulationChanged: entry folder != null
StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEJob: Skip KML generation for cycle 298       *** Writing Day 298 KML
fails.  Thread bottleneck?
GEInterface: Done. cycle=298
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=299
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   1014272K
GE: Memory Used:  789632K
GE: Percent Used: 0.7785213726569401
GEInterface.simulationChanged: Operon KML Option
GEInterface.simulationChanged: entry folder != null
StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEJob: Skip KML generation for cycle 299        *** Writing Day 299
KML fails.  Thread bottleneck?
GEInterface: Done. cycle=299
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=300
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   1006336K
GE: Memory Used:  584730K
GE: Percent Used: 0.581048523430544
GEInterface.simulationChanged: Operon KML Option
GEInterface.simulationChanged: entry folder != null
StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEInterface: Done. cycle=300
GEJob: c:\kml2/operon.kml I 300 false false          *** But Day 300
gets written as KML.  We skip from Day 293 to 300 in the animation (no
changes logged on days 294 to 299).
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=301
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   1006336K
GE: Memory Used:  835396K
GE: Percent Used: 0.8301364740504166
KmlDisplay: write c:\kml2/operon.kml with 2315 entries.
GE: Memory used after GC:  860445K
GE: Percent Used: 0.8519393856115542
GEInterface.simulationChanged: Process for cycle 301 skipped. Memory low
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=305
GEInterface.simulationChanged: Entered processCycle function
GE: Memory Max:   974272K
GE: Memory Used:  619712K
GE: Percent Used: 0.63607766234686
GEInterface.simulationChanged: Operon KML Option
GEInterface.simulationChanged: entry folder != null
StemKml: Output file=c:\kml2/operon.kml
GEInterface.simulationChanged: calling storeAspects()
GEInterface: GenerateKML for id: SIM[0] aspect I
GEInterface.simulationChanged: calling generateKML()
GEJob: Scheduling generation of KML file c:\kml2/operon.kml
GEInterface: Done. cycle=305
GEJob: c:\kml2/operon.kml I 305 false false
GEInterface: SIM[0] State=COMPLETED_CYCLE cycle=306
GEInterface.simulationChanged: Entered processCycle function


Anyway, I'm just looking for some clarity on the expected behavior of
the stem cycle variable.   Does it skip days when there are no changes
to the model's population?  Any ideas?  Could two threads (KmlDisplay
and KmlDisplayCustom) be in a race condition or something?

Thanks

On 2014-10-04 12:55, vruslan@xxxxxxxxxxxxxx wrote:
Hello all,

I've gotten some of the new KML features working in STEM.  The
interesting part is you can import STEM animations directly to Google
Earth -- without Servlets, webserver, etc.  All you need is the single
KML output file.

Screenshot of new STEM KML animation Feature:
http://www.operonlabs.com/sites/default/files/field/image/stem-kml1.png

====

Questions for STEM Developers:

A. Where is the 'county' population data stored?  For example, how do
I go from a key like 'US-NY-36039' (county identifier) to retrieve the
original population in the county and the current population in the
county at a time-step?  (to determine number of fatalities , or total
number of Infections as an absolute value rather than percent).

B. Where can I find a simulation's actual 'start' and 'end' dates in
String or DateTime format?  What object and where is it?

C. Why is my simulation's 'cycle' counter jumping around at the
beginning?  It starts at cycle=0, then goes to cycle=2, then to
cycle=7,8,9,10 etc?

D. Is there a 'proper' way to go about scaling an 'I' value to a
polygon color?  Should I just use the Aspect object?  Any tips on
this?

E. Are there any known bugs with the GE Servlet stuff?  I'm getting
some Servlet exceptions (can't connect to local web server).  Is this
feature broken (the network pipe)?  It's not important -- just
curious.

F. Why are there multiple classes which seem to be involved in writing
KML files... What is the difference between KmlDisplay,
KmlDisplayCustom, and KmlDisplaySelection?   Is any of this
functionality deprecated?

====

Here's what I've done on my local dev branch.

1. Added an Eclipse Google Earth logging option to write a STEM
simulation to a single KML file.  Now natively supports animations by
double-clicking on KML file.

2. The single KML file is iteratively written during a simulation run.
 The KML file is in a newer format , written with new functions, so it
is closely compatible with the Google Earth XML schema.

3. The KML file includes Placemarks which are labeled at the County
level with time-stamped SEIR data.

4. KML Polygon Coloring (manually coded styles) works in Google Earth
... TODO:  New KML coloring needs to be connected with Aspect prefs.

5. TimeSpan was added for each Placemark.   Thus, once loaded into
Google Earth, the KML file result of a STEM Simulation can be 'played'
back at different speeds.  Currently, the TimeSpan is just the
previous to current iteration number.

6. The functionality works pretty well.  The only issue is that a full
2 Year Continental U.S. Ebola simulation at county level results in
lots of polygons,  and a HUGE KML file (>250 MB).  Compressing the KML
to KMZ with winzip reduces the file size to about 10-30 MB, which is
reasonable.  There is, however, lots of data redundancy because the
County-level Polygon coordinates need to be written (duplicated) for
each time step in the simulation.  This is a limitation of KML.

Even still, I was able to import a full continental US level
simulation into Google Earth without too much of a problem on a modern
core i7 laptop.
_______________________________________________
stem-dev mailing list
stem-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/stem-dev
_______________________________________________
stem-dev mailing list
stem-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/stem-dev


Back to the top