Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [mdmbl-dev] Reading MF4 files with org.eclipse.mdm.openatfx.mdf

Hi Viktor,

thanks for the fast answer.

Unfortunately, I can't see any extra files being created next to my test file. Also there seems to be no tabular lookup (`InstanceElement` is null when retrieving the "measurement_lookup").

I did some quick profiling with JVisualVM. Here is the result:

Do you maybe know why the AtfxCache takes up all the time?

Kind regards,

David

Am 29.05.2019 um 08:36 schrieb Stöhr, Viktor:

Hi David,

 

this may happen if your MDF file has content which can not easily be provided in an ODS context:

·         Validation bits for measured values (if existing) – here the library has to lookup each validation bit and export it in an ODS compliant way into a dedicated file „flags.bin“ or „flags_<rnd>.bin“ next to your mdf file.

o   You may disable this feature by setting „wrtite_flags_file“ to false. Now every accessed measured value will be interpreted as valid! [NOT RECOMMENDED]

·         Channels (CN-/CC-blocks) with a rational conversion rule – here we have the same case as with the validation bits. It is not possible to access those measured values in an ODS compliant way, therefore we have to lookup each raw value and calculate its physical representation. If such things happen you will find a file called „rational_conversion.calc“ next to your MDF file.

o   There is no configuration property to disable this behavior.

·         Last but not least we have conversion rules for tabular lookups. Here the library will have to read all mappings and store them in memory (this ist he way the OpenATFX driver does it) which results in increasing memory consumption. If your MDF file has at least one such case you will have an „measurement_lookup“ measurement.

o   There is no configuration property to disable this behavior.

 

I am sorry but there is no easy way to solve your perfomance issue, since it highly depends on the MDF file structure itself and the not fully compliant ATFX layer around it.

 

Regards,

Viktor

 

Von: mdmbl-dev-bounces@xxxxxxxxxxx <mdmbl-dev-bounces@xxxxxxxxxxx> Im Auftrag von David Kolb
Gesendet: Dienstag, 28. Mai 2019 17:30
An: mdmbl-dev@xxxxxxxxxxx
Betreff: Re: [mdmbl-dev] Reading MF4 files with org.eclipse.mdm.openatfx.mdf

 

Hi Viktor,

thanks for the detailed answer. That helped a lot. I managed to read the stored data from my test files. However, I noticed that depending on the size of the used .mdf file, the call to retrieve the `AoSession` from the `MDFConverter` takes a long time. For a 170MB .mf4 file containing 184 CG Block's, the following call:

AoSession session = reader.getAoSessionForMDF(orb, path, props);

took about 2m30s.

Is this expected? I guess it depends on the structure of the MDF file. I used the example code you attached in your last email. Should the file be read in a different way? Are there any properties that should be set to speed it up?

Thanks in advance for your help.

Kind Regards,

David

Am 11.03.2019 um 09:53 schrieb Stöhr, Viktor:

Hi David,

 

the project org.eclipse.mdm.openatfx.mdf generates an ATFX layer around a given MDF3 or MF4 file, which has to be sorted (please see the MDF specification for details regarding sorted MDF files).

Those ATFX files are usually used for imports into so called measurement systems which are driven by ODS servers. Therefore the reading of the measured data may be done by the ODS server, once the ATFX file has been imported.

 

However I made a quick test with one of the MF4 test files (bundled in this project) and was able to read the stored channel data.

 

1.      Is there an example/test case where the actual measurement data is read from the file?

o   See attached file (Example.java), I hope it gets you started J

o   Within the ATFX session you may find one or two measurements, the second one is created one the fly, once needed:

                                                                                      i.      „measurement“ will contain for each data group (DGBLOCK) with exactly one channel group „CGBLOCK“ a so called SubMatrix.

A ValueMatrix is spanned over the SubMatrix and is used to read the stored measured data per colum „CNBLOCK“.

                                                                                    ii.      „measurement_lookup“ will only be created for channels „CNBLOCK“s which have lookup tables (value to value mappings, value range tot ext mappings, etc.).

You have to follow the same rules to access its  data. The given example should print the values for both measurements. Try it with your test files…

o   The library may create additional files next to your MF4 file in the following cases:

                                                                                      i.            Validation bits of measured data are stored in a non ODS compliant way, therefore the library exports them into an additional binary file

                                                                                    ii.            Channel data with a rational conversion can not be represented in an ODS compliant way, therefore they are fullly calculated and stored in an additional binary file

2.      I'm confused about the keys that are used to retrieve ApplicationElements and InstanceElements from the ApplicationStruture. Is there some resource describing how to access the file and where the keys come from? I tried to compare it to the keys shown by the MDFValidator by VECTOR but they do not seem to match.

o   See the comments in example. In short, the keys you see in the MDFValidator are MDF specific and are reinterpreted and stored in an ODS compliant way.

3.      I'm unfamiliar with the ASAM OO-API and I couldn't find a guide to the API. Also I couldn't find sources and JavaDoc for the ods-5.3.0.jar to view this information in Eclipse while looking at the tests.

o   The full specification may be downloaded for ASAM members for free: https://www.asam.net/standards/detail/ods/older/

 

Additional notes:

·         The library may not support all possible channel configurations you may have in your environment (it is not a full implementation of the MDF specifications)

·         You may want to write a JNI wrapper for this library https://www.turbolab.de/mdf_libf.htm

·         You may use the other project org.eclipse.mdm.mdfsorter to sort unsorted MDF files

 

Regards

Viktor

 

Von: mdmbl-dev-bounces@xxxxxxxxxxx <mdmbl-dev-bounces@xxxxxxxxxxx> Im Auftrag von David Kolb
Gesendet: Donnerstag, 7. März 2019 14:08
An: mdmbl-dev@xxxxxxxxxxx
Betreff: Re: [mdmbl-dev] Reading MF4 files with org.eclipse.mdm.openatfx.mdf

 

Hi Angelika,

thanks for the quick answer.

Could you tell us which context you are working?

We are developing a data analysis tool called KNIME Analytics Platform. It has a visual interface where you can build so called workflows that model a data pipeline with operations on the data modelled as nodes. Some users from our community would be interested in a node to read MDF files in our tool and do some analysis on the measurement data. As our tool is written in Java, I was investigating possible Java libraries that we could use to read MDF and that we could integrate.

Are you interested in the openatfx.mdf code or in the underlaying openatfx driver?

Actually, I'm not 100% sure. If I understood correctly, openatfx.mdf provides functionality to read MDF files using the openatfx driver? However, I was having troubles reading the actual measurement data from the a MF4 file. I cloned this repository and had a look at the test cases. Specifically I have the following questions:

1) Is there an example/test case where the actual measurement data is read from the file?

2) I'm confused about the keys that are used to retrieve ApplicationElements and InstanceElements from the ApplicationStruture. Is there some resource describing how to access the file and where the keys come from? I tried to compare it to the keys shown by the MDFValidator by VECTOR but they do not seem to match.

3) I'm unfamiliar with the ASAM OO-API and I couldn't find a guide to the API. Also I couldn't find sources and JavaDoc for the ods-5.3.0.jar to view this information in Eclipse while looking at the tests.

Thanks for your help,

David

Am 07.03.2019 um 09:39 schrieb Angelika Wittek, openMDM(R):

Dear David,

yes this is the right place to ask questions about the Eclipse mdmbl project. Currently we are busy, so an answer will be delayed up to next week.
Are you interested in the openatfx.mdf code or in the underlaying openatfx driver? That library is hosted on sourgeforge by Christian:
https://sourceforge.net/projects/openatfx/

Could you tell us which context you are working? Our project is associated with the Eclipse openMDM Working Group and we are always interested about the use cases of our users.

Thanks
Angelika

-- 
openMDM(R) EWG product manager
 
mail: awittek.openMDM@xxxxxxxxxxxxxxxxxx
phone: +49 1577 1900802 

Am 06.03.19 um 10:21 schrieb David Kolb:

Dear MDM|BL Development Team,

I'm a working student at the company KNIME and I'm currently investigating Java libraries which provide functionality to read MF4 files. We'd like to include this functionality into our open source software.

After some search, I stumbled upon: https://git.eclipse.org/c/mdmbl/org.eclipse.mdm.openatfx.mdf.git/about/ which lead me to this mailing list in the included NOTICE.txt of the Git repository. As the author of the project is marked as Christian Rechner, AUDI AG (here), I was confused who to contact regarding some questions about the usage of the library. Specifically, I am having troubles reading the actual numeric content of some MF4 files, finding examples, and finding documentation about the ASAM OO-API.

If this is not the right place to ask these questions, could you maybe point me into the right direction? Otherwise, I would be happy to elaborate on my questions and very thankful about your support.

Kind regards,

David Kolb




_______________________________________________
mdmbl-dev mailing list
mdmbl-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mdmbl-dev




_______________________________________________
mdmbl-dev mailing list
mdmbl-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mdmbl-dev



_______________________________________________
mdmbl-dev mailing list
mdmbl-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mdmbl-dev

_______________________________________________
mdmbl-dev mailing list
mdmbl-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mdmbl-dev

Back to the top