[linuxtools-dev] Tmf Ctf Parser Scalability issues (1/3)

[Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>]

Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I
have come up with three points where scalability will be an issue. I am
sending 3 emails, each one describing one of the issues so we can
aggregate them more cohesively. First a primer, CTF is a file format
that has traces written in packets, the packets are parts of files in
streams, the streams are files in a directory.

Issue 1: Number of open files
CTF can have an unlimited amount of stream files. The parser opens all
of them simultaneously. This means if you have say over 1024 or whatever
your file limit is, files opened simultaneously, you will hit a wall. An
example of this would be the new Xeon Phi boards that have something
like 255 cores, if you have more than 4 channels per core, you're going
to hit this issue.

Proposed solution:
I envision fixing them by setting up a pool of files and when the limit
is hit, swapping files in and out, incurring a performance hit but still
allowing the program to run. Any objections?