| Re: [rdf4j-dev] Question about parsing large files |
|
Hi Dan,
I agree with Håvard that this would be best discussed on github. Because the current
emails are rather vague and therefore kinda hard to answer.
RIO parsers are streaming, so something else is going on meaning we need a lot more
details.
Please open an discussion and give us some code snippets.
Regards,
Jerven
From: rdf4j-dev <rdf4j-dev-bounces@xxxxxxxxxxx> on behalf of Dan S via rdf4j-dev <rdf4j-dev@xxxxxxxxxxx>
Sent: 17 January 2024 21:38 To: rdf4j developer discussions <rdf4j-dev@xxxxxxxxxxx> Cc: Dan S <danielms853@xxxxxxxxx> Subject: Re: [rdf4j-dev] Question about parsing large files Hi Håvard,
To clarify, we think this email should be related to the internal rdf4j development. The parser takes in a filename/handle and returns an iterator. A major selling point of the iterator abstraction is that it should enable extraction of elements
without necessarily loading all of them into memory at once. I guess what we're asking is whether the parser can be enhanced so that it can read larger files than can be loaded in memory (maybe this is something we may also be able to help with). Presumably
if a large enough lookahead is provided it might be able to parse out the next few triples, and as the iterator is advanced through it can free up triples that have already been read. This might be especially useful as the parser is a standalone component
of RDF4J which can be imported via maven/gradle into other projects. It would be super useful to find a parser for very large files, and we were wondering if the rdf4j one could eventually become the solution.
Thanks,
Dan
On Wed, Jan 17, 2024, 20:00 Håvard Ottestad via rdf4j-dev <rdf4j-dev@xxxxxxxxxxx> wrote:
|