Parsing a large number of files in different RDF serialisations

[fkraeutli]

I had two issues to solve when indexing RDF data with Qlever:

1. The number of files exceeds what can be passed as a command line argument, i.e. cat *.ttl would fail
2. I also wanted to index some files serialised as RDF/XML

I solved this by using the find command to concatenate the files and passing the files through the Raptor RDF Utility.

The relevant lines in my Qleverfile are:

INPUT_FILES     = data/ttl/main data/ttl/additional/external schemas
CAT_INPUT_FILES = find ${INPUT_FILES} -type f -regex ".*\.\(ttl\|rdf\|rdfs\)" -exec rapper -gq -o turtle {} \;
SETTINGS_JSON   = { "num-triples-per-batch": 1000000, "parallel-parsing": false }

(I encountered problems with the parallel-parsing default so I switched it to false)

[aindlq]

Hi @fkraeutli, parallel parsing works only for ntriples not ttl. Or to be more precise ttl that looks like ntriples, so you have a single triple per line with dot in the end, but can use prefixes. We just use ntriples and it works great.

[fkraeutli]

Ah I didn’t get what the error message meant by ‘well behaved’ turtle files. That’s good to know! In that case, indexing could potentially be sped up by letting rapper output ntriples.

[ktk]

While I don't disagree that at some point QLever should provide support for a wider range of RDF serializations, I don't see that as a priority.

I had a discussion last week with Pierre-Antoine Champin from W3C, he plans to create a Rust RDF community group where we could coordinate work & development on "binary" RDF support front. As in binary executable, suitable for scripting and fast.

The use case was converting 60k JSON files in JSON-LD to RDF in a form that it can be ingested by QLever and the main problem is that if this is batched with riot from Jena, it takes way too long for each file to startup the JVM for an operation that is technically super fast. Also some of these JSON files were creating invalid RDF (emoticons in IRIs...) as JSON accepts pretty much everything you put in there.

So the TL;DR statement here is: We should definitely improve native RDF tooling so we have better flexibility in scripting. I use rapper and serdi on cli on a regular base and riot if I have to, for example for JSON-LD. But it would be nice to have that all in one cli tool that supports all of those serializations in one. Like riot but a bit more cli friendly and without the JVM overhead.