 |
simdjson 5.0.2
Ridiculously Fast JSON
|
|
simdjson 5.0.2
Ridiculously Fast JSON
|
When serializing large databases, it is often better to write out many independent JSON documents, instead of one large monolithic document containing many records. The simdjson library provides high-speed access to files or streams containing multiple small JSON documents separated by ASCII white-space characters. Given an input such as
... you want to read the entries (individual JSON documents) as quickly and as conveniently as possible. Importantly, the input might span several gigabytes, but you want to use a small (fixed) amount of memory. Ideally, you'd also like the parallelize the processing (using more than one core) to speed up the process.
The main motivation for this piece of software is to achieve maximum speed and offer a better quality of life in parsing files containing multiple small JSON documents.
The JavaScript Object Notation (JSON) RFC7159 is a handy serialization format. However, when serializing a large sequence of values as an array, or a possibly indeterminate-length or never- ending sequence of values, JSON may be inconvenient.
Consider a sequence of one million values, each possibly one kilobyte when encoded – roughly one gigabyte. It is often desirable to process such a dataset incrementally without having to first read all of it before beginning to produce results.
Before parsing anything, simdjson first preprocesses the JSON text by identifying all structural indexes (i.e. the starting position of any JSON value, as well as any important operators like ,, :, ] or }) and validating UTF8. This stage is referred to stage 1. However, during this process, simdjson has no knowledge of whether parsed a valid document, multiple documents, or even if the document is complete. Then, to iterate through the JSON text during parsing, we use what we call a JSON iterator that will navigate through the text using these structural indexes. This JSON iterator is not visible though, but it is the key component to make parsing work.
Prior to iterate_many, most people who had to parse a multiline JSON file would proceed by reading the file line by line, using a utility function like std::getline or equivalent, and would then use the parse on each of those lines. From a performance point of view, this process is highly inefficient, in that it requires a lot of unnecessary memory allocation and makes use of the getline function, which is fundamentally slow, slower than the act of parsing with simdjson (more on this here).
Unlike the popular parser RapidJson, our DOM does not require the buffer once the parsing job is completed, the DOM and the buffer are completely independent. The drawback of this architecture is that we need to allocate some additional memory to store our ParsedJson data, for every document inside a given file. Memory allocation can be slow and become a bottleneck, therefore, we want to minimize it as much as possible.
To achieve a minimum amount of allocations, we opted for a design where we create only one parser object and therefore allocate its memory once, and then recycle it for every document in a given file. But, knowing that they often have largely varying size, we need to make sure that we allocate enough memory so that all the documents can fit. This value is what we call the batch size. As of right now, we need to manually specify a value for this batch size, it has to be at least as big as the biggest document in your file, but not too big so that it submerges the cached memory. The bigger the batch size, the fewer we need to make allocations. We found that 1MB is somewhat a sweet spot.
iterate_many, we return a document_stream which the user can iterate over to receive parsed documents.document_stream owns a document instance that keeps track of the current document position in the stream using a JSON iterator. To obtain a valid document, the document_stream returns a reference to its document instance.++ to read the next document, the JSON iterator moves to the start the next document.But how can we make use of threads if they are available? We found a pretty cool algorithm that allows us to quickly identify the position of the last JSON document in a given batch. Knowing exactly where the end of the last document in the batch is, we can safely parse through the last document without any worries that it might be incomplete. Therefore, we can run stage 1 on the next batch concurrently while parsing the documents in the current batch. Running stage 1 in a different thread can, in best cases, remove almost entirely its cost and replaces it by the overhead of a thread, which is orders of magnitude cheaper. Ain't that awesome!
Thread support is only active if thread supported is detected in which case the macro SIMDJSON_THREADS_ENABLED is set. You can also manually pass SIMDJSON_THREADS_ENABLED=1 flag to the library. Otherwise the library runs in single-thread mode.
You should be consistent. If you link against the simdjson library built for multithreading (i.e., with SIMDJSON_THREADS_ENABLED), then you should build your application with multithreading system (setting SIMDJSON_THREADS_ENABLED=1 and linking against a thread library).
A document_stream instance uses at most two threads: there is a main thread and a worker thread. That caps the gain near a factor of two.
Documents in a stream are independent, so you can cut the input yourself and parse the pieces on as many threads as you like. simdjson::slice_at does the cutting: it divides the input into blocks and moves each boundary forward to the next delimiter, so a document is never split.
Give each thread its own parser and set parser.threaded = false: the parallelism is across slices, so a stage-1 thread per parser would only oversubscribe.
A slice is empty when its block falls entirely inside one document, which happens only if that document is longer than block_size; skip it and continue. Iterate while i * block_size < view.size() rather than stopping at the first empty slice.
The delimiter must not occur inside a document: a line feed for NDJSON, or a record separator (0x1E) for RFC 7464.
Since we want to offer flexibility and not restrict ourselves to a specific file format, we support any file that contains any amount of valid JSON document, separated by one or more character that is considered whitespace by the JSON spec. Anything that is not whitespace will be parsed as a JSON document and could lead to failure.
Whitespace Characters:
If your documents are all objects or arrays, then you may even have nothing between them. E.g., [1,2]{"32":1} is recognized as two documents.
Some official formats **(non-exhaustive list)**:
Example:
See basics.md for an overview of the API.
When your input is a large NDJSON / JSON-lines file on disk, the most efficient way to feed iterate_many is to use simdjson::padded_memory_map. It returns a padded_string_view with the right amount of trailing padding, so you can hand it straight to iterate_many without ever copying the file contents into your own buffer.
padded_memory_map is available on POSIX systems (Linux, macOS, BSD, ...) by default. On Windows it is an opt-in feature with the following requirements:
-DSIMDJSON_ENABLE_MEMORY_FILE_MAPPING_ON_WINDOWS=ON, or — if you consume simdjson as a pre-built library — define SIMDJSON_ENABLE_MEMORY_FILE_MAPPING_ON_WINDOWS=1, raise NTDDI_VERSION to at least NTDDI_WIN10_RS4 (0x0A000005, Windows 10 version 1803), and add onecore.lib to your link line yourself. The Windows implementation uses the modern memory APIs CreateFileMapping2 / MapViewOfFile3, which are available starting with that version of Windows and are exported by onecore.lib.#include <windows.h> before #include "simdjson.h" in every translation unit where you want to use padded_memory_map. simdjson deliberately does not pull in <windows.h> itself, so the class is only declared when the Win32 types are already visible.If either requirement is not met on Windows, the padded_memory_map class is not declared at all and any code that references it fails to compile with an "unknown identifier" error. The availability of the class can be tested with the macro SIMDJSON_HAS_PADDED_MEMORY_MAP.
On POSIX, padded_memory_map uses mmap to map the file directly into memory with zero copies. On Windows (when enabled), it uses CreateFileMapping2 + MapViewOfFile3 for true zero-copy mapping whenever the file does not end within SIMDJSON_PADDING bytes of a page boundary; for those rare cases, it transparently falls back to reading the file into a heap-allocated padded buffer so that the returned view always has SIMDJSON_PADDING accessible zero bytes after the file content.
Important lifetime rule: the padded_string_view returned by map.view() is only valid while the padded_memory_map instance is alive, so keep map alive for as long as you are iterating the stream.
The file must not be modified while the memory map is in use. If you need a fully independent copy of the data, use simdjson::padded_string::load(...) instead.
If you prefer single-document parsing on a memory-mapped file, the same pattern applies to parser.iterate(...):
From jsonlines.org:
Better than CSV
CSV seems so easy that many programmers have written code to generate it themselves, and almost every implementation is different. Handling broken CSV files is a common and frustrating task. CSV has no standard encoding, no standard column separator and multiple character escaping standards. String is the only type supported for cell values, so some programs attempt to guess the correct types.
JSON Lines handles tabular data cleanly and without ambiguity. Cells may use the standard JSON types.
The biggest missing piece is an import/export filter for popular spreadsheet programs so that non-programmers can use this format.
Some users would like to know where the document they parsed is in the input array of bytes. It is possible to do so by accessing directly the iterator and calling its current_index() method which reports the location (in bytes) of the current document in the input stream. You may also call the source() method to get a std::string_view instance on the document and error() to check if there were any error.
Let us illustrate the idea with code:
This code will print:
Some users may need to work with truncated streams. The simdjson may truncate documents at the very end of the stream that cannot possibly be valid JSON (e.g., they contain unclosed strings, unmatched brackets, unmatched braces). After iterating through the stream, you may query the truncated_bytes() method which tells you how many bytes were truncated. If the stream is made of full (whole) documents, then you should expect truncated_bytes() to return zero.
Consider the following example where a truncated document ({"key":"intentionally unclosed string</tt>) containing 39 bytes has been left within the stream. In such cases, the first two whole documents are parsed and returned, and the <tt>truncated_bytes()</tt> method returns 39.
@icode{cpp}
auto json = R"([1,2,3] {"1":1,"2":3,"4":4} {"key":"intentionally unclosed string )"_padded; simdjson::ondemand::parser parser; simdjson::ondemand::document_stream stream; auto error = parser.iterate_many(json,json.size()).get(stream); if (error) { std::cerr << error << std::endl; return; } for(auto i = stream.begin(); i != stream.end(); ++i) { std::cout << i.source() << std::endl; } std::cout << stream.truncated_bytes() << " bytes "<< std::endl; // returns 39 bytes
This will print:
The value returned by truncated_bytes() is only meaningful when all of the following hold:
Outside these conditions, the value is not merely imprecise: it is arbitrary, and it may exceed size_in_bytes() or wrap around to a huge value such as 4294967295. An empty input (zero bytes) or an input made only of white space contains no document, and truncated_bytes() returns zero for it.
If you need to detect a truncated tail outside those conditions, track it yourself from the last document that parsed successfully: i.current_index() + i.source().size() is the offset just past the end of that document (see Reading a large stream in chunks for a complete example).
We discourage comma-separated documents. They are not a standard format, and simdjson supports them only with significant limitations (see Limitations of the comma-separated mode). If you control the format, prefer a standard such as JSON Lines/NDJSON (stream_format::whitespace_delimited or stream_format::newline_delimited) or RFC 7464 (stream_format::json_sequence).
To parse comma-separated documents like {"a":1},{"b":2},{"c":3}, use the stream_format::comma_delimited parameter:
Whitespace around the commas is allowed:
Nested commas inside objects and arrays are preserved:
Mixed document types are supported:
The comma-separated mode is a convenience for reading such inputs, not a validator. It works by removing the commas found at the top level (outside of any object or array) and then splitting the remaining bytes as if the documents were separated by white space. Top-level commas are identified by counting braces and brackets. This has the following consequences, which apply to both stream_format::comma_delimited and stream_format::comma_delimited_array.
{"a":1} {"b":2}, {"a":1}{"b":2}, 1 2 3 and "a""b" all produce separate documents without error. You cannot use this mode to check that an input is properly comma-separated.,,1,,2,) are skipped silently: they produce neither empty documents nor errors. This matches the legacy allow_comma_separated behavior.{"a":1},{"b":2},{"c":, iteration returns the first two documents and stops without reporting an error. truncated_bytes() does not tell you how many bytes were dropped in this mode (see Incomplete streams).{"a":[1,2},{"b":2},{"c":3}, the parser can no longer tell which commas are at the top level. Iteration may then end without returning any document and without reporting an error, whereas the whitespace-delimited mode reports TAPE_ERROR on the same kind of input. When the input spans several batches, you may instead get a CAPACITY error even though every document is small.batch_size. Furthermore, a run of documents separated only by white space (without commas) is treated like a single document: if such a run is longer than batch_size, you get a CAPACITY error, even though each document is small and the same input parses fine with stream_format::whitespace_delimited.Because of the silent cases above, if you must know that the whole input was consumed, do not rely on the absence of errors. Instead, compare the end of the last document you received (i.current_index() + i.source().size()) with the size of the input: only white space and commas should remain after it.
If your input arrives in pieces (from stdin, from a decompressor, from the network), see Reading a large stream in chunks.
The allow_comma_separated boolean parameter is deprecated. When set to true, it now internally maps to stream_format::comma_delimited.
The old single-batch limitation no longer applies - comma-delimited parsing now supports multi-batch processing and threading for optimal performance on large files.
RFC 7464 defines a format for streaming JSON values using ASCII Record Separator (RS, 0x1E) as a delimiter. Each JSON text is preceded by RS and optionally followed by ASCII Line Feed (LF, 0x0A).
Example input:
To parse JSON text sequences, use the stream_format::json_sequence parameter:
The stream_format enum has the following values:
stream_format::whitespace_delimited (default): Standard NDJSON/JSON Lines formatstream_format::json_sequence: RFC 7464 format with RS delimitersstream_format::comma_delimited: Comma-separated JSON documentsstream_format::comma_delimited_array: A single JSON array whose elements are iterated as comma-delimited documents (see below)stream_format::newline_delimited: NDJSON/JSON Lines where each document occupies exactly one line (documents are separated by line feeds and no document contains a raw line feed). Same inputs as whitespace_delimited, but the stronger guarantee lets ondemand iterate_many find the end of a document without walking it—including skipping an unread remainder without structure-validating that remainder. Use whitespace_delimited if unsure. See Parsing on many threads for multi-thread slicing with slice_at.The trailing LF after each JSON text is optional but recommended by the RFC for robustness.
Sometimes an input is a single, well-formed JSON array — [{"a":1},{"b":2},{"c":3}] — but you want to iterate its elements one at a time without materializing the whole array. Use stream_format::comma_delimited_array:
The parser strips the outer [ and ] plus any surrounding JSON whitespace (space, tab, LF, CR) and then behaves exactly like stream_format::comma_delimited over the remaining bytes. All comma-delimited features are inherited: multi-batch processing, threading, mixed scalar types, and nested commas preserved inside inner objects and arrays.
The whole array, from the opening [ to the closing ], must be in the buffer you pass to iterate_many: comma_delimited_array cannot be used on a piece of an array. If the array is too large to fit in memory, or it arrives in pieces (from stdin, from a decompressor such as gzip, from the network), use stream_format::comma_delimited instead, as described in Reading a large stream in chunks.
If the input does not start with [ and end with ] (ignoring white space), or if it is empty or all white space, iterate_many returns TAPE_ERROR. That is the only check made on the outer array. Content inside the array is not validated up front — individual document parse errors surface when you iterate, just like comma_delimited.
In particular, comma_delimited_array does not check that the input is a valid JSON array. Since the content is handled by the comma-separated mode, all of its limitations apply. For example, the invalid arrays [1,,2], [,1,2,] and [1 2] are all accepted and produce the documents 1 and 2. An input such as [1],[2] passes the initial check, and the problem is reported only during iteration, after the first document has been returned. If you need to validate the array, parse it as a single document with iterate or parse instead.
Positions reported via current_index() are relative to the stripped buffer (the bytes between [ and ]), not the original input, for consistency with the existing BOM-stripping behavior.
The fastest way to use iterate_many is to give it the whole input at once, e.g., a memory-mapped file. Sometimes that is not possible: the input might be compressed (gzip) and decompressed on the fly, or piped through stdin. You then read the input one chunk at a time into a buffer, and a document may straddle two chunks. The approach is the following:
iterate_many on the buffer and process the complete documents. An incomplete document at the end of the buffer is not returned.i.current_index() + i.source().size().Do not use truncated_bytes() for step 3: it does not work with comma_delimited (see Incomplete streams). And do not use comma_delimited_array on the chunks: it requires the complete array. Instead, use comma_delimited and blank out the opening [ and the closing ] yourself (replacing them with a space keeps the offsets unchanged).
There is one subtle case. When a chunk ends right after a number or an atom (e.g., 12 from ...,123,..., or tru from true), there is no way to know that the value is cut short. If your documents may be scalars, leave a scalar that touches the end of the chunk for the next round, as below. Documents that are objects or arrays do not have this problem.
For a complete, standalone project that streams a gzip-compressed file with bounded memory, see simdjson_compressed_demo. It uses NDJSON, where a document never contains a raw newline, so each chunk can simply be cut after its last newline. Comma-separated input has no such delimiter, so you need to find where the last complete document ends, as below.
The following function processes a comma-separated stream of documents or a single JSON array read from any std::istream:
A few remarks:
SIMDJSON_PADDING spare bytes past the data, hence the padded_string_view.iterate_many (here DEFAULT_BATCH_SIZE, 1 MB) must be larger than your largest document.source() could return the wrong range once the document had been partially consumed, or when a scalar document was followed by an incomplete one. A string document followed by an incomplete document could also be reported as a TAPE_ERROR when the padding bytes were not blank. If you must support these versions, call source() before you access the document and zero the padding bytes (std::memset(buffer.data() + len, 0, SIMDJSON_PADDING)) before calling iterate_many.In C++20, the standard introduced the notion of customization point. A customization point is a function or function object that can be customized for different types. It allows library authors to provide default behavior while giving users the ability to override this behavior for specific types.
A tag_invoke function serves as a mechanism for customization points. It is not directly part of the C++ standard library but is often used in libraries that implement customization points. The tag_invoke function is typically a generic function that takes a tag type and additional arguments. The first argument is usually a tag type (often an empty struct) that uniquely identifies the customization point (e.g., deserialization of custom types in simdjson). Users or library providers can specialize tag_invoke for their types by defining it in the appropriate namespace, often inline namespace.
You can deserialize you own data structures conveniently if your system supports C++20. When it is the case, the macro SIMDJSON_SUPPORTS_CONCEPTS will be set to 1 by the simdjson library.
Consider a custom class Car:
You may support deserializing directly from a JSON value or document to your own Car instance by defining a single tag_invoke function:
Importantly, the tag_invoke function must be inside the simdjson namespace. Let us explain each argument of tag_invoke function.
simdjson::deserialize_tag: it is the tag for Customization Point Object (CPO). You may often ignore this parameter. It is used to indicate that you mean to provide a deserialization function for simdjson.var: It receives automatically a simdjson value type (document, value, document_reference).Please see our main documentation (basics.md) under "Use `tag_invoke` for custom types (C++20)" for details about tag_invoke functions.
Given a stream of JSON documents, you can add them to a data structure such as a std::vector<Car> like so if you support exceptions:
Otherwise you may use this longer version for explicit handling of errors:
Performance tip: You will get better performance if you order the attributes (make, model) in the order they appear in the JSON document.
If you compile with a C++26 compiler that has P2996 static reflection enabled, simdjson detects it and can deserialize a stream of JSON documents directly into your own structures without writing any tag_invoke function. The library inspects the non-static public members of your type at compile time and produces the parsing code automatically.
Consider the same Car structure used in the C++20 example, but without any tag_invoke glue:
With C++26 static reflection enabled, you can iterate a stream of cars and push them into a std::vector<Car> directly:
This works for every stream_format value supported by iterate_many. The following examples each parse the same three cars, but laid out using a different streaming convention.
When the input is a single JSON array, you can stream its elements one at a time without materializing the entire array as a std::vector upfront:
In every case, the user-defined type (Car here) does not need a hand-written tag_invoke overload: the library generates the deserialization code from the type's public data members at compile time.
Performance tip: You will get better performance if you order the attributes (make, model) in the order they appear in the JSON document.
1.9.8