 |
simdjson
3.11.0
Ridiculously Fast JSON
|
|
simdjson
3.11.0
Ridiculously Fast JSON
|
An overview of what you need to know to use simdjson, with examples.
Support for AVX-512 require a processor with AVX512-VBMI2 support (Ice Lake or better, AMD Zen 4 or better) under a 64-bit system and a recent compiler (LLVM clang 6 or better, GCC 8 or better, Visual Studio 2019 or better). You need a correspondingly recent assembler such as gas (2.30+) or nasm (2.14+): recent compilers usually come with recent assemblers. If you mix a recent compiler with an incompatible/old assembler (e.g., when using a recent compiler with an old Linux distribution), you may get errors at build time because the compiler produces instructions that the assembler does not recognize: you should update your assembler to match your compiler (e.g., upgrade binutils to version 2.30 or better under Linux) or use an older compiler matching the capabilities of your assembler.
We test the library on a big-endian system (IBM s390x with Linux).
To include simdjson, copy simdjson.h and simdjson.cpp into your project. Then include it in your project with:
Under most systems, you can compile with:
Note:
simdjson.h file along with the source file simdjson.cpp directly in your project, as they are part of every release as assets. In this manner, you only have to compile simdjson.cpp as any other source file: it works well in every development environment. However, you may also use simdjson as a git submodule (example), using FetchContent (example), with ExternalProject_Add (example) or with CPM (example).c++ -std=c++11 myproject.cpp simdjson.cpp).-march-native) if you want your binaries to run everywhere.You can install the simdjson library on your system or in your project using multiple package managers such as MSYS2, the conan package manager, vcpkg, brew, the apt package manager (debian-based Linux systems), the FreeBSD package manager (FreeBSD), and so on. E.g., we provide an complete example with vcpkg that works under Windows. Visit our wiki for more details.
The following Linux distributions provide simdjson packages: Alpine, RedHat, Rocky Linux, Debian, Fedora, and Ubuntu.
You can include the simdjson library as a CMake dependency by including the following lines in your CMakeLists.txt:
You should provide GIT_TAG with the release you need. If you omit GIT_TAG ..., you will work from the main branch of simdjson: we recommend that if you are working on production code, you always work from a release.
Elsewhere in your project, you can declare dependencies on simdjson with lines such as these:
We recommend CMake version 3.15 or better.
See our CMake demonstration. It works under Linux, FreeBSD, macOS and Windows (including Visual Studio).
The CMake build in simdjson can be tailored with a few variables. You can see the available variables and their default values by entering the cmake -LA command.
Users are discouraged from building production code from the project's main branch. The main branch is used for development: it may contain new features but also additional bugs.
Users should pick a release. They should also access the documentation matching the release that they have chosen. Note that new features may be added over time.
Our releases are tagged using semantic versioning: the tags are made of three numbers prefixed by the letter v and separated by periods.
You can always find the latest release at the following hyperlink:
https://github.com/simdjson/simdjson/releases/latest/
The archive you download at this location contains its own corresponding documentation.
You can also choose to browse a specific version of the documentation and the code using GitHub, by appending the version number to the hyperlink, like so:
https://github.com/simdjson/simdjson/blob/vx.y.z/doc/basics.md
where x.y.z should correspond to the version number you have chosen.
The simdjson library allows you to navigate and validate JSON documents (RFC 8259). As required by the standard, your JSON document should be in a Unicode (UTF-8) string. The whole string, from the beginning to the end, needs to be valid: we do not attempt to tolerate bad inputs before or after a document.
For efficiency reasons, simdjson requires a string with a few bytes (simdjson::SIMDJSON_PADDING) at the end, these bytes may be read but their content does not affect the parsing. In practice, it means that the JSON inputs should be stored in a memory region with simdjson::SIMDJSON_PADDING extra bytes at the end. You do not have to set these bytes to specific values though you may want to if you want to avoid runtime warnings with some sanitizers. Advanced users may want to read the section Free Padding in our performance notes.
The simdjson library offers a tree-like API, which you can access by creating a ondemand::parser and calling the iterate() method. The iterate method quickly indexes the input string and may detect some errors. The following example illustrates how to get started with an input JSON file ("twitter.json"):
You can also create a padded string—and call iterate():
If you have a buffer of your own with enough padding already (SIMDJSON_PADDING extra bytes allocated), you can use padded_string_view to pass it in:
The simdjson library will also accept std::string instances. If the provided reference is non-const, it will allocate padding as needed.
You can copy your data directly on a simdjson::padded_string as follows:
Or as follows...
You can then parse the JSON data from the simdjson::padded_string instance:
Whenever you pass an std::string reference to parser::iterate, the parser will access the bytes beyond the end of the string but before the end of the allocated memory (std::string::capacity()). If you are using a sanitizer that checks for reading uninitialized bytes or std::string's container-overflow checks, you may encounter sanitizer warnings. You can safely ignore these warnings. Or you can call simdjson::pad(std::string&) to pad the string with SIMDJSON_PADDING spaces: this function returns a simdjson::padding_string_view which can be be passed to the parser's iterator function:
We recommend against creating many std::string or many std::padding_string instances in your application to store your JSON data. Consider reusing the same buffers and limiting memory allocations.
By default, the simdjson library throws exceptions (simdjson_error) on errors. We omit try-catch clauses from our illustrating examples: if you omit try-catch in your code, an uncaught exception will halt your program. It is also possible to use simdjson without generating exceptions, and you may even build the library without exception support at all. See Error handling for details.
Some users may want to browse code along with the compiled assembly. You want to check out the following lists of examples:
Windows-specific: Windows users who need to read files with non-ANSI characters in the name should set their code page to UTF-8 (65001). This should be the default with Windows 11 and better. Further, they may use the AreFileApisANSI function to determine whether the filename is interpreted using the ANSI or the system default OEM codepage, and they may call SetFileApisToOEM accordingly.
The simdjson library relies on an approach to parsing JSON that we call "On-Demand". A document is not a fully-parsed JSON value; rather, it is an iterator over the JSON text. This means that while you iterate an array, or search for a field in an object, it is actually walking through the original JSON text, merrily reading commas and colons and brackets to make sure you get where you are going. This is the key to On-Demand's performance: since it's just an iterator, it lets you parse values as you use them. And particularly, it lets you skip values you do not want to use. On-Demand is also ideally suited when you want to capture part of the document without parsing it immediately (e.g., see General direct access to the raw JSON string).
We refer to "On-Demand" as a front-end component since it is an interface between the low-level parsing functions and the user. It hides much of the complexity of parsing JSON documents.
For code safety, you should keep (1) the parser instance, (2) the input string and (3) the document instance alive throughout your parsing. Additionally, you should follow the following rules:
parser may have at most one document open at a time, since it holds allocated memory used for the parsing.document instance per JSON document. Thus, if you must pass a document instance to a function, you should avoid passing it by value: choose to pass it by reference instance to avoid the copy. In any case, the document class does not have a copy constructor.During the iterate call, the original JSON text is never modified–only read. After you are done with the document, the source (whether file or string) can be safely discarded.
For best performance, a parser instance should be reused over several files: otherwise you will needlessly reallocate memory, an expensive process. It is also possible to avoid entirely memory allocations during parsing when using simdjson. See our performance notes for details.
If you need to have several documents active at once, you should have several parser instances.
The simdjson library builds on compilers supporting the C++11 standard. It is also a strict requirement: we have no plan to support older C++ compilers.
We represent parsed Unicode (UTF-8) strings in simdjson using the std::string_view class. It avoids the need to copy the data, as would be necessary with the std::string class. It also avoids the pitfalls of null-terminated C strings. It makes it easier for our users to copy the data into their own favorite class instances (e.g., alternatives to std::string).
A std::string_view instance is effectively just a pointer to a region in memory representing a string. In simdjson, we return std::string_view instances that either point within the input string you parsed (see General direct access to the raw JSON string), or to a temporary string buffer inside our parser class instances that is valid until the parser object is destroyed or you use it to parse another document. When using std::string_view instances, it is your responsibility to ensure that std::string_view instance does not outlive the pointed-to memory (e.g., either the input buffer or the parser instance). Furthermore, some operations reset the string buffer inside our parser instances: e.g., when we parse a new document. Thus a std::string_view instance is often best viewed as a temporary string value that is tied to the document you are parsing. At the cost of some memory allocation, you may convert your std::string_view instances for long-term storage into std::string instances: std::string mycopy(view) (C++17) or std::string mycopy(view.begin(), view.end()) (prior to C++17). For convenience, we also allow storing an escaped string directly into an existing string instance.
The std::string_view class has become standard as part of C++17 but it is not always available on compilers which only supports C++11. When we detect that string_view is natively available, we define the macro SIMDJSON_HAS_STRING_VIEW.
When we detect that it is unavailable, we use string-view-lite as a substitute. In such cases, we use the type alias using string_view = nonstd::string_view; to offer the same API, irrespective of the compiler and standard library. The macro SIMDJSON_HAS_STRING_VIEW will be undefined to indicate that we emulate string_view.
Some users prefer to use non-JSON native encoding formats such as UTF-16 or UTF-32. Users may transcode the UTF-8 strings produced by the simdjson library to other formats. See the simdutf library, for example.
We recommend that you first compile and run your code in Debug mode:
_DEBUG macro defined,__OPTIMIZE__ macro undefined.The simdjson code will set SIMDJSON_DEVELOPMENT_CHECKS=1 in debug mode. Alternatively, you can set the macro SIMDJSON_DEVELOPMENT_CHECKS to 1 prior to including the simdjson.h header to enable these additional checks: just make sure you remove the definition once your code has been tested. When SIMDJSON_DEVELOPMENT_CHECKS is set to 1, the simdjson library runs additional (expensive) tests on your code to help ensure that you are using the library in a safe manner.
Once your code has been tested, you can then run it in Release mode: under Visual Studio, it means having the _DEBUG macro undefined, and, for other compilers, it means setting __OPTIMIZE__ to a positive integer. You can also forcefully disable these checks by setting SIMDJSON_DEVELOPMENT_CHECKS to 0. Once your code is tested, we further encourage you to define NDEBUG in your Release builds to disable additional runtime testing and get the best performance.
Once you have a document (simdjson::ondemand::document), you can navigate it with idiomatic C++ iterators, operators and casts. Besides the document instances and native types (double, uint64_t, int64_t, bool), we also access Unicode (UTF-8) strings (std::string_view), objects (simdjson::ondemand::object) and arrays (simdjson::ondemand::array). We also have a generic ephemeral type (simdjson::ondemand::value) which represents a potential array or object, or scalar type (double, uint64_t, int64_t, bool, null, string) inside an array or an object. Both generic types (simdjson::ondemand::document and simdjson::ondemand::value) have a type() method returning a json_type value describing the value (json_type::array, json_type::object, json_type::number, json_type::string, json_type::boolean, json_type::null). A generic value (simdjson::ondemand::value) is only valid temporarily, as soon as you access other values, other keys in objects, etc. it becomes invalid: you should therefore consume the value immediately by converting it to a scalar type, an array or an object.
Advanced users who need to determine the number types (integer or float) dynamically, should review our section dynamic number types. Indeed, we have an additional ondemand::number type which may represent either integers or floating-point values, depending on how the numbers are formatted. floating-point values followed by an integer.
We invite you to keep the following rules in mind:
document instance should remain in scope: it is your "iterator" which keeps track of where you are in the JSON document. By design, there is one and only one document instance per JSON document.The simdjson library makes generous use of std::string_view instances. If you are unfamiliar with std::string_view in C++, make sure to read the section on std::string_view. They behave much like an immutable std::string but they require no memory allocation. You can create a std::string instance from a std::string_view when you need it.
The following specific instructions indicate how to use the JSON when exceptions are enabled, but simdjson has full, idiomatic support for users who avoid exceptions. See the simdjson error handling documentation for more.
iterate, the document is quickly indexed. If it is not a valid Unicode (UTF-8) string or if there is an unclosed string, an error may be reported right away. However, it is not fully validated. On-Demand only fully validates the values you use and the structure leading to it. It means that at every step as you traverse the document, you may encounter an error. You can handle errors either with exceptions or with error codes.double(element). This works for std::string_view, double, uint64_t, int64_t, bool, ondemand::object and ondemand::array. We also have explicit methods such as get_string(), get_double(), get_uint64(), get_int64(), get_bool(), get_object() and get_array(). After a cast or an explicit method, the number, string or boolean will be parsed, or the initial { or [ will be verified for ondemand::object and ondemand::array. An exception may be thrown if the cast is not possible: there error code is simdjson::INCORRECT_TYPE (see Error handling). Importantly, when getting an ondemand::object or ondemand::array instance, its content is not validated: you are only guaranteed that the corresponding initial character ({ or [) is present. Thus, for example, you could have an ondemand::object instance pointing at the invalid JSON { "this is not a valid object" }: the validation occurs as you access the content. The get_string() returns a valid UTF-8 string, after unescaping characters as needed: unmatched surrogate pairs are treated as an error unless you pass true (get_string(true)) as a parameter to get replacement characters where errors occur. If you somehow need to access non-UTF-8 strings in a lossless manner (e.g., if you strings contain unpaired surrogates), you may use the get_wobbly_string() function to get a string in the WTF-8 format. When calling get_uint64() and get_int64(), if the number does not fit in a corresponding 64-bit integer type, it is also considered an error. When parsing numbers or other scalar values, the library checks that the value is followed by an expected character, thus you may get a number parsing error when accessing the digits as an integer in the following strings: {"number":12332a, {"number":12332\0, {"number":12332 (the digits appear at the end). We always abide by the RFC 8259 JSON specification so that, for example, numbers prefixed by the + sign are in error.IMPORTANT NOTE: values can only be parsed once. Since documents are iterators, once you have parsed a value (such as by casting to double), you cannot get at it again. It is an error to call
get_string()twice on an object (or to cast an object twice tostd::string_view).
* Array Iteration: To iterate through an array, use for (auto value : array) { ... }. This will step through each value in the JSON array.
To iterate through an array, you should be at the beginning of the array: to warn you, an OUT_OF_ORDER_ITERATION error is generated when development checks are active. If you need to access an array more than once, you may call reset() on it although we discourage this practice. Keep in mind that you should consume each value at most once.
If you know the type of the value, you can cast it right there, too! for (double value : array) { ... }.
You may also use explicit iterators: for(auto i = array.begin(); i != array.end(); i++) {}. You can check that an array is empty with the condition auto i = array.begin(); if (i == array.end()) {...}.
Object Iteration: You can iterate through an object's fields, as well: for (auto field : object) { ... }. You may also use explicit iterators : for(auto i = object.begin(); i != object.end(); i++) { auto field = *i; .... }. You can check that an object is empty with the condition auto i = object.begin(); if (i == object.end()) {...}.
field.unescaped_key() will get you the unescaped key string as a std::string_view instance. E.g., the JSON string "\u00e1" becomes the Unicode string รก. Optionally, you pass true as a parameter to the unescaped_key method if you want invalid escape sequences to be replaced by a default replacement character (e.g., \ud800\ud801\ud811): otherwise bad escape sequences lead to an immediate error.field.escaped_key() will get you the key string as as a std::string_view instance, but unlike unescaped_key(), the key is not processed, so no unescaping is done. E.g., the JSON string "\u00e1" becomes the Unicode string \u00e1. We expect that escaped_key() is faster than field.unescaped_key().field.value() will get you the value, which you can then use all these other methods on.To iterate through an object, you should be at the beginning of the object: to warn you, an OUT_OF_ORDER_ITERATION error is generated when development checks are active. If you need to access an object more than once, you may call reset() on it although we discourage this practice. Keep in mind that you should consume each value at most once.
array[1]).object["foo"]. This will scan through the object looking for the field with the matching string, doing a character-by-character comparison. It may generate the error simdjson::NO_SUCH_FIELD if there is no such key in the object, it may throw an exception (see Error handling). For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do not do object["foo"] followed by object["foo"] with the same object instance. For best performance, you should try to query the keys in the same order they appear in the document. If you need several keys and you cannot predict the order they will appear in, it is recommended to iterate through all keys for(auto field : object) {...}. Generally, you should not mix and match iterating through an object (for(auto field : object) {...}) and key accesses (object["foo"]): if you need to iterate through an object after a key access, you need to call reset() on the object. Whenever you call reset(), you need to keep in mind that though you can iterate over the array repeatedly, values should be consumedonly once (e.g., repeatedly calling unescaped_key() on the same key is forbidden). Keep in mind that On-Demand does not buffer or save the result of the parsing: if you repeatedly access object["foo"], then it must repeatedly seek the key and parse the content. The library does not provide a distinct function to check if a key is present, instead we recommend you attempt to access the key: e.g., by doing ondemand::value val{}; if (!object["foo"].get(val)) {...}, you have that val contains the requested value inside the if clause. It is your responsibility as a user to temporarily keep a reference to the value (auto v = object["foo"]), or to consume the content and store it in your own data structures. If you consume an object twice: std::string_view(object["foo"] followed by std::string_view(object["foo"] then your code is in error. Furthermore, you can only consume one field at a time, on the same object. The value instance you get from content["bids"] becomes invalid when you call content["asks"]. If you have retrieved content["bids"].get_array() and you later call content["asks"].get_array(), then the first array should no longer be accessed: it would be unsafe to do so. You can detect such mistakes by first compiling and running the code with development checks: an OUT_OF_ORDER_ITERATION error is generated.NOTE: JSON allows you to escape characters in keys. E.g., the key
"date"may be written as"\u0064\u0061\u0074\u0065". By default, simdjson does not unescape keys when matching. Thus if you search for the key"date"and the JSON document uses"\u0064\u0061\u0074\u0065"as a key, it will not be recognized. This is not generally a problem. Nevertheless, if you do need to support escaped keys, the methodunescaped_key()provides the desired unescaped keys by parsing and writing out the unescaped keys to a string buffer and returning astd::string_viewinstance. Theunescaped_keytakes an optional Boolean value: passing it true will decode invalid Unicode sequences with replacement, meaning that the decoding always succeeds but bogus Unicode replacement characters are inserted. In general, you should expect a performance penalty when usingunescaped_key()compared tokey()because of the string processing: thekey()function just points inside the source JSON document. As a compromise, you may useescaped_key()which returns astd::string_view<tt>instance pointing directly in the document, likekey(), although, unlikekey()`, it has to determine the location of the final quote character.{c++}auto json = R"({"k\u0065y": 1})"_padded;ondemand::parser parser;auto doc = parser.iterate(json);ondemand::object object = doc.get_object();for(auto field : object) {// parses and writes out the key, after unescaping it,// to a string buffer. It causes a performance penalty.// If you do not expect that unescaping is useful, you// may replace field.unescaped_key() with// field.escaped_key().std::string_view keyv = field.unescaped_key();if (keyv == "key") { std::cout << uint64_t(field.value()); }}By default, field lookup is order-insensitive, so you can look up values in any order. However, we still encourage you to look up fields in the order you expect them in the JSON, as it is still faster.
If you want to enforce finding fields in order, you can use
object.find_field("foo")instead. This will only look forward, and will fail to find fields in the wrong order: for example, this will fail:{c++}ondemand::parser parser;auto json = R"( { "x": 1, "y": 2 } )"_padded;auto doc = parser.iterate(json);double y = doc.find_field("y"); // The cursor is now after the 2 (at })double x = doc.find_field("x"); // This fails, because there are no more fields after "y"By contrast, using the default (order-insensitive) lookup succeeds:
{c++}ondemand::parser parser;auto json = R"( { "x": 1, "y": 2 } )"_padded;auto doc = parser.iterate(json);double y = doc["y"]; // The cursor is now after the 2 (at })double x = doc["x"]; // Success: [] loops back around to find "x"
* Output to strings: Given a document, a value, an array or an object in a JSON document, you can output a JSON string version suitable to be parsed again as JSON content: simdjson::to_json_string(element). A call to to_json_string consumes fully the element: if you apply it on a document, the JSON pointer is advanced to the end of the document. The simdjson::to_json_string does not allocate memory. The to_json_string function should not be confused with retrieving the value of a string instance which are escaped and represented using a lightweight std::string_view instance pointing at an internal string buffer inside the parser instance. To illustrate, the first of the following two code segments will print the unescaped string "test" complete with the quote whereas the second one will print the escaped content of the string (without the quotes).
```C++ // serialize a JSON to an escaped std::string instance so that it can be parsed again as JSON auto silly_json = R"( { "test": "result" } )"_padded; ondemand::document doc = parser.iterate(silly_json); std::cout << simdjson::to_json_string(doc["test"]) << std::endl; // Requires simdjson 1.0 or better >`
C++ // retrieves an unescaped string value as a string_view instance auto silly_json = R"( { "test": "result" } )"_padded; ondemand::document doc = parser.iterate(silly_json); std::cout << std::string_view(doc["test"]) << std::endl; >```</blockquote> You can useto_json_stringto efficiently extract components of a JSON document to reconstruct a new JSON document, as in the following example: <blockquote>``C++ auto cars_json = R"( [ { "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] }, { "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] }, { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] } ] )"_padded; std::vector<std::string_view> arrays; // We are going to collect string_view instances which point inside thecars_jsonstring // and are therefore valid as long ascars_jsonremains in scope. { ondemand::parser parser; for (ondemand::object car : parser.iterate(cars_json)) { if (uint64_t(car["year"]) > 2000) { arrays.push_back(simdjson::to_json_string(car["tire_pressure"])); } } } // We can now convert to a JSON string: std::ostringstream oss; oss << "["; for(size_t i = 0; i < arrays.size(); i++) { if (i>0) { oss << ","; } oss << arrays[i]; } oss << "]"; auto json_string = oss.str(); // json_string == "[[ 40.1, 39.9, 37.7, 40.4 ],[ 30.1, 31.0, 28.6, 28.7 ]]" >```</blockquote>* **Extracting Values (without exceptions):** You can use a variant usage ofget()with error codes to avoid exceptions. You first declare the variable of the appropriate type (double,uint64_t,int64_t,bool,ondemand::object<tt>andondemand::array<tt>) and pass it by reference toget()` which gives you back an error code: e.g.,{c++}auto abstract_json = R"({ "str" : { "123" : {"abc" : 3.14 } } })"_padded;ondemand::parser parser;double value;auto doc = parser.iterate(abstract_json);auto error = doc["str"]["123"]["abc"].get(value);if (error) { std::cerr << simdjson::error_message(error) << std::endl; return EXIT_FAILURE; }cout << value << endl; // Prints 3.14This examples also show how we can string several operations and only check for the error once, a strategy we call error chaining. Though error chaining makes the code very compact, it also makes error reporting less precise: in this instance, you may get the same error whether the field "str", "123" or "abc" is missing. If you need to break down error handling per operation, avoid error chaining. Furthermore, you should be mindful that chaining that harm performance by encouraging redundancies: writing both
doc["str"]["123"]["abc"].get(value)anddoc["str"]["123"]["zyw"].get(value)in the same program may force multiple accesses to the same keys ("str"and"123").
- Counting elements in arrays: Sometimes it is useful to scan an array to determine its length prior to parsing it. For this purpose,
arrayinstances have acount_elementsmethod. Users should be aware that thecount_elementsmethod can be costly since it requires scanning the whole array. You should only callcount_elementsas a last resort as it may require scanning the document twice or more. In the spirit of On-Demand, thecount_elementsfunction does not validate the values in the array. You may use it as follows if your document is itself an array:{C++}auto cars_json = R"( [ 40.1, 39.9, 37.7, 40.4 ] )"_padded;auto doc = parser.iterate(cars_json);size_t count = doc.count_elements(); // requires simdjson 1.0 or betterstd::vector<double> values(count);size_t index = 0;for(double x : doc) { values[index++] = x; }If you access an array inside a document, you can use the
count_elementsmethod as follow. You should not let the array instance go out of scope before consuming it after calling thecount_elementsmethod:ondemand::parser parser;auto cars_json = R"( { "test":[ { "val1":1, "val2":2 }, { "val1":1, "val2":2 } ] } )"_padded;auto doc = parser.iterate(cars_json);auto test_array = doc.find_field("test").get_array();size_t count = test_array.count_elements(); // requires simdjson 1.0 or betterstd::cout << "Number of elements: " << count << std::endl;for(ondemand::object elem: test_array) {std::cout << simdjson::to_json_string(elem);}
- Counting fields in objects: Other times, it is useful to scan an object to determine the number of fields prior to parsing it. For this purpose,
objectinstances have acount_fieldsmethod. Again, users should be aware that thecount_fieldsmethod can be costly since it requires scanning the whole objects. You should only callcount_fieldsas a last resort as it may require scanning the document twice or more. You may use it as follows if your document is itself an object:{C++}ondemand::parser parser;auto json = R"( { "test":{ "val1":1, "val2":2 } } )"_padded;auto doc = parser.iterate(json);size_t count = doc.count_fields(); // requires simdjson 1.0 or betterstd::cout << "Number of fields: " << count << std::endl; // Prints "Number of fields: 1"Similarly to
count_elements, you should not let an object instance go out of scope before consuming it after calling thecount_fieldsmethod. If you access an object inside a document, you can use thecount_fieldsmethod as follow.ondemand::parser parser;auto json = R"( { "test":{ "val1":1, "val2":2 } } )"_padded;auto doc = parser.iterate(json);auto test_object = doc.find_field("test").get_object();size_t count = test_object.count_fields(); // requires simdjson 1.0 or betterstd::cout << "Number of fields: " << count << std::endl; // Prints "Number of fields: 2"
Tree Walking and JSON Element Types: Sometimes you don't necessarily have a document with a known type, and are trying to generically inspect or walk over JSON elements. You can also represent arbitrary JSON values with
ondemand::valueinstances: it can represent anything except a scalar document (lone number, string, null or Boolean). You can check for scalar documents with the methodscalar(). You can cast a document that is either an array or an object to anondemand::valueinstance immediately after you create the document instance: you cannot create anondemand::valueinstance from a document that has already been accessed as it would mean that you would have two instances of the object or array simultaneously (see rewinding). You can query the type of a document or a value with thetype()method. Thetype()method does not consume or validate documents and values, but it tells you whether they are
- arrays (
json_type::array),- objects (
json_type::object)- numbers (
json_type::number),- strings (
json_type::string),- Booleans (
json_type::boolean),- null (
json_type::null).You must still validate and consume the values (e.g., call
is_null()) after callingtype(). You may also access the raw JSON string. For example, the following is a quick and dirty recursive function that verbosely prints the JSON document as JSON. This example also illustrates lifecycle requirements: thedocumentinstance holds the iterator. The document must remain in scope while you are accessing instances ofvalue,objectandarray.{c++}void recursive_print_json(ondemand::value element) {bool add_comma;switch (element.type()) {case ondemand::json_type::array:cout << "[";add_comma = false;for (auto child : element.get_array()) {if (add_comma) {cout << ",";}// We need the call to value() to get// an ondemand::value type.recursive_print_json(child.value());add_comma = true;}cout << "]";break;case ondemand::json_type::object:cout << "{";add_comma = false;for (auto field : element.get_object()) {if (add_comma) {cout << ",";}// key() returns the key as it appears in the raw// JSON document, if we want the unescaped key,// we should do field.unescaped_key().// We could also use field.escaped_key() if we want// a std::string_view instance, but we do not need// escaping.cout << "\"" << field.key() << "\": ";recursive_print_json(field.value());add_comma = true;}cout << "}\n";break;case ondemand::json_type::number:// assume it fits in a doublecout << element.get_double();break;case ondemand::json_type::string:// get_string() would return escaped string, but// we are happy with unescaped string.cout << "\"" << element.get_raw_json_string() << "\"";break;case ondemand::json_type::boolean:cout << element.get_bool();break;case ondemand::json_type::null:// We check that the value is indeed null// otherwise: an error is thrown.if (element.is_null()) {cout << "null";}break;}}void basics_treewalk() {padded_string json = R"( [{ "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },{ "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },{ "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }] )"_padded;ondemand::parser parser;ondemand::document doc = parser.iterate(json);ondemand::value val = doc;recursive_print_json(val);std::cout << std::endl;}
Let us review these concepts with some additional examples. For simplicity, we omit the include clauses (#include "simdjson.h") as well as namespace-using clauses (using namespace simdjson;).
The first example illustrates how we can chain operations. In this instance, we repeatedly select keys using the bracket operator (doc["str"]) and then finally request a number (using get_double()). It is safe to write code in this manner: if any step causes an error, the error status propagates and an exception is thrown at the end. You do not need to constantly check for errors.
In the following example, we start with a JSON document that contains an array of objects. We iterate through the objects using a for-loop. Within each object, we use the bracket operator (e.g., car["make"]) to select values. We also show how we can iterate through an array, corresponding to the key tire_pressure, that is contained inside each object.
The previous example had an array of objects, but we can use essentially the same approach with an object of objects.
The following example illustrates how you may also iterate through object values, effectively visiting all key-value pairs in the object.
There are 2 main ways provided by simdjson to deserialize a value into a custom type:
simdjson::ondemand::document::get for the whole documentsimdjson::ondemand::value::get for each valuetag_invoke *(the recommended way if your system supports C++20 or better)*We describe both of them in the following sections. Most users who have systems compatible with C++20 or better should skip ahead to using tag_invoke for custom types (C++20) as it is more powerful and simpler.
Suppose you have your own types, such as a Car struct:
You might want to write code that automatically parses the JSON content to your custom type:
We may do so by providing additional template definitions to the ondemand::value type. We may start by providing a definition for std::vector<double> as follows. Observe how we guard the code with #if !SIMDJSON_SUPPORTS_DESERIALIZATION: that is because the necessary code is automatically provided by simdjson if C++20 (and concepts) are available. See Use tag_invoke for custom types if you have C++20 support.
We may then provide support for our Car struct:
And that is all that is needed! The following code is a complete example:
Observe that we require an explicit cast (Car c(val) instead of for (Car c : doc) {): it is by design. We require explicit casting.
If you prefer to avoid exceptions, you may modify the main function as follows:
Our example is limited to ondemand::value instances. If you wish to also be able to map directly the document instance itself to a custom type, you need to provide the definitions to the ondemand::document type. In this instance, we must replace the function with signature simdjson_result<Car> simdjson::ondemand::value::get() with a function having signature simdjson_result<Car> simdjson::ondemand::document::get() &. The following is a complete example:
In C++20, the standard introduced the notion of customization point. A customization point is a function or function object that can be customized for different types. It allows library authors to provide default behavior while giving users the ability to override this behavior for specific types.
A tag_invoke function serves as a mechanism for customization points. It is not directly part of the C++ standard library but is often used in libraries that implement customization points. The tag_invoke function is typically a generic function that takes a tag type and additional arguments. The first argument is usually a tag type (often an empty struct) that uniquely identifies the customization point (e.g., deserialization of custom types in simdjson). Users or library providers can specialize tag_invoke for their types by defining it in the appropriate namespace, often inline namespace.
If your system supports C++20, we recommend that you adopt the tag_invoke approach instead to deserialize custom types. It may prove to be considerably simpler. When simdjson detects the necessary support, it sets the SIMDJSON_SUPPORTS_DESERIALIZATION macro to 1, otherwise it is set to 0.
Consider a custom class Car:
Observe how we defined the class to use types that simdjson does not directly support (float, int). With C++20 support, the library grabs from the JSON the generic type (double, int) and then it casts it automatically.
You may support deserializing directly from a JSON value or document to your own Car instance by defining a single tag_invoke function:
Observe how we call get<std::vector<float>>() even though we never defined support for std::vector<float> in the simdjson library: it is all automated thanks to C++20 concepts.
Importantly, the tag_invoke function must be inside the simdjson namespace. Let us explain each argument of tag_invoke function.
simdjson::deserialize_tag: it is the tag for Customization Point Object (CPO). You may often ignore this parameter. It is used to indicate that you mean to provide a deserialization function for simdjson.var: It receives automatically a simdjson value type (document, value, document_reference).You can use it like so:
Observe how we first get an instance of document and then we cast.
You can also handle errors explicitly:
You can also read instances of Car from an array or an object:
Observe how we first get a generic (val) which we cast to Car. It is by design: we require explicit casting. The cast may throw an exception.
Once more, you can handle errors explicitly:
You can also use the custom Car type as part of a template such as std::vector:
By default, we support a wide range of standard templates such as std::vector, std::list, std::set, std::stack, std:queue, std:deque, std::priority_queue, std::unordered_set, std::multiset, std::unordered_multiset, std::unique_ptr, std::shared_ptr, std::optional, etc. They are handled automatically.
E.g., you can recover an std::unique_ptr<Car> like so:
You may also conditionally fill in std::optional values.
And so forth.
Advanced users may want to overwrite the defaults provided by the simdjson library. Suppose for example that you want to construct an instance of std::list<Car>, but you also want to filter out any car made by Toyota. You may provide your own tag_invoke function:
With this code, deserializing an std::list<Car> instance would capture only the cars that are not made by Toyota.
In some cases, you may have valid JSON strings that you do not wish to parse but that you wish to minify. That is, you wish to remove all unnecessary spaces. We have a fast function for this purpose (simdjson::minify(const char * input, size_t length, const char * output, size_t& new_length)). This function does not validate your content, and it does not parse it. It is much faster than parsing the string and re-serializing it in minified form (simdjson::minify(parser.parse())). Usage is relatively simple. You must pass an input pointer with a length parameter, as well as an output pointer and an output length parameter (by reference). The output length parameter is not read, but written to. The output pointer should point to a valid memory region that is as large as the original string length. The input pointer and input length are read, but not written to.
Though it does not validate the JSON input, it will detect when the document ends with an unterminated string. E.g., it would refuse to minify the string "this string is not terminated because of the missing final quote.
The simdjson library has fast functions to validate UTF-8 strings. They are many times faster than most functions commonly found in libraries. You can use our fast functions, even if you do not care about JSON.
The UTF-8 validation function merely checks that the input is valid UTF-8: it works with strings in general, not just JSON strings.
Your input string does not need any padding. Any string will do. The validate_utf8 function does not do any memory allocation on the heap, and it does not throw exceptions.
If you find yourself needing only fast Unicode functions, consider using the simdutf library instead: https://github.com/simdutf/simdutf
The simdjson library also supports JSON pointer through the at_pointer() method, letting you reach further down into the document in a single call. JSON pointer is supported by both the DOM approach as well as the On-Demand approach.
Note: The On-Demand implementation of JSON pointer relies on find_field which implies that it does not unescape keys when matching.
Consider the following example:
A JSON Pointer path is a sequence of segments each starting with the '/' character. Within arrays, a zero-based integer index allows you to select the indexed node. Within objects, the string value of the key allows you to select the value. If your keys contain the characters '/' or '~', they must be escaped as '~1' and '~0' respectively. An empty JSON Pointer Path refers to the whole document.
For multiple JSON pointer queries on a document, one can call at_pointer multiple times.
In most instances, a JSON Pointer is an ASCII string and the keys in a JSON document are ASCII strings. We support UTF-8 in JSON Pointer, but key values are matched exactly, without unescaping or Unicode normalization. We do a byte-by-byte comparison. The e acute character is considered distinct from its escaped version \u00E9. E.g.,
Note that at_pointer calls rewind to reset the parser at the beginning of the document. Hence, it invalidates all previously parsed values, objects and arrays: make sure to consume the values between each call to at_pointer. Consider the following example where one wants to store each object from the JSON into a vector of struct car_type:
Furthermore, at_pointer calls rewind at the beginning of the call (i.e. the document is not reset after at_pointer). Consider the following example,
When the JSON Pointer Path is the empty string ("") applied to a scalar document (lone string, number, Boolean or null), a SCALAR_DOCUMENT_AS_VALUE error is returned because scalar document cannot be represented as value instances. You can check that a document is a scalar with the method scalar().
The simdjson library supports a subset of JSONPath through the at_path() method, allowing you to reach further into the document in a single call. The subset of JSONPath that is implemented is the subset that is trivially convertible into the JSON Pointer format, using . to access a field and [] to access a specific index.
This implementation relies on at_path() converting its argument to JSON Pointer and then calling at_pointer, which makes use of rewind to reset the parser at the beginning of the document. Hence, it invalidates all previously parsed values, objects and arrays: make sure to consume the values between each call to at_path.
Consider the following example:
A call to at_path(json_path) can result in any of the errors that are returned by the at_pointer method and if the conversion of json_path to JSON Pointer fails, it will lead to an simdjson::INVALID_JSON_POINTERerror.
In most instances, a JSONPath is an ASCII string and the keys in a JSON document are ASCII strings. We support UTF-8 within a JSONPath expression, but key values are matched exactly, without unescaping or Unicode normalization. We do a byte-by-byte comparison. The e acute character is considered distinct from its escaped version \u00E9. E.g.,
We also support the $ prefix. When you start a JSONPath expression with $, you are indicating that the path starts from the root of the JSON document. E.g.,
Error handing with exception and a single try/catch clause makes the code simple, but it gives you little control over errors. For easier debugging or more robust error handling, you may want to consider our exception-free approach.
The entire simdjson API is usable with and without exceptions. All simdjson APIs that can fail return simdjson_result<T>, which is a <value, error_code> pair. You can retrieve the value with .get() without generating an exception, like so:
When there is no error, the error code simdjson::SUCCESSis returned: it evaluates as false as a Boolean. We have several error codes to indicate errors, they all evaluate to true as a Boolean: your software should not generally not depend on exact error codes. We may change the error codes in future releases and the exact error codes could vary depending on your system.
Some errors are recoverable:
simdjson::INCORRECT_TYPE after trying to convert a value to an incorrect type: e.g., you expected a number and try to convert the value to a number, but it is an array.simdjson::NO_SUCH_FIELD: e.g., you call obj["myname"] and the object does not have a key "myname".Other errors (e.g., simdjson::INCOMPLETE_ARRAY_OR_OBJECT) may indicate a fatal error and often follow from the fact that the document is not valid JSON. In which case, it is no longer possible to continue accessing the document: calling the method is_alive() on the document instance returns false. All following accesses will keep returning the same fatal error (e.g., simdjson::INCOMPLETE_ARRAY_OR_OBJECT).
When you use the code without exceptions, it is your responsibility to check for error before using the result: if there is an error, the result value will not be valid and using it will caused undefined behavior. Most compilers should be able to help you if you activate the right set of warnings: they can identify variables that are written to but never otherwise accessed.
Let us illustrate with an example where we try to access a number that is not valid (3.14.1). If we want to proceed without throwing and catching exceptions, we can do so as follows:
Observe how we verify the error variable before accessing the retrieved number (variable x).
The equivalent with exception handling might look as follows.
Notice how we can retrieve the exact error condition (in this instance simdjson::NUMBER_ERROR) from the exception.
We can write a "quick start" example where we attempt to parse the following JSON file and access some data, without triggering exceptions:
Our program loads the file, selects value corresponding to key "search_metadata" which expected to be an object, and then it selects the key "count" within that object.
The following is a similar example where one wants to get the id of the first tweet without triggering exceptions. To do this, we use ["statuses"].at(0)["id"]. We break that expression down:
"statuses" key of the document) using ["statuses"]). The result is expected to be an array..at(0). The result is expected to be an object.Observe how we use the at method when querying an index into an array, and not the bracket operator.
The at method can only be called once on an array. It cannot be used to iterate through the values of an array.
This is how the example in "Using the parsed JSON" could be written using only error code checking (without exceptions):
For safety, you should only use our ondemand instances (e.g., ondemand::object) after you have initialized them and checked that there is no error:
The following examples illustrates how to iterate through the content of an object without having to handle exceptions.
The simdjson can be build with exceptions entirely disabled. It checks the __cpp_exceptions macro at compile time. Even if exceptions are enabled in your compiler, you may still disable exceptions specifically for simdjson, by setting SIMDJSON_EXCEPTIONS to 0 (false) at compile-time when building the simdjson library. If you are building with CMake, to ensure you don't write any code that uses exceptions, you compile with SIMDJSON_EXCEPTIONS=OFF. For example, if including the project via cmake:
Users more comfortable with an exception flow may choose to directly cast the simdjson_result<T> to the desired type:
When used this way, a simdjson_error exception will be thrown if an error occurs, preventing the program from continuing if there was an error.
If one is willing to trigger exceptions, it is possible to write simpler code:
You can do handle errors gracefully as well...
Sometimes, it might be helpful to know the current location in the document during iteration. This is especially useful when encountering errors. The current_location() method on a document instances makes it easy to identify common JSON errors. Users can call the current_location() method on a valid document instance to retrieve a const char * pointer to the current location in the document. This method also works even after an error has invalidated the document and the parser (e.g. TAPE_ERROR, INCOMPLETE_ARRAY_OR_OBJECT). When the input was a padding_string or another null-terminated source, then you may use the const char * pointer as a C string. As an example, consider the following example where we used the exception-free simdjson interface:
You may also use current_location() with exceptions as follows:
In these examples, we tried to access the "integer" key, but since the parser had to go through a value without a key before (false), a TAPE_ERROR error is thrown. The pointer returned by the current_location() method then points at the location of the error. The current_location() may also be used when the error is triggered by a user action, even if the JSON input is valid. Consider the following example:
If the location is invalid (i.e. at the end of a document), the current_location() methods returns an OUT_OF_BOUNDS error. For example:
Conversely, if doc.current_location().error() == simdjson::SUCCESS, then the document has more content.
Finally, the current_location() method may also be used even when no exceptions/errors are thrown. This can be helpful for users that want to know the current state of iteration during parsing. For example:
The current_location() method requires a valid document instance. If the iterate function fails to return a valid document, then you cannot use current_location() to identify the location of an error in the input string. The errors reported by iterate function include EMPTY if no JSON document is detected, UTF8_ERROR if the string is not a valid UTF-8 string, UNESCAPED_CHARS if a string contains control characters that must be escaped and UNCLOSED_STRING if there is an unclosed string in the document. We do not provide location information for these errors.
The parser validates all parsed content, but your code may exhaust the content while not having processed the entire document. Thus, as a final optional step, you may call at_end() on the document instance. If it returns false, then you may conclude that you have trailing content and that your document is not valid JSON. You may then use doc.current_location() to obtain a pointer to the start of the trailing content.
The at_end() method is equivalent to doc.current_location().error() == simdjson::SUCCESS but more convenient.
In some instances, you may need to go through a document more than once. For that purpose, you may call the rewind() method on the document instance. It allows you to restart processing from the beginning without rescanning all of the input data again. It invalidates all values, objects and arrays that you have created so far (including unescaped strings).
In the following example, we print on the screen the number of cars in the JSON input file before printout the data.
Performance note: the On-Demand front-end does not materialize the parsed numbers and other values. If you are accessing everything twice, you may need to parse them twice. Thus the rewind functionality is best suited for cases where the first pass only scans the structure of the document.
Both arrays and objects have a similar method reset(). It is similar to the document rewind() method, except that it does not rewind the internal string buffer. Thus you should consume values only once even if you can iterate through the array or object more than once. If you unescape a string within an array more than once, you have unsafe code.
When processing large inputs (e.g., in the context of data engineering), engineers commonly serialize data into streams of multiple JSON documents. That is, instead of one large (e.g., 2 GB) JSON document containing multiple records, it is often preferable to write out multiple records as independent JSON documents, to be read one-by-one.
The simdjson library also supports multithreaded JSON streaming through a large file containing many smaller JSON documents in either ndjson or JSON lines format. If your JSON documents all contain arrays or objects, we even support direct file concatenation without whitespace. However, if there is content between your JSON documents, it should be exclusively ASCII white-space characters.
The concatenated file has no size restrictions (including larger than 4GB), though each individual document must be no larger than 4 GB.
Here is an example:
Unlike parser.iterate, parser.iterate_many may parse "On-Demand" (lazily). That is, no parsing may have been done before you enter the loop for (auto doc : docs) { and you should expect the parser to only ever fully parse one JSON document at a time.
As with parser.iterate, when calling parser.iterate_many(string), no copy is made of the provided string input. The provided memory buffer may be accessed each time a JSON document is parsed. Calling parser.iterate_many(string) on a temporary string buffer (e.g., docs = parser.parse_many("[1,2,3]"_padded)) is unsafe (and will not compile) because the document_stream instance needs access to the buffer to return the JSON documents.
The iterate_many function can also take an optional parameter size_t batch_size which defines the window processing size. It is set by default to a large value (1000000 corresponding to 1 MB). None of your JSON documents should exceed this window size, or else you will get the error simdjson::CAPACITY. You cannot set this window size larger than 4 GB: you will get the error simdjson::CAPACITY. The smaller the window size is, the less memory the function will use. Setting the window size too small (e.g., less than 100 kB) may also impact performance negatively. Leaving it to 1 MB is expected to be a good choice, unless you have some larger documents.
The following toy examples illustrates how to get capacity errors. It is an artificial example since you should never use a batch_size of 50 bytes (it is far too small).
This example should print out:
If your documents are large (e.g., larger than a megabyte), then the iterate_many function is maybe ill-suited. It is really meant to support reading efficiently streams of relatively small documents (e.g., a few kilobytes each). If you have larger documents, you should use other functions like iterate.
We also provide some support for comma-separated documents and other advanced features. See iterate_many.md for detailed information and design.
Though the JSON specification allows for numbers and string values, many engineers choose to integrate the numbers inside strings, e.g., they prefer {"a":"1.9"} to{"a":1.9}. The simdjson library supports parsing valid numbers inside strings which makes it more convenient for people working with those types of documents. This feature is supported through three methods: get_double_in_string, get_int64_in_string and get_uint64_in_string. However, it is important to note that these methods are not substitute to the regular get_double, get_int64 and get_uint64. The usage of the get_*_in_string methods is solely to parse valid JSON numbers inside strings, and so we expect users to call these methods appropriately. In particular, a valid JSON number has no leading and no trailing whitespaces, and the strings "nan", "1e" and "infinity" will not be accepted as valid numbers. As an example, suppose we have the following JSON text:
Now, suppose that a user wants to get the time stamp from the timestampstr key. One could do the following:
Another thing a user might want to do is extract the markets array and get the market name, price and volume. Here is one way to do so:
Finally, here is an example dealing with errors where the user wants to convert the string "Infinity"("change" key) to a float with infinity value.
It is also important to note that when dealing an invalid number inside a string, simdjson will report a NUMBER_ERROR error if the string begins with a number whereas simdjson will report an INCORRECT_TYPE error otherwise.
The *_in_string methods can also be called on a single document instance: e.g., when your document consist solely of a quoted number.
The JSON standard does not offer strongly typed numbers. It suggests that using the binary64 type (double in C++) is a safe choice, but little else. Given the JSON array [1.0,1], it is not specified whether it is an array of two floating-point numbers, two integers, or one floating-point number followed by an integer.
Given an ondemand::value instance, you may ask whether it is a negative value with the is_negative() method. The function is inexpensive.
To occasionally distinguish between floating-point values and integers given an ondemand::value instance, you may call the is_integer() method. We recognize an integer number by the lack decimal point and the lack of exponential suffix. E.g., 1e1 is always considered to be a floating-point number. The is_integer() method does not consume the value, but it scans the number string. You should avoid calling it repeatedly.
If you need to determine both the type of the number (integer or floating-point) and its value efficiently, you may call the get_number() method on the ondemand::value instance. Upon success, it returns an ondemand::number instance.
An ondemand::number instance may contain an integer value or a floating-point value. Thus it is a dynamically typed number. Before accessing the value, you must determine the detected type:
number.get_number_type() has value number_type::signed_integer if we have a integer in [-9223372036854775808,9223372036854775808). You can recover the value by the get_int64() method applied on the ondemand::number instance. When number.get_number_type() has value number_type::signed_integer, you also have that number.is_int64() is true. Calling get_int64() on the ondemand::number instance when number.get_number_type() is not number_type::signed_integer is unsafe. You may replace get_int64() by a cast to a int64_t value.number.get_number_type() has value number_type::unsigned_integer if we have a integer in [9223372036854775808,18446744073709551616). You can recover the value by the get_uint64() method applied on the ondemand::number instance. When number.get_number_type() has value number_type::unsigned_integer, you also have that number.is_uint64() is true. Calling get_uint64() on the ondemand::number instance when number.get_number_type() is not number_type::unsigned_integer is unsafe. You may replace get_uint64() by a cast to a uint64_t value.number.get_number_type() has value number_type::floating_point_number if we have and we have a floating-point (binary64) number. You can recover the value by the get_double() method applied on the ondemand::number instance. When number.get_number_type() has value number_type::floating_point_number, you also have that number.is_double() is true. Calling get_double() on the ondemand::number instance when number.get_number_type() is not number_type::floating_point_number is unsafe. You may replace get_double() by a cast to a double value.number.get_number_type() has value number_type::big_integer. If you try to parse such a number of get_number(), you get the error BIGINT_ERROR. You can access the underlying string of digits with the function raw_json_token() which returns a std::string_view instance starting at the beginning of the digit. You can also call get_double() to get a floating-point approximation.You must check the type before accessing the value: it is an error to call get_int64() when number.get_number_type() is not number_type::signed_integer and when number.is_int64() is false. You are responsible for this check as the user of the library.
The get_number() function is designed with performance in mind. When calling get_number(), you scan the number string only once, determining efficiently the type and storing it in an efficient manner.
Consider the following example:
It will output:
In the following example, we have an array of integers that are outside the valid range of 64-bit signed or unsigned integers. Calling get_number_type() on the values returns ondemand::number_type::big_integer. You can try to represent these big integers as 64-bit floating-point numbers, though you typically lose precision in the process (as illustrated in the example).
This program might print:
You may get access to the underlying string representing the big integer with raw_json_token() and you may parse the resulting number strings using your own parser.
This code prints the following:
It is sometimes useful to have access to a raw (unescaped) string: we make available a minimalist raw_json_string data type which contains a pointer inside the string in the original document, right after the quote. It is accessible via get_raw_json_string() on a string instance and returned by the key() method on an object's field instance. It is always optional: replacing get_raw_json_string() with get_string() and key() by unescaped_key() or escaped_key() returns an string_view instance of the unescaped/unprocessed string.
You can quickly compare a raw_json_string instance with a target string. You may also unescape the raw_json_string on your own string buffer: parser.unescape(mystr, ptr) advances the provided pointer ptr and returns a string_view instance on the newly serialized string upon success, otherwise it returns an error. When unescaping to your own string buffer, you should ensure that you have sufficient memory space: the total size of the strings plus simdjson::SIMDJSON_PADDING bytes. The following example illustrates how we can unescape JSON string to a user-provided buffer:
Some users might prefer to have a direct access to a std::string_view instance pointing inside the source document. The key_raw_json_token() method serves this purpose. It provides a view on the key, including the starting quote character, and everything up to the next : character after the final quote character. E.g., if the key is "name" then key_raw_json_token() returns a std::string_view which begins with "name" and may containing trailing white-space characters.
If your value is a string, the raw_json_string you get with get_raw_json_string() gives you direct access to the unprocessed string. But the simdjson library allows you to have access to the raw underlying JSON more generally, not just for strings.
The simdjson library makes explicit assumptions about types. For examples, numbers must be integers (up to 64-bit integers) or binary64 floating-point numbers. Some users have different needs. For example, some users might want to support big integers. The library makes this possible by providing a raw_json_token method which returns a std::string_view instance containing the value as a string which you may then parse as you see fit.
The raw_json_token method even works when the JSON value is a string. In such cases, it will return the complete string with the quotes and with eventual escaped sequences as in the source document.
The raw_json_token() should be fast and free of allocation.
Given a quote-deliminated string, you find the string sequence inside the quote with a single line of code:
If your value is an array or an object, raw_json_token() returns effectively a single character ([) or (}) which is not very useful. For arrays and objects, we have another method called raw_json() which consumes (traverses) the array or the object.
Because raw_json() consumes to object or the array, if you want to both have access to the raw string, and also use the array or object, you should call reset().
You can use raw_json() with the values inside an array and object. When calling raw_json() on an untyped value, it acts as raw_json() when the value is an array or an object. Otherwise, it acts as raw_json_token(). It is useful if you do not care for the type of the value and just wants a string representation.
You can use raw_json() to capture the content of some JSON values as std::string_view instances which can be safely used later. The std::string_view instances point inside the original document and do not depend in any way on simdjson. In the following example, we store the std::string_view instances inside a std::vector<std::string_view> instance and print the out after the parsing is concluded:
The simdjson library favours the use of std::string_view instances because it tends to lead to better performance due to causing fewer memory allocations. However, they are cases where you need to store a string result in a std::string instance. You can do so with a templated version of the to_string() method which takes as a parameter a reference to a std::string.
The same routine can be written without exceptions handling:
The std::string instance, once created, is independent. Unlike our std::string_view instances, it does not point at data that is within our parser instance. The same caveat applies: you should only consume a JSON string once.
Because get_string() is a template that requires a type that can be assigned a std::string, you can use it with features such as std::optional:
You should be mindful of the trade-off: allocating multiple std::string instances can become expensive.
We built simdjson with thread safety in mind.
The simdjson library is single-threaded except for `iterate_many` and `parse_many` which may use secondary threads under their control when the library is compiled with thread support.
We recommend using one parser object per thread. When using the On-Demand front-end (our default), you should access the document instances in a single-threaded manner since it acts as an iterator (and is therefore not thread safe).
The CPU detection, which runs the first time parsing is attempted and switches to the fastest parser for your CPU, is transparent and thread-safe. Our runtime dispatching is based on global objects that are instantiated at the beginning of the main thread and may be discarded at the end of the main thread. If you have multiple threads running and some threads use the library while the main thread is cleaning up ressources, you may encounter issues. If you expect such problems, you may consider using std::quick_exit.
In a threaded environment, stack space is often limited. Running code like simdjson in debug mode may require hundreds of kilobytes of stack memory. Thus stack overflows are a possibility. We recommend you turn on optimization when working in an environment where stack space is limited. If you must run your code in debug mode, we recommend you configure your system to have more stack space. We discourage you from running production code based on a debug build.
The simdjson library is fully compliant with the RFC 8259 JSON specification.
01 is not valid JSON document since the specification states that leading zeros are not allowed.long or a C/C++ long long and all 64-bit unsigned integers. When we cannot represent exactly an integer as a signed or unsigned 64-bit value, we reject the JSON document.std::numeric_limits<double>::lowest() to std::numeric_limits<double>::max(), so from -1.7976e308 all the way to 1.7975e308. Extreme values (less or equal to -1e308, greater or equal to 1e308) are rejected: we refuse to parse the input document. Numbers are parsed with a perfect accuracy (ULP 0): the nearest floating-point value is chosen, rounding to even when needed. If you serialized your floating-point numbers with 17 significant digits in a standard compliant manner, the simdjson library is guaranteed to recover the same numbers, exactly.The only header file supported by simdjson is simdjson.h. Older versions of simdjson published a number of other include files such as document.h or ParsedJson.h alongside simdjson.h; these headers may be moved or removed in future versions.
Some users like to have example. The following code samples illustrate how to process specific JSON inputs. For simplicity, we do not include full error support: this code would throw exceptions on error.
std::string_view instancesSIMDJSON_VERBOSE_LOGGING prior to including the simdjson.h header, which enables logging in simdjson. Run your code. It may generate a lot of logging output; adding printouts from your application that show each section may be helpful. The log's output will show step-by-step information on state, buffer pointer position, depth, and key retrieval status. Importantly, unless SIMDJSON_VERBOSE_LOGGING is defined, logging is entirely disabled and thus carries no overhead.count_elements, rewind, reset and similar methods.field in an object, calling field.key() is often faster than field.unescaped_key() so if you do not need an unescaped std::string_view instance, prefer field.key(). Similarly, we expect field.escaped_key() to be faster than field.unescaped_key() even though both return a std::string_view instance.NDEBUG pre-processor directive when compiling the simdjson library. Importantly, using the optimization flags -O2 or -O3 under GCC and LLVM clang does not set the NDEBUG directive, you must set it manually (e.g., -DNDEBUG)."data" key to create an object... ```C++ std::string_view make = o["data"]["make"]; std::string_view model = o["data"]["model"]; std::string_view year = o["data"]["year"]; {"a":1, "b":2, "c":3}, do value1 = data["a"]; value2 = data["b"]; value3 data["c"]; and not value2 = data["b"]; value1 = data["a"]; value3 data["c"];. Of course, it is not always possible to know for sure in which order the keys appear.
1.9.1