How to handle partial json string?

[Mr-LiuDC]

Describe your Issue

JSON structure is clear, I need to handle partial json，How to handle 1.txt to 2.txt?
1.txt

{"
{"name
{"name":"
{"name":"Tom
{"name":"Tom","
{"name":"Tom","age
{"name":"Tom","age":
{"name":"Tom","age":25
{"name":"Tom","age":25,"
{"name":"Tom","age":25,"h
{"name":"Tom","age":25,"hobby
{"name":"Tom","age":25,"hobby":["
{"name":"Tom","age":25,"hobby":["reading
{"name":"Tom","age":25,"hobby":["reading","
{"name":"Tom","age":25,"hobby":["reading","h
{"name":"Tom","age":25,"hobby":["reading","hiking
{"name":"Tom","age":25,"hobby":["reading","hiking"],
{"name":"Tom","age":25,"hobby":["reading","hiking"],"
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""}}

2.txt

{}
{"name":null}
{"name":""}
{"name":"Tom"}
{"name":"Tom"}
{"name":"Tom","age":null}
{"name":"Tom","age":null}
{"name":"Tom","age":25}
{"name":"Tom","age":25}
{"name":"Tom","age":25,"h":null}
{"name":"Tom","age":25,"hobby":null}
{"name":"Tom","age":25,"hobby":[]}
{"name":"Tom","age":25,"hobby":["reading"]}
{"name":"Tom","age":25,"hobby":["reading"]}
{"name":"Tom","age":25,"hobby":["reading","h"]}
{"name":"Tom","age":25,"hobby":["reading","hiking"]}
{"name":"Tom","age":25,"hobby":["reading","hiking"]}
{"name":"Tom","age":25,"hobby":["reading","hiking"]}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address:null}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":null}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":null}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":null}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New"}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York"}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York"}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":null}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":null}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":null}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""}}

[cowtowncoder]

Since this is not an issue but question, will convert to Discussion.

[cowtowncoder]

To handle partial content, you wil probably need to create and use JsonParser, to read tokens one by one with JsonParser.nextToken().
Note that at the point where content is found to be invalid (like end-of-content where another token would be expected for valid JSON documents), you will get an exception that needs handling.

But it all depends on what you are actually trying to achieve. So more details would be needed...

[yawkat]

There's also NonBlockingJsonParser which can be used for partial input, but it really depends on what you're trying to do

[cowtowncoder]

Yes, very good point @yawkat : non-blocking parser is probably the most applicable thing as it will not throw exceptions for incomplete input.

[Mr-LiuDC]

@yawkat @cowtowncoder

Thanks for your reply.

Is there any relevant documentation or examples for using NonBlockingJsonParser? I searched on Google but couldn't find any suitable documentation or usage examples.

[JooHyukKim]

Maybe check out the jackson-core test code?
And also, when you do find documentation or examples, please let us know! (So we may add it somewhere for others to know)

Or how about just go straight up try NonBlockingJsonParser?

[cowtowncoder]

@Mr-LiuDC Documentation is parse, but as @JooHyukKim said, there are tests under src/test/java/.../async. API for reading is just JsonParser, with addition that nextToken() may return JsonToken.NOT_AVAILABLE when there isn't enough input available for full token.

But one thing to be aware of is that none of databind can be used with it, since JsonDeserializers rely on blocking input (that is, nextToken() always returning fully decoded token (or null for end of input).

[Mr-LiuDC]

@JooHyukKim Thanks.

I found some examples in jackson-core unit test codes, here is my example, not perfect, but usable.

@Slf4j
public class PartialJsonParseTest {

    private static final JsonFactory jsonFactory = new JsonFactory();
    private static final ObjectMapper objectMapper = new ObjectMapper();

    @Test
    public void test() {
        List<String> lines = FileUtil.readLines("files/txt/partial-json.txt", StandardCharsets.UTF_8);
        Flux<String> fluxLines = Flux.fromIterable(lines);
        fluxLines
                .map(jsonStr -> {
                    Optional<JsonNode> optional = fixPartialJson(jsonStr);
                    log.info("Original JSON: {}", optional.orElse(null));
                    return optional.map(JsonNode::toString).orElse(jsonStr);
                })
                .subscribe(jsonObj -> log.info("Parsed JSON: {}", jsonObj));
    }

    public static Optional<JsonNode> fixPartialJson(String partialJson) {
        byte[] jsonBytes = partialJson.getBytes(StandardCharsets.UTF_8);
        try (NonBlockingJsonParser jsonParser = (NonBlockingJsonParser) jsonFactory.createNonBlockingByteArrayParser()) {
            jsonParser.feedInput(jsonBytes, 0, jsonBytes.length);
            JsonToken token;
            while ((token = jsonParser.nextToken()) != null) {
                if (token == JsonToken.START_OBJECT) {
                    JsonNode node = objectMapper.readTree(jsonParser);
                    return Optional.of(node);
                }
                if (token == JsonToken.NOT_AVAILABLE) {
                    break;
                }
            }
        } catch (IOException e) {
            // Do nothing
        }
        return Optional.empty();
    }
}

Output result:

{}
{}
null
null
{"name":"Tom"}
{"name":"Tom"}
null
null
{"name":"Tom","age":25}
{"name":"Tom","age":25}
{"name":"Tom","age":25}
null
null
null
null
null
{"name":"Tom","age":25,"hobby":["reading","hiking"]}
{"name":"Tom","age":25,"hobby":["reading","hiking"]}
{"name":"Tom","age":25,"hobby":["reading","hiking"]}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":""}}
null
null
null
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York"}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York"}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":""}}
null
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""}}
{"name":"Tom","age":25,"hobby":["reading","hiking"],"address":{"street":"","city":"New York","state":"","zip":""}}

[yawkat]

I don't think there's any point to using NonBlockingJsonParser in that code. It is basically the approach @cowtowncoder suggested with a normal JsonParser.

[cowtowncoder]

Right -- and ObjectMapper.readTree() cannot be called since it may bump into JsonToken.NOT_AVAILABLE. Non-blocking parser must be used directly, as databind (including ObjectMapper) isn't designed for handling incomplete non-blocking input. API for non-blocking parser is that of JsonParser with just addition of that one more JsonToken` type, and method(s) to feed input instead of reading from blocking source.

But whether it is practical here or not is open question.