Paging

[mgussekloo]

Thanks for this awesome library! I'm parsing a huge JSON with this, and I'd like to do it in chunks. Is there any way to tell this library to iterate over the "first 1000", then the "second 1000", etc.?

Any help appreciated.

[cerbero90]

Hi @mgussekloo, glad this library can help you!

JSON Parser iterates over items one by one.

To process items in chunks, we can collect the items in a temporary array and process the array every time it is filled with 1000 items:

$chunk = [];

foreach (new JsonParser($source) as $key => $value) {
    $chunk[$key] = $value;

    // process and clear the chunk every time it contains 1000 items
    if (count($chunk) === 1000) {
        processChunk($chunk);

        $chunk = [];
    }
}

// process and clear the remaining items in the chunk (the last chunk might contain less than 1000 items)
if ($chunk) {
    processChunk($chunk);

    $chunk = [];
}

[mgussekloo]

Thanks, much appreciate it! I came up with a similar solution, just iterating over the items as quickly as possible using the lazy() method. I was hoping there would be an even faster option (because this gets slow after a few pages).

This is what I came up with for future reference.

static function jsonProcessPaged($json, $page, $perPage, $callback) {
    	$items = [];
        $i = 0; $bailed = false;

	$rows = JsonParser::parse($json)->lazy();

        foreach ($rows as $lazyRow) {
            $i++;

        	if ($i <= $page * $perPage) {
        		continue;
        	}

        	if ($i > $page * $perPage + $perPage) {
        		$bailed = true;
        		break;
        	}

        	$row = (object)$lazyRow->toArray();
            $items = $callback($row, $items);
        }

        return (object)[
        	'completed' => !$bailed,
        	'items' => $items
        ];
}