Fix long JSON preprocessing

[davidalejandroaguilar]

Description

This PR:

• Fixes long JSON preprocessing.
  • The current implementation is clearing the stack too aggressively when it finds a JSON object, which causes it to lose partial JSON data that should be kept for the next object. This is particularly noticeable when parsing larger JSON data.
• Adds FixtureHelpers to spec/support which are automatically loaded in specs, so we can compare fixtures to expected results more easily.

I tried to follow the existing code style, let me know if you'd like to me clean something up.

Example

Previous

When the stack is:

  {
    "title": "Functional Programming Perfection",
    "tweet": "Embrace Ruby\'s functional programming capabilities and write concise, expressive, and maintainable code. #RubyFP #LessCodeMoreFun"
  },
  {

The parsed JSON object is:

{"title"=>"Functional Programming Perfection", "tweet"=>"Embrace Ruby's functional programming capabilities and write concise, expressive, and maintainable code. #RubyFP #LessCodeMoreFun"}

But the resulting stack is:

    "title": "Object-Oriente

So as the stack starts filling up again:

    "title": "Object-Oriented Onslaught",
    "tweet": "Master the art of object-oriented programming in Ruby and create modular, reusable, and scalable software. #RubyOOP #CodeArchitecture"
  },

We'll get an error parsing it:

Anthropic JSON Error (spotted in ruby-anthropic 0.3.2): undefined method `[]' for nil

Now

When the stack is:

 {
    "title": "Functional Programming Perfection",
    "tweet": "Embrace Ruby\'s functional programming capabilities and write concise, expressive, and maintainable code. #RubyFP #LessCodeMoreFun"
  },
  {

The parsed JSON object is:

{"title"=>"Functional Programming Perfection", "tweet"=>"Embrace Ruby's functional programming capabilities and write concise, expressive, and maintainable code. #RubyFP #LessCodeMoreFun"}

And the resulting stack is:

{
    "title": "Object-Oriente

So as the stack starts filling up again:

{
    "title": "Object-Oriented Onslaught",
    "tweet": "Master the art of object-oriented programming in Ruby and create modular, reusable, and scalable software. #RubyOOP #CodeArchitecture"
  },

We'll be able to get the next JSON object:

{"title"=>"Object-Oriented Onslaught", "tweet"=>"Master the art of object-oriented programming in Ruby and create modular, reusable, and scalable software. #RubyOOP #CodeArchitecture"}

All Submissions:

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?
• Have you added an explanation of what your changes do and why you'd like us to include them?

[alexrudall]

Thanks, @davidalejandroaguilar - is there a way this could be a breaking change? I can't imagine it breaking anyone's existing use of the gem, should just make it more reliable basically?

[davidalejandroaguilar]

@alexrudall Appreciate you taking a look.

I also don't think this would be a breaking change, it should indeed make it more reliable.

However, I can add more specs to cover more edge cases, such as an empty JSON, a JSON which is not enclosed in an array, etc. if you're OK with additional VCRs (kinda followed the existing style, didn't want to go overboard). I'm OK paying for the API requests.

[davidalejandroaguilar]

@alexrudall Added additional specs, these are the specs we'd have:

1. Pre-processing small JSON array.
2. Pre-processing large JSON array.
3. Pre-processing empty JSON array.
4. Pre-processing single small JSON object.
5. Pre-processing single large JSON object.
6. Pre-processing single empty JSON object.

---

• With the old approach, number 2 is the only one that fails.
• With the new approach, all succeed.

Let me know if you can think of additional edge cases, but I think we should be OK releasing this as a non-breaking change.

[davidalejandroaguilar]

@alexrudall Hey there, just checking in on this 😃 , is there anything holding it up?

[alexrudall]

@alexrudall Hey there, just checking in on this 😃 , is there anything holding it up?

Apologies, only my own prioritisation! Thank you for chasing, I'll try to get this merged this week.