Skip to content

Conversation

nikooo777
Copy link

@shyba and I are not fluent in Erlang, so pardon us if slop is spotted!
we'll try our best to clean it up during review.

Why

  • Prevent OOM and long latency when serving large data via /{id}/data.
  • Add HTTP Range support for media/partial reads.
  • Preserve HTTPSig semantics for streamed responses.

What's Included

  • Range requests for /{id}/data and /tx/{id}/data
    • Parses Range (bytes=start-end | start- | -suffix).
    • Returns correct 206 with Content‑Range and Accept‑Ranges, or 416 with bytes */total.
    • Small ranges are buffered and fully signed (includes Content‑Digest); large ranges stream.
  • Memory‑efficient streaming
    • Large full GETs and large ranges stream via chunked transfer encoding.
    • Open‑ended ranges (bytes=start-) stream chunk‑by‑chunk; no unbounded buffering.
    • Streamed responses are header‑signed (status + ao-result).
  • Storage‑agnostic reader
    • New hb_data_reader: HEAD‑based metadata discovery with Range fallback; read_range/stream helpers.
    • New hb_http_range: Range parsing and Content‑Range builders.
  • HTTP server integration
    • hb_http_server intercepts data endpoints, decides buffer vs. stream based on size threshold.
    • Adds reply_streamed[_signed]/send_streamed_response in hb_http.
  • HTTPSig improvements
    • dev_codec_httpsig: components serialization now directly uses hb_message:convert (fixes badmatch, normalizes headers/body for tests).
  • Defaults and config
    • hb_opts adds:
      • streaming_threshold (10MB)
      • stream_chunk_size (1MB)
      • range_default_content_type (application/octet-stream)
      • range_requests_enabled (true).
      • stream_signature (off | header)
    • Streaming is disabled unless you opt in.
  • Minor cleanup: rebar.config dedup.

Notes

  • Buffered responses (small ranges/small full GETs): standard HTTPSig response with Content‑Digest.
  • Streamed responses (large full or large ranges): header‑only HTTPSig;
  • CORS headers preserved; Accept‑Ranges: bytes always set for data endpoints.

shyba and others added 2 commits October 4, 2025 01:13
Implement signed streaming responses using HTTPSig for large data and range requests.
@nikooo777 nikooo777 force-pushed the http-range-streaming branch from 0bbf3f9 to 16248cc Compare October 3, 2025 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants