puya-ts superbox: simple box list operations backed by multiple boxes

Overview

Superbox enables performing simple list operations backed by multiple boxes.

Supported write ops:

• append value(s) (tail only)
• delete value(s) (anywhere)

Read ops:

• Get value data by index
• Get value location [box number, byte offset] by index
• Get metadata, e.g. value count, byte length count, etc.

Restrictions:

• Fixed value sizes
• Not able to insert data at middle (see future work)

Status

Beta

Setup

npm i @d13co/superbox

Usage

Import the sb* functions from your puya-ts files, e.g.

import { sbCreate } from "@d13co/superbox";

// ... inside your contract:

public startAddingData() {
  sbCreate("myBox", 2048, 2, "uint16")

  const singleValue = new arc4.Uint16(13).bytes)

  // add one value
  sbAppend("myBox", singleValue)
}

public addMoreData() {
  sbAppend("myBox", arc4.Uint16(13).bytes)

  const multipleValues = new arc4.Uint16(13).bytes
    .concat(new arc4.Uint16(37).bytes)
    .concat(new arc4.Uint16(41).bytes)
    .concat(new arc4.Uint16(60).bytes)

  // add multiple values
  sbAppend("myBox", multipleValues)
}

API Reference

sbCreate(name, maxBoxSize, valueSize, valueSchema)

Create a new superbox metadata record.

Parameters:

• name (string) – Superbox name/prefix.
• maxBoxSize (uint64) – Maximum size of each box in bytes.
• valueSize (uint64) – Size of each stored value in bytes.
• valueSchema (string) – Description of value type (e.g., "uint32").

---

sbAppend(name, data)

Append data to the superbox in multiples of valueSize.

Parameters:

• name (string) – Superbox name/prefix.
• data (bytes) – Data to append. Can be multiple values concatenated together.

Returns: uint64 – New total byte length of the superbox.

---

sbGetMeta(name)

Retrieve metadata for a superbox

Parameters:

• name (string) – Superbox name/prefix.

Returns: SuperboxMeta – Superbox metadata

---

sbGetData(name, valueIndex)

Retrieve a value by index.

Parameters:

• name (string) – Superbox name/prefix.
• valueIndex (uint64) – Value index.

Returns: bytes – Stored value data.

---

sbGetLocation(name, valueIndex)

Get the location of a value in the underlying box. Prefer to use sbGetData when possible.

Parameters:

• name (string) – Superbox name/prefix.
• valueIndex (uint64) – Value index.

Returns: [BoxNum, ByteOffset] – Box index and byte offset.

---

sbDeleteIndex(name, valueIndex)

Delete a single value by its index. Provide the index of the value - not the byte offset.

Parameters:

• name (string) – Superbox name/prefix.
• valueIndex (uint64) – Index of the value to delete.

Returns: uint64 – New total byte length of the superbox.

---

sbDeleteBox(name, boxNum)

Delete an entire box by its number. Only allowed if the box is empty.

Parameters:

• name (string) – Superbox name/prefix.
• boxNum (uint64) – Box number to delete.

Returns: uint64 – New total byte length of the superbox.

---

sbDeleteSuperbox(name)

Delete a superbox entirely. Only allowed if the superbox is empty.

Parameters:

• name (string) – Superbox name/prefix.

---

Error Codes

• ERR:SBEXISTS – Superbox already exists.
• ERR:DATALEN – Data length not a multiple of valueSize.
• ERR:OOB – Out of bounds access.
• ERR:DLTD – Attempt to delete a non-existent box.
• ERR:NEXIST – Superbox does not exist.
• ERR:NEMPTY – Attempt to delete a non-empty superbox.

Use case

The original use case for this is to facilitate very large VRF shuffles, where the options would not fit in a single box.

The superbox abstraction spreads option values over multiple boxes, making it easier to 1) compose the options list, and 2) retrieve the winner value by index.

A stretch goal was to facilitate a straightforward way to do multiple VRF draws without repeated winners, hence the delete operations.

Box naming convention

Boxes are currently named by convention:

Metadata boxes are suffixed with _m.

Data boxes are suffixed with an integer counter starting from 0.

Example: A Superbox named chunky with two data boxes would have the following keys;

• chunky_m
• chunky0
• chunky1

Metadata

Superbox metadata extends the configuration values specified above with some managed values:

/**
 * Metadata struct. Stored per superbox
 */
export class SuperboxMeta extends arc4.Struct<{
  /**
   * Size of individual boxes backing superbox
   */
  boxByteLengths: DynamicArray<UintN16>
  /**
   * Total data in superbox
   */
  totalByteLength: UintN64
  /**
   * Max individual box size to use
   */
  maxBoxSize: UintN64
  /**
   * Byte width of individual value.
   * Used to enforce box/value boundaries & calculate offsets when operating by value index
   */
  valueSize: UintN64
  /**
   * Informational. Schema of value, e.g. `(uint16,uint16)` for Tuple<uint16, uint16>
   */
  valueSchema: Str
}> {}

Future work

For "full marks", this could be changed to allow arbitrary data writes at any point in the superbox.

To enable this we would change one core aspect and provide an optimisation:

Insert anywhere at finite cost ~= arbitrary data box naming

To make it possible to insert at arbitrary points in large superboxes, we want to avoid having to shift the box values "downstream" of our insertion point, as this would cost too many opcodes and would likely not be doable within a single group in many cases.

Superbox operations should be atomic and leave the data in a good state.

We can facilitate this by inserting new data boxes between existing ones: we would keep the convention-based data box naming system for creating new boxes, but we would also store the data box names in their correct order in order to be able to break this convention when inserting in the middle.

E.g. After inserting data after box zero, we could end up with data box names in this order

data0
data4 (new box)
data1
data2
data3

The order of these boxes would be stored in Metadata

Available space optimization

In order to facilitate efficient insertion at arbitrary locations, we could optimize our writing strategy to take two factors into account:

• maxBoxSize: would remain the hard limit of individual box size
• optimalBoxSize: would be the box size at which we stop appending

An example would be a maxBoxSize of 1024, with a smaller optinal size of ~80% / 820 bytes. When appending data, boxes would be kept to 820 bytes, with capacity to grow from arbitrary insertions up to 1024 before a new box is created.

Development Setup - Algokit README

Pre-requisites

• Nodejs 22 or later
• AlgoKit CLI 2.5 or later
• Docker (only required for LocalNet)
• Puya Compiler 4.4.4 or later

For interactive tour over the codebase, download vsls-contrib.codetour extension for VS Code, then open the .codetour.json file in code tour extension.

Initial Setup

1. Clone the Repository

Start by cloning this repository to your local machine.

2. Install Pre-requisites

Ensure the following pre-requisites are installed and properly configured:

• Docker: Required for running a local Algorand network.
• AlgoKit CLI: Essential for project setup and operations. Verify installation with algokit --version, expecting 2.6.0 or later.

3. Bootstrap Your Local Environment

Run the following commands within the project folder:

• Setup Project: Execute algokit project bootstrap all to install dependencies and setup npm dependencies.
• Configure environment: Execute algokit generate env-file -a target_network localnet to create a .env.localnet file with default configuration for localnet.
• Start LocalNet: Use algokit localnet start to initiate a local Algorand network.

Development Workflow

Terminal

Directly manage and interact with your project using AlgoKit commands:

1. Build Contracts: algokit project run build compiles all smart contracts. You can also specify a specific contract by passing the name of the contract folder as an extra argument. For example: algokit project run build -- hello_world will only build the hello_world contract.
2. Deploy: Use algokit project deploy localnet to deploy contracts to the local network. You can also specify a specific contract by passing the name of the contract folder as an extra argument. For example: algokit project deploy localnet -- hello_world will only deploy the hello_world contract.

VS Code

For a seamless experience with breakpoint debugging and other features:

1. Open Project: In VS Code, open the repository root.
2. Install Extensions: Follow prompts to install recommended extensions.
3. Debugging:
   • Use F5 to start debugging.

JetBrains IDEs

While primarily optimized for VS Code, JetBrains IDEs are supported:

1. Open Project: In your JetBrains IDE, open the repository root.
2. Automatic Setup: The IDE should configure the Node.js environment.
3. Debugging: Use Shift+F10 or Ctrl+R to start debugging. Note: Windows users may encounter issues with pre-launch tasks due to a known bug. See JetBrains forums for workarounds.

AlgoKit Workspaces and Project Management

This project supports both standalone and monorepo setups through AlgoKit workspaces. Leverage algokit project run commands for efficient monorepo project orchestration and management across multiple projects within a workspace.

AlgoKit Generators

This template provides a set of algokit generators that allow you to further modify the project instantiated from the template to fit your needs, as well as giving you a base to build your own extensions to invoke via the algokit generate command.

Generate Smart Contract

By default the template creates a single HelloWorld contract under sueprbox folder in the smart_contracts directory. To add a new contract:

1. From the root of the project (../) execute algokit generate smart-contract. This will create a new starter smart contract and deployment configuration file under {your_contract_name} subfolder in the smart_contracts directory.
2. Each contract potentially has different creation parameters and deployment steps. Hence, you need to define your deployment logic in deploy-config.ts file.
3. Technically, you need to reference your contract deployment logic in the index.ts file. However, by default, index.ts will auto import all TypeScript deployment files under smart_contracts directory. If you want to manually import specific contracts, modify the default code provided by the template in index.ts file.

Please note, above is just a suggested convention tailored for the base configuration and structure of this template. The default code supplied by the template in the index.ts file is tailored for the suggested convention. You are free to modify the structure and naming conventions as you see fit.

Generate '.env' files

By default the template instance does not contain any env files to deploy to different networks. Using algokit project deploy against localnet | testnet | mainnet will use default values for algod and indexer unless overwritten via .env or .env.{target_network}.

To generate a new .env or .env.{target_network} file, run algokit generate env-file

Debugging Smart Contracts

This project is optimized to work with AlgoKit AVM Debugger extension. To activate it:

Refer to the commented header in the index.ts file in the smart_contracts folder.Since you have opted in to include VSCode launch configurations in your project, you can also use the Debug TEAL via AlgoKit AVM Debugger launch configuration to interactively select an available trace file and launch the debug session for your smart contract.

For information on using and setting up the AlgoKit AVM Debugger VSCode extension refer here. To install the extension from the VSCode Marketplace, use the following link: AlgoKit AVM Debugger extension.### Continuous Integration / Continuous Deployment (CI/CD)

This project uses GitHub Actions to define CI/CD workflows, which are located in the .github/workflows folder.

Please note, if you instantiated the project with --workspace flag in algokit init it will automatically attempt to move the contents of the .github folder to the root of the workspace.

AlgoKit Workspaces

To define custom algokit project run commands refer to documentation. This allows orchestration of commands spanning across multiple projects within an algokit workspace based project (monorepo).

Setting up GitHub for CI/CD workflow and TestNet deployment

1. Every time you have a change to your smart contract, and when you first initialize the project you need to build the contract and then commit the smart_contracts/artifacts folder so the output stability tests pass
2. Decide what values you want to use for the allowUpdate and allowDelete parameters specified in deploy-config.ts. When deploying to LocalNet these values are both set to true for convenience. But for non-LocalNet networks they are more conservative and use false These default values will allow the smart contract to be deployed initially, but will not allow the app to be updated or deleted if is changed and the build will instead fail. To help you decide it may be helpful to read the AlgoKit Utils app deployment documentation or the AlgoKit smart contract deployment architecture.
3. Create a Github Environment named Test. Note: If you have a private repository and don't have GitHub Enterprise then Environments won't work and you'll need to convert the GitHub Action to use a different approach. Ignore this step if you picked Starter preset.
4. Create or obtain a mnemonic for an Algorand account for use on TestNet to deploy apps, referred to as the DEPLOYER account.
5. Store the mnemonic as a secret DEPLOYER_MNEMONIC in the Test environment created in step 3.
6. The account used to deploy the smart contract will require enough funds to create the app, and also fund it. There are two approaches available here:
   • Either, ensure the account is funded outside of CI/CD. In Testnet, funds can be obtained by using the Algorand TestNet dispenser and we recommend provisioning 50 ALGOs.
   • Or, fund the account as part of the CI/CD process by using a DISPENSER_MNEMONIC GitHub Environment secret to point to a separate DISPENSER account that you maintain ALGOs in (similarly, you need to provision ALGOs into this account using the TestNet dispenser).

Continuous Integration

For pull requests and pushes to main branch against this repository the following checks are automatically performed by GitHub Actions:

• NPM dependencies are audited using better-npm-audit
• Code formatting is performed using Prettier
• Linting is checked using ESLint
• The base framework for testing is vitest, and the project includes two separate kinds of tests:
• • Algorand TypeScript smart contract unit tests, that are run using algorand-typescript-testing, which are executed in a Node.js intepreter emulating major AVM behaviour
• • End-to-end (e2e) AppClient tests that are run against algokit localnet and test the behaviour in a real network environment
• Smart contract artifacts are built
• Smart contract artifacts are checked for output stability.
• Smart contract is deployed to a AlgoKit LocalNet instance

NOTE: By default smart contract artifacts are compiled with --debug-level set to 0, to change this, modify the compiler invocation under the build script in package.json

Continuous Deployment

For pushes to main branch, after the above checks pass, the following deployment actions are performed:

• The smart contract(s) are deployed to TestNet using AlgoNode.

Please note deployment is also performed via algokit deploy command which can be invoked both via CI as seen on this project, or locally. For more information on how to use algokit deploy please see AlgoKit documentation.

Tools

This project makes use of Algorand TypeScript to build Algorand smart contracts. The following tools are in use:

• Algorand - Layer 1 Blockchain; Developer portal, Why Algorand?
• AlgoKit - One-stop shop tool for developers building on the Algorand network; docs, intro tutorial
• Algorand TypeScript - A semantically and syntactically compatible, typed TypeScript language that works with standard TypeScript tooling and allows you to express smart contracts (apps) and smart signatures (logic signatures) for deployment on the Algorand Virtual Machine (AVM); docs, examples
• AlgoKit Utils - A set of core Algorand utilities that make it easier to build solutions on Algorand.
• NPM: TypeScript packaging and dependency management.
• TypeScript: Strongly typed programming language that builds on JavaScript
• ts-node-dev: TypeScript development execution environment- Prettier: A code formatter.- ESLint: A JavaScript / TypeScript linter.
• vitest: Automated testing.
• better-npm-audit: Tool for scanning JavaScript / TypeScript environments for packages with known vulnerabilities.
• pre-commit: A framework for managing and maintaining multi-language pre-commit hooks, to enable pre-commit you need to run pre-commit install in the root of the repository. This will install the pre-commit hooks and run them against modified files when committing. If any of the hooks fail, the commit will be aborted. To run the hooks on all files, use pre-commit run --all-files.

It has also been configured to have a productive dev experience out of the box in VS Code, see the .vscode folder.

It has also been configured to have a productive dev experience out of the box in Jetbrains IDEs, see the .idea folder.