Inconsistent Read Behavior During Concurrent Writes.

[shaj13]

We’re working on a POC to use BoltDB as the underlying database with MultiRaft for a scalable stats infrastructure. Each shard has its own independent BoltDB instance. In the background, the service scrapes stats from target sidecar pods and writes them to BoltDB. However, we're facing an issue when reading the DB concurrently with writes — during active scrapes, reading the DB sometimes returns partial or no key-value data. For example, out of 10 expected keys, we may see only a few or none. When there are no ongoing writes, the full dataset is visible.

Is this behavior expected? To be honest, I've used BoltDB before and never encountered this issue. If I recall correctly, BoltDB internally uses a read-write mutex to manage transactions.

type metricsdb struct {
	mu     sync.Mutex
	groups map[uint64]*bolt.DB
}

func (mdb *metricsdb) Write(gid uint64) (*dbtxn, error) {
	mdb.mu.Lock()
	defer mdb.mu.Unlock()

	txn, err := mdb.groups[gid].Begin(true)
	if err != nil {
		return nil, err
	}

	return &dbtxn{txn: txn}, nil
}

type dbtxn struct {
	txn *bolt.Tx
}

func (t *dbtxn) Put(res metrics.Resource, buckets ...string) error {
	var (
		b   *bolt.Bucket
		err error
	)

	for _, name := range buckets {
		if b == nil {
			b, err = t.txn.CreateBucketIfNotExists([]byte(name))
		} else {
			b, err = b.CreateBucketIfNotExists([]byte(name))
		}

		if err != nil {
			return err
		}
	}

	return b.Put([]byte(res.UID()), flatbuf.Marshal(res))
}

func (t *dbtxn) Commit() error {
	return t.txn.Commit()
}

Dump All DB Data.

				for gid, db := range dbs {
					fmt.Fprintf(w, "\n---\nGroup(Shard): %x\n", gid)
					db.View(func(tx *bolt.Tx) error {
						return tx.ForEach(func(root []byte, b *bolt.Bucket) error {
							return b.ForEachBucket(func(leaf []byte) error {
								return b.Bucket(leaf).ForEach(func(k, v []byte) error {
									res := flatbuf.Unmarshal(v)

									row := []string{string(root)}

									res.Labels.Range(func(key string, value metrics.Value) bool {
										row = append(row, value.String())
										return true
									})

									if res.Value.Kind() != metrics.KindSliceValue {
										return nil
									}

									for _, v := range res.Value.SliceValue() {
										row = append(row, v.String())
									}

									fmt.Fprintf(w, "%s\n", strings.Join(row, ","))
									return nil
								})
							})
						})
					})
				}

				return nil
			})

The second question is about performance. Our goal is to store stats in a key-value database rather than a time-series database because we need to continuously update the same metric and later read the entire DB to aggregate data across targets before sending it to OpenTelemetry. We're planning to start with 64 shards. Do you think BoltDB can handle that volume of reads and writes? I know etcd is built on top of BoltDB and offers good read performance, but I haven't been able to find concrete performance numbers for our use case.

[ahrtr]

Your program of dumping All DB data seems not correct, please refer to example below on how to dump all data,

bbolt/compact.go

Lines 91 to 119 in c0f7c3c

	func walk(db *DB, walkFn walkFunc) error {
	return db.View(func(tx *Tx) error {
	return tx.ForEach(func(name []byte, b *Bucket) error {
	return walkBucket(b, nil, name, nil, b.Sequence(), walkFn)
	})
	})
	}
	func walkBucket(b *Bucket, keypath [][]byte, k, v []byte, seq uint64, fn walkFunc) error {
	// Execute callback.
	if err := fn(keypath, k, v, seq); err != nil {
	return err
	}
	// If this is not a bucket then stop.
	if v != nil {
	return nil
	}
	// Iterate over each child key/value.
	keypath = append(keypath, k)
	return b.ForEach(func(k, v []byte) error {
	if v == nil {
	bkt := b.Bucket(k)
	return walkBucket(bkt, keypath, k, nil, bkt.Sequence(), fn)
	}
	return walkBucket(b, keypath, k, v, b.Sequence(), fn)
	})
	}

In term of performance,

• bbolt is good for read intensive workloads. Sequential write performance is also fast but random writes can be slow. You can use DB.Batch() or add a write-ahead log to help mitigate this issue.
• Several GB data should be fine. Some users also tested dozens of GB.

[shaj13]

Yes, we use a WAL. But how exactly does it improve performance? Or is it simply because all writes go to the WAL first, so we don’t really need to worry about when the actual write to the main storage happens?