Skip to content

Blockchain updates for L2 extension #1430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 99 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
63b159c
Initial draft
esuwu Jun 14, 2024
69e9dbd
Changed a comment
esuwu Jun 14, 2024
b35bea1
moved a file
esuwu Jun 14, 2024
27494ad
rename a folder
esuwu Jun 14, 2024
983babc
Sent the updates channel into block appplier
esuwu Jun 20, 2024
5272899
Added some design comments
esuwu Jun 20, 2024
4fef6e3
Context in the updates function
esuwu Jun 20, 2024
dcd18f3
Merge branch 'master' into node-updates-plugin-l2
esuwu Jul 1, 2024
3cc00d5
Added protobuf
esuwu Jul 1, 2024
6752f8a
Moved proto, added conversion
esuwu Jul 16, 2024
bdc8854
Added state comparison
esuwu Jul 19, 2024
19cc302
Added a draft for retrieving history entries
esuwu Jul 22, 2024
b6fb33c
Renamed a function
esuwu Jul 22, 2024
dca31ee
Fixed most of the linter issues
esuwu Aug 1, 2024
f524fe9
Merge branch 'master' into node-updates-plugin-l2
esuwu Aug 1, 2024
18eab8c
Added paging
esuwu Aug 9, 2024
e2da71e
Added changes detector
esuwu Aug 28, 2024
da48829
Fixed a bug
esuwu Sep 12, 2024
b9318c2
Renamed functions
esuwu Sep 12, 2024
1e91dac
Merged from main
esuwu Sep 12, 2024
4dd8e5a
Merged from master
esuwu Sep 12, 2024
59f75f3
Started writing the filter
esuwu Sep 17, 2024
3a0ce55
Merged from master
esuwu Sep 17, 2024
7b77774
Commented an unused function
esuwu Sep 17, 2024
4d9419f
Added a basic filter
esuwu Oct 3, 2024
6338578
Fixed filtering
esuwu Oct 3, 2024
3d2bf74
Added tests
esuwu Oct 16, 2024
9fe8ed9
Added tests and parameters for nats subscriber
esuwu Oct 16, 2024
e057d2a
Fixed tests and linter errors
esuwu Oct 24, 2024
b2a461c
Merged from master
esuwu Oct 24, 2024
860b830
fixed gosec issues
esuwu Oct 24, 2024
ec15106
fixed one more gosec issue
esuwu Oct 24, 2024
8f70624
Mod clean.
nickeskov Nov 6, 2024
c0652df
Merge branch 'master' into node-updates-plugin-l2
nickeskov Nov 6, 2024
02f82ef
Fix lint for 'runNode' function.
nickeskov Nov 6, 2024
bb3bd64
Fix gosec issues.
nickeskov Nov 6, 2024
47320cf
Deleted stale go protobuf generated code.
nickeskov Nov 7, 2024
fb56f6b
Make some types and functions package private.
nickeskov Nov 7, 2024
89ffaf2
Update 'go.mod' go version.
nickeskov Nov 7, 2024
28086dc
Change location of 'blockchaininfo_test.go'.
nickeskov Nov 7, 2024
2f8c3c0
Add new method 'PartialBlockHeader' for 'ProtobufConverter'.
nickeskov Nov 7, 2024
5f1010c
Refactor 'BUpdatesInfoFromProto' function with unused functions removal.
nickeskov Nov 7, 2024
7dc0748
Refactored a bit 'L2ContractDataEntriesFromProto'.
nickeskov Nov 7, 2024
42bf0ff
Merge branch 'master' into node-updates-plugin-l2
nickeskov Nov 11, 2024
3f9bb83
Added a parse function
esuwu Nov 12, 2024
5d37195
Merge branch 'master' into node-updates-plugin-l2
nickeskov Nov 18, 2024
cf23bdf
Set 'NoSigs' option for nats sever to true.
nickeskov Nov 18, 2024
d7dcd5e
Fix 'runPublisher' in case when updates channel is closed.
nickeskov Nov 18, 2024
d13b940
Add nats sever shutdown.
nickeskov Nov 18, 2024
4dc94d5
Fix deadlock and refactoring.
nickeskov Nov 18, 2024
214bd46
Merge branch 'master' into node-updates-plugin-l2
nickeskov Nov 18, 2024
8fbe2b4
Fixed err not found handling
esuwu Nov 20, 2024
bb23dbc
Fixed a toolchain not found issue (#1545)
esuwu Nov 20, 2024
b65c6d9
Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 (#1546)
dependabot[bot] Nov 25, 2024
68db8b3
Handle blockchain import errors properly (#1548)
nickeskov Nov 26, 2024
d66423a
Bump github.com/elliotchance/orderedmap/v2 from 2.4.0 to 2.5.0 (#1549)
dependabot[bot] Nov 28, 2024
c598d67
Check on leasing state added. (#1550)
alexeykiselev Nov 28, 2024
bd889f5
Prepare timers for go1.23. (#1552)
nickeskov Dec 2, 2024
c24cb57
Peer address parsing refactoring (#1551)
nickeskov Dec 3, 2024
6814098
Merge branch 'master' into node-updates-plugin-l2
esuwu Dec 4, 2024
b87f8ad
Left a todo
esuwu Dec 24, 2024
cdb6083
Merged from master
esuwu Dec 24, 2024
7960470
Pulled the contract updates from snapshots, removed by block filterin…
esuwu Jan 9, 2025
a8fedad
merged from master
esuwu Jan 9, 2025
673f800
Added nats requests functionality
esuwu Jan 12, 2025
a873866
When restarted, the previous state must be nil
esuwu Jan 13, 2025
45523aa
Formatted a line correctly
esuwu Jan 13, 2025
52377b4
Merged from master
esuwu Jan 13, 2025
fb0e42f
Fixed some review issues
esuwu Jan 13, 2025
c750bae
Added an unmarshal function for blockMeta
esuwu Jan 15, 2025
6fd4d59
Write a test for a function that is not used, so the linter doesn't a…
esuwu Jan 15, 2025
91b38a8
Merged from master
esuwu Mar 18, 2025
9783694
Merge branch 'master' into node-updates-plugin-l2
esuwu Mar 25, 2025
c810622
Merge branch 'master' into node-updates-plugin-l2
esuwu Mar 31, 2025
574bea9
Support rollback (#1610)
esuwu Apr 2, 2025
62666ac
Merge branch 'master' into node-updates-plugin-l2
esuwu Apr 2, 2025
4f17cf0
Fixed merging errors
esuwu Apr 8, 2025
056639a
Fixed some issues from review
esuwu Apr 9, 2025
5817157
Added a package level error
esuwu Apr 10, 2025
30e1993
Returned a mock name line into mockery yaml
esuwu Apr 10, 2025
c1c3051
Fixed the rest of the isxues
esuwu Apr 10, 2025
5716cb0
returned the state version back
esuwu Apr 10, 2025
12aacda
Merged from master
esuwu Apr 10, 2025
97ae1f0
Replaced a read Bytes to readFull
esuwu Apr 11, 2025
3f0eb56
updated blockchain info proto scheme
esuwu Apr 28, 2025
e9bb59f
Renamed some functions
esuwu Apr 29, 2025
6229b76
Fixed a ready condition and sending empty data entries
esuwu Apr 30, 2025
5590d85
Changed the updates channel to a bufferized one
esuwu Apr 30, 2025
0d9be2c
Fixed linter issues
esuwu Apr 30, 2025
1853195
Merged from master
esuwu Apr 30, 2025
0a06fe7
Updated the channel buffer size
esuwu Apr 30, 2025
4af1096
Moved a variable initialization
esuwu May 5, 2025
62b1b7d
Merged from master
esuwu May 13, 2025
289346b
Moved some constants, added a test
esuwu May 13, 2025
c1bae62
removed the subscriber
esuwu May 13, 2025
c77aaa3
Merge branch 'master' into node-updates-plugin-l2
esuwu May 14, 2025
019d4ad
Changed serialization
esuwu May 14, 2025
78fb3c7
Merge branch 'master' into node-updates-plugin-l2
esuwu May 14, 2025
87caeba
Merge branch 'master' into node-updates-plugin-l2
nickeskov May 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .mockery.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
quiet: False
dir: "{{.InterfaceDir}}/mocks"
mockname: "Mock{{.InterfaceName}}"
with-expecter: true
issue-845-fix: True
filename: "{{.InterfaceName | snakecase}}.go"

packages:
Expand All @@ -9,3 +11,6 @@ packages:
Header:
Protocol:
Handler:
github.com/wavesplatform/gowaves/pkg/blockchaininfo:
interfaces:
UpdatesPublisherInterface:
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,8 @@ mock:
mockgen -source pkg/p2p/peer/peer.go -destination pkg/mock/peer.go -package mock Peer
mockgen -source pkg/state/api.go -destination pkg/mock/state.go -package mock State
mockgen -source pkg/grpc/server/api.go -destination pkg/mock/grpc.go -package mock GrpcHandlers
mockery --dir=pkg/mock --filename=blockchaininfo_types.go --outpkg=mock # The interface name must be specified in .mockery.yaml, see examples there.


proto:
@protoc --proto_path=pkg/grpc/protobuf-schemas/proto/ --go_out=./ --go_opt=module=$(MODULE) --go-vtproto_out=./ --go-vtproto_opt=features=marshal_strict+unmarshal+size --go-vtproto_opt=module=$(MODULE) pkg/grpc/protobuf-schemas/proto/waves/*.proto
Expand Down
2 changes: 1 addition & 1 deletion cmd/importer/importer.go
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ func runImporter(c *cfg) error {
return err
}

st, err := state.NewState(c.dataDirPath, false, c.params(fds), ss, false)
st, err := state.NewState(c.dataDirPath, false, c.params(fds), ss, false, nil)
if err != nil {
return fmt.Errorf("failed to create state: %w", err)
}
Expand Down
170 changes: 124 additions & 46 deletions cmd/node/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,13 @@ import (

"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus/promhttp"

"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"golang.org/x/sync/errgroup"

"github.com/wavesplatform/gowaves/pkg/api"
"github.com/wavesplatform/gowaves/pkg/blockchaininfo"
"github.com/wavesplatform/gowaves/pkg/crypto"
"github.com/wavesplatform/gowaves/pkg/grpc/server"
"github.com/wavesplatform/gowaves/pkg/libs/microblock_cache"
Expand Down Expand Up @@ -71,51 +73,53 @@ var defaultPeers = map[string]string{
type config struct {
isParsed bool

logLevel zapcore.Level
logDevelopment bool
logNetwork bool
logNetworkData bool
logFSM bool
statePath string
blockchainType string
peerAddresses string
declAddr string
nodeName string
cfgPath string
apiAddr string
apiKey string
apiMaxConnections int
rateLimiterOptions string
grpcAddr string
grpcAPIMaxConnections int
enableMetaMaskAPI bool
enableMetaMaskAPILog bool
enableGrpcAPI bool
blackListResidenceTime time.Duration
buildExtendedAPI bool
serveExtendedAPI bool
buildStateHashes bool
bindAddress string
disableOutgoingConnections bool
minerVoteFeatures string
disableBloomFilter bool
reward int64
obsolescencePeriod time.Duration
walletPath string
walletPassword string
limitAllConnections uint
minPeersMining int
disableMiner bool
profiler bool
prometheus string
metricsID int
metricsURL string
dropPeers bool
dbFileDescriptors uint
newConnectionsLimit int
disableNTP bool
microblockInterval time.Duration
enableLightMode bool
logLevel zapcore.Level
logDevelopment bool
logNetwork bool
logNetworkData bool
logFSM bool
statePath string
blockchainType string
peerAddresses string
declAddr string
nodeName string
cfgPath string
apiAddr string
apiKey string
apiMaxConnections int
rateLimiterOptions string
grpcAddr string
grpcAPIMaxConnections int
enableMetaMaskAPI bool
enableMetaMaskAPILog bool
enableGrpcAPI bool
blackListResidenceTime time.Duration
buildExtendedAPI bool
serveExtendedAPI bool
buildStateHashes bool
bindAddress string
disableOutgoingConnections bool
minerVoteFeatures string
disableBloomFilter bool
reward int64
obsolescencePeriod time.Duration
walletPath string
walletPassword string
limitAllConnections uint
minPeersMining int
disableMiner bool
profiler bool
prometheus string
metricsID int
metricsURL string
dropPeers bool
dbFileDescriptors uint
newConnectionsLimit int
disableNTP bool
microblockInterval time.Duration
enableLightMode bool
enableBlockchainUpdatesPlugin bool
BlockchainUpdatesL2Address string
}

var errConfigNotParsed = stderrs.New("config is not parsed")
Expand Down Expand Up @@ -168,6 +172,7 @@ func (c *config) logParameters() {
zap.S().Debugf("disable-ntp: %t", c.disableNTP)
zap.S().Debugf("microblock-interval: %s", c.microblockInterval)
zap.S().Debugf("enable-light-mode: %t", c.enableLightMode)
zap.S().Debugf("enable-blockchain-updates-plugin: %t", c.enableBlockchainUpdatesPlugin)
}

func (c *config) parse() {
Expand Down Expand Up @@ -265,6 +270,11 @@ func (c *config) parse() {
"Interval between microblocks.")
flag.BoolVar(&c.enableLightMode, "enable-light-mode", false,
"Start node in light mode")

flag.BoolVar(&c.enableBlockchainUpdatesPlugin, "enable-blockchain-info", false,
"Turn on blockchain updates plugin")
flag.StringVar(&c.BlockchainUpdatesL2Address, "l2-contract-address", "",
"Specify the smart contract address from which the updates will be pulled")
Comment on lines +276 to +277
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if I want to receive updates from two or more chain contracts?
Maybe add TODO about it?

flag.Parse()
c.logLevel = *l
}
Expand Down Expand Up @@ -361,6 +371,20 @@ func run(nc *config) (retErr error) {
return nil
}

func initBlockchainUpdatesPlugin(ctx context.Context,
l2addressContract string,
enableBlockchainUpdatesPlugin bool,
updatesChannel chan<- proto.BUpdatesInfo, firstBlock *bool,
) (*proto.BlockchainUpdatesPluginInfo, error) {
l2address, cnvrtErr := proto.NewAddressFromString(l2addressContract)
if cnvrtErr != nil {
return nil, errors.Wrapf(cnvrtErr, "failed to convert L2 contract address %q", l2addressContract)
}
bUpdatesPluginInfo := proto.NewBlockchainUpdatesPluginInfo(ctx, l2address, updatesChannel,
firstBlock, enableBlockchainUpdatesPlugin)
return bUpdatesPluginInfo, nil
}

func runNode(ctx context.Context, nc *config) (_ io.Closer, retErr error) {
cfg, err := blockchainSettings(nc)
if err != nil {
Expand Down Expand Up @@ -392,11 +416,39 @@ func runNode(ctx context.Context, nc *config) (_ io.Closer, retErr error) {
return nil, errors.Wrap(err, "failed to create state parameters")
}

st, err := state.NewState(path, true, params, cfg, nc.enableLightMode)
updatesChannel := make(chan proto.BUpdatesInfo, blockchaininfo.UpdatesBufferedChannelSize)
firstBlock := false
bUpdatesPluginInfo, initErr := initBlockchainUpdatesPlugin(ctx, nc.BlockchainUpdatesL2Address,
nc.enableBlockchainUpdatesPlugin, updatesChannel, &firstBlock)
if initErr != nil {
return nil, errors.Wrap(initErr, "failed to initialize blockchain updates plugin")
}
st, err := state.NewState(path, true, params, cfg, nc.enableLightMode, bUpdatesPluginInfo)
if err != nil {
return nil, errors.Wrap(err, "failed to initialize node's state")
}
defer func() { retErr = closeIfErrorf(st, retErr, "failed to close state") }()
makeExtensionReadyFunc := func() {
bUpdatesPluginInfo.MakeExtensionReady()
}

if nc.enableBlockchainUpdatesPlugin {
bUpdatesExtension, bUErr := initializeBlockchainUpdatesExtension(ctx, cfg, nc.BlockchainUpdatesL2Address,
updatesChannel, &firstBlock, st, makeExtensionReadyFunc)
if bUErr != nil {
bUpdatesExtension.Close()
return nil, errors.Wrap(bUErr, "failed to run blockchain updates plugin")
}
go func() {
publshrErr := bUpdatesExtension.RunBlockchainUpdatesPublisher(ctx,
cfg.AddressSchemeCharacter)
if publshrErr != nil {
zap.S().Fatalf("Failed to run blockchain updates publisher: %v", publshrErr)
}
}()
zap.S().Info("The blockchain info extension started pulling info from smart contract address",
nc.BlockchainUpdatesL2Address)
}

features, err := minerFeatures(st, nc.minerVoteFeatures)
if err != nil {
Expand Down Expand Up @@ -794,6 +846,32 @@ func runAPIs(
return nil
}

func initializeBlockchainUpdatesExtension(
ctx context.Context,
cfg *settings.BlockchainSettings,
l2ContractAddress string,
updatesChannel chan proto.BUpdatesInfo,
firstBlock *bool,
state state.State,
makeExtensionReady func(),
) (*blockchaininfo.BlockchainUpdatesExtension, error) {
bUpdatesExtensionState, err := blockchaininfo.NewBUpdatesExtensionState(
blockchaininfo.StoreBlocksLimit,
cfg.AddressSchemeCharacter,
l2ContractAddress,
state,
)
if err != nil {
return nil, errors.Wrap(err, "failed to initialize blockchain updates extension state")
}
l2address, cnvrtErr := proto.NewAddressFromString(l2ContractAddress)
if cnvrtErr != nil {
return nil, errors.Wrapf(cnvrtErr, "failed to convert L2 contract address %q", l2ContractAddress)
}
return blockchaininfo.NewBlockchainUpdatesExtension(ctx, l2address, updatesChannel,
bUpdatesExtensionState, firstBlock, makeExtensionReady), nil
}

func FromArgs(scheme proto.Scheme, c *config) func(s *settings.NodeSettings) error {
return func(s *settings.NodeSettings) error {
s.DeclaredAddr = c.declAddr
Expand Down
2 changes: 1 addition & 1 deletion cmd/rollback/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ func main() {
params.BuildStateHashes = *buildStateHashes
params.StoreExtendedApiData = *buildExtendedAPI

s, err := state.NewState(*statePath, true, params, cfg, false)
s, err := state.NewState(*statePath, true, params, cfg, false, nil)
if err != nil {
zap.S().Error(err)
return
Expand Down
2 changes: 1 addition & 1 deletion cmd/statehash/statehash.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ func run() error {
params.BuildStateHashes = true
params.ProvideExtendedApi = false

st, err := state.NewState(statePath, false, params, ss, false)
st, err := state.NewState(statePath, false, params, ss, false, nil)
if err != nil {
zap.S().Errorf("Failed to open state at '%s': %v", statePath, err)
return err
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ require (
github.com/jinzhu/copier v0.4.0
github.com/mr-tron/base58 v1.2.0
github.com/nats-io/nats-server/v2 v2.11.3
github.com/nats-io/nats.go v1.41.2
github.com/neilotoole/slogt v1.1.0
github.com/ory/dockertest/v3 v3.12.0
github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2
Expand Down
105 changes: 105 additions & 0 deletions pkg/blockchaininfo/blockchaininfo.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
package blockchaininfo

import (
"bytes"
"encoding/binary"
"strconv"
"strings"

"github.com/pkg/errors"
"github.com/wavesplatform/gowaves/pkg/proto"
)

const EpochKeyPrefix = "epoch_"
const blockMeta0xKeyPrefix = "block_0x"
Comment on lines +13 to +14
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be package private.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is package private?


// Helper function to read uint64 from bytes.
func readInt64(data *bytes.Reader) (int64, error) {
var num int64
err := binary.Read(data, binary.BigEndian, &num)
if err != nil {
return 0, err
}
return num, nil
}

// Decode base64 and extract blockHeight and height.
func extractEpochFromBlockMeta(blockMetaValue []byte) (int64, error) {
var blockMeta BlockMeta
err := blockMeta.UnmarshalBinary(blockMetaValue)
if err != nil {
return 0, errors.Errorf("failed to unmarshal blockMeta, %v", err)
}

return blockMeta.BlockEpoch, nil
}

func filterEpochEntry(entry proto.DataEntry, beforeHeight uint64) ([]proto.DataEntry, error) {
key := entry.GetKey()
// Extract the part after "epoch_"
epochStr := key[len(EpochKeyPrefix):]

epochNumber, err := strconv.ParseUint(epochStr, 10, 64)
if err != nil {
return nil, err
}

// Return this entry only if epochNumber is greater than beforeHeight
if epochNumber > beforeHeight {
return []proto.DataEntry{entry}, nil
}
return nil, nil
}

func filterBlock0xEntry(entry proto.DataEntry, beforeHeight uint64) ([]proto.DataEntry, error) {
// Extract blockHeight and height from base64.
binaryEntry, ok := entry.(*proto.BinaryDataEntry)
if !ok {
return nil, errors.New("failed to convert block meta key to binary data entry")
}
epoch, err := extractEpochFromBlockMeta(binaryEntry.Value)
if err != nil {
return nil, errors.Errorf("failed to filter data entries, %v", err)
}

if epoch < 0 {
return nil, errors.New("epoch is less than 0")
}
// Return this entry only if epochNumber is greater than beforeHeight
if uint64(epoch) > beforeHeight {
return []proto.DataEntry{entry}, nil
}
return nil, nil
}

func filterDataEntries(beforeHeight uint64, dataEntries []proto.DataEntry) ([]proto.DataEntry, error) {
var filteredDataEntries []proto.DataEntry

for _, entry := range dataEntries {
key := entry.GetKey()

switch {
// Filter "epoch_" prefixed keys.
case strings.HasPrefix(key, EpochKeyPrefix):
entryOrNil, err := filterEpochEntry(entry, beforeHeight)
if err != nil {
return nil, err
}
filteredDataEntries = append(filteredDataEntries, entryOrNil...)

// Filter block_0x binary entries.
case strings.HasPrefix(key, blockMeta0xKeyPrefix):
entryOrNil, err := filterBlock0xEntry(entry, beforeHeight)
if err != nil {
return nil, err
}
filteredDataEntries = append(filteredDataEntries, entryOrNil...)

// Default case to handle non-epoch and non-base64 entries.
default:
filteredDataEntries = append(filteredDataEntries, entry)
}
}

return filteredDataEntries, nil
}
Loading
Loading