Add signer balance metric #137
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| package metrics | ||
|
|
||
| import ( | ||
| "context" | ||
| "math/big" | ||
| "time" | ||
|
|
||
| ecommon "github.com/ethereum/go-ethereum/common" | ||
| "github.com/prometheus/client_golang/prometheus" | ||
| "github.com/rs/zerolog/log" | ||
| "github.com/shutter-network/rolling-shutter/rolling-shutter/medley/service" | ||
| ) | ||
|
|
||
| const ( | ||
| balancePollInterval = 60 * time.Second | ||
| balancePollTimeout = 10 * time.Second | ||
| ) | ||
|
|
||
| // weiPerEther is the divisor turning a wei balance into ether. | ||
| var weiPerEther = new(big.Float).SetFloat64(1e18) | ||
|
|
||
| var SignerBalanceEther = newSignerBalanceGauge() | ||
|
|
||
| // A GaugeVec with no labels, so the series stays absent until a balance has | ||
| // been read. A plain Gauge would be registered holding 0, and a restart while | ||
| // the RPC endpoint is down would publish 0 ether and fire the low-balance | ||
| // alert. The exposed series is the same either way. | ||
| func newSignerBalanceGauge() *prometheus.GaugeVec { | ||
| return prometheus.NewGaugeVec( | ||
| prometheus.GaugeOpts{ | ||
| Namespace: "shutter_api", | ||
| Name: "signer_balance_ether", | ||
| Help: "Balance of the signer account in ether.", | ||
| }, | ||
| []string{}, | ||
| ) | ||
| } | ||
|
|
||
| func initBalanceMetrics() { | ||
| prometheus.MustRegister(SignerBalanceEther) | ||
| } | ||
|
|
||
| // BalanceReader reads an account balance from the chain. It is the single | ||
| // method of ethclient.Client that the poller needs. | ||
| type BalanceReader interface { | ||
| BalanceAt(ctx context.Context, account ecommon.Address, blockNumber *big.Int) (*big.Int, error) | ||
| } | ||
|
|
||
| // BalancePoller publishes the signer's balance as a gauge. The signer pays gas | ||
| // for every identity registration, so an empty account takes registration down; | ||
| // without this the service spends from an account it cannot observe. | ||
| type BalancePoller struct { | ||
| client BalanceReader | ||
| address ecommon.Address | ||
| } | ||
|
|
||
| func NewBalancePoller(client BalanceReader, address ecommon.Address) *BalancePoller { | ||
| return &BalancePoller{client: client, address: address} | ||
| } | ||
|
|
||
| func (p *BalancePoller) Start(ctx context.Context, runner service.Runner) error { | ||
| runner.Go(func() error { | ||
| ticker := time.NewTicker(balancePollInterval) | ||
| defer ticker.Stop() | ||
|
|
||
| // Publish once up front so the series exists from the first scrape | ||
| // rather than only after a full interval. | ||
| p.poll(ctx) | ||
|
|
||
| for { | ||
| select { | ||
| case <-ctx.Done(): | ||
| return ctx.Err() | ||
| case <-ticker.C: | ||
| p.poll(ctx) | ||
| } | ||
| } | ||
| }) | ||
| return nil | ||
| } | ||
|
|
||
| // poll reads the balance and updates the gauge. A failed read leaves the gauge | ||
| // at its previous value: publishing a zero would look like an empty account and | ||
| // fire the very alert this metric exists to raise. Errors are never returned, | ||
| // because the poller shares an error group with the API and a transient RPC | ||
| // failure must not shut the service down. | ||
| func (p *BalancePoller) poll(ctx context.Context) { | ||
| ctx, cancel := context.WithTimeout(ctx, balancePollTimeout) | ||
| defer cancel() | ||
|
|
||
| wei, err := p.client.BalanceAt(ctx, p.address, nil) | ||
| if err != nil { | ||
|
Collaborator
There was a problem hiding this comment. This counts shutdown as an RPC failure. A cancelled parent context lands in the same error branch as a real failure, so a redeploy that catches a poll in flight logs an error and increments FailedRPCCalls. DeadlineExceeded should stay counted though, a node that doesn't answer in 10s is a genuine failure. So guard on cancellation only: |
||
| log.Err(err).Str("address", p.address.Hex()).Msg("failed to query signer balance") | ||
| FailedRPCCalls.Inc() | ||
| return | ||
| } | ||
|
|
||
| ether, _ := new(big.Float).Quo(new(big.Float).SetInt(wei), weiPerEther).Float64() | ||
| SignerBalanceEther.WithLabelValues().Set(ether) | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| package metrics | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "math/big" | ||
| "testing" | ||
| "time" | ||
|
|
||
| ecommon "github.com/ethereum/go-ethereum/common" | ||
| "github.com/prometheus/client_golang/prometheus/testutil" | ||
| "github.com/stretchr/testify/mock" | ||
| "github.com/stretchr/testify/require" | ||
| ) | ||
|
|
||
| // mockBalanceReader is local to this test: the poller needs a single method, | ||
| // while tests/mock.MockEthClient serves the usecase's wider interface. | ||
| type mockBalanceReader struct { | ||
| mock.Mock | ||
| } | ||
|
|
||
| func (m *mockBalanceReader) BalanceAt(ctx context.Context, account ecommon.Address, blockNumber *big.Int) (*big.Int, error) { | ||
| args := m.Called(ctx, account, blockNumber) | ||
| balance, _ := args.Get(0).(*big.Int) | ||
| return balance, args.Error(1) | ||
| } | ||
|
|
||
| // Deliberately not a real signer address. | ||
| var testSignerAddress = ecommon.HexToAddress("0x1111111111111111111111111111111111111111") | ||
|
|
||
| // A registered-but-never-read balance must expose nothing at all. Publishing a | ||
| // zero would be read as an empty signer account. | ||
| func TestBalanceIsAbsentBeforeTheFirstRead(t *testing.T) { | ||
| require.Zero(t, testutil.CollectAndCount(newSignerBalanceGauge())) | ||
| } | ||
|
|
||
| func TestPollPublishesBalanceInEther(t *testing.T) { | ||
| // 1.5 ether, to catch a big.Int quotient truncating the fraction away. | ||
| wei := new(big.Int).Add( | ||
| new(big.Int).SetUint64(1e18), | ||
| new(big.Int).SetUint64(5e17), | ||
| ) | ||
|
|
||
| client := &mockBalanceReader{} | ||
| client.On("BalanceAt", mock.Anything, testSignerAddress, (*big.Int)(nil)).Return(wei, nil) | ||
|
|
||
| NewBalancePoller(client, testSignerAddress).poll(context.Background()) | ||
|
|
||
| require.InDelta(t, 1.5, testutil.ToFloat64(SignerBalanceEther.WithLabelValues()), 1e-9) | ||
| client.AssertExpectations(t) | ||
| } | ||
|
|
||
| func TestPollLeavesGaugeUntouchedOnError(t *testing.T) { | ||
| SignerBalanceEther.WithLabelValues().Set(2) | ||
| failuresBefore := testutil.ToFloat64(FailedRPCCalls) | ||
|
|
||
| client := &mockBalanceReader{} | ||
| client.On("BalanceAt", mock.Anything, testSignerAddress, (*big.Int)(nil)). | ||
| Return(nil, errors.New("rpc unavailable")) | ||
|
|
||
| NewBalancePoller(client, testSignerAddress).poll(context.Background()) | ||
|
|
||
| // A zero here would read as an empty account and fire a false alert. | ||
| require.InDelta(t, 2.0, testutil.ToFloat64(SignerBalanceEther.WithLabelValues()), 1e-9) | ||
| require.Equal(t, failuresBefore+1, testutil.ToFloat64(FailedRPCCalls)) | ||
| } | ||
|
|
||
| // The read must carry its own deadline, so a hung RPC cannot stall the loop | ||
| // and stop every later poll. | ||
| func TestPollBoundsTheReadWithATimeout(t *testing.T) { | ||
| var got context.Context | ||
|
|
||
| client := &mockBalanceReader{} | ||
| client.On("BalanceAt", mock.Anything, testSignerAddress, (*big.Int)(nil)). | ||
| Run(func(args mock.Arguments) { got = args.Get(0).(context.Context) }). | ||
| Return(big.NewInt(0), nil) | ||
|
|
||
| NewBalancePoller(client, testSignerAddress).poll(context.Background()) | ||
|
|
||
| deadline, ok := got.Deadline() | ||
| require.True(t, ok, "read was made without a deadline") | ||
| require.LessOrEqual(t, time.Until(deadline), balancePollTimeout) | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This gives the wrong reason for swallowing errors. Swallowing them is correct, but the stated reason isn't: the comment says the poller "shares an error group with the API and a transient RPC failure must not shut the service down". It shares the group at main.go:206 with the metrics server, and that group's error is only logged at :209, never cancelled.
The actual reason is: errgroup.WithContext cancels the group on the first non-nil error, so returning one would take metricsServer down with the poller and we'd lose the metrics entirely over a transient RPC blip. Suggested: