!5303ci/flake-sweep-2026-10-26 → developf441987d54

Weekly flaky test sweep: three bifrost race fixes

AI-generatedEverything on this page was written by an AI agent from the diff: the summaries, the narration, the diagrams, the flags and the glossary. It can be wrong or incomplete, so check it against the code before you rely on it.

3commits
3files
52lines changed
52lines to read
0production lines
3flaky tests fixed
278failed jobs swept

Where the diff is

1-minute explainer

What changed

Poll for unhealthy

Health flips after the hook; poll 1 s instead of reading once.

hook firestest readshealthy=false
  • poll every 1 ms, 1 s deadline
  • no-retry-before-backoff check kept

Blocking read stream

Read blocks until Reset, so only ctx.Done can win select.

bytes.Buffer → EOFblockingReadStream

old 47/100 fail with 1 ms injected sleep · new 0/100

FROST timeout headroom

Coordinator timeout 100 ms → 1 s; it bounds CPU time only.

100 ms→ 1 s

96 parallel -race copies: 3–4 fail → 0

Look here first

Poll now precedes the no-retry checkRetry check runs ≤1 s + 30 ms after the hook; still inside the 200 ms backoff that starts after the Store(false).blockscanner_rewind_test.go:670 ↗careful
648661
649662 scanner.Start(make(chan stypes.TxIn, 4), make(chan common.NetworkFee, 4))
650663 select {
651664 case <-fetcher.failureObserved:
652665 case <-time.After(time.Second):
653666 c.Fatalf("mode %s did not reach persistence failure", mode)
654667 }
668+ // The failure hook runs before the scanner marks itself unhealthy, so
669+ // poll rather than read health right after the hook fires.
670+ c.Check(waitForUnhealthy(scanner), Equals, true, Commentf("mode %s reported healthy", mode))
655671 time.Sleep(30 * time.Millisecond)
656672 c.Check(fetcher.calls.Load(), Equals, int64(1), Commentf("mode %s retried without backoff", mode))
657− c.Check(scanner.IsHealthy(), Equals, false, Commentf("mode %s reported healthy", mode))
658673 scanner.Stop()
659674 }
660675 }
661676
662677 type terminalRecoveryFetcher struct {
663678 calls atomic.Int64
664679 asValue bool
665680 }
666681
667682 func (f *terminalRecoveryFetcher) FetchMemPool(int64) (stypes.TxIn, error) {
waitForUnhealthy: 1 s deadline, 1 ms pollHealthy starts false but is stored true before FetchTxs, so the poll can't pass on the initial value.blockscanner_rewind_test.go:447 ↗look
438438 }
439439 time.Sleep(5 * time.Millisecond)
440440 }
441441 return false
442442 }
443443
444+// waitForUnhealthy polls because backoffAfterPersistenceFailure marks the
445+// scanner unhealthy only after the fetcher's failure hook has returned.
446+func waitForUnhealthy(scanner *BlockScanner) bool {
447+ deadline := time.Now().Add(time.Second)
448+ for time.Now().Before(deadline) {
449+ if !scanner.IsHealthy() {
450+ return true
451+ }
452+ time.Sleep(time.Millisecond)
453+ }
454+ return false
455+}
456+
444457 func (s *BlockScannerExtraSuite) TestTypedCursorRewindClearsStalePrefetch(c *C) {
445458 server, bridge := s.newRewindTestBridge(c)
446459 defer server.Close()
447460
Read ignores buffer, blocks until ResetReadStreamWithBufferWithContext resets on ctx.Done, closing the channel, so the reader goroutine never leaks.stream_helper_extra_test.go:148 ↗look
139+ *MockNetworkStream
140+ reset chan struct{}
141+ resetOnce sync.Once
142+}
143+
144+func newBlockingReadStream() *blockingReadStream {
145+ return &blockingReadStream{MockNetworkStream: NewMockNetworkStream(), reset: make(chan struct{})}
146+}
147+
148+func (m *blockingReadStream) Read([]byte) (int, error) {
149+ <-m.reset
150+ return 0, errors.New("stream reset")
151+}
152+
153+func (m *blockingReadStream) Reset() error {
154+ m.resetOnce.Do(func() { close(m.reset) })
155+ return nil
156+}
157+
136158 func TestReadStreamWithBufferWithContext(t *testing.T) {
137159 ApplyDeadline.Store(false)
138160
139161 t.Run("happy path", func(t *testing.T) {
Old header-write setup removedbinary import still used elsewhere in the file; subtest now only asserts context.Canceled.stream_helper_extra_test.go:176 ↗look
151173 t.Run("context cancelled", func(t *testing.T) {
152− // Use a stream that blocks on read (empty stream)
153− s := NewMockNetworkStream()
154− // Write header indicating we need 100 bytes but provide none
155− buf := make([]byte, LengthHeader)
156− binary.LittleEndian.PutUint32(buf, 100)
157− s.Buffer.Write(buf)
174+ // The read must block: an empty buffer returns EOF at once, and if the
175+ // reader goroutine finishes first select may pick its short-read error.
176+ s := newBlockingReadStream()
158177
159178 ctx, cancel := context.WithCancel(context.Background())
160179 cancel() // cancel immediately
161180
162181 _, err := ReadStreamWithBufferWithContext(ctx, s)
163182 assert.Error(t, err)
164183 assert.Equal(t, context.Canceled, err)
165184 })
166185
167186 t.Run("context deadline", func(t *testing.T) {
168187 ApplyDeadline.Store(true)
169188 defer ApplyDeadline.Store(false)
170189
Timeout 100 ms → 1 sAll messages pre-queued; loop exits on completion, so a pass is no slower. Failure path waits up to 1 s.frost_keygen_test.go:1499 ↗look
14911491 round2TranscriptHash := testFrostKeygenTranscriptHash(t, "msg-id", frostDKGRound2,
14921492 frostKeygenTranscriptEntry{participantIndex: localIndex, payloadHash: localRound2Hash},
14931493 frostKeygenTranscriptEntry{participantIndex: remoteIndex, payloadHash: remoteRound2Hash},
14941494 )
14951495 enqueueVerifyMessage(t, inbox, "msg-id", testPubKeys[1], frostDKGRound2, round2TranscriptHash)
14961496
1497−coord := newTestFrostKeygenCoordinator(testPubKeys[0], "msg-id", inbox, 100*time.Millisecond)
1497+// Every message is already queued, so the timeout only bounds CPU time; a
1498+// 100ms bound timed out under -race on a loaded runner.
1499+coord := newTestFrostKeygenCoordinator(testPubKeys[0], "msg-id", inbox, time.Second)
14981500 initTestFrostKeygenRoundState(t, coord, activeSet, participantIndexMap)
14991501 shares, failure := coord.exchangeRound2(activeSet, participantIndexMap, peerIDs, localRound2)
15001502 if failure != nil {
15011503 t.Fatalf("round2 failed under phase skew: %+v", failure)
15021504 }
15031505 if shares[remoteIndexKey][localIndexKey] != "remote-to-local" {
15041506 t.Fatalf("round2 lost the remote encrypted share: %+v", shares)
15051507 }
15061508 round3Verifier, err := coord.roundState.verifier(frostDKGRound3)
15071509 if err != nil {
15081510 t.Fatalf("round3 verifier: %v", err)
15091511 }
15101512 if got := len(round3Verifier.payloadHashByTarget); got != 0 {

Ask an AI about this change

Chat assistantClaude, ChatGPT, … (reads a link)

Coding agentClaude Code, Cursor, … (reads the repo)

The context file is context.md. Use Copy with context for an AI that can't open links; it pastes the whole file in.

Glossary

-race
Go's data-race detector; it slows code several times, which exposed the 100 ms timeout.
backoffAfterPersistenceFailure
Logs a persistence failure, marks the scanner unhealthy, then waits BlockHeightDiscoverBackoff before retrying. blockscanner.go ↗
Blockscanner
Bifrost's per-chain loop that fetches blocks, persists observations and reports health to the signer. blockscanner.go ↗
FROST
Threshold Schnorr signature scheme; this test drives its distributed key generation round 2 exchange.
ReadStreamWithBufferWithContext
Reads a length-prefixed libp2p message in a goroutine; returns ctx.Err and resets the stream if the context ends first. stream_helper.go ↗