* diag: thread-safety step1 - json marshal under mutex (erigontech#11134) * diag: thread-safety step2 - unlock mutex in defer (erigontech#11135) * diag: thread-safety step4 - remove dedicated shutdown listener goroutine (erigontech#11137) reason: - we already have 1 goroutine for saving data: ``` func (d *DiagnosticClient) runSaveProcess(rootCtx context.Context) { ticker := time.NewTicker(5 * time.Minute) go func() { for { select { case <-ticker.C: d.SaveData() case <-rootCtx.Done(): ticker.Stop() return } } }() } ``` no reason to save it from one more goroutine. just save it right here - in `case <-rootCtx.Done()` section. less concurrency - better. rootContext already subscribed to sigterm * diag: thread-safety step3 - `PeerStatistics.Clone()` and `PeerStats.mutex` (erigontech#11136) Co-authored-by: dvovk <vovk.dimon@gmail.com> * dl: manifest-verify green CI (erigontech#11142) - skip `erigon2-v2` buckets - until erigontech#10967 * bor: fix race in `LockedMilestoneIDs` access (erigontech#11139) for erigontech#11129 * Diagnostics: refactor bulk execution thread safety (erigontech#11143) * Call UnwindTo with tx instead of nil in sync_test.go (erigontech#11133) Co-authored-by: antonis19 <antonis19@users.noreply.github.com> * Caplin: Tweaked CGO_Flags (erigontech#11144) * diag: thread-safety step5 - race in speedtest (erigontech#11138) - attempt to upgrade speedtest - to fix race: showwin/speedtest-go#109 (comment) It didn't help. Created: showwin/speedtest-go#223 I see: ``` ================== WARNING: DATA RACE Write at 0x00c2167c2088 by goroutine 70275: github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1.1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:203 +0x84 sync.(*Once).doSlow() /usr/local/go/src/sync/once.go:74 +0xf0 sync.(*Once).Do() /usr/local/go/src/sync/once.go:65 +0x44 github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:200 +0xb1 Previous read at 0x00c2167c2088 by goroutine 69927: github.com/showwin/speedtest-go/speedtest.(*DataChunk).Read() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:452 +0x64 io.(*nopCloser).Read() <autogenerated>:1 +0x6c io.(*LimitedReader).Read() /usr/local/go/src/io/io.go:479 +0xc5 io.copyBuffer() /usr/local/go/src/io/io.go:429 +0x29a io.Copy() /usr/local/go/src/io/io.go:388 +0x6f net.genericReadFrom() /usr/local/go/src/net/net.go:689 +0x12 net.(*TCPConn).readFrom() /usr/local/go/src/net/tcpsock_posix.go:54 +0xc9 net.(*TCPConn).ReadFrom() /usr/local/go/src/net/tcpsock.go:130 +0x64 io.copyBuffer() /usr/local/go/src/io/io.go:415 +0x22e io.Copy() /usr/local/go/src/io/io.go:388 +0x95 net/http.persistConnWriter.ReadFrom() /usr/local/go/src/net/http/transport.go:1824 +0x12 bufio.(*Writer).ReadFrom() /usr/local/go/src/bufio/bufio.go:794 +0x2b0 io.copyBuffer() /usr/local/go/src/io/io.go:415 +0x22e io.CopyBuffer() /usr/local/go/src/io/io.go:402 +0x8f net/http.(*transferWriter).doBodyCopy() /usr/local/go/src/net/http/transfer.go:416 +0x144 net/http.(*transferWriter).writeBody() /usr/local/go/src/net/http/transfer.go:371 +0x75c net/http.(*Request).write() /usr/local/go/src/net/http/request.go:755 +0x1413 net/http.(*persistConn).writeLoop() /usr/local/go/src/net/http/transport.go:2447 +0x379 net/http.(*Transport).dialConn.gowrap3() /usr/local/go/src/net/http/transport.go:1800 +0x33 Goroutine 70275 (running) created at: github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:267 +0x3ef github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.gowrap1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:279 +0x41 Goroutine 69927 (running) created at: net/http.(*Transport).dialConn() /usr/local/go/src/net/http/transport.go:1800 +0x27fe net/http.(*Transport).dialConnFor() /usr/local/go/src/net/http/transport.go:1485 +0x124 net/http.(*Transport).queueForDial.gowrap1() /usr/local/go/src/net/http/transport.go:1449 +0x44 ================== ================== WARNING: DATA RACE Write at 0x00c2167c2088 by goroutine 63832: github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1.1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:203 +0x84 sync.(*Once).doSlow() /usr/local/go/src/sync/once.go:74 +0xf0 sync.(*Once).Do() /usr/local/go/src/sync/once.go:65 +0x44 github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:200 +0xb1 Previous read at 0x00c2167c2088 by goroutine 57836: github.com/showwin/speedtest-go/speedtest.(*DataChunk).DownloadHandler() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:408 +0x3ee github.com/showwin/speedtest-go/speedtest.downloadRequest() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:183 +0x624 github.com/showwin/speedtest-go/speedtest.(*Server).downloadTestContext.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:119 +0x85 github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca Goroutine 63832 (running) created at: time.goFunc() /usr/local/go/src/time/sleep.go:177 +0x44 Goroutine 57836 (running) created at: github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a github.com/showwin/speedtest-go/speedtest.(*Server).downloadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:122 +0x2e8 github.com/showwin/speedtest-go/speedtest.(*Server).DownloadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:109 +0x1ee github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:66 +0x1d2 github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2 ================== ================== WARNING: DATA RACE Write at 0x00c2167c2068 by goroutine 63836: github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247 github.com/showwin/speedtest-go/speedtest.uploadRequest() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113 github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85 github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca Previous read at 0x00c2167c2068 by goroutine 63840: github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:438 +0x179 github.com/showwin/speedtest-go/speedtest.uploadRequest() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113 github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85 github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca Goroutine 63836 (running) created at: github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8 github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240 github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2 Goroutine 63840 (running) created at: github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8 github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240 github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2 ================== ================== WARNING: DATA RACE Write at 0x00c2167c2068 by goroutine 63840: github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247 github.com/showwin/speedtest-go/speedtest.uploadRequest() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113 github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85 github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca Previous write at 0x00c2167c2068 by goroutine 63835: github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247 github.com/showwin/speedtest-go/speedtest.uploadRequest() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113 github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85 github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca Goroutine 63840 (running) created at: github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8 github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240 github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2 Goroutine 63835 (running) created at: github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8 github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240 github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2 ================== ================== WARNING: DATA RACE Read at 0x00c2167c2100 by goroutine 63833: github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.func1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:258 +0x1c6 github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.gowrap1() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:279 +0x41 Previous write at 0x00c2167c2100 by goroutine 63888: sync/atomic.AddInt64() /usr/local/go/src/runtime/race_amd64.s:289 +0xb sync/atomic.AddInt64() <autogenerated>:1 +0x15 io.(*nopCloser).Read() <autogenerated>:1 +0x6c io.(*LimitedReader).Read() /usr/local/go/src/io/io.go:479 +0xc5 io.copyBuffer() /usr/local/go/src/io/io.go:429 +0x29a io.Copy() /usr/local/go/src/io/io.go:388 +0x6f net.genericReadFrom() /usr/local/go/src/net/net.go:689 +0x12 net.(*TCPConn).readFrom() /usr/local/go/src/net/tcpsock_posix.go:54 +0xc9 net.(*TCPConn).ReadFrom() /usr/local/go/src/net/tcpsock.go:130 +0x64 io.copyBuffer() /usr/local/go/src/io/io.go:415 +0x22e io.Copy() /usr/local/go/src/io/io.go:388 +0x95 net/http.persistConnWriter.ReadFrom() /usr/local/go/src/net/http/transport.go:1824 +0x12 bufio.(*Writer).ReadFrom() /usr/local/go/src/bufio/bufio.go:794 +0x2b0 io.copyBuffer() /usr/local/go/src/io/io.go:415 +0x22e io.CopyBuffer() /usr/local/go/src/io/io.go:402 +0x8f net/http.(*transferWriter).doBodyCopy() /usr/local/go/src/net/http/transfer.go:416 +0x144 net/http.(*transferWriter).writeBody() /usr/local/go/src/net/http/transfer.go:371 +0x75c net/http.(*Request).write() /usr/local/go/src/net/http/request.go:755 +0x1413 net/http.(*persistConn).writeLoop() /usr/local/go/src/net/http/transport.go:2447 +0x379 net/http.(*Transport).dialConn.gowrap3() /usr/local/go/src/net/http/transport.go:1800 +0x33 Goroutine 63833 (running) created at: github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:253 +0x464 github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:195 +0x4a4 github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8 github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext() /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240 github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1() /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2 Goroutine 63888 (running) created at: net/http.(*Transport).dialConn() /usr/local/go/src/net/http/transport.go:1800 +0x27fe net/http.(*Transport).dialConnFor() /usr/local/go/src/net/http/transport.go:1485 +0x124 net/http.(*Transport).queueForDial.gowrap1() /usr/local/go/src/net/http/transport.go:1449 +0x44 ================== ``` - also i don't understand why do we need cached client object erigontech#10259 * rlpgen: step 1 (erigontech#11112) - no behavior changes - move encbuffer and encreader to `encbuffer.go` - copy-paste rlpgen package * bor: finality race - step 2 (erigontech#11151) ``` ================== WARNING: DATA RACE Write at 0x00c015bf2208 by goroutine 110: github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).RemoveMilestoneID() /home/ubuntu/erigon/polygon/bor/finality/whitelist/milestone.go:190 +0x164 github.com/ledgerwatch/erigon/polygon/bor/finality.handleNoAckMilestone() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:237 +0x12c github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:159 +0x5c4 github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:117 +0x57 github.com/ledgerwatch/erigon/polygon/bor/finality.startNoAckMilestoneService() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:102 +0x2a github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap3() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:65 +0x17 Previous read at 0x00c015bf2208 by goroutine 109: github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).ProcessFutureMilestone() /home/ubuntu/erigon/polygon/bor/finality/whitelist/milestone.go:277 +0x166 github.com/ledgerwatch/erigon/polygon/bor/finality.handleMilestone() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:208 +0x1e5 github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:159 +0x5c4 github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:117 +0x5c github.com/ledgerwatch/erigon/polygon/bor/finality.startMilestoneWhitelistService() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:93 +0x2a github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap2() /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:64 +0x17 ``` * integration: wait for caplin snap open finish (erigontech#11124) * rpc bottleneck: block files mutex (e3) (erigontech#11156) for: erigontech#11090 thank you [tholcman](https://github.com/tholcman) for finding * dl: use native `filepath.IsLocal` and `filepath.Clean` funcs (erigontech#11141) they was vendored because of go1.19 compatibility which we dropped * chore: confusing log (erigontech#11043) * execution spec tests update for devnet-1 (erigontech#11127) updating to version [devnet-1@v1.3.0](https://github.com/ethereum/execution-spec-tests/releases/tag/devnet-1%40v1.3.0) - update to 7702: some extra json fields displayed in fixtures (rather than just rlps) - some refactoring to fixture json ordering means that a huge number of test files were changed - [refactor requested](erigontech#10812 (comment)) by Alex done in this pr [here](https://github.com/ledgerwatch/erigon/pull/11127/files#diff-8c398ef0a79f97ba6d497a99247a815b3a51918de9e16954215b6073f907c92e). * Delete as not needed cmd/release/. (erigontech#11161) Discussed with @AskAlexSharov * Fix downloader completion set and races (erigontech#11182) fixes: erigontech#11060 Also fixes several races, which should include: erigontech#11123 erigontech#11102 * chore: fix some comments (erigontech#11170) Signed-off-by: stellrust <gohunter@foxmail.com> * tests: auto-close temporal db (erigontech#11160) * discovery table: revalidate race fix (erigontech#11159) for erigontech#11158 * Caplin: Added POST `Validators` (erigontech#11152) close erigontech#11150 * diagnostics: refactor network peers mutex (erigontech#11178) Refactor to pattern which was suggested by @AskAlexSharov - move business-logic inside private methods - move mutex locking inside public methods - call private methods from public. don't call public methods from private. * move interfaces from ledgerwatch to erigontech (erigontech#11194) * increase snapshot semaphore default limit (erigontech#11193) * p2p receipts (erigontech#11010) closes erigontech#10320 and closes erigontech#11014 --------- Co-authored-by: JkLondon <ilya@mikheev.fun> Co-authored-by: alex.sharov <AskAlexSharov@gmail.com> * Caplin: Optimization and Parallelization of processes and reduction of Goroutines (erigontech#11058) Optimizations: 1) Single goroutine for tracking expiry of gossip subscriptions 2) tweaking of parameters on operations retention in cache 3) moving the dumping of BeaconState after forkchoice, so that we do not mess up block times with I/O * Caplin: Look for peers if not avaiable within subnet (erigontech#11057) Actually look for peers of needed subnet for better performance on holesky and mainnet --------- Co-authored-by: Kewei <kewei.train@gmail.com> * Downloader: Bump up defaults (erigontech#11197) * `erigon_getLatestLogs` add limit parameter (erigontech#11095) If user provides logCount param the API should returns logCount records, even if the block contains more logs that satisfy the filter. The logCount param is already supported, this PR avoids to return the log records in excess. Moreover I have created 14 integration tests for this API (run with success on erigon2). * refine ProcessBlock and ProcessBlindedBlock (erigontech#10923) - Apply generic interface `GenericBlock` for blinded block and normal block. - Refine function `ProcessBlock()` so as to remove duplicated code. * less logs on CI (erigontech#11122) * revert logging of binary data (erigontech#11199) * pool: do fsync by non-empty update (e3) (erigontech#11198) for erigontech#11163 * engineapi: Fix req list check (erigontech#11191) * make `latest seen blocks` visible earlier (erigontech#11204) * move secp256k1 to erigontech (erigontech#11209) * move erigon-snapshot to erigontech (erigontech#11213) * replaced speedtest lib with our fork (erigontech#11217) Replaced speed test lib with our for which uses erigon cloudflare webbed to test download speed. I decided to change speedtest servers to erigon servers as users can complain why there is traffic servers which is not related to erigon infrastructure. Our for of speedtest https://github.com/erigontech/speedtest * Caplin: Better old-state pruning (erigontech#11219) * Caplin: Remove TmpDB from `BlockCollector` (erigontech#11215) Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com> * upstrem * Attempt to use go21 (erigontech#11207) * post validator balances (erigontech#11218) impl: erigontech#11055 * eth/stagedsync: polygon sync stage to use new heimdall service (erigontech#11196) part 1 of erigontech#11186 * polygon/heimdall: remove old duplicate heimdall component (erigontech#11214) part 2 of erigontech#11186 * enable `madv_normal` for .kv files of > 0 lvl (erigontech#11223) enable `KV_MADV_NORMAL_NO_LAST_LVL=accounts,storage,code,commitment` by default * Move from ledgerwatch to erigontech (erigontech#11224) * add log * add log * erigon-lib: tidy up slices,generics,cmp commons (erigontech#11216) * Update module path of erigonwatch to erigontech (erigontech#11226) * Fix previous download completion processing (erigontech#11227) This fixes downloads stalling if a locally produced file appears not downloaded It also will re-complete hashes without re-downloading if the download db is removed It should also complete locally produced files when they are verified, if not it will self mend on restart --------- Co-authored-by: Giulio <giulio.rebuffo@gmail.com> Co-authored-by: alex.sharov <AskAlexSharov@gmail.com> * Bring back dirs to BaseAPI (erigontech#11228) This PR brings back `datadir.Dirs` to `BaseAPI`, and removes a redundant `Aggregator` argument to `EngineServer.Start` . --------- Co-authored-by: antonis19 <antonis19@users.noreply.github.com> * Logs checking (erigontech#11229) changed erigon api logs to e3 closes erigontech#11117 --------- Co-authored-by: JkLondon <ilya@mikheev.fun> Co-authored-by: alex.sharov <AskAlexSharov@gmail.com> * ots: nil-ptr in rpc (erigontech#11232) * txpool changes for set_code_tx support (erigontech#11235) * Erigon 3.0: Smarter full and minimal nodes (erigontech#11220) * add log * add log * added var inside WriteMap for chain-like interface (erigontech#11241) closes erigontech#11202 --------- Co-authored-by: JkLondon <ilya@mikheev.fun> * tracer: add support bailout on evm.create() (erigontech#11237) Add bailout mgt on evm.create() method (it is already mgt on evm.cal()). When approved I will create PR on rel 2.60.x * added rpc info feature (erigontech#11242) closes erigontech#11157 but a bit of scared that we probably don't have some MetadataFromContext method like in ethereum/go-ethereum#24255 Co-authored-by: JkLondon <ilya@mikheev.fun> * [test] remove unused variables (erigontech#10938) remove unused variables * force fsync after notifications sent (erigontech#11244) * Fixed nil pointer exception (erigontech#11253) * fix panic due to nil validator set (erigontech#11260) fix erigontech#11027 root cause: specific slot state data not found leads to empty validator set * qa-tests: increase test time of sync-from-scratch for minimal node (erigontech#11256) the last few runs of the test did not complete on time * qa-tests: fix sync-from-scratch test result uploading (erigontech#11245) The test results are uploaded to the github actions test run page at the end of the test. As this test has 2 jobs, we need to give the uploaded test results different names to avoid clashes. * Caplin: Add support for beacon snapshots (also stops relying on Engine API) (erigontech#11250) I had to: * Add Caplin snapshot download * Fix Snapshot Downloader on Holesky * Fixed Holesky's chainspec * add log * fix chapel bt hash * rm prints (erigontech#11261) * fix chapel bt hash * Enable `madv_normal` for level0 .kv (erigontech#11265) * HexOrDecimal - to accept unquoted numbers - in json (e3) (erigontech#11262) accept in `genesis.json`:` "nonce": 0,` now see: ``` Fatal: invalid genesis file: json: cannot unmarshal number into Go struct field Genesis.alloc of type *math.HexOrDecimal64 ``` See also `https://github.com/ethereum/go-ethereum/pull/26758` * e3: make getLatest layers visible in pprof (erigontech#11266) * hack to prevent early download finish (erigontech#11267) * add log * add log * add log * add log * add log * add log * add log * add log * add log * Wire OverridePragueTime into txpool (erigontech#11234) also extract common logic from `(p *TxPool) isShanghai()`, `isCancun()`, `isPrague()`. * readme update (erigontech#11275) * diag: race in updateIndexingStatus (erigontech#11274) for erigontech#11268 * Bump version to 3.0.0-alpha1 (erigontech#11276) Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com> * fix body data * fix goreleaser after update (erigontech#11281) Fixes https://github.com/erigontech/erigon/actions/runs/10041015251/job/27748204997#step:9:72 (caused by PR erigontech#10726 apparently). See also https://github.com/goreleaser/goreleaser-cross-example/blob/master/Makefile * fix body data * fix body data * bor: loopbreaker - to stop check `LoopBlockLimit` and rely on stage_headers progress (erigontech#11286) * Special logs for near-chain-tip execution (erigontech#11288) * fix upstream * e3 use same goreleaser-cross version as in e2 (erigontech#11285) - switched to `v1.21.5` - added to UI checkbox "Publish Artifacts" - which is disabled by default. if not set: `make release-dry-run` * add bool variable to auto-release (erigontech#11290) * on chain-tip: if batch is full - stop execution stage - to allow commit and reduce db size (erigontech#11287) * add log * PIP-35: enforce 25gwei gas config for all polygon chains (erigontech#11294) Remove the checks for amoy added previously to prepare for mainnet release. Sets `txpool.pricelimit`, `miner.gasprice` and `gpo.ignoreprice` to 25gwei for all polygon based networks. * refactor: rename fields in hex patricia trie (erigontech#11296) Small refactor to improve the readability of some fields in the `hex_patricia_hashed.go` . Co-authored-by: antonis19 <antonis19@users.noreply.github.com> * add log * add log * add log * new dirWalk for test purposes (erigontech#11277) closes erigontech#10086 but it uses old lib (https://github.com/karrick/godirwalk). This branch could be used for tests for someone who experiences troubles with RAM with RemoveContents func. (For example for this guy https://discord.com/channels/687972960811745322/1233600171821240380) Maybe we should fork this lib :) thing for future milestone --------- Co-authored-by: JkLondon <ilya@mikheev.fun> Co-authored-by: alex.sharov <AskAlexSharov@gmail.com> * chore: fix some comments (erigontech#11273) Signed-off-by: yingshanghuangqiao <yingshanghuangqiao@foxmail.com> * remove useless logs --------- Signed-off-by: stellrust <gohunter@foxmail.com> Signed-off-by: yingshanghuangqiao <yingshanghuangqiao@foxmail.com> Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com> Co-authored-by: dvovk <vovk.dimon@gmail.com> Co-authored-by: antonis19 <antonis19@mail.com> Co-authored-by: antonis19 <antonis19@users.noreply.github.com> Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com> Co-authored-by: chuwt <weitaochu@gmail.com> Co-authored-by: sudeep <sudeepdino008@gmail.com> Co-authored-by: lystopad <oleksandr.lystopad@erigon.tech> Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com> Co-authored-by: stellrust <gohunter@foxmail.com> Co-authored-by: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com> Co-authored-by: awskii <awskii@users.noreply.github.com> Co-authored-by: Ilya Mikheev <54912776+JkLondon@users.noreply.github.com> Co-authored-by: JkLondon <ilya@mikheev.fun> Co-authored-by: Kewei <kewei.train@gmail.com> Co-authored-by: lupin012 <58134934+lupin012@users.noreply.github.com> Co-authored-by: Somnath <snb895@outlook.com> Co-authored-by: milen <94537774+taratorio@users.noreply.github.com> Co-authored-by: LEE <Leeyqiongyuan@gmail.com> Co-authored-by: Michelangelo Riccobene <michelangelo.riccobene@gmail.com> Co-authored-by: VBulikov <vlad.bulikov@gmail.com> Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com> Co-authored-by: Manav Darji <manavdarji.india@gmail.com> Co-authored-by: yingshanghuangqiao <yingshanghuangqiao@foxmail.com>

dl: mdbxPieceCompletion.Set(false) must unset bit in completion bitmap #11060

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

dl: `mdbxPieceCompletion.Set(false)` must unset bit in completion bitmap #11060