Originally created by: leighmcculloch
Originally owned by: dmkozh
Running stellar-core normally connected to testnet results in periodic "Catchup material failed verification - hash mismatch" errors.
Unknown at this time. This has occurred three times when running stellar-core in the quickstart image connected to the testnet network as a watcher node.
docker run --name stellar -p 8000:8000 stellar/quickstart:testing --testnet --enable-soroban-rpc
No other action has been required to my knowledge.
$ stellar-core version
Warning: soroban-env-host-curr is running a pre-release version 20.0.0-rc1
v20.0.0rc2
rust version: rustc 1.72.1 (d5c2e9c34 2023-09-13)
soroban-env-host:
curr:
package version: 20.0.0-rc1
git version: f19ef13363a1e0cbff7b100c0599a1d63dea88a6
ledger protocol version: 20
pre-release version: 57
rs-stellar-xdr:
package version: 20.0.0-rc1
git version: d5ce0c9e7aa83461773a6e81662067f35d39e4c1
base XDR git version: 9ac02641139e6717924fdad716f6e958d0168491
uname -a)console
$ uname -a
Linux 781545c4c81d 6.3.13-linuxkit [#1] SMP PREEMPT Thu Sep 7 07:48:47 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
Both. Command line inside a container running inside a VM managed by Docker for Mac on an Apple Silicon M2 Mac.
The stellar/quickstart:testing image has stellar-core compiled with the --disable-tests option.
The stellar/quickstart:testing image is a multi-platform image and when this issue occurred I was running it using the linux/arm64 platform image. To run specifically that image you can pass to docker --platform linux/arm64 or if using a container runtime that doesn't support specifying the platform the same image is available as a single-platform image under the name stellar/quickstart:testing-arm64.
The configuration file template used for testnet in quickstart is here:
https://github.com/stellar/quickstart/blob/140ee88feb0d265120255827f0e2e1dae04d6da5/testnet/core/etc/stellar-core.cfg#L3
Once the template is fully rendered it is:
```toml
HTTP_PORT=11626
PUBLIC_HTTP_PORT=true
LOG_FILE_PATH=""
MANUAL_CLOSE=false
NETWORK_PASSPHRASE="Test SDF Network ; September 2015"
KNOWN_CURSORS=["HORIZON"]
DATABASE="postgresql://dbname=core host=localhost user=stellar password=..."
CATCHUP_RECENT=100
UNSAFE_QUORUM=true
FAILURE_SAFETY=1
[[HOME_DOMAINS]]
HOME_DOMAIN="testnet.stellar.org"
QUALITY="HIGH"
[[VALIDATORS]]
NAME="sdf_testnet_1"
HOME_DOMAIN="testnet.stellar.org"
PUBLIC_KEY="GDKXE2OZMJIPOSLNA6N6F2BVCI3O777I2OOC4BV7VOYUEHYX7RTRYA7Y"
ADDRESS="core-testnet1.stellar.org"
HISTORY="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_001/{0} -o {1}"
[[VALIDATORS]]
NAME="sdf_testnet_2"
HOME_DOMAIN="testnet.stellar.org"
PUBLIC_KEY="GCUCJTIYXSOXKBSNFGNFWW5MUQ54HKRPGJUTQFJ5RQXZXNOLNXYDHRAP"
ADDRESS="core-testnet2.stellar.org"
HISTORY="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_002/{0} -o {1}"
[[VALIDATORS]]
NAME="sdf_testnet_3"
HOME_DOMAIN="testnet.stellar.org"
PUBLIC_KEY="GC2V2EFSXN6SQTWVYA5EPJPBWWIMSD2XQNKUOHGEKB535AQE2I6IXV2Z"
ADDRESS="core-testnet3.stellar.org"
HISTORY="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_003/{0} -o {1}"
```
2023-09-20T21:36:16.487 GDHDX [History ERROR] Catchup material failed verification - hash mismatch, propagating failure
2023-09-20T21:36:16.487 GDHDX [History ERROR] One or more of history archives may be corrupted. Update HISTORY configuration entry to only contain valid ones
Full log of stellar-core stderr output is attached: full-log-2023-09-20.txt (github.com).
Catching up; Catching up to ledger 1619071: downloading ledger files 1/1 (100%)
Catching up; Catching up to ledger 1619071: Failed: download-verify-ledgers-seq
Catching up; Catching up to ledger 1619071: Succeeded: batch-download-ledger-0018b47f-0018b4bf : 1/1 children completed
Catching up; Catching up to ledger 1619071: Succeeded: batch-download-ledger-0018b47f-0018b4bf : 2/2 children completed
Catching up; Catching up to ledger 1619071: Succeeded: batch-download-ledger-0018b47f-0018b4bf : 3/3 children completed
Catching up; Catching up to ledger 1619071: Succeeded: batch-download-ledger-0018b47f-0018b4bf : 4/4 children completed
Catching up; Catching up to ledger 1619071: Succeeded: batch-download-ledger-0018b47f-0018b4bf : 5/5 children completed
There has been some offline discussion on Slack about this issue:
Originally posted by: leighmcculloch
The same issue occurred again for me on October 2nd, although I failed to capture which ledger.
Originally posted by: leighmcculloch
The same issue occurred again for me on October 3rd, when attempting to catchup ledger 1834623. I saw the following status on the info endpoint:
Originally posted by: MonsieurNicolas
I ran into this as well. Similar config just with bucketsdb enabled (so faster to repro), and on Windows.
Similar situation, where the node replayed a few ledgers from archive, then crashed replaying buffered ledgers:
stellar-testnet-bdb - corrupt.log (github.com)
All I did was run
new-dbfollowed byrun.This does not seem to happen all the time, and does not happen when replaying the exact same range using offline catchup. This may indicate that somehow the txset gets borked by catchup?
Originally posted by: MonsieurNicolas
OK I could repro it some other time, and could capture data on both a "good" (replaying from history) and a "bad" (from consensus).
I got a divergence on ledger 1850632 that I could narrow down to the last transaction applied, in the way fees got applied before even applying the transaction (9461a5cd2a008d5d82d686c6df65658e4c070f366b96431ec205783670217a89).
Transaction is
good:
bad:
only difference is that the account went from 477031939487 :
This probably points to an issue with the way the effective fee is calculated for Soroban transactions.
Here is the full meta of that ledger (which includes the full txset) from the "good" replay:
good-meta.txt (github.com)
Originally posted by: dmkozh
That's really concerning, especially given that the difference is pretty high. Unless there is some weird non-determinism in how we pass the tx resources to fee computations, it would point at some config-induced non-determinism, e.g. due to using the stale cached value after the config upgrade or bucket list size change.
Originally posted by: MonsieurNicolas
configs should not have any impact on the fee charged before applying the transaction, right (only on validity/refunds)?
Originally posted by: MonsieurNicolas
oh I see it's possible that if we have the wrong settings (like if they're very low), we end up overestimating the inclusion fee, that ends up being 0 (as there is no contention)
Originally posted by: dmkozh
No, we charge the
resource_fee + discounted_inclusion_feepre-apply.resource_feeis a function of configuration and BL size.Originally posted by: msteiner96
Had the same issues today
Originally posted by: MonsieurNicolas
So would the bug be an issue with
mSorobanResourceFeeor with the base fee cached improperly inTxSetFrame?Originally posted by: dmkozh
Could be either - hard to tell without knowing what's the
TxSetFramewas. I'm more suspicious of config (because it's new/used in a lot of places), but who knows.Originally posted by: dmkozh
With some help from @marta-lokhova we've identified at least one issue that could cause the catchup issues: during catchup we build the transaction set objects (which includes fee computation) for the incoming tx sets using the current ledger state (which can be arbitrarily old), which might result in using a stale config. I'm not 100% sure that's the cause for the initial issue (ledger 1619071 seems a bit too far back?), but it's definitely an issue.
Originally posted by: MonsieurNicolas
as we're going to have rc1/rc2 running on testnet for still some time.
The workaround here is:
new-dbyour instance (start from fresh)