#3952 TxMeta for ledger 48263265 is not reproducible (Version: v19.13.0)

[Anonymous]

Originally created by: tamirms
Originally owned by: graydon

Issue Description

I am getting inconsistent txmeta every time I run stellar-core --conf stellar-core.conf --console catchup 48263265/1 on a new db. Specifically, the txmeta for ledger 48263265 seems to be different when running the catchup command multiple times.

Steps to Reproduce

1. stellar-core --conf stellar-core.conf --console new-db
2. stellar-core --conf stellar-core.conf --console catchup 48263265/1
3. Open a sqlite shell: sqlite3 stellar.db
4. Select rows from txhistory: select txindex, txmeta from txhistory where ledgerseq = 48263265 order by txindex asc;
5. Save the output from the sql query and wipe out the sqlite db.
6. Repeat steps 1-5 and you will find the SQL output from the second attempt differs from the first attempt.

Expected Result

I expected the output from the sql query (select txindex, txmeta from txhistory where ledgerseq = 48263265 order by txindex asc) to be identical when rerunning the catchup command.

Actual Result

Every time I run the catchup command on a new db the txhistory rows for ledgerseq 48263265 appear to be different (see supporting files).

Your Environment and Setup

stellar-core Version

v19.13.0

Environment

Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:24 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6000 arm64

Supporting Files

stellar-core.conf (gist.github.com)
sql dump from first attempt
sql dump from second attempt
catchup command log output

[Anonymous]

Originally posted by: tamirms

I am able to reproduce the issue with stellar-core v20.0.0rc2 as well

[Anonymous]

Originally posted by: mbsdf

This is not something specific to ledger 48263265. Ledger 48263265 is just an example. Similarly, it is not something specific to v19.13.0 version. Stellar core v19.13.0 is just an example. It could be any ledger or it could be any previous stellar-core version.

[Anonymous]

Originally posted by: graydon

Reproduced. Though you'll notice that the differences don't show up if you do dump-xdr, just if you look at the raw bytes of the XDR (or their base64 encoding).

This .. may or may not be something we want to consider a bug, but if it's a bug it's not a serious one. The issue is that txmeta reflects the order of changes as they occur in a (randomized for DoS-resistance) hashtable, inside the ltx subsystem, and we don't happen to normalize it (i.e. "sort it into a stable order") before emitting it to the txmeta stream or database.

We do sort it in other contexts, eg. when doing testing or doing dump-xdr on the command line, and it's just a single function call to normalize it. So we could do so when writing txmeta to the stream, if that's important to consumers. It's a little more CPU time and memory to do that normalization, but it probably won't kill us to do so.

Anyway, it's not like corrupt data or a memory error or something. You're just observing nondeterminism in a place it's allowed to be (in fact intended to be) nondeterministic.

[Anonymous]

Originally posted by: tamirms

It would be more convenient for horizon if core could normalize txmeta before writing it to the stream (or at least provide a configuration option to do so). In the mean time, we will reimplement https://github.com/stellar/stellar-core/blob/master/src/util/MetaUtils.cpp in horizon.

[Anonymous]

Originally posted by: graydon

Posted a plausible fix in https://github.com/stellar/stellar-core/pull/3954

[Anonymous]

• assigned_to: graydon