Calls to debug_executionWitness cause future block execution to fail (BAD BLOCK)

[max-sanchez]

System information

Geth version: 1.16.7-stable-b9f3a3d9
CL client & version: Prysm/v7.1.2/7950a249266a692551e5a910adb9a82a02c92040
OS & Version: Linux
Commit hash : b9f3a3d

Expected behaviour

An RPC call to debug_executionWitness should succeed and geth should continue processing new blocks.

Actual behaviour

RPC call to debug_executionWitness succeeds, but geth fails to process future blocks afterwards with a BAD BLOCK log message indicating one of the transactions in the block had a nonce too high.

Note that debug_executionWitness calls for the current tip block do not seem to ever cause a failure, indicating that the query for historical state may be rolling back existing state on which future block execution is run.

Steps to reproduce the behaviour

Perform a debug_executionWitness RPC query for a block that is behind geth's current tip.

For the below logs, my geth node was at block 24199456 when I ran a debug_executionWitness call for the previous block 24199455:

curl <RPC>   -X POST   -H "Content-Type: application/json"   --data '{"method":"debug_executionWitness","params":["0x171411F"],"id":1,"jsonrpc":"2.0"}'

Backtrace / Logs

Jan 09 20:16:13 geth-mainnet geth[21361]: INFO [01-09|20:16:13.955] Imported new potential chain segment     number=24,199,455 hash=e08927..33c1b1 blocks=1 txs=651 mgas=37.907 elapsed=111.755ms   mgasps=339.200 triediffs=345.89MiB triedirty=165.16MiB
Jan 09 20:16:14 geth-mainnet geth[21361]: INFO [01-09|20:16:14.014] Chain head was updated                   number=24,199,455 hash=e08927..33c1b1 root=19ae03..b292d1 elapsed=5.903636ms
Jan 09 20:16:24 geth-mainnet geth[21361]: INFO [01-09|20:16:24.501] Imported new potential chain segment     number=24,199,456 hash=1699a8..c4fd66 blocks=1 txs=258 mgas=18.909 elapsed=56.509ms    mgasps=334.615 triediffs=344.90MiB triedirty=165.80MiB
Jan 09 20:16:24 geth-mainnet geth[21361]: INFO [01-09|20:16:24.573] Chain head was updated                   number=24,199,456 hash=1699a8..c4fd66 root=1630ab..4c9ec8 elapsed=2.404465ms
Jan 09 20:16:36 geth-mainnet geth[21361]: ERROR[01-09|20:16:36.449] Unexpected trie node                     location=diff owner=0x37d65eaa92c6bc4c13a5ec45527f0c18ea8932588728769ec7aecfe6d9f32e42 path=[]                                       expect=0xae9a1d41989e2d91bcc217a55456d1065888aa93c3516b81931ae14f79bcc44a got=0x49dbdd991d2c6fd5ed870f8f22912e0df2a12f1926056d7775f7ebe3da1c031a blob=0xf90211a04df648158b4235c122119c7a4d556461c4ddac4f215cd17d5dd6836c5ff93c61a0eced7440c908be5382c6f2c757fe1b81bf10b37119e650e4666b9f6a4d38bbd2a0643dc349b4c2931ace00920cd1f3a4b89b97c5835089df3438c9db237d173ac4a0188749f3cac295e9154bdc3f3fa1f0f06ad92e75598d8f8fa0a4edc859541d28a0b62d4f7d2d6c1bce5d52827c68057751c1278860b3b97c73f6fcfc81c02e235fa0cb08b48c9c219e8d2815a756bbba263f471416236d94ed81a6225f7e4153baaca0ffb582a605f0d7304258b5d9fbdf643a91f029f5da43b4413f7189dfeba13578a063bee28c4940686301664c9eb8fd1698fd011b447b2d7b9e002c1454c40463c8a0f15c18ff23b203ba20d10b2a1b7416996afa97a3570221d3cdd6e9e3c5d65314a0a7d747cb298464b8dc22a42c590bc0070df553f0b6afb55e10e5e9cac6c4e937a0b27ac3a008ce8586e4d3f44fb3c317458fce3134239d9c8d8e78c850000f5dc2a06718d7fb95d0cc6bae296b90f9ab03d70d80cfe5ffe31fb5c29e01c6008f2aefa0f780606f941b766beea975f47712c698846e6bcdde4915afb811a5de862eefa0a00f511bf987a776db55055a5f4d9b469139b3c528be1fea108be88da1afb85f9aa01c1413e9089b6949f5ea3a9f392d5e0626123b28257daa185ac3f81ff688185ba0ca9a5693c761349877d88b6a38f62eb415b002180816199bc1444009e4a63ba280
Jan 09 20:16:36 geth-mainnet geth[21361]: WARN [01-09|20:16:36.449] Trie prefetcher failed opening storage trie root=ae9a1d..bcc44a err="missing trie node ae9a1d41989e2d91bcc217a55456d1065888aa93c3516b81931ae14f79bcc44a (owner 37d65eaa92c6bc4c13a5ec45527f0c18ea8932588728769ec7aecfe6d9f32e42) (path ) unexpected node: (37d65eaa92c6bc4c13a5ec45527f0c18ea8932588728769ec7aecfe6d9f32e42 []), ae9a1d41989e2d91bcc217a55456d1065888aa93c3516b81931ae14f79bcc44a!=49dbdd991d2c6fd5ed870f8f22912e0df2a12f1926056d7775f7ebe3da1c031a, loc: diff, depth: 0, blob: 0xf90211a04df648158b4235c122119c7a4d556461c4ddac4f215cd17d5dd6836c5ff93c61a0eced7440c908be5382c6f2c757fe1b81bf10b37119e650e4666b9f6a4d38bbd2a0643dc349b4c2931ace00920cd1f3a4b89b97c5835089df3438c9db237d173ac4a0188749f3cac295e9154bdc3f3fa1f0f06ad92e75598d8f8fa0a4edc859541d28a0b62d4f7d2d6c1bce5d52827c68057751c1278860b3b97c73f6fcfc81c02e235fa0cb08b48c9c219e8d2815a756bbba263f471416236d94ed81a6225f7e4153baaca0ffb582a605f0d7304258b5d9fbdf643a91f029f5da43b4413f7189dfeba13578a063bee28c4940686301664c9eb8fd1698fd011b447b2d7b9e002c1454c40463c8a0f15c18ff23b203ba20d10b2a1b7416996afa97a3570221d3cdd6e9e3c5d65314a0a7d747cb298464b8dc22a42c590bc0070df553f0b6afb55e10e5e9cac6c4e937a0b27ac3a008ce8586e4d3f44fb3c317458fce3134239d9c8d8e78c850000f5dc2a06718d7fb95d0cc6bae296b90f9ab03d70d80cfe5ffe31fb5c29e01c6008f2aefa0f780606f941b766beea975f47712c698846e6bcdde4915afb811a5de862eefa0a00f511bf987a776db55055a5f4d9b469139b3c528be1fea108be88da1afb85f9aa01c1413e9089b6949f5ea3a9f392d5e0626123b28257daa185ac3f81ff688185ba0ca9a5693c761349877d88b6a38f62eb415b002180816199bc1444009e4a63ba280"
Jan 09 20:16:36 geth-mainnet geth[21361]: ERROR[01-09|20:16:36.451] Unexpected trie node                     location=diff owner=0x6c9d57be05dd69371c4dd2e871bce6e9f4124236825bb612ee18a45e5675be51 path=[]                                       expect=0x630cf2851109d254adf5845a852ab3318105740c99c8ccd90e2a0a9a4478a817 got=0x773621231e37d2ea570b1f551bef9859e2751b51e07da398a459ad68c10152e3 blob=0xf90211a0fcf4ec8653511b2b3083fcd5c739f7fb62a733edb5d70f9687b7c8f213983fb9a0ac2f9ea5f3491727c2668cc39f33bada1ce3aedba7e94f061475d9ad4438b693a07a9e575990c1d9a682326fdbd8b2f341227db06aab31d830b7644f96fb57b6aca0899f6fdb3ebabc8dd22a057f0168085cea3255ac6f979ad2502993e554391faaa0dfbeb0ac81a8b8ff35f34692de16873aa6d68d3315216b4623391d7f02a76e0aa049ccbf10330ced1b83cb6879d08dfaac3b14c2b173d8ffdf04528e72dae824bca08eac7ce48beed052f2b2615ee4c0778c2b731728b295142ac9569c5b4be3ad93a08bccacaefb1138f0c4a81c658f9136dbfd6e3bd7451aadd48695b5aa43dcdbaba03bdae5c4904f4de30eb1976e5a02235a25e11de9a7aeeddf24270aee3b92c60fa01b8793442b5f6ae82c1c99b14dc19d1128f03414d65d4708c6fcd0806f3f5159a02c296e31b5a3ea87da446ac206037ada9e863b5a751c4fd6a086590eb4d03716a0d20a96e39f3cef96218652aeff388d27d3d0824ba14ec715c5eb27d7d101b4b0a0cee20deec40e8e8afe065d2a9826c71bb4b2ca45c57d946aa35ea96cbf4e0e8ea07a4f58f11556e9354d152e747ae67a75ffbc4ba30149b5cffc452102592499b3a06d0197fd0a55d8fea6100fde0aeaab9adad224738f7b4d88e231a2ad0fe6663ca047535f63ef1ad602945cd8a57ad787a7a70470b40ddb4ffcafcca8509d06782480
Jan 09 20:16:36 geth-mainnet geth[21361]: WARN [01-09|20:16:36.451] Trie prefetcher failed opening storage trie root=630cf2..78a817 err="missing trie node 630cf2851109d254adf5845a852ab3318105740c99c8ccd90e2a0a9a4478a817 (owner 6c9d57be05dd69371c4dd2e871bce6e9f4124236825bb612ee18a45e5675be51) (path ) unexpected node: (6c9d57be05dd69371c4dd2e871bce6e9f4124236825bb612ee18a45e5675be51 []), 630cf2851109d254adf5845a852ab3318105740c99c8ccd90e2a0a9a4478a817!=773621231e37d2ea570b1f551bef9859e2751b51e07da398a459ad68c10152e3, loc: diff, depth: 0, blob: 0xf90211a0fcf4ec8653511b2b3083fcd5c739f7fb62a733edb5d70f9687b7c8f213983fb9a0ac2f9ea5f3491727c2668cc39f33bada1ce3aedba7e94f061475d9ad4438b693a07a9e575990c1d9a682326fdbd8b2f341227db06aab31d830b7644f96fb57b6aca0899f6fdb3ebabc8dd22a057f0168085cea3255ac6f979ad2502993e554391faaa0dfbeb0ac81a8b8ff35f34692de16873aa6d68d3315216b4623391d7f02a76e0aa049ccbf10330ced1b83cb6879d08dfaac3b14c2b173d8ffdf04528e72dae824bca08eac7ce48beed052f2b2615ee4c0778c2b731728b295142ac9569c5b4be3ad93a08bccacaefb1138f0c4a81c658f9136dbfd6e3bd7451aadd48695b5aa43dcdbaba03bdae5c4904f4de30eb1976e5a02235a25e11de9a7aeeddf24270aee3b92c60fa01b8793442b5f6ae82c1c99b14dc19d1128f03414d65d4708c6fcd0806f3f5159a02c296e31b5a3ea87da446ac206037ada9e863b5a751c4fd6a086590eb4d03716a0d20a96e39f3cef96218652aeff388d27d3d0824ba14ec715c5eb27d7d101b4b0a0cee20deec40e8e8afe065d2a9826c71bb4b2ca45c57d946aa35ea96cbf4e0e8ea07a4f58f11556e9354d152e747ae67a75ffbc4ba30149b5cffc452102592499b3a06d0197fd0a55d8fea6100fde0aeaab9adad224738f7b4d88e231a2ad0fe6663ca047535f63ef1ad602945cd8a57ad787a7a70470b40ddb4ffcafcca8509d06782480"
<snip>
Jan 09 20:16:36 geth-mainnet geth[21361]: ERROR[01-09|20:16:36.470] Unexpected trie node                     location=diff owner=0x54af0c0062d4664f72bbdd2b504bcd8d9582ce6a4ca9986666a3ee4d0241a7a7 path=[]                                       expect=0x4c199e4e77596e0352fe0b798bd94623c80e56f595d1a2e21a32d82dad3c2b94 got=0xe414482c7d25b71ba2b149cae8c82641cebdf85694cd72e894100f92d0f51939 blob=0xf90211a0d6aa40a8eadb7093db50c1ea54bff988a4b98b3b54c5880d0ca76058ba01275da090218d33b85c04a37ef8848493d94791e050d94dfb81dfcc3bb1378a4d051ff9a0e981db2c4fcad5706d53ce1610c42cbdbe09787a9526ac6b33dd6ad01691c46aa045a5fa1ae30823bfec48319442071128e9a5e78e9b31b2b3a6448bda459bf2c5a0b4598550e212fb48d11a7d602508f95ec0ca1eaadfdcb97e82e414f91cf7be72a04b5047e3aab399857877f584430900bb3096a9615cfbcc3af272852d668a6583a021a75467730adafffe4416d74c555da996bf58019663a5f2b7be2afb726b7e37a01b4fa8bd02e5eebb98fcc629e4e73befa4dc9fe2e6b94bcb1fcf1c47e7ddeb6da069fcae3b8590de2a676ce79dc6b14cff549c2dfd05f7f8f77dd9d53105929113a001522efea5e3e0fc89f5853fdca5920c158955ee8d93a48ddbd30627d36c7e13a06b9923c1bde8330a1da8a5f8325254f04b8758d8126f0522c7ad57da0e77c59da053c3772f0c7feb167011c4862bd22a4e01adc6f0b3e32465b23b8ed287d62f28a05e4a250f7d601affd6300a3bcf854d5e7a2dbe7514c1498cb6611f4a9dccc1aaa0c4b97215fbc26552778b51045f36f3607dce60ab3fed1189cc3c442c5b853c26a04f7ede64116af16c811f75da34a82065021816dc9a0fdda412de5f24978a2398a0e10c652a4b29a5b9cdf3121564f0833f309c9cae9b8e482f1227d1966f4cda0980
Jan 09 20:16:36 geth-mainnet geth[21361]: WARN [01-09|20:16:36.470] Trie prefetcher failed opening storage trie root=4c199e..3c2b94 err="missing trie node 4c199e4e77596e0352fe0b798bd94623c80e56f595d1a2e21a32d82dad3c2b94 (owner 54af0c0062d4664f72bbdd2b504bcd8d9582ce6a4ca9986666a3ee4d0241a7a7) (path ) unexpected node: (54af0c0062d4664f72bbdd2b504bcd8d9582ce6a4ca9986666a3ee4d0241a7a7 []), 4c199e4e77596e0352fe0b798bd94623c80e56f595d1a2e21a32d82dad3c2b94!=e414482c7d25b71ba2b149cae8c82641cebdf85694cd72e894100f92d0f51939, loc: diff, depth: 0, blob: 0xf90211a0d6aa40a8eadb7093db50c1ea54bff988a4b98b3b54c5880d0ca76058ba01275da090218d33b85c04a37ef8848493d94791e050d94dfb81dfcc3bb1378a4d051ff9a0e981db2c4fcad5706d53ce1610c42cbdbe09787a9526ac6b33dd6ad01691c46aa045a5fa1ae30823bfec48319442071128e9a5e78e9b31b2b3a6448bda459bf2c5a0b4598550e212fb48d11a7d602508f95ec0ca1eaadfdcb97e82e414f91cf7be72a04b5047e3aab399857877f584430900bb3096a9615cfbcc3af272852d668a6583a021a75467730adafffe4416d74c555da996bf58019663a5f2b7be2afb726b7e37a01b4fa8bd02e5eebb98fcc629e4e73befa4dc9fe2e6b94bcb1fcf1c47e7ddeb6da069fcae3b8590de2a676ce79dc6b14cff549c2dfd05f7f8f77dd9d53105929113a001522efea5e3e0fc89f5853fdca5920c158955ee8d93a48ddbd30627d36c7e13a06b9923c1bde8330a1da8a5f8325254f04b8758d8126f0522c7ad57da0e77c59da053c3772f0c7feb167011c4862bd22a4e01adc6f0b3e32465b23b8ed287d62f28a05e4a250f7d601affd6300a3bcf854d5e7a2dbe7514c1498cb6611f4a9dccc1aaa0c4b97215fbc26552778b51045f36f3607dce60ab3fed1189cc3c442c5b853c26a04f7ede64116af16c811f75da34a82065021816dc9a0fdda412de5f24978a2398a0e10c652a4b29a5b9cdf3121564f0833f309c9cae9b8e482f1227d1966f4cda0980"
Jan 09 20:16:36 geth-mainnet geth[21361]: ERROR[01-09|20:16:36.470] Failed to prefetch slots                 addr=0xdAC17F958D2ee523a2206206994597C13D831ec7 slots=2 err="fetcher is already terminated"
<snip>
Jan 09 20:16:36 geth-mainnet geth[21361]: ERROR[01-09|20:16:36.474] Failed to prefetch slots                 addr=0xdAC17F958D2ee523a2206206994597C13D831ec7 slots=2 err="fetcher is already terminated"
Jan 09 20:16:36 geth-mainnet geth[21361]: ERROR[01-09|20:16:36.483]
Jan 09 20:16:36 geth-mainnet geth[21361]: ########## BAD BLOCK #########
Jan 09 20:16:36 geth-mainnet geth[21361]: Block: 24199457 (0x17745c71ffda1963b9812b0bae00f3f6184a06b741dc543b03a9d4239cc16979)
Jan 09 20:16:36 geth-mainnet geth[21361]: Error: could not apply tx 71 [0xdfc3366aa86ba0bf7b31ea53647889b371d075b853094bdfcc4d51a9465c55b0]: nonce too high: address 0x62823EE33597787873815a79749650B966Ebee2C, tx: 1636 state: 1635
Jan 09 20:16:36 geth-mainnet geth[21361]: Platform: geth v1.16.7 go1.25.1 amd64 linux
Jan 09 20:16:36 geth-mainnet geth[21361]: VCS: b9f3a3d9-20251104
Jan 09 20:16:36 geth-mainnet geth[21361]: Chain config: &params.ChainConfig{ChainID:1, HomesteadBlock:1150000, DAOForkBlock:1920000, DAOForkSupport:true, EIP150Block:2463000, EIP155Block:2675000, EIP158Block:2675000, ByzantiumBlock:4370000, ConstantinopleBlock:7280000, PetersburgBlock:7280000, IstanbulBlock:9069000, MuirGlacierBlock:9200000, BerlinBlock:12244000, LondonBlock:12965000, ArrowGlacierBlock:13773000, GrayGlacierBlock:15050000, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(0xc0003a2030), CancunTime:(*uint64)(0xc0003a2038), PragueTime:(*uint64)(0xc0003a2040), OsakaTime:(*uint64)(0xc0003a2048), BPO1Time:(*uint64)(0xc0003a2050), BPO2Time:(*uint64)(0xc0003a2058), BPO3Time:(*uint64)(nil), BPO4Time:(*uint64)(nil), BPO5Time:(*uint64)(nil), AmsterdamTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:58750000000000000000000, DepositContractAddress:0x00000000219ab540356cBB839Cbe05303d7705Fa, EnableVerkleAtGenesis:false, Ethash:(*params.EthashConfig)(0x3429a00), Clique:(*params.CliqueConfig)(nil), BlobScheduleConfig:(*params.BlobScheduleConfig)(0x33f78e0)}
Jan 09 20:16:36 geth-mainnet geth[21361]: Receipts:
Jan 09 20:16:36 geth-mainnet geth[21361]: ##############################
Jan 09 20:16:36 geth-mainnet geth[21361]: WARN [01-09|20:16:36.483] NewPayload: inserting block failed       error="could not apply tx 71 [0xdfc3366aa86ba0bf7b31ea53647889b371d075b853094bdfcc4d51a9465c55b0]: nonce too high: address 0x62823EE33597787873815a79749650B966Ebee2C, tx: 1636 state: 1635"

Additional Information

Network: Mainnet
Sync Mode: Snap (issue encountered after snap sync has completed)

Restarting geth restores it to an operable state and is able to continue progressing state past the failed block after the debug_executionWitness call

[max-sanchez]

Fix proposed in #33574

[islishude]

I've encountered the same bug.

I think the following code changes should be correct for now.

we don't need to make witness and save states at the same time.

	// Don't write the block if we only want to generate the witness
	if !makeWitness {
		if !setHead {
			// Don't set the head, only insert the block
			err = bc.writeBlockWithState(block, res.Receipts, statedb)
		} else {
			status, err = bc.writeBlockAndSetHead(block, res.Receipts, res.Logs, statedb, false)
		}
		if err != nil {
			return nil, err
		}
	}