My node is failed to start and showing the `panic: runtime error`

i created the testnet node using the following docs and the node is failed to start.
the error i got is below:

sh[16600]: 2:14PM INF ABCI Replay Blocks appHeight=11862373 module=consensus server=node stateHeight=11862373 storeHeight=11862374 sh[16600]: 2:14PM INF Replay last block using real app module=consensus server=node sh[16600]: panic: runtime error: index out of range [7] with length 1 sh[16600]: goroutine 1 [running]:

1 Like

could you provide more logs like last 500 lines as well as the output of fxcored doctor command?

sure @lancelai ,

Dec 19 04:46:10  systemd[1]: Started fxcore service.
Dec 19 04:46:11  sh[50547]: 4:46AM INF Unlocking keyring
Dec 19 04:46:11  sh[50547]: 4:46AM INF starting ABCI with Tendermint
Dec 19 04:46:22  sh[50547]: 4:46AM INF service start impl=multiAppConn module=proxy msg={} server=node
Dec 19 04:46:22  sh[50547]: 4:46AM INF service start connection=query impl=localClient module=abci-client msg={} server=node
Dec 19 04:46:22  sh[50547]: 4:46AM INF service start connection=snapshot impl=localClient module=abci-client msg={} server=node
Dec 19 04:46:22  sh[50547]: 4:46AM INF service start connection=mempool impl=localClient module=abci-client msg={} server=node
Dec 19 04:46:22  sh[50547]: 4:46AM INF service start connection=consensus impl=localClient module=abci-client msg={} server=node
Dec 19 04:46:22  sh[50547]: 4:46AM INF service start impl=EventBus module=events msg={} server=node
Dec 19 04:46:22  sh[50547]: 4:46AM INF service start impl=PubSub module=pubsub msg={} server=node
Dec 19 04:46:22  sh[50547]: 4:46AM INF service start impl=IndexerService module=txindex msg={} server=node
Dec 19 04:46:22  sh[50547]: 4:46AM INF ABCI Handshake App Info hash="J�����,�\x14�kVQo:|�n��%)a\x03���b�+�A" height=11862373 module=consensus protocol-version=0 server=node software-version=release/v3.1.x-8c9b7b73d0a1e148ba4e86f409320a04a9e5749d
Dec 19 04:46:22  sh[50547]: 4:46AM INF ABCI Replay Blocks appHeight=11862373 module=consensus server=node stateHeight=11862373 storeHeight=11862374
Dec 19 04:46:22  sh[50547]: 4:46AM INF Replay last block using real app module=consensus server=node
Dec 19 04:46:22  sh[50547]: panic: runtime error: index out of range [7] with length 1
Dec 19 04:46:22  sh[50547]: goroutine 1 [running]:
Dec 19 04:46:22  sh[50547]: encoding/binary.bigEndian.Uint64(...)
Dec 19 04:46:22  sh[50547]:         encoding/binary/binary.go:179
Dec 19 04:46:22  sh[50547]: github.com/cosmos/cosmos-sdk/types.BigEndianToUint64(...)
Dec 19 04:46:22  sh[50547]:         github.com/cosmos/cosmos-sdk@v0.45.11/types/utils.go:66
Dec 19 04:46:22  sh[50547]: github.com/cosmos/cosmos-sdk/x/upgrade/keeper.Keeper.GetLastCompletedUpgrade({{0x7fff84c28e64, 0x14}, 0xc002d8c630, {0x2db1920, 0xc002dbf950}, {0x2ddb230, 0xc000564320}, 0xc002e3e180, {0x2da92c0, 0xc000ecb500}, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/cosmos/cosmos-sdk@v0.45.11/x/upgrade/keeper/keeper.go:250 +0x385
Dec 19 04:46:22  sh[50547]: github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7fff84c28e64, 0x14}, 0xc002d8c630, {0x2db1920, 0xc002dbf950}, {0x2ddb230, 0xc000564320}, 0xc002e3e180, {0x2da92c0, 0xc000ecb500}, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/cosmos/cosmos-sdk@v0.45.11/x/upgrade/abci.go:30 +0x298
Dec 19 04:46:22  sh[50547]: github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
Dec 19 04:46:22  sh[50547]:         github.com/cosmos/cosmos-sdk@v0.45.11/x/upgrade/module.go:130
Dec 19 04:46:22  sh[50547]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2dca848, 0xc000054060}, {0x2ddd640, 0xc008c50300}, {{0xb, 0x0}, {0xc008bccc30, 0xa}, 0xb50166, ...}, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/cosmos/cosmos-sdk@v0.45.11/types/module/module.go:491 +0x1db
Dec 19 04:46:22  sh[50547]: github.com/functionx/fx-core/v3/app.(*App).BeginBlocker(...)
Dec 19 04:46:22  sh[50547]:         github.com/functionx/fx-core/v3/app/app.go:182
Dec 19 04:46:22  sh[50547]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc01217a520, 0x20, 0x20}, {{0xb, 0x0}, {0xc008bccc30, 0xa}, 0xb50166, {0x1d5bed24, ...}, ...}, ...})
Dec 19 04:46:22  sh[50547]:         github.com/cosmos/cosmos-sdk@v0.45.11/baseapp/abci.go:177 +0x97b
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc01217a520, 0x20, 0x20}, {{0xb, 0x0}, {0xc008bccc30, 0xa}, 0xb50166, {0x1d5bed24, ...}, ...}, ...})
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/abci/client/local_client.go:280 +0x118
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc01217a520, 0x20, 0x20}, {{0xb, 0x0}, {0xc008bccc30, 0xa}, 0xb50166, {0x1d5bed24, ...}, ...}, ...})
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/proxy/app_conn.go:81 +0x55
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2dcbbc0?, 0xc000e0b920}, {0x2dd3ec0, 0xc0035f5530}, 0xc002e461e0, {0x2ddc640, 0xc00d32c1c8}, 0xb50165?)
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/state/execution.go:307 +0x51d
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0035efc10, 0x7}}, {0xc0035efc20, 0xa}, 0x1, 0xb50165, {{0xc0121508c0, ...}, ...}, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/state/execution.go:140 +0x171
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc0035efc10, 0x7}}, {0xc0035efc20, 0xa}, 0x1, 0xb50165, {{0xc0121508c0, ...}, ...}, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/consensus/replay.go:503 +0x23c
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc0035efc10, 0x7}}, {0xc0035efc20, 0xa}, 0x1, 0xb50165, {{0xc0121508c0, ...}, ...}, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/consensus/replay.go:416 +0x7ae
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc008cb5500, {0x2dde388, 0xc00132ea90})
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/consensus/replay.go:268 +0x3d4
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc0035efc10, 0x7}}, {0xc0035efc20, 0xa}, 0x1, 0xb50165, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/node/node.go:329 +0x1b8
Dec 19 04:46:22  sh[50547]: github.com/tendermint/tendermint/node.NewNode(0xc000578f00, {0x2dc7240, 0xc00062f360}, 0xc0002c8540, {0x2dab360, 0xc009af5ec0}, 0x0?, 0x0?, 0xc0002c8680, {0x2dcbbc0, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/tendermint/tendermint@v0.34.23/node/node.go:777 +0x597
Dec 19 04:46:22  sh[50547]: github.com/functionx/fx-core/v3/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2ded830, 0xc000620630}, {0x0, 0x0}, {0x2dd0458, 0xc000564320}, ...}, ...)
Dec 19 04:46:22  sh[50547]:         github.com/functionx/fx-core/v3/server/start.go:307 +0xd13
Dec 19 04:46:22  sh[50547]: github.com/functionx/fx-core/v3/server.StartCmd.func2(0xc001512900?, {0xc001517b80?, 0x1?, 0xa?})
Dec 19 04:46:22  sh[50547]:         github.com/functionx/fx-core/v3/server/start.go:116 +0x228
Dec 19 04:46:22  sh[50547]: github.com/spf13/cobra.(*Command).execute(0xc001512900, {0xc001517a40, 0xa, 0xa})
Dec 19 04:46:22  sh[50547]:         github.com/spf13/cobra@v1.6.0/command.go:916 +0x862
Dec 19 04:46:22  sh[50547]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000004f00)
Dec 19 04:46:22  sh[50547]:         github.com/spf13/cobra@v1.6.0/command.go:1040 +0x3bd
Dec 19 04:46:22  sh[50547]: github.com/spf13/cobra.(*Command).Execute(...)
Dec 19 04:46:22  sh[50547]:         github.com/spf13/cobra@v1.6.0/command.go:968
Dec 19 04:46:22  sh[50547]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Dec 19 04:46:22  sh[50547]:         github.com/spf13/cobra@v1.6.0/command.go:961
Dec 19 04:46:22  sh[50547]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc0001231e8, 0x14})
Dec 19 04:46:22  sh[50547]:         github.com/cosmos/cosmos-sdk@v0.45.11/server/cmd/execute.go:36 +0x1eb
Dec 19 04:46:22  sh[50547]: main.main()
Dec 19 04:46:22  sh[50547]:         github.com/functionx/fx-core/v3/cmd/fxcored/main.go:13 +0x2c
Dec 19 04:46:22  systemd[1]: fxcore.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 19 04:46:22  systemd[1]: fxcore.service: Failed with result 'exit-code'.
^C

hi, i need the output of fxcored doctor command as well

actually im using the binary not docker

yes binary has the command fxcored doctor since v4.0.0 , we need the output to troubleshoot, I am guessing you are using the wrong version of binary. You can try to compile with release/v6.0.x and use latest snapshot to run again

you can also check the version by typing fxcored version

@lancelai yes my bad i used old version, i updated the new version. and the output you asked is,

Please note that these warnings are just used to help the fxCore maintainers
If everything you use "fxcored" for is working fine: please don't worry; 
just ignore this. Thanks!

Computer Info:
  OS/Arch: linux/amd64
  CPU: 8
  Memory Total: 15.17 GB, Available: 14.02 GB, UsedPercent: 5.37%
  Disk Total: 968.99 GB, Free: 544.05 GB, UsedPercent: 43.85%
    Path: /home/ubuntu/.fxcore
fxcored Info:
  Version: release/v5.0.x-v5.0.0
  Git Commit: ab04d667f2fb52deb5101fbc193b32dc9f99b9f8
  Build Tags: netgo,ledger
  Go Version: go1.20.3
  Cosmos SDK Version: v0.46.13
  Tendermint Version: 0.34.28
Genesis:
  File: /home/ubuntu/.fxcore/config/genesis.json
  Network: Mainnet
Upgrade Info:
  File: /home/ubuntu/.fxcore/data/upgrade-info.json
  Name: v6.0.x
  Height: 11701000
Blockchain Data:
  Remote Node: tcp://localhost:26657 
  Chain ID: dhobyghaut
  Block Height: 11862374
  Syncing: true
  Node Info: 
    Version: release/v5.0.x-v5.0.0
    Git Commit: ab04d667f2fb52deb5101fbc193b32dc9f99b9f8
    Build Tags: netgo,ledger
    Go Version: go version go1.20.3 linux/amd64
    Cosmos SDK Version: v0.46.13
    Warning: The remote node chainId(dhobyghaut) does not match the local genesis chainId(fxcore)
App Config:
  File: /home/ubuntu/.fxcore/config/app.toml
Tendermint Config:
  File: /home/ubuntu/.fxcore/config/config.toml
Cosmovisor: not found

and the logs are

gger":{}}
Dec 19 08:52:45 sh[105988]: 08:52:45AM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg="Stopping Peer service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}
Dec 19 08:52:45 sh[105988]: 08:52:45AM ERR error while stopping peer error="already stopped" module=p2p
Dec 19 08:52:47 sh[105988]: 08:52:47AM INF Error reconnecting to peer. Trying again addr={"id":"96f04dffc25ffcce11e179581d2a3ab6cb5535d5","ip":"54.164.97.154","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=10
Dec 19 08:52:48 sh[105988]: 08:52:48AM INF Error reconnecting to peer. Trying again addr={"id":"836ded83bac83a4ac8511826fa1ad4ca2238f960","ip":"35.172.96.99","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=10
Dec 19 08:52:51 sh[105988]: 08:52:51AM INF Error reconnecting to peer. Trying again addr={"id":"0fee38117655b6961319950d6beb929fb194217c","ip":"34.234.202.223","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=11
Dec 19 08:52:51 sh[105988]: 08:52:51AM INF Error reconnecting to peer. Trying again addr={"id":"6e8818051a2ca9b8be67a6f2ba48c33d8c489d5c","ip":"52.202.15.88","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=10
Dec 19 08:52:52 sh[105988]: 08:52:52AM INF Error reconnecting to peer. Trying again addr={"id":"7c7a260eeefda37eac896ae423e78cf345a2ef70","ip":"52.4.101.2","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=11
Dec 19 08:52:55 sh[105988]: 08:52:55AM INF Error reconnecting to peer. Trying again addr={"id":"96f04dffc25ffcce11e179581d2a3ab6cb5535d5","ip":"54.164.97.154","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=11
Dec 19 08:52:56 sh[105988]: 08:52:56AM INF Error reconnecting to peer. Trying again addr={"id":"836ded83bac83a4ac8511826fa1ad4ca2238f960","ip":"35.172.96.99","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=11
Dec 19 08:52:57 sh[105988]: 08:52:57AM INF Error reconnecting to peer. Trying again addr={"id":"0fee38117655b6961319950d6beb929fb194217c","ip":"34.234.202.223","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=12
Dec 19 08:52:57 sh[105988]: 08:52:57AM INF Error reconnecting to peer. Trying again addr={"id":"6e8818051a2ca9b8be67a6f2ba48c33d8c489d5c","ip":"52.202.15.88","port":26656} err="incompatible: peer is on a different network. Got fxcore, expected dhobyghaut" module=p2p tries=11
Dec 19 09:02:46 sh[106046]: 09:02:46AM INF Reconnecting to peer addr={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656} module=p2p
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF service start impl="Peer{MConn{54.80.190.189:26656} e922b34e660976a64d6024bde495666752141992 out}" module=p2p msg="Starting Peer service" peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF service start impl=MConn{54.80.190.189:26656} module=p2p msg="Starting MConnection service" peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF service start impl="Peer{MConn{44.205.223.158:26656} d22e741b4e8e2586dbe38fd348d3de8dfbb889a0 out}" module=p2p msg="Starting Peer service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF service start impl=MConn{44.205.223.158:26656} module=p2p msg="Starting MConnection service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF Connection is closed @ recvRoutine (likely by the other side) conn={"Logger":{}} module=p2p peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF service stop impl={"Logger":{}} module=p2p msg="Stopping MConnection service" peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 19 09:02:47 sh[106046]: 09:02:47AM ERR Stopping peer for error err=EOF module=p2p peer={"Data":{},"Logger":{}}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg="Stopping Peer service" peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF Reconnecting to peer addr={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656} module=p2p
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF invalid peer blockHeight=11862663 module=blockchain peer=d22e741b4e8e2586dbe38fd348d3de8dfbb889a0
Dec 19 09:02:47 sh[106046]: 09:02:47AM ERR Stopping peer for error err="error with peer d22e741b4e8e2586dbe38fd348d3de8dfbb889a0: invalid peer" module=p2p peer={"Data":{},"Logger":{}}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg="Stopping Peer service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}
Dec 19 09:02:47 sh[106046]: 09:02:47AM INF service stop impl={"Logger":{}} module=p2p msg="Stopping MConnection service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}
Dec 19 09:02:48 sh[106046]: 09:02:48AM INF service start impl="Peer{MConn{54.80.190.189:26656} e922b34e660976a64d6024bde495666752141992 out}" module=p2p msg="Starting Peer service" peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 19 09:02:48 sh[106046]: 09:02:48AM INF service start impl=MConn{54.80.190.189:26656} module=p2p msg="Starting MConnection service" peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 19 09:02:48 sh[106046]: 09:02:48AM INF Connection is closed @ recvRoutine (likely by the other side) conn={"Logger":{}} module=p2p peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}

for peer to peer error, refer to below

@lancelai i have added the same in the config.toml file,

# Comma separated list of seed nodes to connect to
seeds = "e922b34e660976a64d6024bde495666752141992@dhobyghaut-seed-node-1.functionx.io:26656,a817685c010402703820be2b5a90d9e07bc5c2d3@dhobyghaut-node-1.functionx.io:26656"

# Comma separated list of nodes to keep persistent connections to
persistent_peers = "e922b34e660976a64d6024bde495666752141992@dhobyghaut-seed-node-1.functionx.io:26656,a817685c010402703820be2b5a90d9e07bc5c2d3@dhobyghaut-node-1.functionx.io:26656"
2 Likes

@lancelai still the issue is not resolved any solution for it?

Dec 21 05:23:02 sh[109576]: 05:23:02AM ERR Stopping peer for error err="read tcp 172.31.14.10:52836->54.83.49.110:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
Dec 21 05:23:02 sh[109576]: 05:23:02AM ERR error while stopping peer error="already stopped" module=p2p
Dec 21 05:23:02 sh[109576]: 05:23:02AM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg="Stopping Peer service" peer={"id":"a817685c010402703820be2b5a90d9e07bc5c2d3","ip":"54.83.49.110","port":26656}
Dec 21 05:23:09 sh[109576]: 05:23:09AM ERR Error in validation err="wrong Block.Header.AppHash.  Expected 9A0465017668A551FA290B75EEE90CC2C847977D041EC5EEBBEF03193CF290ED, got 93575FD900D68D7B3A31E9CA7F7E85F371EB58B4BA249CA483CCD16C3977316F" module=blockchain
Dec 21 05:23:09 sh[109576]: 05:23:09AM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected 9A0465017668A551FA290B75EEE90CC2C847977D041EC5EEBBEF03193CF290ED, got 93575FD900D68D7B3A31E9CA7F7E85F371EB58B4BA249CA483CCD16C3977316F" module=p2p peer={"Data":{},"Logger":{}}
Dec 21 05:23:09 sh[109576]: 05:23:09AM INF service stop impl={"Logger":{}} module=p2p msg="Stopping MConnection service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}
Dec 21 05:23:09 sh[109576]: 05:23:09AM ERR Stopping peer for error err="read tcp 172.31.14.10:49242->44.205.223.158:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
Dec 21 05:23:09 sh[109576]: 05:23:09AM ERR error while stopping peer error="already stopped" module=p2p
Dec 21 05:23:09 sh[109576]: 05:23:09AM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg="Stopping Peer service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}
Dec 21 05:23:30 sh[109576]: 05:23:30AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10

Hello @lancelai any update on my issue above?

Please provide the output of fxcored doctor again after using the new version

And did you use any snapshot?

Please note that these warnings are just used to help the fxCore maintainers
If everything you use "fxcored" for is working fine: please don't worry; 
just ignore this. Thanks!

Computer Info:
  OS/Arch: linux/amd64
  CPU: 8
  Memory Total: 15.17 GB, Available: 13.97 GB, UsedPercent: 5.63%
  Disk Total: 968.99 GB, Free: 543.80 GB, UsedPercent: 43.88%
    Path: /home/ubuntu/.fxcore
fxcored Info:
  Version: release/v5.0.x-v5.0.0
  Git Commit: ab04d667f2fb52deb5101fbc193b32dc9f99b9f8
  Build Tags: netgo,ledger
  Go Version: go1.20.3
  Cosmos SDK Version: v0.46.13
  Tendermint Version: 0.34.28
Genesis:
  File: /home/ubuntu/.fxcore/config/genesis.json
  Network: Testnet
Upgrade Info:
  File: /home/ubuntu/.fxcore/data/upgrade-info.json
  Name: v6.0.x
  Height: 11701000
Blockchain Data:
  Remote Node: tcp://localhost:26657 
  Chain ID: dhobyghaut
  Block Height: 11862374
  Syncing: true
  Node Info: 
    Version: release/v5.0.x-v5.0.0
    Git Commit: ab04d667f2fb52deb5101fbc193b32dc9f99b9f8
    Build Tags: netgo,ledger
    Go Version: go version go1.20.3 linux/amd64
    Cosmos SDK Version: v0.46.13
App Config:
  File: /home/ubuntu/.fxcore/config/app.toml
Tendermint Config:
  File: /home/ubuntu/.fxcore/config/config.toml
Cosmovisor: not found

use latest snapshot

used the latest pruned snapshot

","ip":"54.80.190.189","port":26656}
Dec 22 06:52:28 ip-172-31-14-10 sh[116083]: 06:52:28AM INF service stop impl={"Logger":{}} module=p2p msg="Stopping MConnection service" peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 22 06:52:28 ip-172-31-14-10 sh[116083]: 06:52:28AM ERR Stopping peer for error err=EOF module=p2p peer={"Data":{},"Logger":{}}
Dec 22 06:52:28 ip-172-31-14-10 sh[116083]: 06:52:28AM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg="Stopping Peer service" peer={"id":"e922b34e660976a64d6024bde495666752141992","ip":"54.80.190.189","port":26656}
Dec 22 06:52:33 ip-172-31-14-10 sh[116083]: 06:52:33AM ERR Error in validation err="wrong Block.Header.AppHash.  Expected 5C6E74AC21DFFE435DA2BD4744AC9F9E280C2982C2A948FC7158785177B559D2, got DA73455650D756C256794B3E9E664F85A234768072A3B49F7A41B202130991FB" module=blockchain
Dec 22 06:52:33 ip-172-31-14-10 sh[116083]: 06:52:33AM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected 5C6E74AC21DFFE435DA2BD4744AC9F9E280C2982C2A948FC7158785177B559D2, got DA73455650D756C256794B3E9E664F85A234768072A3B49F7A41B202130991FB" module=p2p peer={"Data":{},"Logger":{}}
Dec 22 06:52:33 ip-172-31-14-10 sh[116083]: 06:52:33AM INF service stop impl={"Logger":{}} module=p2p msg="Stopping MConnection service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}
Dec 22 06:52:33 ip-172-31-14-10 sh[116083]: 06:52:33AM ERR Stopping peer for error err="read tcp 172.31.14.10:41702->44.205.223.158:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
Dec 22 06:52:33 ip-172-31-14-10 sh[116083]: 06:52:33AM ERR error while stopping peer error="already stopped" module=p2p
Dec 22 06:52:33 ip-172-31-14-10 sh[116083]: 06:52:33AM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg="Stopping Peer service" peer={"id":"d22e741b4e8e2586dbe38fd348d3de8dfbb889a0","ip":"44.205.223.158","port":26656}

hi, from the output of fxcored doctor we identified that the fxcore version you are using are v5, but the snapshot is the latest so v6 binary is required

1 Like

@lancelai hey its worked Thanks a lot. :raised_hands:

1 Like