En panic 오류에 대해 질문드립니다

한국어🇰🇷 Kaia
1

안녕하세요 EN노드를 운영하고 있습니다.
ken 1.9.X 버전대에서 ken 1.10.0 버전으로 올리니 아래의 오류들이 1분에 1번씩 발생해 En 노드의 정상적인 운영이 불가능합니다.

다시 1.9.X 버전대로 돌아가도 마찬가지 입니다…
한번 확인 부탁드립니다. 감사합니다.

문제가 발생한 ken version

Klaytn v1.9.1+8664bc8831
Klaytn v1.10.0+a7ac680ffc

kend.conf

# Configuration file for the kend

# cypress, baobab is only available if you don't specify NETWORK_ID.
NETWORK="cypress"
# if you specify NETWORK_ID, a private network is created.
NETWORK_ID=

PORT=32323

SERVER_TYPE="fasthttp"
SYNCMODE="full"
VERBOSITY=3
MAXCONNECTIONS=10

# txpool options setting
TXPOOL_EXEC_SLOTS_ALL=4096
TXPOOL_NONEXEC_SLOTS_ALL=4096
TXPOOL_EXEC_SLOTS_ACCOUNT=4096
TXPOOL_NONEXEC_SLOTS_ACCOUNT=4096
TXPOOL_LIFE_TIME="30m"

# rpc options setting
RPC_ENABLE=0 # if this is set, the following options will be used
RPC_API="klay,net,web3,rpc,eth" # available apis: admin,debug,klay,miner,net,personal,rpc,txpool,web3
RPC_PORT=80
RPC_ADDR="0.0.0.0"
RPC_CORSDOMAIN="*"
RPC_VHOSTS="*"
RPC_CONCURRENCYLIMIT=3000
RPC_READ_TIMEOUT=30
RPC_WRITE_TIMEOUT=30
RPC_IDLE_TIMEOUT=120
RPC_EXECUTION_TIMEOUT=30

# ws options setting
WS_ENABLE=0 # if this is set, the following options will be used
WS_API="klay,net,web3" # available apis: admin,debug,klay,miner,net,personal,rpc,txpool,web3
WS_ADDR="0.0.0.0"
WS_PORT=8552
WS_ORIGINS="*"

# service chain options setting
SC_MAIN_BRIDGE=0 # if this is set, the following options will be used.
SC_MAIN_BRIDGE_PORT=50505
SC_MAIN_BRIDGE_INDEXING=0  # this option will be deprecated.

# Setting 1 is to enable options, otherwise disabled.
AUTO_RESTART=0
METRICS=0
PROMETHEUS=0
DB_NO_PARALLEL_WRITE=0
MULTICHANNEL=1
SUBPORT=$((PORT + 1)) # used for multi channel option

# discover options
NO_DISCOVER=0 # setting 1 to disable discovery
BOOTNODES=""

# Raw options e.g) "--txpool.nolocals"
ADDITIONAL=""

# auto restart options
AUTO_RESTART_NODE=1
AUTO_RESTART_INTERVAL=0.1

DATA_DIR=/klay/cypress
LOG_DIR=$DATA_DIR/logs

시스템 정보

Instance Type:  m5a.2xlarge (AWS)
OS: Ubuntu 22.04.1 LTS

오류 1

INFO[01/02,04:41:10 Z] [18] Setting connection type                   nodetype=en conntype=1
INFO[01/02,04:41:10 Z] [18] Cypress bootnodes are set                
INFO[01/02,04:41:10 Z] [18] Bootnode - Add Seed                       Node="kni://18b36118cce093673499fc6e9aa196f047fe17a0de35b6f2a76a4557802f6abf9f89aa5e7330e93c9014b714b9df6378393611efe39aec9d3d831d6aa9d617ae@52.78.22.32:32323?ntype=bn"
INFO[01/02,04:41:10 Z] [18] Bootnode - Add Seed                       Node="kni://63f1c96874da85140ecca3ce24875cb5ef28fa228bc3572e16f690db4a48fc8067502d2f6e8f0c66fb558276a5ada1e4906852c7ae42b0003e9f9f25d1e123b1@52.194.200.217:32323?ntype=bn"
INFO[01/02,04:41:10 Z] [18] Bootnode - Add Seed                       Node="kni://94cc15e2014b86584908707de55800c0a2ea8a24dc5550dcb507043e4cf18ff04f21dc86ed17757dc63b1fa85bb418b901e5e24e4197ad4bbb0d96cd9389ed98@18.136.251.28:32323?ntype=bn"
INFO[01/02,04:41:10 Z] [18] Setting MaxPhysicalConnections            MaxPhysicalConnections=10
INFO[01/02,04:41:10 Z] [18] Cypress network ID is set                 networkid=8217
INFO[01/02,04:41:10 Z] [18] Set the concurrency limit of RPC-HTTP server  limit=3000
INFO[01/02,04:41:10 Z] [18] Raised fd limit to process's maximum value  fd=1048576
INFO[01/02,04:41:10 Z] [18] Cypress network ID is set                 networkid=8217
INFO[01/02,04:41:10 Z] [18] Archiving mode of this node               isArchiveMode=false
INFO[01/02,04:41:10 Z] [40] Starting peer-to-peer node                instance=Klaytn/v1.9.1+8664bc8831/linux-amd64/go1.15.7
INFO[01/02,04:41:10 Z] [48] Non-single database is used for persistent storage  DBType=LevelDB
INFO[01/02,04:41:10 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/misc levelDBCacheSize=16 openFilesLimit=10485 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:10 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/header levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:11 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/body   levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:11 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/receipts levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:11 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/statetrie_migrated_87145087/0 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:12 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/statetrie_migrated_87145087/1 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:13 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/statetrie_migrated_87145087/2 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:14 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/statetrie_migrated_87145087/3 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:15 Z] [48] Created a sharded database                dbType=statetrie numShards=4
INFO[01/02,04:41:15 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/txlookup                      levelDBCacheSize=16 openFilesLimit=10485 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:15 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/bridgeservice                 levelDBCacheSize=16 openFilesLimit=5242  useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:15 Z] [48] LevelDB configurations                    path=/klay/cypress/chaindata/snapshot                      levelDBCacheSize=22 openFilesLimit=15728 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[01/02,04:41:16 Z] [30] Successfully loaded governance state from database  blockNumber=110725120
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x14b8709]

goroutine 1 [running]:
github.com/klaytn/klaytn/governance.(*Governance).ReadGovernance(0xc0001a50e0, 0x6998900, 0x691e4eac1c7eb14, 0x95fcd6b1fd74fdd4, 0x883cb4414bfba42c, 0xc0041cccd8)
	/ext-go/1/src/github.com/klaytn/klaytn/governance/default.go:814 +0x29
github.com/klaytn/klaytn/governance.(*Governance).initializeCache(0xc0001a50e0, 0x201dd60, 0xc00021be00)
	/ext-go/1/src/github.com/klaytn/klaytn/governance/default.go:717 +0x428
github.com/klaytn/klaytn/governance.NewGovernanceInitialize(0x2d0cfa0, 0x201dd60, 0xc00021be00, 0x0)
	/ext-go/1/src/github.com/klaytn/klaytn/governance/default.go:450 +0xa5
github.com/klaytn/klaytn/governance.newMixedEngine(0x2d0cfa0, 0x201dd60, 0xc00021be00, 0x2001, 0x2010000)
	/ext-go/1/src/github.com/klaytn/klaytn/governance/mixed.go:45 +0xec
github.com/klaytn/klaytn/governance.NewMixedEngine(...)
	/ext-go/1/src/github.com/klaytn/klaytn/governance/mixed.go:58
github.com/klaytn/klaytn/node/cn.New(0xc0008f60c0, 0xc0004d5800, 0x1b6fe00, 0x1c1d501, 0xc0008f60c0)
	/ext-go/1/src/github.com/klaytn/klaytn/node/cn/backend.go:218 +0x198
github.com/klaytn/klaytn/cmd/utils.RegisterCNService.func1(0xc0008f60c0, 0xc0004fd6e0, 0xc00046d5c8, 0x33, 0xc00046d5d8)
	/ext-go/1/src/github.com/klaytn/klaytn/cmd/utils/flags.go:1988 +0x65
github.com/klaytn/klaytn/node.(*Node).initService(0xc00003d800, 0xc0000c28e8, 0x1, 0x1, 0xc0004fd6e0, 0x2, 0x1cdabcf)
	/ext-go/1/src/github.com/klaytn/klaytn/node/node.go:298 +0x102
github.com/klaytn/klaytn/node.(*Node).Start(0xc00003d800, 0x0, 0x0)
	/ext-go/1/src/github.com/klaytn/klaytn/node/node.go:199 +0x585
github.com/klaytn/klaytn/cmd/utils.StartNode(0xc00003d800)
	/ext-go/1/src/github.com/klaytn/klaytn/cmd/utils/cmd.go:46 +0x2f
github.com/klaytn/klaytn/cmd/utils/nodecmd.startNode(0xc0001a6580, 0xc00003d800)
	/ext-go/1/src/github.com/klaytn/klaytn/cmd/utils/nodecmd/defaultcmd.go:65 +0x7e
github.com/klaytn/klaytn/cmd/utils/nodecmd.RunKlaytnNode(0xc0001a6580, 0x0, 0x0)
	/ext-go/1/src/github.com/klaytn/klaytn/cmd/utils/nodecmd/defaultcmd.go:53 +0x45
gopkg.in/urfave/cli%2ev1.HandleAction(0x19bd240, 0x1d86990, 0xc0001a6580, 0xc000276de0, 0x0)
	/go/pkg/mod/gopkg.in/urfave/cli.v1@v1.20.0/app.go:490 +0x82
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc00010e820, 0xc00003c300, 0x30, 0x30, 0x0, 0x0)
	/go/pkg/mod/gopkg.in/urfave/cli.v1@v1.20.0/app.go:264 +0x5f5
main.main()
	/ext-go/1/src/github.com/klaytn/klaytn/cmd/ken/main.go:91 +0x7b

오류 2

panic: runtime error: slice bounds out of range [:4560] with capacity 4096

goroutine 185507 [running]:
bufio.(*Reader).Read(0xc058e6ad20, {0xc054b87ff7?, 0x9?, 0x4000?})
	/usr/local/src/bufio/bufio.go:249 +0x2ed
github.com/valyala/fasthttp.(*requestStream).Read(0xc053331590, {0xc054b87ff7?, 0xc01016d540?, 0xc05457a000?})
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/streaming.go:75 +0x2e5
github.com/valyala/bytebufferpool.(*ByteBuffer).ReadFrom(0xc056060780, {0x1e2eae0, 0xc053331590})
	/go/pkg/mod/github.com/valyala/bytebufferpool@v1.0.0/bytebuffer.go:45 +0x13e
io.copyBuffer({0x1e2e920, 0xc056060780}, {0x1e2eae0, 0xc053331590}, {0xc03fa08000, 0x1000, 0x1000})
	/usr/local/src/io/io.go:412 +0x14b
io.CopyBuffer({0x1e2e920?, 0xc056060780?}, {0x1e2eae0?, 0xc053331590?}, {0xc03fa08000?, 0xc0534a2dc0?, 0x0?})
	/usr/local/src/io/io.go:399 +0x3c
github.com/valyala/fasthttp.copyZeroAlloc({0x1e2e920, 0xc056060780}, {0x1e2eae0, 0xc053331590})
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/http.go:1981 +0x7e
github.com/valyala/fasthttp.(*Request).bodyBytes(0xc047d10000)
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/http.go:371 +0xec
github.com/valyala/fasthttp.(*Request).Body(0xe0ed25?)
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/http.go:671 +0x9e
github.com/valyala/fasthttp.(*RequestCtx).PostBody(...)
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/server.go:1415
github.com/valyala/fasthttp/fasthttpadaptor.ConvertRequest(0xc047d10000, 0xc03f10eea0, 0x1)
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/fasthttpadaptor/request.go:15 +0x37
github.com/valyala/fasthttp/fasthttpadaptor.NewFastHTTPHandler.func1(0xc047d10000)
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/fasthttpadaptor/adaptor.go:50 +0x85
github.com/valyala/fasthttp.TimeoutWithCodeHandler.func1.1()
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/server.go:471 +0x33
created by github.com/valyala/fasthttp.TimeoutWithCodeHandler.func1
	/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/server.go:470 +0x16b
2

안녕하세요, 질문하신 내용에 궁금한점이 있어 질문 남깁니다.

먼저 v1.9.1 → v1.10.0으로 업그레이드 하시고나서 오류2가 1분마다 발생해서 다시 1.9로 롤백하셨다는 의미가 맞나요?

4

안녕하세요 @jack_jin
오류 1, 오류 2가 번갈아가면서 나타나고 있고, v1.9.X 버전대로 롤백해도 동일한 오류가 지속해서 나타나고 있어 EN 노드를 중지했습니다!

5

v1.10으로 업그레이드 하셨을때, 오류 1,2가 번갈아 가면서 나타났고 롤백후에도 동일하게 번갈아가면서 나타난다는 말씀이신거죠?

6

네 맞습니다.
현재 그 증상이고 블록 싱크가 되다가도 갑작스레 오류 1, 2번이 번갈아가며 나타나서 ken이 재시작됩니다.

7

@yaucto 우선 DB가 꼬인것 같아서 최신 스냅샷을 데이터부터 다시 싱크를 진행하시는 것을 추천드립니다.

그리고 디버깅이 조금 필요할 것 같은데 혹시 디스코드에서 저에게 한번만 멘션해주실 수 있으신가요?