Bug #45634
closedv15.2.1 vstart.sh：failed run crimson-osd on ARM
0%
Description
MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh n -x --without-dashboard --memstore --crimson --nodaemon --redirect-output --osd-args "--memory 4G --debug" /home/ceph/build/dev/mon.a
ceph-mgr dashboard not built - disabling.
rm -f core*
hostname localhost
ip 90.90.26.251
port 40412
/home/ceph/build/bin/ceph-authtool --create-keyring --gen-key --name=mon. /home/ceph/build/keyring --cap mon 'allow *'
creating /home/ceph/build/keyring
/home/ceph/build/bin/ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /home/ceph/build/keyring
/home/ceph/build/bin/monmaptool --create --clobber --addv a [v2:90.90.26.251:40412,v1:90.90.26.251:40413] --print /tmp/ceph_monmap.14682
/home/ceph/build/bin/monmaptool: monmap file /tmp/ceph_monmap.14682
/home/ceph/build/bin/monmaptool: generated fsid 4f768ee5-6c79-461f-bcc7-03db65326676
epoch 0
fsid 4f768ee5-6c79-461f-bcc7-03db65326676
last_changed 2020-05-23T22:57:02.135190+0800
created 2020-05-23T22:57:02.135190+0800
min_mon_release 0 (unknown)
0: [v2:90.90.26.251:40412/0,v1:90.90.26.251:40413/0] mon.a
/home/ceph/build/bin/monmaptool: writing epoch 0 to /tmp/ceph_monmap.14682 (1 monitors)
rm -rf -
mkdir p /home/ceph/build/dev/mon.a /tmp/ceph_monmap.14682
/home/ceph/build/bin/ceph-mon --mkfs -c /home/ceph/build/ceph.conf -i a --monmap=/tmp/ceph_monmap.14682 --keyring=/home/ceph/build/keyring
rm -
Populating config ...
[mgr]
mgr/telemetry/enable = false
mgr/telemetry/nag = false
creating /home/ceph/build/dev/mgr.x/keyring
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring -i /home/ceph/build/dev/mgr.x/keyring auth add mgr.x mon 'allow profile mgr' mds 'allow ' osd 'allow *'
added key for mgr.x
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring config set mgr mgr/prometheus/x/server_port 9283 --force
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring config set mgr mgr/restful/x/server_port 42412 --force
Starting mgr.x
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring -h
waiting for mgr restful module to start
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring -h
waiting for mgr restful module to start
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring -h
waiting for mgr restful module to start
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring -h
waiting for mgr restful module to start
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring -h
waiting for mgr restful module to start
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring -h
waiting for mgr restful module to start
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring -h
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring restful create-self-signed-cert
Restarting RESTful API server...
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring restful create-key admin -o /tmp/tmp.GNEdvIUuex
add osd0 39049ddb-d022-49ce-96c0-28777d85009c
/home/ceph/build/bin/ceph -c /home/ceph/build/ceph.conf -k /home/ceph/build/keyring osd new 39049ddb-d022-49ce-96c0-28777d85009c -i /home/ceph/build/dev/osd0/new.json
0
INFO 2020-05-23 22:57:14,212 [shard 0] objclass - <cls> /home/ceph/src/cls/log/cls_log.cc:301: Loaded log class!
INFO 2020-05-23 22:57:14,213 [shard 0] objclass - <cls> /home/ceph/src/cls/cas/cls_cas.cc:237: Loaded cas class!
INFO 2020-05-23 22:57:14,213 [shard 0] objclass - <cls> /home/ceph/src/cls/version/cls_version.cc:214: Loaded version class!
INFO 2020-05-23 22:57:14,213 [shard 0] objclass - <cls> /home/ceph/src/cls/user/cls_user.cc:429: Loaded user class!
INFO 2020-05-23 22:57:14,214 [shard 0] objclass - <cls> /home/ceph/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class!
WARN 2020-05-23 22:57:14,214 [shard 0] objclass - <cls> /home/ceph/src/cls/sdk/cls_sdk.cc:120: loading cls_sdk
WARN 2020-05-23 22:57:14,215 [shard 0] objclass - <cls> /home/ceph/src/cls/hello/cls_hello.cc:312: loading cls_hello
WARN 2020-05-23 22:57:14,215 [shard 0] objclass - <cls> /home/ceph/src/cls/cephfs/cls_cephfs.cc:198: loading cephfs
WARN 2020-05-23 22:57:14,216 [shard 0] osd - _load_class could not open class /home/ceph/build/lib/libcls_queue.so (dlopen failed): /home/ceph/build/lib/libcls_queue.so: undefined symbol: _Z18cls_cxx_write_zeroPvii
WARN 2020-05-23 22:57:14,216 [shard 0] osd - OSD warning: got an error loading one or more classes: (5) Input/output error
INFO 2020-05-23 22:57:14,295 [shard 0] osd - operator() writing superblock cluster_fsid cff9620e-eada-44fc-9d7b-9297fc0812dd osd_fsid 39049ddb-d022-49ce-96c0-28777d85009c
DEBUG 2020-05-23 22:57:14,295 [shard 0] filestore - _write meta #-1:7b3f43c4:::osd_superblock:0# 0 ~ 523
created object store /home/ceph/build/dev/osd0 for osd.0 fsid cff9620e-eada-44fc-9d7b-9297fc0812dd
INFO 2020-05-23 22:57:14,329 [shard 0] osd - stop
start osd.0
OSDs started
vstart cluster complete. Use stop.sh to stop. See out/ (e.g. 'tail -f out/????') for debug output.
restful urls: https://90.90.26.251:42412
w/ user/pass: admin / 0c8e6eae-f879-4bb2-947b-6f5b33704c97
export PYTHONPATH=/home/ceph/src/pybind:/home/ceph/build/lib/cython_modules/lib.3:/home/ceph/src/python-common:$PYTHONPATH
export LD_LIBRARY_PATH=/home/ceph/build/lib:$LD_LIBRARY_PATH
alias cephfs-shell=/home/ceph/src/tools/cephfs/cephfs-shell
CEPH_DEV=1
Files
Updated by pengfei wang almost 4 years ago
- File osd.0.stdout osd.0.stdout added
out/osd.0.stdout
/home/ceph/src/ceph-run /home/ceph/build/bin/crimson-osd --smp 1 --cpuset 0 --memory 4G --debug i 0 -c /home/ceph/build/ceph.conf -f &>is_mode_secure()
INFO 2020-05-23 22:57:14,800 [shard 0] objclass - <cls> /home/ceph/src/cls/log/cls_log.cc:301: Loaded log class!
INFO 2020-05-23 22:57:14,801 [shard 0] objclass - <cls> /home/ceph/src/cls/cas/cls_cas.cc:237: Loaded cas class!
INFO 2020-05-23 22:57:14,801 [shard 0] objclass - <cls> /home/ceph/src/cls/version/cls_version.cc:214: Loaded version class!
INFO 2020-05-23 22:57:14,801 [shard 0] objclass - <cls> /home/ceph/src/cls/user/cls_user.cc:429: Loaded user class!
INFO 2020-05-23 22:57:14,802 [shard 0] objclass - <cls> /home/ceph/src/cls/timeindex/cls_timeindex.cc:242: Loaded timeindex class!
WARN 2020-05-23 22:57:14,802 [shard 0] objclass - <cls> /home/ceph/src/cls/sdk/cls_sdk.cc:120: loading cls_sdk
WARN 2020-05-23 22:57:14,802 [shard 0] objclass - <cls> /home/ceph/src/cls/hello/cls_hello.cc:312: loading cls_hello
WARN 2020-05-23 22:57:14,803 [shard 0] objclass - <cls> /home/ceph/src/cls/cephfs/cls_cephfs.cc:198: loading cephfs
WARN 2020-05-23 22:57:14,804 [shard 0] osd - _load_class could not open class /home/ceph/build/lib/libcls_queue.so (dlopen failed): /home/ceph/build/lib/libcls_queue.so: undefined symbol: _Z18cls_cxx_write_zeroPvii
WARN 2020-05-23 22:57:14,804 [shard 0] osd - OSD warning: got an error loading one or more classes: (5) Input/output error
INFO 2020-05-23 22:57:14,804 [shard 0] osd - start
DEBUG 2020-05-23 22:57:14,804 [shard 0] filestore - read meta #-1:7b3f43c4:::osd_superblock:0# 0~0
WARN 2020-05-23 22:57:14,804 [shard 0] osd - ignoring unrecognized collection: meta
ERROR 2020-05-23 22:57:14,804 [shard 0] none - Falling back to public interface
INFO 2020-05-23 22:57:14,804 [shard 0] osd - picked address v2:0.0.0.0:0/0
INFO 2020-05-23 22:57:14,804 [shard 0] osd - picked address v1:0.0.0.0:0/0
DEBUG 2020-05-23 22:57:14,804 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6800/15388] do_bind: try listen 0.0.0.0:6800...
INFO 2020-05-23 22:57:14,804 [shard 0] osd - picked address v2:0.0.0.0:0/0
INFO 2020-05-23 22:57:14,804 [shard 0] osd - picked address v1:0.0.0.0:0/0
DEBUG 2020-05-23 22:57:14,804 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6800/15388] do_bind: try listen 0.0.0.0:6800...
DEBUG 2020-05-23 22:57:14,804 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6801/15388] do_bind: try listen 0.0.0.0:6801...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6801/15388] do_bind: try listen 0.0.0.0:6801...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6802/15388] do_bind: try listen 0.0.0.0:6802...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6803/15388] do_bind: try listen 0.0.0.0:6803...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6804/15388] do_bind: try listen 0.0.0.0:6804...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6805/15388] do_bind: try listen 0.0.0.0:6805...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6802/15388] do_bind: try listen 0.0.0.0:6802...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6803/15388] do_bind: try listen 0.0.0.0:6803...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6804/15388] do_bind: try listen 0.0.0.0:6804...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6805/15388] do_bind: try listen 0.0.0.0:6805...
DEBUG 2020-05-23 22:57:14,805 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6806/15388] do_bind: try listen 0.0.0.0:6806...
DEBUG 2020-05-23 22:57:14,806 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6807/15388] do_bind: try listen 0.0.0.0:6807...
DEBUG 2020-05-23 22:57:14,806 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6808/15388] do_bind: try listen 0.0.0.0:6808...
INFO 2020-05-23 22:57:14,806 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6808/15388] try_bind: done
DEBUG 2020-05-23 22:57:14,806 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6806/15388] do_bind: try listen 0.0.0.0:6806...
DEBUG 2020-05-23 22:57:14,806 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6807/15388] do_bind: try listen 0.0.0.0:6807...
DEBUG 2020-05-23 22:57:14,806 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6808/15388] do_bind: try listen 0.0.0.0:6808...
DEBUG 2020-05-23 22:57:14,806 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388] do_bind: try listen 0.0.0.0:6809...
INFO 2020-05-23 22:57:14,806 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388] try_bind: done
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding auth protocol: cephx
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding auth protocol: cephx
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding auth protocol: cephx
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: secure
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: crc
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: secure
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: crc
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: secure
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: crc
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: crc
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: secure
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: crc
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: secure
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: crc
DEBUG 2020-05-23 22:57:14,806 [shard 0] auth - AuthRegistry(0xe000000dc408) adding con mode: secure
INFO 2020-05-23 22:57:14,806 [shard 0] auth - auth: KeyRing::load: loaded key file /home/ceph/build/dev/osd0/keyring
INFO 2020-05-23 22:57:14,807 [shard 0] monc - reopen_session to mon.-1
INFO 2020-05-23 22:57:14,807 [shard 0] monc - connecting to mon.0
INFO 2020-05-23 22:57:14,807 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] ProtocolV2::start_connect(): peer_addr=v2:90.90.26.251:40412/0, peer_type=mon, cc=4412757272530161755 policy(lossy=1, server=0, standby=0, resetcheck=0)
DEBUG 2020-05-23 22:57:14,807 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] TRIGGER CONNECTING, was NONE
DEBUG 2020-05-23 22:57:14,807 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] UPDATE: gs=1 for connect
INFO 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] write_event: delay ...
DEBUG 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] socket connected
DEBUG 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] SEND banner: len_payload=16, supported=0, required=0, banner="ceph v2
"
DEBUG 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] RECV banner: "ceph v2
"
DEBUG 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] GOT banner: payload_len=16
DEBUG 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] RECV banner features: supported=0 required=0
DEBUG 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] WRITE HelloFrame: my_type=osd, peer_addr=v2:90.90.26.251:40412/0
DEBUG 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6809/15388 >> mon.? v2:90.90.26.251:40412/0] GOT HelloFrame: my_type=mon peer_addr=v2:90.90.26.251:56837/0
INFO 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:90.90.26.251:6809/15388@56837 >> mon.? v2:90.90.26.251:40412/0] learned myaddr=v2:90.90.26.251:6809/15388 (blank IP) from v2:90.90.26.251:56837/0
INFO 2020-05-23 22:57:14,808 [shard 0] monc - get_auth_request(con=[osd.0(client) v2:90.90.26.251:6809/15388@56837 >> mon.? v2:90.90.26.251:40412/0], auth_method=0)
INFO 2020-05-23 22:57:14,808 [shard 0] monc - method 2 preferred_modes {2, 1}
DEBUG 2020-05-23 22:57:14,808 [shard 0] ms - [osd.0(client) v2:90.90.26.251:6809/15388@56837 >> mon.? v2:90.90.26.251:40412/0] WRITE AuthRequestFrame: method=2, preferred_modes={2, 1}, payload_len=18
DEBUG 2020-05-23 22:57:14,853 [shard 0] ms - [osd.0(client) v2:90.90.26.251:6809/15388@56837 >> mon.? v2:90.90.26.251:40412/0] GOT AuthReplyMoreFrame: payload_len=9
INFO 2020-05-23 22:57:14,853 [shard 0] monc - responding with 36 bytes
DEBUG 2020-05-23 22:57:14,853 [shard 0] ms - [osd.0(client) v2:90.90.26.251:6809/15388@56837 >> mon.? v2:90.90.26.251:40412/0] WRITE AuthRequestMoreFrame: payload_len=36
DEBUG 2020-05-23 22:57:14,853 [shard 0] ms - [osd.0(client) v2:90.90.26.251:6809/15388@56837 >> mon.? v2:90.90.26.251:40412/0] GOT AuthDoneFrame: gid=4134, con_mode=secure, payload_len=995
ERROR 2020-05-23 22:57:14,853 [shard 0] none - /home/ceph/src/crimson/net/ProtocolV2.cc:671 : In function 'crimson::net::ProtocolV2::handle_auth_reply()::<lambda(ceph::msgr::v2::Tag)>::<lambda()>', ceph_assert(%s)
!auth_meta
Aborting on shard 0.
Backtrace:
0x0000000000f48213
0x0000000000f0ecf3
0x0000000000f0edb7
0x0000000000f0eeff
0x000000000000066b
/lib64/libc.so.6+0x0000000000035237
/lib64/libc.so.6+0x00000000000368af
0x00000000007b9553
0x00000000007b956f
0x0000000000d1b85f
0x0000000000d2749b
0x0000000000d2822b
0x0000000000f0c677
0x0000000000f0c9e3
0x0000000000f325e7
0x0000000000ee470f
0x00000000007bc413
/lib64/libc.so.6+0x0000000000021723
0x00000000007ecfcb
/home/ceph/src/ceph-run: line 21: 15388 Aborted "$@"
ceph-run: /home/ceph/build/bin/crimson-osd dumped core, restarting in 5 seconds...
Updated by Deepika Upadhyay almost 4 years ago
- Related to Subtask #45535: crimson: crimson-osd failure in ceph-container added
Updated by Matan Breizman 3 months ago
- Status changed from New to Closed
Feel free to re-open if relevant