Bug #8207
"[ERR] 3.6 missing primary copy.." in upgrade:dumpling-x:stress-split-firefly---basic-plana
Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
This cold be a duplicate of #7976
Error in @ubuntu@plana68.front.sepia.ceph.com/log/ceph-osd.1.log.*
151754699-2014-04-24 07:48:53.683321 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] recover_object: 4b4dd5a6/plana7317904-366/head//3 151754988-2014-04-24 07:48:53.683340 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] prep_push_to_replica: 4b4dd5a6/plana7317904-366/head//3 v20'55 size 2414549 to osd.4 151755312-2014-04-24 07:48:53.683354 7f07eecc4700 15 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] push_to_replica snapset is 0=[]:[]+head 151755591-2014-04-24 07:48:53.683366 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] calc_head_subsets 4b4dd5a6/plana7317904-366/head//3 clone_overlap {} 151755899-2014-04-24 07:48:53.683380 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] calc_head_subsets 4b4dd5a6/plana7317904-366/head//3 data_subset [0~2414549] clone_subsets {} 151756233-2014-04-24 07:48:53.683408 7f07eecc4700 7 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] send_push_op 4b4dd5a6/plana7317904-366/head//3 v 20'55 size 2414549 recovery_info: ObjectRecoveryInfo(4b4dd5a6/plana7317904-366/head//3@20'55, copy_subset: [0~2414549], clone_subset: {}) 151756659-2014-04-24 07:48:53.683427 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) omap_get_header 3.6_head/4b4dd5a6/plana7317904-366/head//3 151756797-2014-04-24 07:48:53.683541 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattrs 3.6_head/4b4dd5a6/plana7317904-366/head//3 151756928-2014-04-24 07:48:53.683570 7f07eecc4700 20 filestore(/var/lib/ceph/osd/ceph-1) fgetattrs 81 getting '_' 151757032-2014-04-24 07:48:53.683581 7f07eecc4700 20 filestore(/var/lib/ceph/osd/ceph-1) fgetattrs 81 getting 'snapset' 151757142-2014-04-24 07:48:53.683678 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) getattrs 3.6_head/4b4dd5a6/plana7317904-366/head//3 = 0 151757277-2014-04-24 07:48:53.683695 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) get_omap_iterator 3.6_head/4b4dd5a6/plana7317904-366/head//3 151757417-2014-04-24 07:48:53.683782 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) read 3.6_head/4b4dd5a6/plana7317904-366/head//3 0~1048576 151757554-2014-04-24 07:48:53.684120 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) FileStore::read 3.6_head/4b4dd5a6/plana7317904-366/head//3 0~1048576/1048576 151757710-2014-04-24 07:48:53.684139 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] recover_replicas: recover_object_replicas(2a2af326/plana7317904-384/head//3) 151758026-2014-04-24 07:48:53.684158 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] prep_object_replica_pushes: on 2a2af326/plana7317904-384/head//3 151758330-2014-04-24 07:48:53.684172 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/2a2af326/plana7317904-384/head//3 '_' 151758464-2014-04-24 07:48:53.684203 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) error opening file /var/lib/ceph/osd/ceph-1/current/3.6_head/plana7317904-384__head_2A2AF326__3 with flags=2: (2) No such file or directory 151758683-2014-04-24 07:48:53.684216 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/2a2af326/plana7317904-384/head//3 '_' = -2 151758822-2014-04-24 07:48:53.684220 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] get_object_context: no obc for soid 2a2af326/plana7317904-384/head//3 and !can_create 151759147:2014-04-24 07:48:53.684245 7f07eecc4700 0 log [ERR] : 3.6 missing primary copy of 2a2af326/plana7317904-384/head//3, unfound 151759273-2014-04-24 07:48:53.684251 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] recover_replicas: recover_object_replicas(4bba5ea6/plana7317904-426/head//3) 151759593-2014-04-24 07:48:53.684281 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] prep_object_replica_pushes: on 4bba5ea6/plana7317904-426/head//3 151759901-2014-04-24 07:48:53.684308 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/4bba5ea6/plana7317904-426/head//3 '_' 151760035-2014-04-24 07:48:53.684352 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/4bba5ea6/plana7317904-426/head//3 '_' = 235 151760175-2014-04-24 07:48:53.684370 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/4bba5ea6/plana7317904-426/head//3 'snapset' 151760315-2014-04-24 07:48:53.684384 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/4bba5ea6/plana7317904-426/head//3 'snapset' = 31 151760460-2014-04-24 07:48:53.684390 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] populate_obc_watchers 4bba5ea6/plana7317904-426/head//3 151760759-2014-04-24 07:48:53.684419 7f07eecc4700 20 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] ReplicatedPG::check_blacklisted_obc_watchers for obc 4bba5ea6/plana7317904-426/head//3 151761089-2014-04-24 07:48:53.684447 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] get_object_context: creating obc from disk: 0x2f07340 151761386-2014-04-24 07:48:53.684473 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] get_object_context: 0x2f07340 4bba5ea6/plana7317904-426/head//3 rwstate(none n=0 w=0) oi: 4bba5ea6/plana7317904-426/head//3(21'64 client.4202.0:1118 wrlock_by=unknown.0.0:0 dirty s 57 uv64) ssc: 0x33188c0 snapset: 0=[]:[]+head 151761856-2014-04-24 07:48:53.684504 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] start_recovery_op 4bba5ea6/plana7317904-426/head//3 151762151-2014-04-24 07:48:53.684533 7f07eecc4700 10 osd.1 231 start_recovery_op pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] 4bba5ea6/plana7317904-426/head//3 (6/15 rops) 151762448-2014-04-24 07:48:53.684560 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] recover_object: 4bba5ea6/plana7317904-426/head//3 151762741-2014-04-24 07:48:53.684592 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] prep_push_to_replica: 4bba5ea6/plana7317904-426/head//3 v21'64 size 57 to osd.4 151763064-2014-04-24 07:48:53.684622 7f07eecc4700 15 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] push_to_replica snapset is 0=[]:[]+head 151763347-2014-04-24 07:48:53.684648 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] calc_head_subsets 4bba5ea6/plana7317904-426/head//3 clone_overlap {} 151763659-2014-04-24 07:48:53.684674 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] calc_head_subsets 4bba5ea6/plana7317904-426/head//3 data_subset [0~57] clone_subsets {} 151763992-2014-04-24 07:48:53.684708 7f07eecc4700 7 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] send_push_op 4bba5ea6/plana7317904-426/head//3 v 21'64 size 57 recovery_info: ObjectRecoveryInfo(4bba5ea6/plana7317904-426/head//3@21'64, copy_subset: [0~57], clone_subset: {}) 151764412-2014-04-24 07:48:53.684740 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) omap_get_header 3.6_head/4bba5ea6/plana7317904-426/head//3 151764550-2014-04-24 07:48:53.684829 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattrs 3.6_head/4bba5ea6/plana7317904-426/head//3
archive_path: /var/lib/teuthworker/archive/teuthology-2014-04-23_19:55:03-upgrade:dumpling-x:stress-split-firefly---basic-plana/212166 description: upgrade/dumpling-x/stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/readwrite.yaml 6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rados_api_tests.yaml rbd-python.yaml rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml} email: null job_id: '212166' last_in_suite: false machine_type: plana name: teuthology-2014-04-23_19:55:03-upgrade:dumpling-x:stress-split-firefly---basic-plana nuke-on-error: true os_type: ubuntu os_version: '12.04' overrides: admin_socket: branch: firefly ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on legacy crush tunables: false osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 20 log-whitelist: - slow request - wrongly marked me down - objects unfound and apparently lost - log bound mismatch sha1: 2708c3c559d99e6f3b557ee1d223efa3745f655c ceph-deploy: branch: dev: firefly conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 osd default pool size: 2 install: ceph: sha1: 2708c3c559d99e6f3b557ee1d223efa3745f655c s3tests: branch: master workunit: sha1: 2708c3c559d99e6f3b557ee1d223efa3745f655c owner: scheduled_teuthology@teuthology roles: - - mon.a - mon.b - mds.a - osd.0 - osd.1 - osd.2 - - osd.3 - osd.4 - osd.5 - mon.c - - client.0 targets: ubuntu@plana68.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCk8EYGKdzsq6sKetc5IXfxLMcta+UDLEDk+a9OAih+iRON3upQ6JJqu2WHq4wNk7W+YjXyAN9u8FkJ/WzDX4tmhT8MeK//Ejy+A5tYIxePYrcJ7ujVZHj3wusN+DwpyfXD6UWc2jT0f8ejJ+Cb7bR1or+/TpxT7YmLmd6vQKZUJzHpwsM2gyB3+63dRA7D6lyiqB3OBYfzOdVf4kpKRepLCIk2r3Ai+JpxnXV+LSbWoqbzfRBBmQlI3YbtOd/sEAbgf8iwgG2eiK1PN45yTSxcZAJ9cNUzJW81JVw02cQmtWTt9kKRTnvyfcriB8RrhsXKOQsJLgmeeWJJ0oYLidDx ubuntu@plana73.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMMFyMv1oFsyfKlALIqBCgjBnJ+kN4Hjg995rfea6OT9lATRiWTTH4dPiAdDrK0xaA5sHcxIeIBAuFQ0TEeeLOyMtCwjNFSc5w5IekUPh+JKGuYxfOZMAbU1lZoz3w5ofV8U/u5JhX2i5CPXMH18KstLFhr6XhXRcFuPrS/MxXavjfTOTk6FME2hcSaGQiyWn+6RmJglrX+dxbn662UCwcsqMRBUZPZYCSGG2viF2ONnqDsxgpG6KN5TzLboms70ZHhBxah87FkOZ24mRQxAWJrAYcSKU/mJmtsK1i6htmtdyXRycuaeL4sLOcH0G1x9yf7CVrJs71Pg+AgmFHRZ/j ubuntu@plana74.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3xL+6EJiFQ0H0hqw1CS8NXMgy5Rojc65kcaHG7MsK6rrNFfipBooIhfCcUVJfsjQtJqhPJr4XQ69DX/ph1V5h/kvQBbOEurYXR5vZtcJpTIDCD83g8SeN0oZo/oZefBScnV9I0saqcvrL5hbn34RwuiARmQZLQckMXNcbcAjD+ev9QncitthMI6l4tEchCOBlPLPiHQyaJCCxJCoE+8GLmKGuNGBdU81QG9D/jot9KHZ2eSvD1phxaZslARpr+aaii832hBtVE9Kf/eeD465ECCKswzNsaFPnhNY2fGdCDOotUu8g6CZetMaQQKvGMvU8JCM8OD5BKiukhz/Fp5V tasks: - internal.lock_machines: - 3 - plana - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - internal.check_ceph_data: null - internal.vm_setup: null - internal.base: null - internal.archive: null - internal.coredump: null - internal.sudo: null - internal.syslog: null - internal.timer: null - chef: null - clock.check: null - install: branch: dumpling - ceph: fs: xfs - install.upgrade: osd.0: null - ceph.restart: daemons: - osd.0 - osd.1 - osd.2 - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 thrash_primary_affinity: false timeout: 1200 - ceph.restart: daemons: - mon.a wait-for-healthy: false wait-for-osds-up: true - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 - ceph.restart: daemons: - mon.b wait-for-healthy: false wait-for-osds-up: true - workunit: branch: dumpling clients: client.0: - rbd/test_librbd.sh - install.upgrade: mon.c: null - ceph.restart: daemons: - mon.c wait-for-healthy: false wait-for-osds-up: true - ceph.wait_for_mon_quorum: - a - b - c - workunit: branch: dumpling clients: client.0: - rados/test-upgrade-firefly.sh - workunit: branch: dumpling clients: client.0: - rbd/test_librbd_python.sh - rgw: client.0: idle_timeout: 300 - swift: client.0: rgw_server: client.0 - rados: clients: - client.0 objects: 500 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: firefly verbose: true worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.6494
description: upgrade/dumpling-x/stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/readwrite.yaml 6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rados_api_tests.yaml rbd-python.yaml rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml} duration: 6701.551152944565 failure_reason: '"2014-04-24 07:48:53.684247 osd.1 10.214.132.10:6813/27655 3 : [ERR] 3.6 missing primary copy of 2a2af326/plana7317904-384/head//3, unfound" in cluster log' flavor: basic owner: scheduled_teuthology@teuthology success: false
Related issues
History
#1 Updated by Samuel Just almost 10 years ago
- Status changed from New to Duplicate