Project

General

Profile

Bug #8207

"[ERR] 3.6 missing primary copy.." in upgrade:dumpling-x:stress-split-firefly---basic-plana

Added by Yuri Weinstein almost 10 years ago. Updated almost 10 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This cold be a duplicate of #7976

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-04-23_19:55:03-upgrade:dumpling-x:stress-split-firefly---basic-plana/212166/

Error in @/log/ceph-osd.1.log.*

151754699-2014-04-24 07:48:53.683321 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] recover_object: 4b4dd5a6/plana7317904-366/head//3
151754988-2014-04-24 07:48:53.683340 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] prep_push_to_replica: 4b4dd5a6/plana7317904-366/head//3 v20'55 size 2414549 to osd.4
151755312-2014-04-24 07:48:53.683354 7f07eecc4700 15 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] push_to_replica snapset is 0=[]:[]+head
151755591-2014-04-24 07:48:53.683366 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] calc_head_subsets 4b4dd5a6/plana7317904-366/head//3 clone_overlap {}
151755899-2014-04-24 07:48:53.683380 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] calc_head_subsets 4b4dd5a6/plana7317904-366/head//3  data_subset [0~2414549]  clone_subsets {}
151756233-2014-04-24 07:48:53.683408 7f07eecc4700  7 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] send_push_op 4b4dd5a6/plana7317904-366/head//3 v 20'55 size 2414549 recovery_info: ObjectRecoveryInfo(4b4dd5a6/plana7317904-366/head//3@20'55, copy_subset: [0~2414549], clone_subset: {})
151756659-2014-04-24 07:48:53.683427 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) omap_get_header 3.6_head/4b4dd5a6/plana7317904-366/head//3
151756797-2014-04-24 07:48:53.683541 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattrs 3.6_head/4b4dd5a6/plana7317904-366/head//3
151756928-2014-04-24 07:48:53.683570 7f07eecc4700 20 filestore(/var/lib/ceph/osd/ceph-1) fgetattrs 81 getting '_'
151757032-2014-04-24 07:48:53.683581 7f07eecc4700 20 filestore(/var/lib/ceph/osd/ceph-1) fgetattrs 81 getting 'snapset'
151757142-2014-04-24 07:48:53.683678 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) getattrs 3.6_head/4b4dd5a6/plana7317904-366/head//3 = 0
151757277-2014-04-24 07:48:53.683695 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) get_omap_iterator 3.6_head/4b4dd5a6/plana7317904-366/head//3
151757417-2014-04-24 07:48:53.683782 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) read 3.6_head/4b4dd5a6/plana7317904-366/head//3 0~1048576
151757554-2014-04-24 07:48:53.684120 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) FileStore::read 3.6_head/4b4dd5a6/plana7317904-366/head//3 0~1048576/1048576
151757710-2014-04-24 07:48:53.684139 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] recover_replicas: recover_object_replicas(2a2af326/plana7317904-384/head//3)
151758026-2014-04-24 07:48:53.684158 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] prep_object_replica_pushes: on 2a2af326/plana7317904-384/head//3
151758330-2014-04-24 07:48:53.684172 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/2a2af326/plana7317904-384/head//3 '_'
151758464-2014-04-24 07:48:53.684203 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) error opening file /var/lib/ceph/osd/ceph-1/current/3.6_head/plana7317904-384__head_2A2AF326__3 with flags=2: (2) No such file or directory
151758683-2014-04-24 07:48:53.684216 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/2a2af326/plana7317904-384/head//3 '_' = -2
151758822-2014-04-24 07:48:53.684220 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering] get_object_context: no obc for soid 2a2af326/plana7317904-384/head//3 and !can_create
151759147:2014-04-24 07:48:53.684245 7f07eecc4700  0 log [ERR] : 3.6 missing primary copy of 2a2af326/plana7317904-384/head//3, unfound
151759273-2014-04-24 07:48:53.684251 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] recover_replicas: recover_object_replicas(4bba5ea6/plana7317904-426/head//3)
151759593-2014-04-24 07:48:53.684281 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] prep_object_replica_pushes: on 4bba5ea6/plana7317904-426/head//3
151759901-2014-04-24 07:48:53.684308 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/4bba5ea6/plana7317904-426/head//3 '_'
151760035-2014-04-24 07:48:53.684352 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/4bba5ea6/plana7317904-426/head//3 '_' = 235
151760175-2014-04-24 07:48:53.684370 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/4bba5ea6/plana7317904-426/head//3 'snapset'
151760315-2014-04-24 07:48:53.684384 7f07eecc4700 10 filestore(/var/lib/ceph/osd/ceph-1) getattr 3.6_head/4bba5ea6/plana7317904-426/head//3 'snapset' = 31
151760460-2014-04-24 07:48:53.684390 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] populate_obc_watchers 4bba5ea6/plana7317904-426/head//3
151760759-2014-04-24 07:48:53.684419 7f07eecc4700 20 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] ReplicatedPG::check_blacklisted_obc_watchers for obc 4bba5ea6/plana7317904-426/head//3
151761089-2014-04-24 07:48:53.684447 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] get_object_context: creating obc from disk: 0x2f07340
151761386-2014-04-24 07:48:53.684473 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] get_object_context: 0x2f07340 4bba5ea6/plana7317904-426/head//3 rwstate(none n=0 w=0) oi: 4bba5ea6/plana7317904-426/head//3(21'64 client.4202.0:1118 wrlock_by=unknown.0.0:0 dirty s 57 uv64) ssc: 0x33188c0 snapset: 0=[]:[]+head
151761856-2014-04-24 07:48:53.684504 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=1 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] start_recovery_op 4bba5ea6/plana7317904-426/head//3
151762151-2014-04-24 07:48:53.684533 7f07eecc4700 10 osd.1 231 start_recovery_op pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] 4bba5ea6/plana7317904-426/head//3 (6/15 rops)
151762448-2014-04-24 07:48:53.684560 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] recover_object: 4bba5ea6/plana7317904-426/head//3
151762741-2014-04-24 07:48:53.684592 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] prep_push_to_replica: 4bba5ea6/plana7317904-426/head//3 v21'64 size 57 to osd.4
151763064-2014-04-24 07:48:53.684622 7f07eecc4700 15 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] push_to_replica snapset is 0=[]:[]+head
151763347-2014-04-24 07:48:53.684648 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] calc_head_subsets 4bba5ea6/plana7317904-426/head//3 clone_overlap {}
151763659-2014-04-24 07:48:53.684674 7f07eecc4700 10 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] calc_head_subsets 4bba5ea6/plana7317904-426/head//3  data_subset [0~57]  clone_subsets {}
151763992-2014-04-24 07:48:53.684708 7f07eecc4700  7 osd.1 pg_epoch: 231 pg[3.6( v 229'158 (0'0,229'158] local-les=231 n=15 ec=11 les/c 231/224 230/230/230) [1,4] r=0 lpr=230 pi=41-229/19 rops=2 crt=128'150 lcod 229'157 mlcod 0'0 active+recovering m=1] send_push_op 4bba5ea6/plana7317904-426/head//3 v 21'64 size 57 recovery_info: ObjectRecoveryInfo(4bba5ea6/plana7317904-426/head//3@21'64, copy_subset: [0~57], clone_subset: {})
151764412-2014-04-24 07:48:53.684740 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) omap_get_header 3.6_head/4bba5ea6/plana7317904-426/head//3
151764550-2014-04-24 07:48:53.684829 7f07eecc4700 15 filestore(/var/lib/ceph/osd/ceph-1) getattrs 3.6_head/4bba5ea6/plana7317904-426/head//3
archive_path: /var/lib/teuthworker/archive/teuthology-2014-04-23_19:55:03-upgrade:dumpling-x:stress-split-firefly---basic-plana/212166
description: upgrade/dumpling-x/stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/readwrite.yaml
  6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rados_api_tests.yaml
  rbd-python.yaml rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml}
email: null
job_id: '212166'
last_in_suite: false
machine_type: plana
name: teuthology-2014-04-23_19:55:03-upgrade:dumpling-x:stress-split-firefly---basic-plana
nuke-on-error: true
os_type: ubuntu
os_version: '12.04'
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - wrongly marked me down
    - objects unfound and apparently lost
    - log bound mismatch
    sha1: 2708c3c559d99e6f3b557ee1d223efa3745f655c
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 2708c3c559d99e6f3b557ee1d223efa3745f655c
  s3tests:
    branch: master
  workunit:
    sha1: 2708c3c559d99e6f3b557ee1d223efa3745f655c
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mon.b
  - mds.a
  - osd.0
  - osd.1
  - osd.2
- - osd.3
  - osd.4
  - osd.5
  - mon.c
- - client.0
targets:
  ubuntu@plana68.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCk8EYGKdzsq6sKetc5IXfxLMcta+UDLEDk+a9OAih+iRON3upQ6JJqu2WHq4wNk7W+YjXyAN9u8FkJ/WzDX4tmhT8MeK//Ejy+A5tYIxePYrcJ7ujVZHj3wusN+DwpyfXD6UWc2jT0f8ejJ+Cb7bR1or+/TpxT7YmLmd6vQKZUJzHpwsM2gyB3+63dRA7D6lyiqB3OBYfzOdVf4kpKRepLCIk2r3Ai+JpxnXV+LSbWoqbzfRBBmQlI3YbtOd/sEAbgf8iwgG2eiK1PN45yTSxcZAJ9cNUzJW81JVw02cQmtWTt9kKRTnvyfcriB8RrhsXKOQsJLgmeeWJJ0oYLidDx
  ubuntu@plana73.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMMFyMv1oFsyfKlALIqBCgjBnJ+kN4Hjg995rfea6OT9lATRiWTTH4dPiAdDrK0xaA5sHcxIeIBAuFQ0TEeeLOyMtCwjNFSc5w5IekUPh+JKGuYxfOZMAbU1lZoz3w5ofV8U/u5JhX2i5CPXMH18KstLFhr6XhXRcFuPrS/MxXavjfTOTk6FME2hcSaGQiyWn+6RmJglrX+dxbn662UCwcsqMRBUZPZYCSGG2viF2ONnqDsxgpG6KN5TzLboms70ZHhBxah87FkOZ24mRQxAWJrAYcSKU/mJmtsK1i6htmtdyXRycuaeL4sLOcH0G1x9yf7CVrJs71Pg+AgmFHRZ/j
  ubuntu@plana74.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3xL+6EJiFQ0H0hqw1CS8NXMgy5Rojc65kcaHG7MsK6rrNFfipBooIhfCcUVJfsjQtJqhPJr4XQ69DX/ph1V5h/kvQBbOEurYXR5vZtcJpTIDCD83g8SeN0oZo/oZefBScnV9I0saqcvrL5hbn34RwuiARmQZLQckMXNcbcAjD+ev9QncitthMI6l4tEchCOBlPLPiHQyaJCCxJCoE+8GLmKGuNGBdU81QG9D/jot9KHZ2eSvD1phxaZslARpr+aaii832hBtVE9Kf/eeD465ECCKswzNsaFPnhNY2fGdCDOotUu8g6CZetMaQQKvGMvU8JCM8OD5BKiukhz/Fp5V
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: dumpling
- ceph:
    fs: xfs
- install.upgrade:
    osd.0: null
- ceph.restart:
    daemons:
    - osd.0
    - osd.1
    - osd.2
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    thrash_primary_affinity: false
    timeout: 1200
- ceph.restart:
    daemons:
    - mon.a
    wait-for-healthy: false
    wait-for-osds-up: true
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 10
      read: 45
      write: 45
    ops: 4000
- ceph.restart:
    daemons:
    - mon.b
    wait-for-healthy: false
    wait-for-osds-up: true
- workunit:
    branch: dumpling
    clients:
      client.0:
      - rbd/test_librbd.sh
- install.upgrade:
    mon.c: null
- ceph.restart:
    daemons:
    - mon.c
    wait-for-healthy: false
    wait-for-osds-up: true
- ceph.wait_for_mon_quorum:
  - a
  - b
  - c
- workunit:
    branch: dumpling
    clients:
      client.0:
      - rados/test-upgrade-firefly.sh
- workunit:
    branch: dumpling
    clients:
      client.0:
      - rbd/test_librbd_python.sh
- rgw:
    client.0:
      idle_timeout: 300
- swift:
    client.0:
      rgw_server: client.0
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 100
    ops: 4000
teuthology_branch: firefly
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.6494
description: upgrade/dumpling-x/stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/readwrite.yaml
  6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rados_api_tests.yaml
  rbd-python.yaml rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml}
duration: 6701.551152944565
failure_reason: '"2014-04-24 07:48:53.684247 osd.1 10.214.132.10:6813/27655 3 : [ERR]
  3.6 missing primary copy of 2a2af326/plana7317904-384/head//3, unfound" in cluster
  log'
flavor: basic
owner: scheduled_teuthology@teuthology
success: false

Related issues

Duplicates Ceph - Bug #8162: osd: dumpling advances last_backfill prematurely Resolved 04/19/2014

History

#1 Updated by Samuel Just almost 10 years ago

  • Status changed from New to Duplicate

Also available in: Atom PDF