Project

General

Profile

Actions
Copy link

Bug #6627

closed

segfault at PGMap::dirty_all during upgrade mon

Added by Tamilarasi muthamizhan over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Joao Eduardo Luis
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

logs: ubuntu@teuthology:/a/teuthology-2013-10-22_01:30:02-upgrade-next-testing-basic-plana/64049

2013-10-22T08:58:24.590 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]: mon/PGMap.cc: In function 'void PGMap::dirty_all(PGMap::Incremental&)' thread 7fc97e94f700 time 2013-10-22 08:58:24.588009
2013-10-22T08:58:24.590 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]: mon/PGMap.cc: 535: FAILED assert(inc.get_osd_epochs().count(p->first))
2013-10-22T08:58:24.608 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  ceph version 0.71-224-gad4553a (ad4553a4fddfa01129d84da1f9e166a93260bd3e)
2013-10-22T08:58:24.608 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  1: (PGMap::dirty_all(PGMap::Incremental&)+0x36b) [0x61944b]
2013-10-22T08:58:24.608 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  2: (PGMonitor::upgrade_format()+0x110) [0x5f0c00]
2013-10-22T08:58:24.608 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  3: (PaxosService::_active()+0x5d9) [0x59dba9]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  4: (Context::complete(int)+0x9) [0x56fb79]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  5: (finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x95) [0x572465]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  6: (Paxos::handle_last(MMonPaxos*)+0xd68) [0x596048]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  7: (Paxos::dispatch(PaxosServiceMessage*)+0x29b) [0x59668b]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  8: (Monitor::dispatch(MonSession*, Message*, bool)+0x51d) [0x56f92d]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  9: (Monitor::_ms_dispatch(Message*)+0x204) [0x56db54]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  10: (Monitor::ms_dispatch(Message*)+0x32) [0x5882c2]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  11: (DispatchQueue::entry()+0x549) [0x7ef259]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  12: (DispatchQueue::DispatchThread::entry()+0xd) [0x71ccfd]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  13: (()+0x7e9a) [0x7fc983470e9a]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  14: (clone()+0x6d) [0x7fc981a2dccd]
2013-10-22T08:58:24.609 INFO:teuthology.task.ceph.mon.b.err:[10.214.133.39]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/teuthology-2013-10-22_01:30:02-upgrade-next-testing-basic-plana/64049$ cat config.yaml 
archive_path: /var/lib/teuthworker/archive/teuthology-2013-10-22_01:30:02-upgrade-next-testing-basic-plana/64049
description: upgrade/mixed-mons/{0-cluster/start.yaml 1-cuttlefish-install/cuttlefish.yaml
  2-cuttlefish-workload/cephtool.yaml 3-partial-mon-upgrade/dumpling.yaml 4-mon-restart/restart.yaml
  5-mixed-workload/cephtool.yaml 6-rest/rest.yaml}
email: null
job_id: '64049'
kernel: &id001
  kdb: true
  sha1: b3efdaef9e1ac82e95ade43861f625b732dab06d
last_in_suite: false
machine_type: plana
name: teuthology-2013-10-22_01:30:02-upgrade-next-testing-basic-plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug ms: 1
        debug osd: 5
    log-whitelist:
    - slow request
    - wrongly marked me down
    - had wrong client addr
    - had wrong cluster addr
    sha1: ad4553a4fddfa01129d84da1f9e166a93260bd3e
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: ad4553a4fddfa01129d84da1f9e166a93260bd3e
  s3tests:
    branch: next
  workunit:
    sha1: ad4553a4fddfa01129d84da1f9e166a93260bd3e
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
targets:
  ubuntu@plana75.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCt7qJ60OT5FpVznjR8YnQxQK6yWWDQ/i0Jeo4WJXuGA+GwSy92BcDS/jj13oLuoS3awmOktg8bPicMwZA/HBeNi1TwIAZTNKh7C2dObC7yk/1AfIvn/QtgMFFR9o3mkAueRokCo1HwI5FU6TYMWc1RRUVZ+jff/33rWRqGY6gRIgnApirHiO0ImaqLPnX6LdWN37y2VgZRerOKhf0xVmamW37WlcC39C13+kt0ooJCHvLRgng14SmYEQjCPbWxywJ2A9M1jhOMV4xgB/JxOGMZ/goRKB38+vrDuU4cSLMlxX26mF83xRrX4LLU2bQxdv/e3s8GGRhGmAKcQdq6IvCZ
  ubuntu@plana76.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBLCcylPkP+7OnxDTMxZcJVG81J6QvbrqqGsZiBhFLf2axZSoJusO+YW6I+0W3FIZeH1C5Lq5g0+Us8iAG4XNkIxkBph60wsyv8yZw5VSLFXV8uL0bXcN7Fu3DLN9RdfqgdQ/RIXq0DMz5fDLxb65k28spbb3iMOsX8y1QAz6WBpyxyB7Rd40/MLLNQVF799Odhi7rfBfFkcuimf4StDt8yFUduvUIjPnePvz6JtnTY7cAT8V7pih0mj1HByxPB7D2twvfy3AXLfGbbhNygFM7wrPXoZ83FO6Hf6IL2igdaWLZVt6LzuCuMY8CJ6/rzXHSI1y9k9JXqk4WPiioRRuj
  ubuntu@plana77.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkn2oRTDwq/cR7G1MUH0AdFa2GTDbxo2BHHiVBZ1d25KqcemcLzUTQoZbY63xvGN4OhHl9cXbyyiIBcKEdBGJF7buCX0OmlrPq+tB5SJxZOjuyTJZa50Oans+DAJzHsRBS3d4xjf3pDn/FoVg6zbPU+mnppIvS5GYk8vtyFhkBIIBtu8lcatPs4oWDk7RTypPkZbaQ0FeYnhI5lh0vTq4wFU6J2sSLJTeBtIu8QU5O0hfAjiiG4ri0NghRb1waFJL69cOJfw6PlblVSrRr0p6x/cn1sbCyQ/cu0dkmFjUY1ae8H0HrEOeA3YPlJt1p52Lf+FvAPibeZo3P70cluhX/
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: cuttlefish
- ceph: null
- workunit:
    branch: cuttlefish
    clients:
      all:
      - cephtool/test.sh
      - mon/pool_ops.sh
- install.upgrade:
    mon.a:
      branch: dumpling
- ceph.restart:
    daemons:
    - mon.a
    wait-for-healthy: false
    wait-for-osds-up: true
- workunit:
    branch: cuttlefish
    clients:
      all:
      - cephtool/test.sh
      - mon/pool_ops.sh
- install.upgrade:
    client.0:
      branch: dumpling
    mon.b:
      branch: dumpling
- ceph.restart:
    daemons:
    - mon.b
    - mon.c
    - osd.0
    - osd.1
    - osd.2
    - osd.3
- workunit:
    branch: dumpling
    clients:
      all:
      - cephtool/test.sh
      - mon/pool_ops.sh
Actions
Copy link
 #1

Updated by Tamilarasi muthamizhan over 10 years ago

hi Joao, reproduced this issue when upgrading monitors one at a time from cuttlefish to next with debugs on - logs are available in ubuntu@mira056:~/bug_6627

This doesn't happen with upgrading from cuttlefish to dumpling branch though [logs: :~/up_cuttlefish_dumpling]

here is the yaml file i used to reproduce this.

tamil@ubuntu:~/tam_final/teuthology/up_bug_next$ cat orig.config.yaml 
overrides:
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug ms: 1
        debug osd: 5
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
targets:
  ubuntu@mira056.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGA68+hmpO5e6pXfZyxksE/JowkX45ddPm4qYZc4anqwGkbOeTcBZACbMJYTyJQWccouVrHZe9Y7eqFU6iRF4ja/zp1IdvoHfJnBwcMVvJK7HQuGtZcMgzQ0sLvfWbYU3iWNsh9bWFa39hNFd/xD/TK7aOkvvxjyDUqa07O8G9npV8+ukI+NMBf2lsGm3Qy1HnRGRhw1fhd6H7bbpe8oSGKZyfY5spEKK4yhebviZWKJhBWDHkp8ePQomMuv4fSZ+YE7Thnj+1EKPH/zXHwTFhpVucXE+ctioX6VqUdY9cngaymLHzlgYHMeemCYaabQkrQKl4ehQWey8dN7SaT9ed
  ubuntu@mira074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuzTGrQ9CaYud6lfhAXosbRh8/P1xCeTQfxuj5QYWYJf079r2b4IPlhW+rOc2ZfK5HkOatZH0+eV6eMZREYMLZNn8n+S3jQclWpyyoI6U0B0TP65ByYRtI2f+wvab5TGBWXHasGLNQh7zzxadhLWMVQ9AT/7c5oJTEHe1+BRIfvR0dBpK/cCrOlVjwcGUYkZn6s/My216zbVVuENHXa62NJBAlmNEWJsJHRh9IEDB+Cl+PmD+qD5zAWgJr2e2OtOWh+9v8v6YWOyO3KEhg/BKKxmBevkdcKZTcybjARDjU2IMu9nyeOhH1F+8xQQJ7dDRQ5TA7DYH6lKO9iEHD8YFr
  ubuntu@mira080.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGnNwnG4C8IU3gTmWaNB9YU0gbNeoBAcyD18JRuLJlLMKaZgvD2qvDgjis/4n1Fn0w7yY9YILNAI+fRlifaZRjg0nNyjIB3MpYbK/7oB12sO3R/fpNhA8FU6bt0V/9XQWrdWLe2s1PlTVgucMOVEJnp+3eFSR+3thfR9XXHqjdpOJ7Q1Ra/dLzjk/SP94i0EshvBlPl4kClyEWqKLlkMvZNGzKeJj+J9g8jagvsSJ65fdyi9qcaLVvicuOeL6T4ZGRypPaYfUNLRsPGRjTlqE2IZxC8RBtuyL4sVdFl2hBGT3HhOF4IQuFYTWgWzh9fUDJdwCVI7FH8O3id0QLirE7
tasks:
- chef: null
- install:
    branch: cuttlefish
- ceph: null
- workunit:
    branch: cuttlefish
    clients:
      all:
      - cephtool/test.sh
      - mon/pool_ops.sh
- install.upgrade:
    mon.a:
      branch: next
- ceph.restart:
    daemons:
    - mon.a
    wait-for-healthy: false
    wait-for-osds-up: true
- workunit:
    branch: cuttlefish
    clients:
      all:
      - cephtool/test.sh
      - mon/pool_ops.sh
- install.upgrade:
    client.0:
      branch: next
    mon.b:
      branch: next
- ceph.restart:
    daemons:
    - mon.b
    - mon.c
    - osd.0
    - osd.1
    - osd.2
    - osd.3
- workunit:
    branch: next
    clients:
      all:
      - cephtool/test.sh
      - mon/pool_ops.sh
Actions
Copy link
 #2

Updated by Samuel Just over 10 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF