Ceph

when upgrading from bobtail to next branch, seeing repeated wrong node messages in the osd logs.

2013-06-04 12:21:01.505350 7f351729a700  1 mds.0.2  waiting for osdmap 7 (which blacklists prior instance)
2013-06-04 12:21:01.522313 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53185 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.32:6803/21481 not 10.214.1
31.32:6803/20680 - wrong node!
2013-06-04 12:21:01.522388 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53185 s=1 pgs=0 cs=0 l=1).fault
2013-06-04 12:21:01.522680 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :45979 s=1 pgs=0 cs=0 l=1).connect claims to be 0.0.0.0:6803/26492 not 10.214.131.31:
6803/25734 - wrong node!
2013-06-04 12:21:01.522742 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :45979 s=1 pgs=0 cs=0 l=1).fault
2013-06-04 12:21:01.522823 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53188 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.32:6803/21481 not 10.214.131.32:6803/20680 - wrong node!
2013-06-04 12:21:01.522876 7f3514792700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6800/25733 pipe(0x16a6780 sd=19 :44142 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6800/26492 not 10.214.131.31:6800/25733 - wrong node!
2013-06-04 12:21:01.522951 7f3514792700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6800/25733 pipe(0x16a6780 sd=19 :44142 s=1 pgs=0 cs=0 l=1).fault
2013-06-04 12:21:01.523499 7f3514792700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6800/25733 pipe(0x16a6780 sd=19 :44145 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6800/26492 not 10.214.131.31:6800/25733 - wrong node!
2013-06-04 12:21:01.523541 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :45982 s=1 pgs=0 cs=0 l=1).connect claims to be 0.0.0.0:6803/26492 not 10.214.131.31:6803/25734 - wrong node!
2013-06-04 12:21:01.723426 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53191 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.32:6803/21481 not 10.214.131.32:6803/20680 - wrong node!
2013-06-04 12:21:01.724294 7f3514792700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6800/25733 pipe(0x16a6780 sd=19 :44147 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6800/26492 not 10.214.131.31:6800/25733 - wrong node!
2013-06-04 12:21:01.724380 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :45986 s=1 pgs=0 cs=0 l=1).connect claims to be 0.0.0.0:6803/26492 not 10.214.131.31:6803/25734 - wrong node!
2013-06-04 12:21:02.124057 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53198 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.32:6803/21481 not 10.214.131.32:6803/20680 - wrong node!
2013-06-04 12:21:02.125059 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :45993 s=1 pgs=0 cs=0 l=1).connect claims to be 0.0.0.0:6803/26492 not 10.214.131.31:6803/25734 - wrong node!
2013-06-04 12:21:02.125126 7f3514792700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6800/25733 pipe(0x16a6780 sd=19 :44154 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6800/26492 not 10.214.131.31:6800/25733 - wrong node!
2013-06-04 12:21:02.924607 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53206 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.32:6803/21481 not 10.214.131.32:6803/20680 - wrong node!
2013-06-04 12:21:02.925946 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :46000 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6803/26492 not 10.214.131.31:6803/25734 - wrong node!
2013-06-04 12:21:02.925995 7f3514792700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6800/25733 pipe(0x16a6780 sd=19 :44163 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6800/26492 not 10.214.131.31:6800/25733 - wrong node!
2013-06-04 12:21:04.525170 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53216 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.32:6803/21481 not 10.214.131.32:6803/20680 - wrong node!
2013-06-04 12:21:04.526761 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :46010 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6803/26492 not 10.214.131.31:6803/25734 - wrong node!
2013-06-04 12:21:04.526810 7f3514792700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6800/25733 pipe(0x16a6780 sd=19 :44173 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6800/26492 not 10.214.131.31:6800/25733 - wrong node!
2013-06-04 12:21:07.725852 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53232 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.32:6803/21481 not 10.214.131.32:6803/20680 - wrong node!
2013-06-04 12:21:07.727634 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :46026 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6803/26492 not 10.214.131.31:6803/25734 - wrong node!
2013-06-04 12:21:07.727640 7f3514792700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6800/25733 pipe(0x16a6780 sd=19 :44189 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6800/26492 not 10.214.131.31:6800/25733 - wrong node!
2013-06-04 12:21:14.126429 7f351ba16700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.32:6803/20680 pipe(0x16a6280 sd=17 :53241 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.32:6803/21481 not 10.214.131.32:6803/20680 - wrong node!
2013-06-04 12:21:14.128540 7f3514893700  0 -- 10.214.131.32:6800/21468 >> 10.214.131.31:6803/25734 pipe(0x168b000 sd=18 :46035 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.131.31:6803/26492 not 10.214.131.31:6803/25734 - wrong node!

to reproduce:
tamil@ubuntu:/tmp/up$ cat config.yaml 
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
  - client.0
targets:
  ubuntu@plana08.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqqzUa6OfNvqirlipO2jY32KanSfk2mSstJJdakPuKVijDoaDHvC7L14A9iNMChI/+OgVJRpBF4DI+QwrO00yj4jSGSuq/dtp6vxkX5fw1+g21uqZG7Sl6S89ytRkRs+NFoNY1jhWR0Qo4opEim9qApVSxlouG61L++IEv7zhZ62ogpknTMQhkgpHJ4w146silaCh6vnmoNsTBt+eIuVE/7vhMQep4REpw5uVZR6PVUBsJDJAkJuyAkNu3Xva7KC4W22sjzSqHKWqzNeAmPxQ8Ywvu5PWQulOtA/LF9gAVsJjbKE7+ZsXVYvTfpZtEKfOduss8dB3lP8Xez6CbUzsv
  ubuntu@plana09.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsROY/kuvJRrw3yOos6aHDtveOLPgZ+mBs4ul/O6LTm0swWIpeiilmIeILba12z98XmkoqIGTAgZqcGx6FfNpwrOg8P2pdDg9YKT9YrdESlFZryHyu3Rjyn+lZcskMdHAoXhTUzRkHh7t9/vM5cuhq+BoO5+nAebtFCQ1mcn9w8jn6ZqnXkQiplB93UBmXqGKhbqnaGok6xJfYtV3NtmXOFBJ0keja0rOT4ylnSBExG3YBdZ4qSEaWVNeBtoDslu6K0hq5/mmNtZwJf2iXQWOkhfM3tVUbcMfwaGVEe77gzg3CeU0lT7ag1oJan9MAxzVBqOXO1ZS2h8w4zBrX7eYt
tasks:
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- install:
    branch: bobtail
- ceph: null
- workunit:
    clients:
      all:
      - misc/trivial_sync.sh
- install.upgrade:
    all:
      branch: next
- ceph.restart:
  - mon.a
  - mon.b
  - mon.c
  - mds.a
  - osd.0
  - osd.1
  - osd.2
  - osd.3
- workunit:
    clients:
      all:
      - misc/trivial_sync.sh