Bug #5951: osd: next: EEXIST on mkcoll - Ceph - Ceph

Actions

Copy link

Bug #5951

closed

osd: next: EEXIST on mkcoll

Added by Sage Weil over 10 years ago. Updated over 10 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Samuel Just

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

   -30> 2013-08-13 03:09:19.151279 7fc2ad82b700  0 filestore(/var/lib/ceph/osd/ceph-4)  error (17) File exists not handled on operation 20 (4621.0.0, or op 0, counting from 0)
   -29> 2013-08-13 03:09:19.151299 7fc2ad82b700  0 filestore(/var/lib/ceph/osd/ceph-4) unexpected error code
   -28> 2013-08-13 03:09:19.151301 7fc2ac028700  5 --OSD::tracker-- reqid: client.4120.0:3830, seq: 698, time: 2013-08-13 03:09:19.151301, event: commit_sent, request: osd_sub_op(client.4120.0:3830 3.0 9e70cbab/plana2011975-257/head//3 [] v 112'3159 snapset=0=[]:[] snapc=0=[]) v7
   -27> 2013-08-13 03:09:19.151323 7fc2ac028700  1 -- 10.214.131.20:6810/18454 --> osd.0 10.214.131.21:6813/30793 -- osd_sub_op_reply(client.4120.0:3830 3.0 9e70cbab/plana2011975-257/head//3 [] ondisk, result = 0) v1 -- ?+0 0x2d05a00
   -26> 2013-08-13 03:09:19.151301 7fc2ad82b700  0 filestore(/var/lib/ceph/osd/ceph-4)  transaction dump:
{ "ops": [
        { "op_num": 0,
          "op_name": "mkcoll",
          "collection": "1.e_head"},
        { "op_num": 1,
          "op_name": "collection_setattr",
          "collection": "1.e_head",
          "name": "info",
          "length": 1},
        { "op_num": 2,
          "op_name": "omap_setkeys",
          "collection": "meta",
          "oid": "16ef7597\/infos\/head\/\/-1",
          "attr_lens": { "1.e_biginfo": 55,
              "1.e_epoch": 4,
              "1.e_info": 568}},
        { "op_num": 3,
          "op_name": "touch",
          "collection": "meta",
          "oid": "103a8f2f\/pglog_1.e\/0\/\/-1"},
        { "op_num": 4,
          "op_name": "omap_setkeys",
          "collection": "meta",
          "oid": "103a8f2f\/pglog_1.e\/0\/\/-1",
          "attr_lens": {}}]}

ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2013-08-13_01:00:13-rados-next-testing-basic-plana/104707$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 68d04c9a036bea12d76fdd51e926aef771e8e041
machine_type: plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      global:
        ms inject delay max: 1
        ms inject delay probability: 0.005
        ms inject delay type: osd
        ms inject internal delays: 0.002
        ms inject socket failures: 2500
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
    fs: ext4
    log-whitelist:
    - slow request
    sha1: 2632846e24e3c26139e982e0a569951d25e1589b
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: 2632846e24e3c26139e982e0a569951d25e1589b
  s3tests:
    branch: next
  workunit:
    sha1: 2632846e24e3c26139e982e0a569951d25e1589b
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 10
      read: 45
      write: 45
    ops: 4000
teuthology_branch: next

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Samuel Just over 10 years ago

./virtualenv/bin/teuthology-schedule --name "samuelj-5951-0" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-0.jobs
Job scheduled with ID 105406
Job scheduled with ID 105407
Job scheduled with ID 105408
Job scheduled with ID 105409
Job scheduled with ID 105410
Job scheduled with ID 105411
Job scheduled with ID 105412
Job scheduled with ID 105413
Job scheduled with ID 105414
Job scheduled with ID 105415
Job scheduled with ID 105416
Job scheduled with ID 105417
Job scheduled with ID 105418
Job scheduled with ID 105419
Job scheduled with ID 105420
Job scheduled with ID 105421
Job scheduled with ID 105422
Job scheduled with ID 105423
Job scheduled with ID 105424
Job scheduled with ID 105425
Job scheduled with ID 105426
Job scheduled with ID 105427
Job scheduled with ID 105428
Job scheduled with ID 105429
Job scheduled with ID 105430
Job scheduled with ID 105431
Job scheduled with ID 105432
Job scheduled with ID 105433
Job scheduled with ID 105434
Job scheduled with ID 105435
Job scheduled with ID 105436
Job scheduled with ID 105437
Job scheduled with ID 105438
Job scheduled with ID 105439
Job scheduled with ID 105440
Job scheduled with ID 105441
Job scheduled with ID 105442
Job scheduled with ID 105443
Job scheduled with ID 105444
Job scheduled with ID 105445
Job scheduled with ID 105446
Job scheduled with ID 105447
Job scheduled with ID 105448
Job scheduled with ID 105449
Job scheduled with ID 105450
Job scheduled with ID 105451
Job scheduled with ID 105452
Job scheduled with ID 105453
Job scheduled with ID 105454
Job scheduled with ID 105455

Actions

Copy link

Updated by Samuel Just over 10 years ago

failed due to github:
~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-1" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-1.jobs
Job scheduled with ID 106882
Job scheduled with ID 106883
Job scheduled with ID 106884
Job scheduled with ID 106885
Job scheduled with ID 106886
Job scheduled with ID 106887
Job scheduled with ID 106888
Job scheduled with ID 106889
Job scheduled with ID 106890
Job scheduled with ID 106891
Job scheduled with ID 106892
Job scheduled with ID 106893
Job scheduled with ID 106894
Job scheduled with ID 106895
Job scheduled with ID 106896
Job scheduled with ID 106897
Job scheduled with ID 106898
Job scheduled with ID 106899
Job scheduled with ID 106900
Job scheduled with ID 106901
Job scheduled with ID 106902
Job scheduled with ID 106903
Job scheduled with ID 106904
Job scheduled with ID 106905
Job scheduled with ID 106906
Job scheduled with ID 106907
Job scheduled with ID 106908
Job scheduled with ID 106909
Job scheduled with ID 106910
Job scheduled with ID 106911
Job scheduled with ID 106912
Job scheduled with ID 106913
Job scheduled with ID 106914
Job scheduled with ID 106915
Job scheduled with ID 106916
Job scheduled with ID 106917
Job scheduled with ID 106918
Job scheduled with ID 106919
Job scheduled with ID 106920
Job scheduled with ID 106921
Job scheduled with ID 106922
Job scheduled with ID 106923
Job scheduled with ID 106924
Job scheduled with ID 106925
Job scheduled with ID 106926
Job scheduled with ID 106927
Job scheduled with ID 106928
Job scheduled with ID 106929
Job scheduled with ID 106930
Job scheduled with ID 106931

Actions

Copy link

Updated by Samuel Just over 10 years ago

2 failures in previous set, 1 mon clock, 1 stuck wait_backfill. Didn't have logging. D'oh. Rerunning.

~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-2" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-2.jobs
Job scheduled with ID 108265
Job scheduled with ID 108266
Job scheduled with ID 108267
Job scheduled with ID 108268
Job scheduled with ID 108269
Job scheduled with ID 108270
Job scheduled with ID 108271
Job scheduled with ID 108272
Job scheduled with ID 108273
Job scheduled with ID 108274
Job scheduled with ID 108275
Job scheduled with ID 108276
Job scheduled with ID 108277
Job scheduled with ID 108278
Job scheduled with ID 108279
Job scheduled with ID 108280
Job scheduled with ID 108281
Job scheduled with ID 108282
Job scheduled with ID 108283
Job scheduled with ID 108284
Job scheduled with ID 108285
Job scheduled with ID 108286
Job scheduled with ID 108287
Job scheduled with ID 108288
Job scheduled with ID 108289
Job scheduled with ID 108290
Job scheduled with ID 108291
Job scheduled with ID 108292
Job scheduled with ID 108293
Job scheduled with ID 108294
Job scheduled with ID 108295
Job scheduled with ID 108296
Job scheduled with ID 108297
Job scheduled with ID 108298
Job scheduled with ID 108299
Job scheduled with ID 108300
Job scheduled with ID 108301
Job scheduled with ID 108302
Job scheduled with ID 108303
Job scheduled with ID 108304
Job scheduled with ID 108305
Job scheduled with ID 108306
Job scheduled with ID 108307
Job scheduled with ID 108308
Job scheduled with ID 108309
Job scheduled with ID 108310
Job scheduled with ID 108311
Job scheduled with ID 108312
Job scheduled with ID 108313
Job scheduled with ID 108314

Actions

Copy link

Updated by Sage Weil over 10 years ago

Status changed from New to 12

ubuntu@teuthology:/a/teuthology-2013-08-19_01:00:13-rados-master-testing-basic-plana/969

Actions

Copy link

Updated by Samuel Just over 10 years ago

~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-3" -n 20 --owner samuelj@slider testruns/5951.yaml | tee 5951-3.jobs
Job scheduled with ID 1709
Job scheduled with ID 1710
Job scheduled with ID 1711
Job scheduled with ID 1712
Job scheduled with ID 1713
Job scheduled with ID 1714
Job scheduled with ID 1715
Job scheduled with ID 1716
Job scheduled with ID 1717
Job scheduled with ID 1718
Job scheduled with ID 1719
Job scheduled with ID 1720
Job scheduled with ID 1721
Job scheduled with ID 1722
Job scheduled with ID 1723
Job scheduled with ID 1724
Job scheduled with ID 1725
Job scheduled with ID 1726
Job scheduled with ID 1727
Job scheduled with ID 1728

Actions

Copy link

Updated by Samuel Just over 10 years ago

no failures in the last run
~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-4" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-4.jobs
Job scheduled with ID 1850
Job scheduled with ID 1851
Job scheduled with ID 1852
Job scheduled with ID 1853
Job scheduled with ID 1854
Job scheduled with ID 1855
Job scheduled with ID 1856
Job scheduled with ID 1857
Job scheduled with ID 1858
Job scheduled with ID 1859
Job scheduled with ID 1860
Job scheduled with ID 1861
Job scheduled with ID 1862
Job scheduled with ID 1863
Job scheduled with ID 1864
Job scheduled with ID 1865
Job scheduled with ID 1866
Job scheduled with ID 1867
Job scheduled with ID 1868
Job scheduled with ID 1869
Job scheduled with ID 1870
Job scheduled with ID 1871
Job scheduled with ID 1872
Job scheduled with ID 1873
Job scheduled with ID 1874
Job scheduled with ID 1875
Job scheduled with ID 1876
Job scheduled with ID 1877
Job scheduled with ID 1878
Job scheduled with ID 1879
Job scheduled with ID 1880
Job scheduled with ID 1881
Job scheduled with ID 1882
Job scheduled with ID 1883
Job scheduled with ID 1884
Job scheduled with ID 1885
Job scheduled with ID 1886
Job scheduled with ID 1887
Job scheduled with ID 1888
Job scheduled with ID 1889
Job scheduled with ID 1890
Job scheduled with ID 1891
Job scheduled with ID 1892
Job scheduled with ID 1893
Job scheduled with ID 1894
Job scheduled with ID 1895
Job scheduled with ID 1896
Job scheduled with ID 1897
Job scheduled with ID 1898
Job scheduled with ID 1899

Actions

Copy link

Updated by Samuel Just over 10 years ago

no failures in previous run
~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-5" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-5.jobs
Job scheduled with ID 2375
Job scheduled with ID 2376
Job scheduled with ID 2377
Job scheduled with ID 2378
Job scheduled with ID 2379
Job scheduled with ID 2380
Job scheduled with ID 2381
Job scheduled with ID 2382
Job scheduled with ID 2383
Job scheduled with ID 2384
Job scheduled with ID 2385
Job scheduled with ID 2386
Job scheduled with ID 2387
Job scheduled with ID 2388
Job scheduled with ID 2389
Job scheduled with ID 2390
Job scheduled with ID 2391
Job scheduled with ID 2392
Job scheduled with ID 2393
Job scheduled with ID 2394
Job scheduled with ID 2395
Job scheduled with ID 2396
Job scheduled with ID 2397
Job scheduled with ID 2398
Job scheduled with ID 2399
Job scheduled with ID 2400
Job scheduled with ID 2401
Job scheduled with ID 2402
Job scheduled with ID 2403
Job scheduled with ID 2404
Job scheduled with ID 2405
Job scheduled with ID 2406
Job scheduled with ID 2407
Job scheduled with ID 2408
Job scheduled with ID 2409
Job scheduled with ID 2410
Job scheduled with ID 2411
Job scheduled with ID 2412
Job scheduled with ID 2413
Job scheduled with ID 2414
Job scheduled with ID 2415
Job scheduled with ID 2416
Job scheduled with ID 2417
Job scheduled with ID 2418
Job scheduled with ID 2419
Job scheduled with ID 2420
Job scheduled with ID 2421
Job scheduled with ID 2422
Job scheduled with ID 2423
Job scheduled with ID 2424

Actions

Copy link

Updated by Sage Weil over 10 years ago

ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2013-08-19_20:00:16-rados-dumpling-testing-basic-plana/2087

Actions

Copy link

Updated by Sage Weil over 10 years ago

ubuntu@teuthology:/a/teuthology-2013-08-20_01:00:13-rados-next-testing-basic-plana/2690

Actions

Copy link

#10

Updated by Samuel Just over 10 years ago

~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-6" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-6.jobs
Job scheduled with ID 3542
Job scheduled with ID 3543
Job scheduled with ID 3544
Job scheduled with ID 3545
Job scheduled with ID 3546
Job scheduled with ID 3547
Job scheduled with ID 3548
Job scheduled with ID 3549
Job scheduled with ID 3550
Job scheduled with ID 3551
Job scheduled with ID 3552
Job scheduled with ID 3553
Job scheduled with ID 3554
Job scheduled with ID 3555
Job scheduled with ID 3556
Job scheduled with ID 3557
Job scheduled with ID 3558
Job scheduled with ID 3559
Job scheduled with ID 3560
Job scheduled with ID 3561
Job scheduled with ID 3562
Job scheduled with ID 3563
Job scheduled with ID 3564
Job scheduled with ID 3565
Job scheduled with ID 3566
Job scheduled with ID 3567
Job scheduled with ID 3568
Job scheduled with ID 3569
Job scheduled with ID 3570
Job scheduled with ID 3571
Job scheduled with ID 3572
Job scheduled with ID 3573
Job scheduled with ID 3574
Job scheduled with ID 3575
Job scheduled with ID 3576
Job scheduled with ID 3577
Job scheduled with ID 3578
Job scheduled with ID 3579
Job scheduled with ID 3580
Job scheduled with ID 3581
Job scheduled with ID 3582
Job scheduled with ID 3583
Job scheduled with ID 3584
Job scheduled with ID 3585
Job scheduled with ID 3586
Job scheduled with ID 3587
Job scheduled with ID 3588
Job scheduled with ID 3589
Job scheduled with ID 3590
Job scheduled with ID 3591

Actions

Copy link

#11

Updated by Samuel Just over 10 years ago

~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-7" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-7.jobs
Job scheduled with ID 4028
Job scheduled with ID 4029
Job scheduled with ID 4030
Job scheduled with ID 4031
Job scheduled with ID 4032
Job scheduled with ID 4033
Job scheduled with ID 4034
Job scheduled with ID 4035
Job scheduled with ID 4036
Job scheduled with ID 4037
Job scheduled with ID 4038
Job scheduled with ID 4039
Job scheduled with ID 4040
Job scheduled with ID 4041
Job scheduled with ID 4042
Job scheduled with ID 4043
Job scheduled with ID 4044
Job scheduled with ID 4045
Job scheduled with ID 4046
Job scheduled with ID 4047
Job scheduled with ID 4048
Job scheduled with ID 4049
Job scheduled with ID 4050
Job scheduled with ID 4051
Job scheduled with ID 4052
Job scheduled with ID 4053
Job scheduled with ID 4054
Job scheduled with ID 4055
Job scheduled with ID 4056
Job scheduled with ID 4057
Job scheduled with ID 4058
Job scheduled with ID 4059
Job scheduled with ID 4060
Job scheduled with ID 4061
Job scheduled with ID 4062
Job scheduled with ID 4063
Job scheduled with ID 4064
Job scheduled with ID 4065
Job scheduled with ID 4066
Job scheduled with ID 4067
Job scheduled with ID 4068
Job scheduled with ID 4069
Job scheduled with ID 4070
Job scheduled with ID 4071
Job scheduled with ID 4072
Job scheduled with ID 4073
Job scheduled with ID 4074
Job scheduled with ID 4075
Job scheduled with ID 4076
Job scheduled with ID 4077

Actions

Copy link

#12

Updated by Samuel Just over 10 years ago

No failures again. Switched yaml to xfs, more failures so far with xfs than with ext4.

~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-8" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-8.jobs
Job scheduled with ID 5590
Job scheduled with ID 5591
Job scheduled with ID 5592
Job scheduled with ID 5593
Job scheduled with ID 5594
Job scheduled with ID 5595
Job scheduled with ID 5596
Job scheduled with ID 5597
Job scheduled with ID 5598
Job scheduled with ID 5599
Job scheduled with ID 5600
Job scheduled with ID 5601
Job scheduled with ID 5602
Job scheduled with ID 5603
Job scheduled with ID 5604
Job scheduled with ID 5605
Job scheduled with ID 5606
Job scheduled with ID 5607
Job scheduled with ID 5608
Job scheduled with ID 5609
Job scheduled with ID 5610
Job scheduled with ID 5611
Job scheduled with ID 5612
Job scheduled with ID 5613
Job scheduled with ID 5614
Job scheduled with ID 5615
Job scheduled with ID 5616
Job scheduled with ID 5617
Job scheduled with ID 5618
Job scheduled with ID 5619
Job scheduled with ID 5620
Job scheduled with ID 5621
Job scheduled with ID 5622
Job scheduled with ID 5623
Job scheduled with ID 5624
Job scheduled with ID 5625
Job scheduled with ID 5626
Job scheduled with ID 5627
Job scheduled with ID 5628
Job scheduled with ID 5629
Job scheduled with ID 5630
Job scheduled with ID 5631
Job scheduled with ID 5632
Job scheduled with ID 5633
Job scheduled with ID 5634
Job scheduled with ID 5635
Job scheduled with ID 5636
Job scheduled with ID 5637
Job scheduled with ID 5638
Job scheduled with ID 5639

Actions

Copy link

#13

Updated by Samuel Just over 10 years ago

Nothing useful from the last one.

~/teuthology [mine?] » ./virtualenv/bin/teuthology-schedule --name "samuelj-5951-9" -n 50 --owner samuelj@slider testruns/5951.yaml | tee 5951-9.jobs
Job scheduled with ID 983
Job scheduled with ID 984
Job scheduled with ID 985
Job scheduled with ID 986
Job scheduled with ID 987
Job scheduled with ID 988
Job scheduled with ID 989
Job scheduled with ID 990
Job scheduled with ID 991
Job scheduled with ID 992
Job scheduled with ID 993
Job scheduled with ID 994
Job scheduled with ID 995
Job scheduled with ID 996
Job scheduled with ID 997
Job scheduled with ID 998
Job scheduled with ID 999
Job scheduled with ID 1000
Job scheduled with ID 1001
Job scheduled with ID 1002
Job scheduled with ID 1003
Job scheduled with ID 1004
Job scheduled with ID 1005
Job scheduled with ID 1006
Job scheduled with ID 1007
Job scheduled with ID 1008
Job scheduled with ID 1009
Job scheduled with ID 1010
Job scheduled with ID 1011
Job scheduled with ID 1012
Job scheduled with ID 1013
Job scheduled with ID 1014
Job scheduled with ID 1015
Job scheduled with ID 1016
Job scheduled with ID 1017
Job scheduled with ID 1018
Job scheduled with ID 1019
Job scheduled with ID 1020
Job scheduled with ID 1021
Job scheduled with ID 1022
Job scheduled with ID 1023
Job scheduled with ID 1024
Job scheduled with ID 1025
Job scheduled with ID 1026
Job scheduled with ID 1027
Job scheduled with ID 1028
Job scheduled with ID 1029
Job scheduled with ID 1030
Job scheduled with ID 1031
Job scheduled with ID 1032