Project

General

Profile

Main » History » Version 197

Patrick Donnelly, 10/20/2023 01:51 PM

1 79 Venky Shankar
h1. MAIN
2
3 148 Rishabh Dave
h3. NEW ENTRY BELOW
4
5 195 Venky Shankar
h3. 18 Oct 2023
6
7
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231018.065603
8
9
* https://tracker.ceph.com/issues/52624
10
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
11
* https://tracker.ceph.com/issues/57676
12
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
13
* https://tracker.ceph.com/issues/63233
14
    mon|client|mds: valgrind reports possible leaks in the MDS
15
* https://tracker.ceph.com/issues/63141
16
    qa/cephfs: test_idem_unaffected_root_squash fails
17
* https://tracker.ceph.com/issues/59531
18
  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 
19
* https://tracker.ceph.com/issues/62658
20
    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
21
* https://tracker.ceph.com/issues/62580
22
    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
23
* https://tracker.ceph.com/issues/62067
24
    ffsb.sh failure "Resource temporarily unavailable"
25
* https://tracker.ceph.com/issues/57655
26
    qa: fs:mixed-clients kernel_untar_build failure
27
* https://tracker.ceph.com/issues/62036
28
    src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty())
29
* https://tracker.ceph.com/issues/58945
30
    qa: xfstests-dev's generic test suite has 20 failures with fuse client
31
* https://tracker.ceph.com/issues/62847
32
    mds: blogbench requests stuck (5mds+scrub+snaps-flush)
33
34 193 Venky Shankar
h3. 13 Oct 2023
35
36
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231013.093215
37
38
* https://tracker.ceph.com/issues/52624
39
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
40
* https://tracker.ceph.com/issues/62936
41
    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
42
* https://tracker.ceph.com/issues/47292
43
    cephfs-shell: test_df_for_valid_file failure
44
* https://tracker.ceph.com/issues/63141
45
    qa/cephfs: test_idem_unaffected_root_squash fails
46
* https://tracker.ceph.com/issues/62081
47
    tasks/fscrypt-common does not finish, timesout
48 1 Patrick Donnelly
* https://tracker.ceph.com/issues/58945
49
    qa: xfstests-dev's generic test suite has 20 failures with fuse client
50 194 Venky Shankar
* https://tracker.ceph.com/issues/63233
51
    mon|client|mds: valgrind reports possible leaks in the MDS
52 193 Venky Shankar
53 190 Patrick Donnelly
h3. 16 Oct 2023
54
55
https://pulpito.ceph.com/?branch=wip-batrick-testing-20231016.203825
56
57 192 Patrick Donnelly
Infrastructure issues:
58
* /teuthology/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432286/teuthology.log
59
    Host lost.
60
61 196 Patrick Donnelly
One followup fix:
62
* https://pulpito.ceph.com/pdonnell-2023-10-20_00:33:29-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/
63
64 192 Patrick Donnelly
Failures:
65
66
* https://tracker.ceph.com/issues/56694
67
    qa: avoid blocking forever on hung umount
68
* https://tracker.ceph.com/issues/63089
69
    qa: tasks/mirror times out
70
* https://tracker.ceph.com/issues/52624
71
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
72
* https://tracker.ceph.com/issues/59531
73
  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 
74
* https://tracker.ceph.com/issues/57676
75
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
76
* https://tracker.ceph.com/issues/62658 
77
    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
78
* https://tracker.ceph.com/issues/61243
79
  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
80
* https://tracker.ceph.com/issues/57656
81
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
82
* https://tracker.ceph.com/issues/63233
83
  mon|client|mds: valgrind reports possible leaks in the MDS
84 197 Patrick Donnelly
* https://tracker.ceph.com/issues/63278
85
  kclient: may wrongly decode session messages and believe it is blocklisted (dead jobs)
86 192 Patrick Donnelly
87 189 Rishabh Dave
h3. 9 Oct 2023
88
89
https://pulpito.ceph.com/rishabh-2023-10-06_11:56:52-fs-rishabh-cephfs-mon-testing-default-smithi/
90
91
* https://tracker.ceph.com/issues/54460
92
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
93
* https://tracker.ceph.com/issues/63141
94
  test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails
95
* https://tracker.ceph.com/issues/62937
96
  logrotate doesn't support parallel execution on same set of logfiles
97
* https://tracker.ceph.com/issues/61400
98
  valgrind+ceph-mon issues
99
* https://tracker.ceph.com/issues/57676
100
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
101
* https://tracker.ceph.com/issues/55805
102
  error during scrub thrashing reached max tries in 900 secs
103
104 188 Venky Shankar
h3. 26 Sep 2023
105
106
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230926.081818
107
108
* https://tracker.ceph.com/issues/52624
109
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
110
* https://tracker.ceph.com/issues/62873
111
    qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
112
* https://tracker.ceph.com/issues/61400
113
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
114
* https://tracker.ceph.com/issues/57676
115
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
116
* https://tracker.ceph.com/issues/62682
117
    mon: no mdsmap broadcast after "fs set joinable" is set to true
118
* https://tracker.ceph.com/issues/63089
119
    qa: tasks/mirror times out
120
121 185 Rishabh Dave
h3. 22 Sep 2023
122
123
https://pulpito.ceph.com/rishabh-2023-09-12_12:12:15-fs-wip-rishabh-2023sep12-b2-testing-default-smithi/
124
125
* https://tracker.ceph.com/issues/59348
126
  qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota
127
* https://tracker.ceph.com/issues/59344
128
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
129
* https://tracker.ceph.com/issues/59531
130
  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 
131
* https://tracker.ceph.com/issues/61574
132
  build failure for mdtest project
133
* https://tracker.ceph.com/issues/62702
134
  fsstress.sh: MDS slow requests for the internal 'rename' requests
135
* https://tracker.ceph.com/issues/57676
136
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
137
138
* https://tracker.ceph.com/issues/62863 
139
  deadlock in ceph-fuse causes teuthology job to hang and fail
140
* https://tracker.ceph.com/issues/62870
141
  test_cluster_info (tasks.cephfs.test_nfs.TestNFS)
142
* https://tracker.ceph.com/issues/62873
143
  test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
144
145 186 Venky Shankar
h3. 20 Sep 2023
146
147
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230920.072635
148
149
* https://tracker.ceph.com/issues/52624
150
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
151
* https://tracker.ceph.com/issues/61400
152
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
153
* https://tracker.ceph.com/issues/61399
154
    libmpich: undefined references to fi_strerror
155
* https://tracker.ceph.com/issues/62081
156
    tasks/fscrypt-common does not finish, timesout
157
* https://tracker.ceph.com/issues/62658 
158
    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
159
* https://tracker.ceph.com/issues/62915
160
    qa/suites/fs/nfs: No orchestrator configured (try `ceph orch set backend`) while running test cases
161
* https://tracker.ceph.com/issues/59531
162
    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
163
* https://tracker.ceph.com/issues/62873
164
  qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
165
* https://tracker.ceph.com/issues/62936
166
    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
167
* https://tracker.ceph.com/issues/62937
168
    Command failed on smithi027 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
169
* https://tracker.ceph.com/issues/62510
170
    snaptest-git-ceph.sh failure with fs/thrash
171
* https://tracker.ceph.com/issues/62081
172
    tasks/fscrypt-common does not finish, timesout
173
* https://tracker.ceph.com/issues/62126
174
    test failure: suites/blogbench.sh stops running
175 187 Venky Shankar
* https://tracker.ceph.com/issues/62682
176
    mon: no mdsmap broadcast after "fs set joinable" is set to true
177 186 Venky Shankar
178 184 Milind Changire
h3. 19 Sep 2023
179
180
http://pulpito.front.sepia.ceph.com/mchangir-2023-09-12_05:40:22-fs-wip-mchangir-testing-20230908.140927-testing-default-smithi/
181
182
* https://tracker.ceph.com/issues/58220#note-9
183
  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure
184
* https://tracker.ceph.com/issues/62702
185
  Command failed (workunit test suites/fsstress.sh) on smithi124 with status 124
186
* https://tracker.ceph.com/issues/57676
187
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
188
* https://tracker.ceph.com/issues/59348
189
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
190
* https://tracker.ceph.com/issues/52624
191
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
192
* https://tracker.ceph.com/issues/51964
193
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
194
* https://tracker.ceph.com/issues/61243
195
  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
196
* https://tracker.ceph.com/issues/59344
197
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
198
* https://tracker.ceph.com/issues/62873
199
  qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
200
* https://tracker.ceph.com/issues/59413
201
  cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"
202
* https://tracker.ceph.com/issues/53859
203
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
204
* https://tracker.ceph.com/issues/62482
205
  qa: cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)
206
207 178 Patrick Donnelly
208 177 Venky Shankar
h3. 13 Sep 2023
209
210
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230908.065909
211
212
* https://tracker.ceph.com/issues/52624
213
      qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
214
* https://tracker.ceph.com/issues/57655
215
    qa: fs:mixed-clients kernel_untar_build failure
216
* https://tracker.ceph.com/issues/57676
217
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
218
* https://tracker.ceph.com/issues/61243
219
    qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed
220
* https://tracker.ceph.com/issues/62567
221
    postgres workunit times out - MDS_SLOW_REQUEST in logs
222
* https://tracker.ceph.com/issues/61400
223
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
224
* https://tracker.ceph.com/issues/61399
225
    libmpich: undefined references to fi_strerror
226
* https://tracker.ceph.com/issues/57655
227
    qa: fs:mixed-clients kernel_untar_build failure
228
* https://tracker.ceph.com/issues/57676
229
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
230
* https://tracker.ceph.com/issues/51964
231
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
232
* https://tracker.ceph.com/issues/62081
233
    tasks/fscrypt-common does not finish, timesout
234 178 Patrick Donnelly
235 179 Patrick Donnelly
h3. 2023 Sep 12
236 178 Patrick Donnelly
237
https://pulpito.ceph.com/pdonnell-2023-09-12_14:07:50-fs-wip-batrick-testing-20230912.122437-distro-default-smithi/
238 1 Patrick Donnelly
239 181 Patrick Donnelly
A few failures caused by qa refactoring in https://github.com/ceph/ceph/pull/48130 ; notably:
240
241 182 Patrick Donnelly
* Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) caused by fragmentation from config changes.
242 181 Patrick Donnelly
243
Failures:
244
245 179 Patrick Donnelly
* https://tracker.ceph.com/issues/59348
246
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
247
* https://tracker.ceph.com/issues/57656
248
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
249
* https://tracker.ceph.com/issues/55805
250
  error scrub thrashing reached max tries in 900 secs
251
* https://tracker.ceph.com/issues/62067
252
    ffsb.sh failure "Resource temporarily unavailable"
253
* https://tracker.ceph.com/issues/59344
254
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
255
* https://tracker.ceph.com/issues/61399
256 180 Patrick Donnelly
  libmpich: undefined references to fi_strerror
257
* https://tracker.ceph.com/issues/62832
258
  common: config_proxy deadlock during shutdown (and possibly other times)
259
* https://tracker.ceph.com/issues/59413
260 1 Patrick Donnelly
  cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"
261 181 Patrick Donnelly
* https://tracker.ceph.com/issues/57676
262
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
263
* https://tracker.ceph.com/issues/62567
264
  Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"
265
* https://tracker.ceph.com/issues/54460
266
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
267
* https://tracker.ceph.com/issues/58220#note-9
268
  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure
269
* https://tracker.ceph.com/issues/59348
270
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
271 183 Patrick Donnelly
* https://tracker.ceph.com/issues/62847
272
    mds: blogbench requests stuck (5mds+scrub+snaps-flush)
273
* https://tracker.ceph.com/issues/62848
274
    qa: fail_fs upgrade scenario hanging
275
* https://tracker.ceph.com/issues/62081
276
    tasks/fscrypt-common does not finish, timesout
277 177 Venky Shankar
278 176 Venky Shankar
h3. 11 Sep 2023
279 175 Venky Shankar
280
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230830.153114
281
282
* https://tracker.ceph.com/issues/52624
283
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
284
* https://tracker.ceph.com/issues/61399
285
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
286
* https://tracker.ceph.com/issues/57655
287
    qa: fs:mixed-clients kernel_untar_build failure
288
* https://tracker.ceph.com/issues/61399
289
    ior build failure
290
* https://tracker.ceph.com/issues/59531
291
    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
292
* https://tracker.ceph.com/issues/59344
293
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
294
* https://tracker.ceph.com/issues/59346
295
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
296
* https://tracker.ceph.com/issues/59348
297
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
298
* https://tracker.ceph.com/issues/57676
299
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
300
* https://tracker.ceph.com/issues/61243
301
  qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed
302
* https://tracker.ceph.com/issues/62567
303
  postgres workunit times out - MDS_SLOW_REQUEST in logs
304
305
306 174 Rishabh Dave
h3. 6 Sep 2023 Run 2
307
308
https://pulpito.ceph.com/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/ 
309
310
* https://tracker.ceph.com/issues/51964
311
  test_cephfs_mirror_restart_sync_on_blocklist failure
312
* https://tracker.ceph.com/issues/59348
313
  test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
314
* https://tracker.ceph.com/issues/53859
315
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
316
* https://tracker.ceph.com/issues/61892
317
  test_strays.TestStrays.test_snapshot_remove failed
318
* https://tracker.ceph.com/issues/54460
319
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
320
* https://tracker.ceph.com/issues/59346
321
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
322
* https://tracker.ceph.com/issues/59344
323
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
324
* https://tracker.ceph.com/issues/62484
325
  qa: ffsb.sh test failure
326
* https://tracker.ceph.com/issues/62567
327
  Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"
328
  
329
* https://tracker.ceph.com/issues/61399
330
  ior build failure
331
* https://tracker.ceph.com/issues/57676
332
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
333
* https://tracker.ceph.com/issues/55805
334
  error scrub thrashing reached max tries in 900 secs
335
336 172 Rishabh Dave
h3. 6 Sep 2023
337 171 Rishabh Dave
338 173 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-08-10_20:16:46-fs-wip-rishabh-2023Aug1-b4-testing-default-smithi/
339 171 Rishabh Dave
340 1 Patrick Donnelly
* https://tracker.ceph.com/issues/53859
341
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
342 173 Rishabh Dave
* https://tracker.ceph.com/issues/51964
343
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
344 1 Patrick Donnelly
* https://tracker.ceph.com/issues/61892
345 173 Rishabh Dave
  test_snapshot_remove (test_strays.TestStrays) failed
346
* https://tracker.ceph.com/issues/59348
347
  qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota
348
* https://tracker.ceph.com/issues/54462
349
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
350
* https://tracker.ceph.com/issues/62556
351
  test_acls: xfstests_dev: python2 is missing
352
* https://tracker.ceph.com/issues/62067
353
  ffsb.sh failure "Resource temporarily unavailable"
354
* https://tracker.ceph.com/issues/57656
355
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
356 1 Patrick Donnelly
* https://tracker.ceph.com/issues/59346
357
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
358 171 Rishabh Dave
* https://tracker.ceph.com/issues/59344
359 173 Rishabh Dave
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
360
361 171 Rishabh Dave
* https://tracker.ceph.com/issues/61399
362
  ior build failure
363
* https://tracker.ceph.com/issues/57676
364
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
365
* https://tracker.ceph.com/issues/55805
366
  error scrub thrashing reached max tries in 900 secs
367 173 Rishabh Dave
368
* https://tracker.ceph.com/issues/62567
369
  Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"
370
* https://tracker.ceph.com/issues/62702
371
  workunit test suites/fsstress.sh on smithi066 with status 124
372 170 Rishabh Dave
373
h3. 5 Sep 2023
374
375
https://pulpito.ceph.com/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/
376
orch:cephadm suite run: http://pulpito.front.sepia.ceph.com/rishabh-2023-09-05_12:16:09-orch:cephadm-wip-rishabh-2023aug3-b5-testing-default-smithi/
377
  this run has failures but acc to Adam King these are not relevant and should be ignored
378
379
* https://tracker.ceph.com/issues/61892
380
  test_snapshot_remove (test_strays.TestStrays) failed
381
* https://tracker.ceph.com/issues/59348
382
  test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota
383
* https://tracker.ceph.com/issues/54462
384
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
385
* https://tracker.ceph.com/issues/62067
386
  ffsb.sh failure "Resource temporarily unavailable"
387
* https://tracker.ceph.com/issues/57656 
388
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
389
* https://tracker.ceph.com/issues/59346
390
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
391
* https://tracker.ceph.com/issues/59344
392
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
393
* https://tracker.ceph.com/issues/50223
394
  client.xxxx isn't responding to mclientcaps(revoke)
395
* https://tracker.ceph.com/issues/57655
396
  qa: fs:mixed-clients kernel_untar_build failure
397
* https://tracker.ceph.com/issues/62187
398
  iozone.sh: line 5: iozone: command not found
399
 
400
* https://tracker.ceph.com/issues/61399
401
  ior build failure
402
* https://tracker.ceph.com/issues/57676
403
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
404
* https://tracker.ceph.com/issues/55805
405
  error scrub thrashing reached max tries in 900 secs
406 169 Venky Shankar
407
408
h3. 31 Aug 2023
409
410
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230824.045828
411
412
* https://tracker.ceph.com/issues/52624
413
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
414
* https://tracker.ceph.com/issues/62187
415
    iozone: command not found
416
* https://tracker.ceph.com/issues/61399
417
    ior build failure
418
* https://tracker.ceph.com/issues/59531
419
    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
420
* https://tracker.ceph.com/issues/61399
421
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
422
* https://tracker.ceph.com/issues/57655
423
    qa: fs:mixed-clients kernel_untar_build failure
424
* https://tracker.ceph.com/issues/59344
425
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
426
* https://tracker.ceph.com/issues/59346
427
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
428
* https://tracker.ceph.com/issues/59348
429
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
430
* https://tracker.ceph.com/issues/59413
431
    cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"
432
* https://tracker.ceph.com/issues/62653
433
    qa: unimplemented fcntl command: 1036 with fsstress
434
* https://tracker.ceph.com/issues/61400
435
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
436
* https://tracker.ceph.com/issues/62658
437
    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
438
* https://tracker.ceph.com/issues/62188
439
    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
440 168 Venky Shankar
441
442
h3. 25 Aug 2023
443
444
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.064807
445
446
* https://tracker.ceph.com/issues/59344
447
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
448
* https://tracker.ceph.com/issues/59346
449
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
450
* https://tracker.ceph.com/issues/59348
451
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
452
* https://tracker.ceph.com/issues/57655
453
    qa: fs:mixed-clients kernel_untar_build failure
454
* https://tracker.ceph.com/issues/61243
455
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
456
* https://tracker.ceph.com/issues/61399
457
    ior build failure
458
* https://tracker.ceph.com/issues/61399
459
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
460
* https://tracker.ceph.com/issues/62484
461
    qa: ffsb.sh test failure
462
* https://tracker.ceph.com/issues/59531
463
    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
464
* https://tracker.ceph.com/issues/62510
465
    snaptest-git-ceph.sh failure with fs/thrash
466 167 Venky Shankar
467
468
h3. 24 Aug 2023
469
470
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.060131
471
472
* https://tracker.ceph.com/issues/57676
473
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
474
* https://tracker.ceph.com/issues/51964
475
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
476
* https://tracker.ceph.com/issues/59344
477
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
478
* https://tracker.ceph.com/issues/59346
479
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
480
* https://tracker.ceph.com/issues/59348
481
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
482
* https://tracker.ceph.com/issues/61399
483
    ior build failure
484
* https://tracker.ceph.com/issues/61399
485
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
486
* https://tracker.ceph.com/issues/62510
487
    snaptest-git-ceph.sh failure with fs/thrash
488
* https://tracker.ceph.com/issues/62484
489
    qa: ffsb.sh test failure
490
* https://tracker.ceph.com/issues/57087
491
    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure
492
* https://tracker.ceph.com/issues/57656
493
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
494
* https://tracker.ceph.com/issues/62187
495
    iozone: command not found
496
* https://tracker.ceph.com/issues/62188
497
    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
498
* https://tracker.ceph.com/issues/62567
499
    postgres workunit times out - MDS_SLOW_REQUEST in logs
500 166 Venky Shankar
501
502
h3. 22 Aug 2023
503
504
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230809.035933
505
506
* https://tracker.ceph.com/issues/57676
507
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
508
* https://tracker.ceph.com/issues/51964
509
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
510
* https://tracker.ceph.com/issues/59344
511
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
512
* https://tracker.ceph.com/issues/59346
513
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
514
* https://tracker.ceph.com/issues/59348
515
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
516
* https://tracker.ceph.com/issues/61399
517
    ior build failure
518
* https://tracker.ceph.com/issues/61399
519
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
520
* https://tracker.ceph.com/issues/57655
521
    qa: fs:mixed-clients kernel_untar_build failure
522
* https://tracker.ceph.com/issues/61243
523
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
524
* https://tracker.ceph.com/issues/62188
525
    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
526
* https://tracker.ceph.com/issues/62510
527
    snaptest-git-ceph.sh failure with fs/thrash
528
* https://tracker.ceph.com/issues/62511
529
    src/mds/MDLog.cc: 299: FAILED ceph_assert(!mds_is_shutting_down)
530 165 Venky Shankar
531
532
h3. 14 Aug 2023
533
534
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230808.093601
535
536
* https://tracker.ceph.com/issues/51964
537
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
538
* https://tracker.ceph.com/issues/61400
539
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
540
* https://tracker.ceph.com/issues/61399
541
    ior build failure
542
* https://tracker.ceph.com/issues/59348
543
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
544
* https://tracker.ceph.com/issues/59531
545
    cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold
546
* https://tracker.ceph.com/issues/59344
547
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
548
* https://tracker.ceph.com/issues/59346
549
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
550
* https://tracker.ceph.com/issues/61399
551
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
552
* https://tracker.ceph.com/issues/59684 [kclient bug]
553
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
554
* https://tracker.ceph.com/issues/61243 (NEW)
555
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
556
* https://tracker.ceph.com/issues/57655
557
    qa: fs:mixed-clients kernel_untar_build failure
558
* https://tracker.ceph.com/issues/57656
559
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
560 163 Venky Shankar
561
562
h3. 28 JULY 2023
563
564
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230725.053049
565
566
* https://tracker.ceph.com/issues/51964
567
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
568
* https://tracker.ceph.com/issues/61400
569
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
570
* https://tracker.ceph.com/issues/61399
571
    ior build failure
572
* https://tracker.ceph.com/issues/57676
573
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
574
* https://tracker.ceph.com/issues/59348
575
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
576
* https://tracker.ceph.com/issues/59531
577
    cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold
578
* https://tracker.ceph.com/issues/59344
579
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
580
* https://tracker.ceph.com/issues/59346
581
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
582
* https://github.com/ceph/ceph/pull/52556
583
    task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' (see note #4)
584
* https://tracker.ceph.com/issues/62187
585
    iozone: command not found
586
* https://tracker.ceph.com/issues/61399
587
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
588
* https://tracker.ceph.com/issues/62188
589 164 Rishabh Dave
    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
590 158 Rishabh Dave
591
h3. 24 Jul 2023
592
593
https://pulpito.ceph.com/rishabh-2023-07-13_21:35:13-fs-wip-rishabh-2023Jul13-testing-default-smithi/
594
https://pulpito.ceph.com/rishabh-2023-07-14_10:26:42-fs-wip-rishabh-2023Jul13-testing-default-smithi/
595
There were few failure from one of the PRs under testing. Following run confirms that removing this PR fixes these failures -
596
https://pulpito.ceph.com/rishabh-2023-07-18_02:11:50-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/
597
One more extra run to check if blogbench.sh fail every time:
598
https://pulpito.ceph.com/rishabh-2023-07-21_17:58:19-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/
599
blogbench.sh failure were seen on above runs for first time, following run with main branch that confirms that "blogbench.sh" was not related to any of the PRs that are under testing -
600 161 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-07-21_21:30:53-fs-wip-rishabh-2023Jul13-base-2-testing-default-smithi/
601
602
* https://tracker.ceph.com/issues/61892
603
  test_snapshot_remove (test_strays.TestStrays) failed
604
* https://tracker.ceph.com/issues/53859
605
  test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
606
* https://tracker.ceph.com/issues/61982
607
  test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
608
* https://tracker.ceph.com/issues/52438
609
  qa: ffsb timeout
610
* https://tracker.ceph.com/issues/54460
611
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
612
* https://tracker.ceph.com/issues/57655
613
  qa: fs:mixed-clients kernel_untar_build failure
614
* https://tracker.ceph.com/issues/48773
615
  reached max tries: scrub does not complete
616
* https://tracker.ceph.com/issues/58340
617
  mds: fsstress.sh hangs with multimds
618
* https://tracker.ceph.com/issues/61400
619
  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
620
* https://tracker.ceph.com/issues/57206
621
  libcephfs/test.sh: ceph_test_libcephfs_reclaim
622
  
623
* https://tracker.ceph.com/issues/57656
624
  [testing] dbench: write failed on handle 10010 (Resource temporarily unavailable)
625
* https://tracker.ceph.com/issues/61399
626
  ior build failure
627
* https://tracker.ceph.com/issues/57676
628
  error during scrub thrashing: backtrace
629
  
630
* https://tracker.ceph.com/issues/38452
631
  'sudo -u postgres -- pgbench -s 500 -i' failed
632 158 Rishabh Dave
* https://tracker.ceph.com/issues/62126
633 157 Venky Shankar
  blogbench.sh failure
634
635
h3. 18 July 2023
636
637
* https://tracker.ceph.com/issues/52624
638
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
639
* https://tracker.ceph.com/issues/57676
640
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
641
* https://tracker.ceph.com/issues/54460
642
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
643
* https://tracker.ceph.com/issues/57655
644
    qa: fs:mixed-clients kernel_untar_build failure
645
* https://tracker.ceph.com/issues/51964
646
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
647
* https://tracker.ceph.com/issues/59344
648
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
649
* https://tracker.ceph.com/issues/61182
650
    cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds
651
* https://tracker.ceph.com/issues/61957
652
    test_client_limits.TestClientLimits.test_client_release_bug
653
* https://tracker.ceph.com/issues/59348
654
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
655
* https://tracker.ceph.com/issues/61892
656
    test_strays.TestStrays.test_snapshot_remove failed
657
* https://tracker.ceph.com/issues/59346
658
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
659
* https://tracker.ceph.com/issues/44565
660
    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())
661
* https://tracker.ceph.com/issues/62067
662
    ffsb.sh failure "Resource temporarily unavailable"
663 156 Venky Shankar
664
665
h3. 17 July 2023
666
667
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230704.040136
668
669
* https://tracker.ceph.com/issues/61982
670
    Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
671
* https://tracker.ceph.com/issues/59344
672
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
673
* https://tracker.ceph.com/issues/61182
674
    cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds
675
* https://tracker.ceph.com/issues/61957
676
    test_client_limits.TestClientLimits.test_client_release_bug
677
* https://tracker.ceph.com/issues/61400
678
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
679
* https://tracker.ceph.com/issues/59348
680
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
681
* https://tracker.ceph.com/issues/61892
682
    test_strays.TestStrays.test_snapshot_remove failed
683
* https://tracker.ceph.com/issues/59346
684
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
685
* https://tracker.ceph.com/issues/62036
686
    src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty())
687
* https://tracker.ceph.com/issues/61737
688
    coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific'
689
* https://tracker.ceph.com/issues/44565
690
    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())
691 155 Rishabh Dave
692 1 Patrick Donnelly
693 153 Rishabh Dave
h3. 13 July 2023 Run 2
694 152 Rishabh Dave
695
696
https://pulpito.ceph.com/rishabh-2023-07-08_23:33:40-fs-wip-rishabh-2023Jul9-testing-default-smithi/
697
https://pulpito.ceph.com/rishabh-2023-07-09_20:19:09-fs-wip-rishabh-2023Jul9-testing-default-smithi/
698
699
* https://tracker.ceph.com/issues/61957
700
  test_client_limits.TestClientLimits.test_client_release_bug
701
* https://tracker.ceph.com/issues/61982
702
  Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
703
* https://tracker.ceph.com/issues/59348
704
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
705
* https://tracker.ceph.com/issues/59344
706
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
707
* https://tracker.ceph.com/issues/54460
708
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
709
* https://tracker.ceph.com/issues/57655
710
  qa: fs:mixed-clients kernel_untar_build failure
711
* https://tracker.ceph.com/issues/61400
712
  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
713
* https://tracker.ceph.com/issues/61399
714
  ior build failure
715
716 151 Venky Shankar
h3. 13 July 2023
717
718
https://pulpito.ceph.com/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/
719
720
* https://tracker.ceph.com/issues/54460
721
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
722
* https://tracker.ceph.com/issues/61400
723
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
724
* https://tracker.ceph.com/issues/57655
725
    qa: fs:mixed-clients kernel_untar_build failure
726
* https://tracker.ceph.com/issues/61945
727
    LibCephFS.DelegTimeout failure
728
* https://tracker.ceph.com/issues/52624
729
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
730
* https://tracker.ceph.com/issues/57676
731
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
732
* https://tracker.ceph.com/issues/59348
733
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
734
* https://tracker.ceph.com/issues/59344
735
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
736
* https://tracker.ceph.com/issues/51964
737
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
738
* https://tracker.ceph.com/issues/59346
739
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
740
* https://tracker.ceph.com/issues/61982
741
    Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
742 150 Rishabh Dave
743
744
h3. 13 Jul 2023
745
746
https://pulpito.ceph.com/rishabh-2023-07-05_22:21:20-fs-wip-rishabh-2023Jul5-testing-default-smithi/
747
https://pulpito.ceph.com/rishabh-2023-07-06_19:33:28-fs-wip-rishabh-2023Jul5-testing-default-smithi/
748
749
* https://tracker.ceph.com/issues/61957
750
  test_client_limits.TestClientLimits.test_client_release_bug
751
* https://tracker.ceph.com/issues/59348
752
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
753
* https://tracker.ceph.com/issues/59346
754
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
755
* https://tracker.ceph.com/issues/48773
756
  scrub does not complete: reached max tries
757
* https://tracker.ceph.com/issues/59344
758
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
759
* https://tracker.ceph.com/issues/52438
760
  qa: ffsb timeout
761
* https://tracker.ceph.com/issues/57656
762
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
763
* https://tracker.ceph.com/issues/58742
764
  xfstests-dev: kcephfs: generic
765
* https://tracker.ceph.com/issues/61399
766 148 Rishabh Dave
  libmpich: undefined references to fi_strerror
767 149 Rishabh Dave
768 148 Rishabh Dave
h3. 12 July 2023
769
770
https://pulpito.ceph.com/rishabh-2023-07-05_18:32:52-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/
771
https://pulpito.ceph.com/rishabh-2023-07-06_19:46:43-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/
772
773
* https://tracker.ceph.com/issues/61892
774
  test_strays.TestStrays.test_snapshot_remove failed
775
* https://tracker.ceph.com/issues/59348
776
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
777
* https://tracker.ceph.com/issues/53859
778
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
779
* https://tracker.ceph.com/issues/59346
780
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
781
* https://tracker.ceph.com/issues/58742
782
  xfstests-dev: kcephfs: generic
783
* https://tracker.ceph.com/issues/59344
784
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
785
* https://tracker.ceph.com/issues/52438
786
  qa: ffsb timeout
787
* https://tracker.ceph.com/issues/57656
788
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
789
* https://tracker.ceph.com/issues/54460
790
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
791
* https://tracker.ceph.com/issues/57655
792
  qa: fs:mixed-clients kernel_untar_build failure
793
* https://tracker.ceph.com/issues/61182
794
  cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds
795
* https://tracker.ceph.com/issues/61400
796
  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
797 147 Rishabh Dave
* https://tracker.ceph.com/issues/48773
798 146 Patrick Donnelly
  reached max tries: scrub does not complete
799
800
h3. 05 July 2023
801
802
https://pulpito.ceph.com/pdonnell-2023-07-05_03:38:33-fs:libcephfs-wip-pdonnell-testing-20230705.003205-distro-default-smithi/
803
804 137 Rishabh Dave
* https://tracker.ceph.com/issues/59346
805 143 Rishabh Dave
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
806
807
h3. 27 Jun 2023
808
809
https://pulpito.ceph.com/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/
810 144 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/
811
812
* https://tracker.ceph.com/issues/59348
813
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
814
* https://tracker.ceph.com/issues/54460
815
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
816
* https://tracker.ceph.com/issues/59346
817
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
818
* https://tracker.ceph.com/issues/59344
819
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
820
* https://tracker.ceph.com/issues/61399
821
  libmpich: undefined references to fi_strerror
822
* https://tracker.ceph.com/issues/50223
823
  client.xxxx isn't responding to mclientcaps(revoke)
824 143 Rishabh Dave
* https://tracker.ceph.com/issues/61831
825
  Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
826 142 Venky Shankar
827
828
h3. 22 June 2023
829
830
* https://tracker.ceph.com/issues/57676
831
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
832
* https://tracker.ceph.com/issues/54460
833
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
834
* https://tracker.ceph.com/issues/59344
835
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
836
* https://tracker.ceph.com/issues/59348
837
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
838
* https://tracker.ceph.com/issues/61400
839
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
840
* https://tracker.ceph.com/issues/57655
841
    qa: fs:mixed-clients kernel_untar_build failure
842
* https://tracker.ceph.com/issues/61394
843
    qa/quincy: cluster [WRN] evicting unresponsive client smithi152 (4298), after 303.726 seconds" in cluster log
844
* https://tracker.ceph.com/issues/61762
845
    qa: wait_for_clean: failed before timeout expired
846
* https://tracker.ceph.com/issues/61775
847
    cephfs-mirror: mirror daemon does not shutdown (in mirror ha tests)
848
* https://tracker.ceph.com/issues/44565
849
    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())
850
* https://tracker.ceph.com/issues/61790
851
    cephfs client to mds comms remain silent after reconnect
852
* https://tracker.ceph.com/issues/61791
853
    snaptest-git-ceph.sh test timed out (job dead)
854 139 Venky Shankar
855
856
h3. 20 June 2023
857
858
https://pulpito.ceph.com/vshankar-2023-06-15_04:58:28-fs-wip-vshankar-testing-20230614.124123-testing-default-smithi/
859
860
* https://tracker.ceph.com/issues/57676
861
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
862
* https://tracker.ceph.com/issues/54460
863
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
864 140 Venky Shankar
* https://tracker.ceph.com/issues/54462
865 1 Patrick Donnelly
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
866 141 Venky Shankar
* https://tracker.ceph.com/issues/58340
867 139 Venky Shankar
  mds: fsstress.sh hangs with multimds
868
* https://tracker.ceph.com/issues/59344
869
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
870
* https://tracker.ceph.com/issues/59348
871
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
872
* https://tracker.ceph.com/issues/57656
873
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
874
* https://tracker.ceph.com/issues/61400
875
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
876
* https://tracker.ceph.com/issues/57655
877
    qa: fs:mixed-clients kernel_untar_build failure
878
* https://tracker.ceph.com/issues/44565
879
    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())
880
* https://tracker.ceph.com/issues/61737
881 138 Rishabh Dave
    coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific'
882
883
h3. 16 June 2023
884
885 1 Patrick Donnelly
https://pulpito.ceph.com/rishabh-2023-05-16_10:39:13-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
886 145 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-05-17_11:09:48-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
887 138 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-05-18_10:01:53-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
888 1 Patrick Donnelly
(bins were rebuilt with a subset of orig PRs) https://pulpito.ceph.com/rishabh-2023-06-09_10:19:22-fs-wip-rishabh-2023Jun9-1308-testing-default-smithi/
889
890
891
* https://tracker.ceph.com/issues/59344
892
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
893 138 Rishabh Dave
* https://tracker.ceph.com/issues/59348
894
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
895 145 Rishabh Dave
* https://tracker.ceph.com/issues/59346
896
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
897
* https://tracker.ceph.com/issues/57656
898
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
899
* https://tracker.ceph.com/issues/54460
900
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
901 138 Rishabh Dave
* https://tracker.ceph.com/issues/54462
902
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
903 145 Rishabh Dave
* https://tracker.ceph.com/issues/61399
904
  libmpich: undefined references to fi_strerror
905
* https://tracker.ceph.com/issues/58945
906
  xfstests-dev: ceph-fuse: generic 
907 138 Rishabh Dave
* https://tracker.ceph.com/issues/58742
908 136 Patrick Donnelly
  xfstests-dev: kcephfs: generic
909
910
h3. 24 May 2023
911
912
https://pulpito.ceph.com/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/
913
914
* https://tracker.ceph.com/issues/57676
915
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
916
* https://tracker.ceph.com/issues/59683
917
    Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests
918
* https://tracker.ceph.com/issues/61399
919
    qa: "[Makefile:299: ior] Error 1"
920
* https://tracker.ceph.com/issues/61265
921
    qa: tasks.cephfs.fuse_mount:process failed to terminate after unmount
922
* https://tracker.ceph.com/issues/59348
923
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
924
* https://tracker.ceph.com/issues/59346
925
    qa/workunits/fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
926
* https://tracker.ceph.com/issues/61400
927
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
928
* https://tracker.ceph.com/issues/54460
929
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
930
* https://tracker.ceph.com/issues/51964
931
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
932
* https://tracker.ceph.com/issues/59344
933
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
934
* https://tracker.ceph.com/issues/61407
935
    mds: abort on CInode::verify_dirfrags
936
* https://tracker.ceph.com/issues/48773
937
    qa: scrub does not complete
938
* https://tracker.ceph.com/issues/57655
939
    qa: fs:mixed-clients kernel_untar_build failure
940
* https://tracker.ceph.com/issues/61409
941 128 Venky Shankar
    qa: _test_stale_caps does not wait for file flush before stat
942
943
h3. 15 May 2023
944 130 Venky Shankar
945 128 Venky Shankar
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.090020
946
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.090020-6
947
948
* https://tracker.ceph.com/issues/52624
949
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
950
* https://tracker.ceph.com/issues/54460
951
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
952
* https://tracker.ceph.com/issues/57676
953
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
954
* https://tracker.ceph.com/issues/59684 [kclient bug]
955
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
956
* https://tracker.ceph.com/issues/59348
957
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
958 131 Venky Shankar
* https://tracker.ceph.com/issues/61148
959
    dbench test results in call trace in dmesg [kclient bug]
960 133 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/58340
961 134 Kotresh Hiremath Ravishankar
    mds: fsstress.sh hangs with multimds
962 125 Venky Shankar
963
 
964 129 Rishabh Dave
h3. 11 May 2023
965
966
https://pulpito.ceph.com/yuriw-2023-05-10_18:21:40-fs-wip-yuri7-testing-2023-05-10-0742-distro-default-smithi/
967
968
* https://tracker.ceph.com/issues/59684 [kclient bug]
969
  Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
970
* https://tracker.ceph.com/issues/59348
971
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
972
* https://tracker.ceph.com/issues/57655
973
  qa: fs:mixed-clients kernel_untar_build failure
974
* https://tracker.ceph.com/issues/57676
975
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
976
* https://tracker.ceph.com/issues/55805
977
  error during scrub thrashing reached max tries in 900 secs
978
* https://tracker.ceph.com/issues/54460
979
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
980
* https://tracker.ceph.com/issues/57656
981
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
982
* https://tracker.ceph.com/issues/58220
983
  Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:
984 1 Patrick Donnelly
* https://tracker.ceph.com/issues/58220#note-9
985
  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure
986 134 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/59342
987
  qa/workunits/kernel_untar_build.sh failed when compiling the Linux source
988 135 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/58949
989
    test_cephfs.test_disk_quota_exceeeded_error - AssertionError: DiskQuotaExceeded not raised by write
990 129 Rishabh Dave
* https://tracker.ceph.com/issues/61243 (NEW)
991
  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
992
993 125 Venky Shankar
h3. 11 May 2023
994 127 Venky Shankar
995
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.054005
996 126 Venky Shankar
997 125 Venky Shankar
(no fsstress job failure [https://tracker.ceph.com/issues/58340] since https://github.com/ceph/ceph/pull/49553
998
 was included in the branch, however, the PR got updated and needs retest).
999
1000
* https://tracker.ceph.com/issues/52624
1001
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1002
* https://tracker.ceph.com/issues/54460
1003
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1004
* https://tracker.ceph.com/issues/57676
1005
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1006
* https://tracker.ceph.com/issues/59683
1007
    Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests
1008
* https://tracker.ceph.com/issues/59684 [kclient bug]
1009
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
1010
* https://tracker.ceph.com/issues/59348
1011 124 Venky Shankar
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1012
1013
h3. 09 May 2023
1014
1015
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230506.143554
1016
1017
* https://tracker.ceph.com/issues/52624
1018
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1019
* https://tracker.ceph.com/issues/58340
1020
    mds: fsstress.sh hangs with multimds
1021
* https://tracker.ceph.com/issues/54460
1022
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1023
* https://tracker.ceph.com/issues/57676
1024
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1025
* https://tracker.ceph.com/issues/51964
1026
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1027
* https://tracker.ceph.com/issues/59350
1028
    qa: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ERROR
1029
* https://tracker.ceph.com/issues/59683
1030
    Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests
1031
* https://tracker.ceph.com/issues/59684 [kclient bug]
1032
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
1033
* https://tracker.ceph.com/issues/59348
1034 123 Venky Shankar
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1035
1036
h3. 10 Apr 2023
1037
1038
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230330.105356
1039
1040
* https://tracker.ceph.com/issues/52624
1041
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1042
* https://tracker.ceph.com/issues/58340
1043
    mds: fsstress.sh hangs with multimds
1044
* https://tracker.ceph.com/issues/54460
1045
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1046
* https://tracker.ceph.com/issues/57676
1047
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1048 119 Rishabh Dave
* https://tracker.ceph.com/issues/51964
1049 120 Rishabh Dave
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1050 121 Rishabh Dave
1051 120 Rishabh Dave
h3. 31 Mar 2023
1052 122 Rishabh Dave
1053
run: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/
1054 120 Rishabh Dave
re-run (some PRs removed from batch): http://pulpito.front.sepia.ceph.com/rishabh-2023-03-11_05:54:03-fs-wip-rishabh-2023Mar10-1727-testing-default-smithi/
1055
re-run (some PRs removed from batch): http://pulpito.front.sepia.ceph.com/rishabh-2023-03-23_08:27:28-fs-wip-rishabh-2023Mar20-2250-testing-default-smithi/
1056
1057
There were many more re-runs for "failed+dead" jobs as well as for individual jobs. half of the PRs from the batch were removed (gradually over subsequent re-runs).
1058
1059
* https://tracker.ceph.com/issues/57676
1060
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
1061
* https://tracker.ceph.com/issues/54460
1062
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1063
* https://tracker.ceph.com/issues/58220
1064
  Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
1065
* https://tracker.ceph.com/issues/58220#note-9
1066
  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure
1067
* https://tracker.ceph.com/issues/56695
1068
  Command failed (workunit test suites/pjd.sh)
1069
* https://tracker.ceph.com/issues/58564 
1070
  workuit dbench failed with error code 1
1071
* https://tracker.ceph.com/issues/57206
1072
  libcephfs/test.sh: ceph_test_libcephfs_reclaim
1073
* https://tracker.ceph.com/issues/57580
1074
  Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
1075
* https://tracker.ceph.com/issues/58940
1076
  ceph osd hit ceph_abort
1077
* https://tracker.ceph.com/issues/55805
1078 118 Venky Shankar
  error scrub thrashing reached max tries in 900 secs
1079
1080
h3. 30 March 2023
1081
1082
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230315.085747
1083
1084
* https://tracker.ceph.com/issues/58938
1085
    qa: xfstests-dev's generic test suite has 7 failures with kclient
1086
* https://tracker.ceph.com/issues/51964
1087
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1088
* https://tracker.ceph.com/issues/58340
1089 114 Venky Shankar
    mds: fsstress.sh hangs with multimds
1090
1091 115 Venky Shankar
h3. 29 March 2023
1092 114 Venky Shankar
1093
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230317.095222
1094
1095
* https://tracker.ceph.com/issues/56695
1096
    [RHEL stock] pjd test failures
1097
* https://tracker.ceph.com/issues/57676
1098
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1099
* https://tracker.ceph.com/issues/57087
1100
    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure
1101 116 Venky Shankar
* https://tracker.ceph.com/issues/58340
1102
    mds: fsstress.sh hangs with multimds
1103 114 Venky Shankar
* https://tracker.ceph.com/issues/57655
1104
    qa: fs:mixed-clients kernel_untar_build failure
1105 117 Venky Shankar
* https://tracker.ceph.com/issues/59230
1106
    Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage)
1107 114 Venky Shankar
* https://tracker.ceph.com/issues/58938
1108 113 Venky Shankar
    qa: xfstests-dev's generic test suite has 7 failures with kclient
1109
1110
h3. 13 Mar 2023
1111
1112
* https://tracker.ceph.com/issues/56695
1113
    [RHEL stock] pjd test failures
1114
* https://tracker.ceph.com/issues/57676
1115
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1116
* https://tracker.ceph.com/issues/51964
1117
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1118
* https://tracker.ceph.com/issues/54460
1119
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1120
* https://tracker.ceph.com/issues/57656
1121 112 Venky Shankar
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1122
1123
h3. 09 Mar 2023
1124
1125
https://pulpito.ceph.com/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/
1126
https://pulpito.ceph.com/vshankar-2023-03-08_15:12:36-fs-wip-vshankar-testing-20230308.112059-testing-default-smithi/
1127
1128
* https://tracker.ceph.com/issues/56695
1129
    [RHEL stock] pjd test failures
1130
* https://tracker.ceph.com/issues/57676
1131
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1132
* https://tracker.ceph.com/issues/51964
1133
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1134
* https://tracker.ceph.com/issues/54460
1135
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1136
* https://tracker.ceph.com/issues/58340
1137
    mds: fsstress.sh hangs with multimds
1138
* https://tracker.ceph.com/issues/57087
1139 111 Venky Shankar
    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure
1140
1141
h3. 07 Mar 2023
1142
1143
https://pulpito.ceph.com/vshankar-2023-03-02_09:21:58-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/
1144
https://pulpito.ceph.com/vshankar-2023-03-07_05:15:12-fs-wip-vshankar-testing-20230307.030510-testing-default-smithi/
1145
1146
* https://tracker.ceph.com/issues/56695
1147
    [RHEL stock] pjd test failures
1148
* https://tracker.ceph.com/issues/57676
1149
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1150
* https://tracker.ceph.com/issues/51964
1151
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1152
* https://tracker.ceph.com/issues/57656
1153
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1154
* https://tracker.ceph.com/issues/57655
1155
    qa: fs:mixed-clients kernel_untar_build failure
1156
* https://tracker.ceph.com/issues/58220
1157
    Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:
1158
* https://tracker.ceph.com/issues/54460
1159
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1160
* https://tracker.ceph.com/issues/58934
1161 109 Venky Shankar
    snaptest-git-ceph.sh failure with ceph-fuse
1162
1163
h3. 28 Feb 2023
1164
1165
https://pulpito.ceph.com/vshankar-2023-02-24_02:11:45-fs-wip-vshankar-testing-20230222.025426-testing-default-smithi/
1166
1167
* https://tracker.ceph.com/issues/56695
1168
    [RHEL stock] pjd test failures
1169
* https://tracker.ceph.com/issues/57676
1170
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1171 110 Venky Shankar
* https://tracker.ceph.com/issues/56446
1172 109 Venky Shankar
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1173
1174 107 Venky Shankar
(teuthology infra issues causing testing delays - merging PRs which have tests passing)
1175
1176
h3. 25 Jan 2023
1177
1178
https://pulpito.ceph.com/vshankar-2023-01-25_07:57:32-fs-wip-vshankar-testing-20230125.055346-testing-default-smithi/
1179
1180
* https://tracker.ceph.com/issues/52624
1181
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1182
* https://tracker.ceph.com/issues/56695
1183
    [RHEL stock] pjd test failures
1184
* https://tracker.ceph.com/issues/57676
1185
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1186
* https://tracker.ceph.com/issues/56446
1187
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1188
* https://tracker.ceph.com/issues/57206
1189
    libcephfs/test.sh: ceph_test_libcephfs_reclaim
1190
* https://tracker.ceph.com/issues/58220
1191
    Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:
1192
* https://tracker.ceph.com/issues/58340
1193
  mds: fsstress.sh hangs with multimds
1194
* https://tracker.ceph.com/issues/56011
1195
    fs/thrash: snaptest-snap-rm-cmp.sh fails in mds5sum comparison
1196
* https://tracker.ceph.com/issues/54460
1197 101 Rishabh Dave
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1198
1199
h3. 30 JAN 2023
1200
1201
run: http://pulpito.front.sepia.ceph.com/rishabh-2022-11-28_08:04:11-fs-wip-rishabh-testing-2022Nov24-1818-testing-default-smithi/
1202
re-run: http://pulpito.front.sepia.ceph.com/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/
1203 105 Rishabh Dave
re-run of re-run: http://pulpito.front.sepia.ceph.com/rishabh-2023-01-23_18:53:32-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/
1204
1205 101 Rishabh Dave
* https://tracker.ceph.com/issues/52624
1206
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1207
* https://tracker.ceph.com/issues/56695
1208
  [RHEL stock] pjd test failures
1209
* https://tracker.ceph.com/issues/57676
1210
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
1211
* https://tracker.ceph.com/issues/55332
1212
  Failure in snaptest-git-ceph.sh
1213
* https://tracker.ceph.com/issues/51964
1214
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1215
* https://tracker.ceph.com/issues/56446
1216
  Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1217
* https://tracker.ceph.com/issues/57655 
1218
  qa: fs:mixed-clients kernel_untar_build failure
1219
* https://tracker.ceph.com/issues/54460
1220
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1221 103 Rishabh Dave
* https://tracker.ceph.com/issues/58340
1222
  mds: fsstress.sh hangs with multimds
1223 101 Rishabh Dave
* https://tracker.ceph.com/issues/58219
1224 102 Rishabh Dave
  Command crashed: 'ceph-dencoder type inode_backtrace_t import - decode dump_json'
1225
1226
* "Failed to load ceph-mgr modules: prometheus" in cluster log"
1227 106 Rishabh Dave
  http://pulpito.front.sepia.ceph.com/rishabh-2023-01-23_18:53:32-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7134086
1228
  Acc to Venky this was fixed in https://github.com/ceph/ceph/commit/cf6089200d96fc56b08ee17a4e31f19823370dc8
1229 102 Rishabh Dave
* Created https://tracker.ceph.com/issues/58564
1230 100 Venky Shankar
  workunit test suites/dbench.sh failed error code 1
1231
1232
h3. 15 Dec 2022
1233
1234
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221215.112736
1235
1236
* https://tracker.ceph.com/issues/52624
1237
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1238
* https://tracker.ceph.com/issues/56695
1239
    [RHEL stock] pjd test failures
1240
* https://tracker.ceph.com/issues/58219
1241
* https://tracker.ceph.com/issues/57655
1242
* qa: fs:mixed-clients kernel_untar_build failure
1243
    Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
1244
* https://tracker.ceph.com/issues/57676
1245
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1246
* https://tracker.ceph.com/issues/58340
1247 96 Venky Shankar
    mds: fsstress.sh hangs with multimds
1248
1249
h3. 08 Dec 2022
1250 99 Venky Shankar
1251 96 Venky Shankar
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221130.043104
1252
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221209.043803
1253
1254
(lots of transient git.ceph.com failures)
1255
1256
* https://tracker.ceph.com/issues/52624
1257
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1258
* https://tracker.ceph.com/issues/56695
1259
    [RHEL stock] pjd test failures
1260
* https://tracker.ceph.com/issues/57655
1261
    qa: fs:mixed-clients kernel_untar_build failure
1262
* https://tracker.ceph.com/issues/58219
1263
    Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
1264
* https://tracker.ceph.com/issues/58220
1265
    Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:
1266 97 Venky Shankar
* https://tracker.ceph.com/issues/57676
1267
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1268 98 Venky Shankar
* https://tracker.ceph.com/issues/53859
1269
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1270
* https://tracker.ceph.com/issues/54460
1271
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1272 96 Venky Shankar
* https://tracker.ceph.com/issues/58244
1273 95 Venky Shankar
    Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan)
1274
1275
h3. 14 Oct 2022
1276
1277
https://pulpito.ceph.com/vshankar-2022-10-12_04:56:59-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/
1278
https://pulpito.ceph.com/vshankar-2022-10-14_04:04:57-fs-wip-vshankar-testing-20221014-072608-testing-default-smithi/
1279
1280
* https://tracker.ceph.com/issues/52624
1281
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1282
* https://tracker.ceph.com/issues/55804
1283
    Command failed (workunit test suites/pjd.sh)
1284
* https://tracker.ceph.com/issues/51964
1285
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1286
* https://tracker.ceph.com/issues/57682
1287
    client: ERROR: test_reconnect_after_blocklisted
1288 90 Rishabh Dave
* https://tracker.ceph.com/issues/54460
1289 91 Rishabh Dave
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1290
1291
h3. 10 Oct 2022
1292 92 Rishabh Dave
1293 91 Rishabh Dave
http://pulpito.front.sepia.ceph.com/rishabh-2022-09-30_19:45:21-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
1294
1295
reruns
1296
* fs-thrash, passed: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-04_13:19:47-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
1297 94 Rishabh Dave
* fs-verify, passed: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-05_12:25:37-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
1298 91 Rishabh Dave
* cephadm failures also passed after many re-runs: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-06_13:50:51-fs-wip-rishabh-testing-30Sep2022-2-testing-default-smithi/
1299 93 Rishabh Dave
    ** needed this PR to be merged in ceph-ci branch - https://github.com/ceph/ceph/pull/47458
1300 91 Rishabh Dave
1301
known bugs
1302
* https://tracker.ceph.com/issues/52624
1303
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1304
* https://tracker.ceph.com/issues/50223
1305
  client.xxxx isn't responding to mclientcaps(revoke
1306
* https://tracker.ceph.com/issues/57299
1307
  qa: test_dump_loads fails with JSONDecodeError
1308
* https://tracker.ceph.com/issues/57655 [Exist in main as well]
1309
  qa: fs:mixed-clients kernel_untar_build failure
1310
* https://tracker.ceph.com/issues/57206
1311 90 Rishabh Dave
  libcephfs/test.sh: ceph_test_libcephfs_reclaim
1312
1313
h3. 2022 Sep 29
1314
1315
http://pulpito.front.sepia.ceph.com/rishabh-2022-09-14_12:48:43-fs-wip-rishabh-testing-2022Sep9-1708-testing-default-smithi/
1316
1317
* https://tracker.ceph.com/issues/55804
1318
  Command failed (workunit test suites/pjd.sh)
1319
* https://tracker.ceph.com/issues/36593
1320
  Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
1321
* https://tracker.ceph.com/issues/52624
1322
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1323
* https://tracker.ceph.com/issues/51964
1324
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1325
* https://tracker.ceph.com/issues/56632
1326
  Test failure: test_subvolume_snapshot_clone_quota_exceeded
1327
* https://tracker.ceph.com/issues/50821
1328 88 Patrick Donnelly
  qa: untar_snap_rm failure during mds thrashing
1329
1330
h3. 2022 Sep 26
1331
1332
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220923.171109
1333
1334
* https://tracker.ceph.com/issues/55804
1335
    qa failure: pjd link tests failed
1336
* https://tracker.ceph.com/issues/57676
1337
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1338
* https://tracker.ceph.com/issues/52624
1339
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1340
* https://tracker.ceph.com/issues/57580
1341
    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
1342
* https://tracker.ceph.com/issues/48773
1343
    qa: scrub does not complete
1344
* https://tracker.ceph.com/issues/57299
1345
    qa: test_dump_loads fails with JSONDecodeError
1346
* https://tracker.ceph.com/issues/57280
1347
    qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman
1348
* https://tracker.ceph.com/issues/57205
1349
    Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
1350
* https://tracker.ceph.com/issues/57656
1351
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1352
* https://tracker.ceph.com/issues/57677
1353
    qa: "1 MDSs behind on trimming (MDS_TRIM)"
1354
* https://tracker.ceph.com/issues/57206
1355
    libcephfs/test.sh: ceph_test_libcephfs_reclaim
1356
* https://tracker.ceph.com/issues/57446
1357
    qa: test_subvolume_snapshot_info_if_orphan_clone fails
1358 89 Patrick Donnelly
* https://tracker.ceph.com/issues/57655 [Exist in main as well]
1359
    qa: fs:mixed-clients kernel_untar_build failure
1360 88 Patrick Donnelly
* https://tracker.ceph.com/issues/57682
1361
    client: ERROR: test_reconnect_after_blocklisted
1362 87 Patrick Donnelly
1363
1364
h3. 2022 Sep 22
1365
1366
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220920.234701
1367
1368
* https://tracker.ceph.com/issues/57299
1369
    qa: test_dump_loads fails with JSONDecodeError
1370
* https://tracker.ceph.com/issues/57205
1371
    Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
1372
* https://tracker.ceph.com/issues/52624
1373
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1374
* https://tracker.ceph.com/issues/57580
1375
    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
1376
* https://tracker.ceph.com/issues/57280
1377
    qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman
1378
* https://tracker.ceph.com/issues/48773
1379
    qa: scrub does not complete
1380
* https://tracker.ceph.com/issues/56446
1381
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1382
* https://tracker.ceph.com/issues/57206
1383
    libcephfs/test.sh: ceph_test_libcephfs_reclaim
1384
* https://tracker.ceph.com/issues/51267
1385
    CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:...
1386
1387
NEW:
1388
1389
* https://tracker.ceph.com/issues/57656
1390
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1391
* https://tracker.ceph.com/issues/57655 [Exist in main as well]
1392
    qa: fs:mixed-clients kernel_untar_build failure
1393
* https://tracker.ceph.com/issues/57657
1394
    mds: scrub locates mismatch between child accounted_rstats and self rstats
1395
1396
Segfault probably caused by: https://github.com/ceph/ceph/pull/47795#issuecomment-1255724799
1397 80 Venky Shankar
1398 79 Venky Shankar
1399
h3. 2022 Sep 16
1400
1401
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220905-132828
1402
1403
* https://tracker.ceph.com/issues/57446
1404
    qa: test_subvolume_snapshot_info_if_orphan_clone fails
1405
* https://tracker.ceph.com/issues/57299
1406
    qa: test_dump_loads fails with JSONDecodeError
1407
* https://tracker.ceph.com/issues/50223
1408
    client.xxxx isn't responding to mclientcaps(revoke)
1409
* https://tracker.ceph.com/issues/52624
1410
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1411
* https://tracker.ceph.com/issues/57205
1412
    Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
1413
* https://tracker.ceph.com/issues/57280
1414
    qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman
1415
* https://tracker.ceph.com/issues/51282
1416
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
1417
* https://tracker.ceph.com/issues/48203
1418
  https://tracker.ceph.com/issues/36593
1419
    qa: quota failure
1420
    qa: quota failure caused by clients stepping on each other
1421
* https://tracker.ceph.com/issues/57580
1422 77 Rishabh Dave
    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
1423
1424 76 Rishabh Dave
1425
h3. 2022 Aug 26
1426
1427
http://pulpito.front.sepia.ceph.com/rishabh-2022-08-22_17:49:59-fs-wip-rishabh-testing-2022Aug19-testing-default-smithi/
1428
http://pulpito.front.sepia.ceph.com/rishabh-2022-08-24_11:56:51-fs-wip-rishabh-testing-2022Aug19-testing-default-smithi/
1429
1430
* https://tracker.ceph.com/issues/57206
1431
  libcephfs/test.sh: ceph_test_libcephfs_reclaim
1432
* https://tracker.ceph.com/issues/56632
1433
  Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
1434
* https://tracker.ceph.com/issues/56446
1435
  Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1436
* https://tracker.ceph.com/issues/51964
1437
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1438
* https://tracker.ceph.com/issues/53859
1439
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1440
1441
* https://tracker.ceph.com/issues/54460
1442
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1443
* https://tracker.ceph.com/issues/54462
1444
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
1445
* https://tracker.ceph.com/issues/54460
1446
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1447
* https://tracker.ceph.com/issues/36593
1448
  Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
1449
1450
* https://tracker.ceph.com/issues/52624
1451
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1452
* https://tracker.ceph.com/issues/55804
1453
  Command failed (workunit test suites/pjd.sh)
1454
* https://tracker.ceph.com/issues/50223
1455
  client.xxxx isn't responding to mclientcaps(revoke)
1456 75 Venky Shankar
1457
1458
h3. 2022 Aug 22
1459
1460
https://pulpito.ceph.com/vshankar-2022-08-12_09:34:24-fs-wip-vshankar-testing1-20220812-072441-testing-default-smithi/
1461
https://pulpito.ceph.com/vshankar-2022-08-18_04:30:42-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/ (drop problematic PR and re-run)
1462
1463
* https://tracker.ceph.com/issues/52624
1464
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1465
* https://tracker.ceph.com/issues/56446
1466
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1467
* https://tracker.ceph.com/issues/55804
1468
    Command failed (workunit test suites/pjd.sh)
1469
* https://tracker.ceph.com/issues/51278
1470
    mds: "FAILED ceph_assert(!segments.empty())"
1471
* https://tracker.ceph.com/issues/54460
1472
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1473
* https://tracker.ceph.com/issues/57205
1474
    Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
1475
* https://tracker.ceph.com/issues/57206
1476
    ceph_test_libcephfs_reclaim crashes during test
1477
* https://tracker.ceph.com/issues/53859
1478
  Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1479
* https://tracker.ceph.com/issues/50223
1480 72 Venky Shankar
    client.xxxx isn't responding to mclientcaps(revoke)
1481
1482
h3. 2022 Aug 12
1483
1484
https://pulpito.ceph.com/vshankar-2022-08-10_04:06:00-fs-wip-vshankar-testing-20220805-190751-testing-default-smithi/
1485
https://pulpito.ceph.com/vshankar-2022-08-11_12:16:58-fs-wip-vshankar-testing-20220811-145809-testing-default-smithi/ (drop problematic PR and re-run)
1486
1487
* https://tracker.ceph.com/issues/52624
1488
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1489
* https://tracker.ceph.com/issues/56446
1490
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1491
* https://tracker.ceph.com/issues/51964
1492
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1493
* https://tracker.ceph.com/issues/55804
1494
    Command failed (workunit test suites/pjd.sh)
1495
* https://tracker.ceph.com/issues/50223
1496
    client.xxxx isn't responding to mclientcaps(revoke)
1497
* https://tracker.ceph.com/issues/50821
1498 73 Venky Shankar
    qa: untar_snap_rm failure during mds thrashing
1499 72 Venky Shankar
* https://tracker.ceph.com/issues/54460
1500 71 Venky Shankar
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1501
1502
h3. 2022 Aug 04
1503
1504
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220804-123835 (only mgr/volumes, mgr/stats)
1505
1506 69 Rishabh Dave
Unrealted teuthology failure on rhel
1507 68 Rishabh Dave
1508
h3. 2022 Jul 25
1509
1510
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-22_11:34:20-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/
1511
1512 74 Rishabh Dave
1st re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_03:51:19-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi
1513
2nd re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_08:53:36-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/
1514 68 Rishabh Dave
3rd re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_08:53:36-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/
1515
4th (final) re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-28_03:59:01-fs-wip-rishabh-testing-2022Jul28-0143-testing-default-smithi/
1516
1517
* https://tracker.ceph.com/issues/55804
1518
  Command failed (workunit test suites/pjd.sh)
1519
* https://tracker.ceph.com/issues/50223
1520
  client.xxxx isn't responding to mclientcaps(revoke)
1521
1522
* https://tracker.ceph.com/issues/54460
1523
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1524 1 Patrick Donnelly
* https://tracker.ceph.com/issues/36593
1525 74 Rishabh Dave
  Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
1526 68 Rishabh Dave
* https://tracker.ceph.com/issues/54462
1527 67 Patrick Donnelly
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128~
1528
1529
h3. 2022 July 22
1530
1531
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220721.235756
1532
1533
MDS_HEALTH_DUMMY error in log fixed by followup commit.
1534
transient selinux ping failure
1535
1536
* https://tracker.ceph.com/issues/56694
1537
    qa: avoid blocking forever on hung umount
1538
* https://tracker.ceph.com/issues/56695
1539
    [RHEL stock] pjd test failures
1540
* https://tracker.ceph.com/issues/56696
1541
    admin keyring disappears during qa run
1542
* https://tracker.ceph.com/issues/56697
1543
    qa: fs/snaps fails for fuse
1544
* https://tracker.ceph.com/issues/50222
1545
    osd: 5.2s0 deep-scrub : stat mismatch
1546
* https://tracker.ceph.com/issues/56698
1547
    client: FAILED ceph_assert(_size == 0)
1548
* https://tracker.ceph.com/issues/50223
1549
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1550 66 Rishabh Dave
1551 65 Rishabh Dave
1552
h3. 2022 Jul 15
1553
1554
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-08_23:53:34-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/
1555
1556
re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-15_06:42:04-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/
1557
1558
* https://tracker.ceph.com/issues/53859
1559
  Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1560
* https://tracker.ceph.com/issues/55804
1561
  Command failed (workunit test suites/pjd.sh)
1562
* https://tracker.ceph.com/issues/50223
1563
  client.xxxx isn't responding to mclientcaps(revoke)
1564
* https://tracker.ceph.com/issues/50222
1565
  osd: deep-scrub : stat mismatch
1566
1567
* https://tracker.ceph.com/issues/56632
1568
  Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
1569
* https://tracker.ceph.com/issues/56634
1570
  workunit test fs/snaps/snaptest-intodir.sh
1571
* https://tracker.ceph.com/issues/56644
1572
  Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
1573
1574 61 Rishabh Dave
1575
1576
h3. 2022 July 05
1577 62 Rishabh Dave
1578 64 Rishabh Dave
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/
1579
1580
On 1st re-run some jobs passed - http://pulpito.front.sepia.ceph.com/rishabh-2022-07-03_15:10:28-fs-wip-rishabh-testing-20220702-1631-distro-default-smithi/
1581
1582
On 2nd re-run only few jobs failed -
1583 62 Rishabh Dave
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-06_05:24:29-fs-wip-rishabh-testing-20220705-2132-distro-default-smithi/
1584
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-06_05:24:29-fs-wip-rishabh-testing-20220705-2132-distro-default-smithi/
1585
1586
* https://tracker.ceph.com/issues/56446
1587
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1588
* https://tracker.ceph.com/issues/55804
1589
    Command failed (workunit test suites/pjd.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/
1590
1591
* https://tracker.ceph.com/issues/56445
1592 63 Rishabh Dave
    Command failed on smithi080 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"
1593
* https://tracker.ceph.com/issues/51267
1594
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
1595 62 Rishabh Dave
* https://tracker.ceph.com/issues/50224
1596
    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
1597 61 Rishabh Dave
1598 58 Venky Shankar
1599
1600
h3. 2022 July 04
1601
1602
https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/
1603
(rhel runs were borked due to: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/JSZQFUKVLDND4W33PXDGCABPHNSPT6SS/, tests ran with --filter-out=rhel)
1604
1605
* https://tracker.ceph.com/issues/56445
1606 59 Rishabh Dave
    Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"
1607
* https://tracker.ceph.com/issues/56446
1608
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1609
* https://tracker.ceph.com/issues/51964
1610 60 Rishabh Dave
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1611 59 Rishabh Dave
* https://tracker.ceph.com/issues/52624
1612 57 Venky Shankar
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1613
1614
h3. 2022 June 20
1615
1616
https://pulpito.ceph.com/vshankar-2022-06-15_04:03:39-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/
1617
https://pulpito.ceph.com/vshankar-2022-06-19_08:22:46-fs-wip-vshankar-testing1-20220619-102531-testing-default-smithi/
1618
1619
* https://tracker.ceph.com/issues/52624
1620
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1621
* https://tracker.ceph.com/issues/55804
1622
    qa failure: pjd link tests failed
1623
* https://tracker.ceph.com/issues/54108
1624
    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"
1625
* https://tracker.ceph.com/issues/55332
1626 56 Patrick Donnelly
    Failure in snaptest-git-ceph.sh (it's an async unlink/create bug)
1627
1628
h3. 2022 June 13
1629
1630
https://pulpito.ceph.com/pdonnell-2022-06-12_05:08:12-fs:workload-wip-pdonnell-testing-20220612.004943-distro-default-smithi/
1631
1632
* https://tracker.ceph.com/issues/56024
1633
    cephadm: removes ceph.conf during qa run causing command failure
1634
* https://tracker.ceph.com/issues/48773
1635
    qa: scrub does not complete
1636
* https://tracker.ceph.com/issues/56012
1637
    mds: src/mds/MDLog.cc: 283: FAILED ceph_assert(!mds->is_ any_replay())
1638 55 Venky Shankar
1639 54 Venky Shankar
1640
h3. 2022 Jun 13
1641
1642
https://pulpito.ceph.com/vshankar-2022-06-07_00:25:50-fs-wip-vshankar-testing-20220606-223254-testing-default-smithi/
1643
https://pulpito.ceph.com/vshankar-2022-06-10_01:04:46-fs-wip-vshankar-testing-20220609-175550-testing-default-smithi/
1644
1645
* https://tracker.ceph.com/issues/52624
1646
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1647
* https://tracker.ceph.com/issues/51964
1648
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1649
* https://tracker.ceph.com/issues/53859
1650
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1651
* https://tracker.ceph.com/issues/55804
1652
    qa failure: pjd link tests failed
1653
* https://tracker.ceph.com/issues/56003
1654
    client: src/include/xlist.h: 81: FAILED ceph_assert(_size == 0)
1655
* https://tracker.ceph.com/issues/56011
1656
    fs/thrash: snaptest-snap-rm-cmp.sh fails in mds5sum comparison
1657
* https://tracker.ceph.com/issues/56012
1658 53 Venky Shankar
    mds: src/mds/MDLog.cc: 283: FAILED ceph_assert(!mds->is_ any_replay())
1659
1660
h3. 2022 Jun 07
1661
1662
https://pulpito.ceph.com/vshankar-2022-06-06_21:25:41-fs-wip-vshankar-testing1-20220606-230129-testing-default-smithi/
1663
https://pulpito.ceph.com/vshankar-2022-06-07_10:53:31-fs-wip-vshankar-testing1-20220607-104134-testing-default-smithi/ (rerun after dropping a problematic PR)
1664
1665
* https://tracker.ceph.com/issues/52624
1666
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1667
* https://tracker.ceph.com/issues/50223
1668
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1669
* https://tracker.ceph.com/issues/50224
1670 51 Venky Shankar
    qa: test_mirroring_init_failure_with_recovery failure
1671
1672
h3. 2022 May 12
1673 52 Venky Shankar
1674 51 Venky Shankar
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20220509-125847
1675
https://pulpito.ceph.com/vshankar-2022-05-13_17:09:16-fs-wip-vshankar-testing-20220513-120051-testing-default-smithi/ (drop prs + rerun)
1676
1677
* https://tracker.ceph.com/issues/52624
1678
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1679
* https://tracker.ceph.com/issues/50223
1680
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1681
* https://tracker.ceph.com/issues/55332
1682
    Failure in snaptest-git-ceph.sh
1683
* https://tracker.ceph.com/issues/53859
1684 1 Patrick Donnelly
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1685 52 Venky Shankar
* https://tracker.ceph.com/issues/55538
1686
    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
1687 51 Venky Shankar
* https://tracker.ceph.com/issues/55258
1688 49 Venky Shankar
    lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs (cropss up again, though very infrequent)
1689
1690 50 Venky Shankar
h3. 2022 May 04
1691
1692
https://pulpito.ceph.com/vshankar-2022-05-01_13:18:44-fs-wip-vshankar-testing1-20220428-204527-testing-default-smithi/
1693 49 Venky Shankar
https://pulpito.ceph.com/vshankar-2022-05-02_16:58:59-fs-wip-vshankar-testing1-20220502-201957-testing-default-smithi/ (after dropping PRs)
1694
1695
* https://tracker.ceph.com/issues/52624
1696
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1697
* https://tracker.ceph.com/issues/50223
1698
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1699
* https://tracker.ceph.com/issues/55332
1700
    Failure in snaptest-git-ceph.sh
1701
* https://tracker.ceph.com/issues/53859
1702
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1703
* https://tracker.ceph.com/issues/55516
1704
    qa: fs suite tests failing with "json.decoder.JSONDecodeError: Extra data: line 2 column 82 (char 82)"
1705
* https://tracker.ceph.com/issues/55537
1706
    mds: crash during fs:upgrade test
1707
* https://tracker.ceph.com/issues/55538
1708 48 Venky Shankar
    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
1709
1710
h3. 2022 Apr 25
1711
1712
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20220420-113951 (owner vshankar)
1713
1714
* https://tracker.ceph.com/issues/52624
1715
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1716
* https://tracker.ceph.com/issues/50223
1717
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1718
* https://tracker.ceph.com/issues/55258
1719
    lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs
1720
* https://tracker.ceph.com/issues/55377
1721 47 Venky Shankar
    kclient: mds revoke Fwb caps stuck after the kclient tries writebcak once
1722
1723
h3. 2022 Apr 14
1724
1725
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220411-144044
1726
1727
* https://tracker.ceph.com/issues/52624
1728
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1729
* https://tracker.ceph.com/issues/50223
1730
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1731
* https://tracker.ceph.com/issues/52438
1732
    qa: ffsb timeout
1733
* https://tracker.ceph.com/issues/55170
1734
    mds: crash during rejoin (CDir::fetch_keys)
1735
* https://tracker.ceph.com/issues/55331
1736
    pjd failure
1737
* https://tracker.ceph.com/issues/48773
1738
    qa: scrub does not complete
1739
* https://tracker.ceph.com/issues/55332
1740
    Failure in snaptest-git-ceph.sh
1741
* https://tracker.ceph.com/issues/55258
1742 45 Venky Shankar
    lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs
1743
1744 46 Venky Shankar
h3. 2022 Apr 11
1745 45 Venky Shankar
1746
https://pulpito.ceph.com/?branch=wip-vshankar-testing-55110-20220408-203242
1747
1748
* https://tracker.ceph.com/issues/48773
1749
    qa: scrub does not complete
1750
* https://tracker.ceph.com/issues/52624
1751
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1752
* https://tracker.ceph.com/issues/52438
1753
    qa: ffsb timeout
1754
* https://tracker.ceph.com/issues/48680
1755
    mds: scrubbing stuck "scrub active (0 inodes in the stack)"
1756
* https://tracker.ceph.com/issues/55236
1757
    qa: fs/snaps tests fails with "hit max job timeout"
1758
* https://tracker.ceph.com/issues/54108
1759
    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"
1760
* https://tracker.ceph.com/issues/54971
1761
    Test failure: test_perf_stats_stale_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
1762
* https://tracker.ceph.com/issues/50223
1763
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1764
* https://tracker.ceph.com/issues/55258
1765 44 Venky Shankar
    lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs
1766 42 Venky Shankar
1767 43 Venky Shankar
h3. 2022 Mar 21
1768
1769
https://pulpito.ceph.com/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/
1770
1771
Run didn't go well, lots of failures - debugging by dropping PRs and running against master branch. Only merging unrelated PRs that pass tests.
1772
1773
1774 42 Venky Shankar
h3. 2022 Mar 08
1775
1776
https://pulpito.ceph.com/vshankar-2022-02-28_04:32:15-fs-wip-vshankar-testing-20220226-211550-testing-default-smithi/
1777
1778
rerun with
1779
- (drop) https://github.com/ceph/ceph/pull/44679
1780
- (drop) https://github.com/ceph/ceph/pull/44958
1781
https://pulpito.ceph.com/vshankar-2022-03-06_14:47:51-fs-wip-vshankar-testing-20220304-132102-testing-default-smithi/
1782
1783
* https://tracker.ceph.com/issues/54419 (new)
1784
    `ceph orch upgrade start` seems to never reach completion
1785
* https://tracker.ceph.com/issues/51964
1786
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1787
* https://tracker.ceph.com/issues/52624
1788
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1789
* https://tracker.ceph.com/issues/50223
1790
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1791
* https://tracker.ceph.com/issues/52438
1792
    qa: ffsb timeout
1793
* https://tracker.ceph.com/issues/50821
1794
    qa: untar_snap_rm failure during mds thrashing
1795 41 Venky Shankar
1796
1797
h3. 2022 Feb 09
1798
1799
https://pulpito.ceph.com/vshankar-2022-02-05_17:27:49-fs-wip-vshankar-testing-20220201-113815-testing-default-smithi/
1800
1801
rerun with
1802
- (drop) https://github.com/ceph/ceph/pull/37938
1803
- (drop) https://github.com/ceph/ceph/pull/44335
1804
- (drop) https://github.com/ceph/ceph/pull/44491
1805
- (drop) https://github.com/ceph/ceph/pull/44501
1806
https://pulpito.ceph.com/vshankar-2022-02-08_14:27:29-fs-wip-vshankar-testing-20220208-181241-testing-default-smithi/
1807
1808
* https://tracker.ceph.com/issues/51964
1809
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1810
* https://tracker.ceph.com/issues/54066
1811
    test_subvolume_no_upgrade_v1_sanity fails with `AssertionError: 1000 != 0`
1812
* https://tracker.ceph.com/issues/48773
1813
    qa: scrub does not complete
1814
* https://tracker.ceph.com/issues/52624
1815
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1816
* https://tracker.ceph.com/issues/50223
1817
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1818
* https://tracker.ceph.com/issues/52438
1819 40 Patrick Donnelly
    qa: ffsb timeout
1820
1821
h3. 2022 Feb 01
1822
1823
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220127.171526
1824
1825
* https://tracker.ceph.com/issues/54107
1826
    kclient: hang during umount
1827
* https://tracker.ceph.com/issues/54106
1828
    kclient: hang during workunit cleanup
1829
* https://tracker.ceph.com/issues/54108
1830
    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"
1831
* https://tracker.ceph.com/issues/48773
1832
    qa: scrub does not complete
1833
* https://tracker.ceph.com/issues/52438
1834
    qa: ffsb timeout
1835 36 Venky Shankar
1836
1837
h3. 2022 Jan 13
1838 39 Venky Shankar
1839 36 Venky Shankar
https://pulpito.ceph.com/vshankar-2022-01-06_13:18:41-fs-wip-vshankar-testing-20220106-145819-testing-default-smithi/
1840 38 Venky Shankar
1841
rerun with:
1842 36 Venky Shankar
- (add) https://github.com/ceph/ceph/pull/44570
1843
- (drop) https://github.com/ceph/ceph/pull/43184
1844
https://pulpito.ceph.com/vshankar-2022-01-13_04:42:40-fs-wip-vshankar-testing-20220106-145819-testing-default-smithi/
1845
1846
* https://tracker.ceph.com/issues/50223
1847
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1848
* https://tracker.ceph.com/issues/51282
1849
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
1850
* https://tracker.ceph.com/issues/48773
1851
    qa: scrub does not complete
1852
* https://tracker.ceph.com/issues/52624
1853
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1854
* https://tracker.ceph.com/issues/53859
1855 34 Venky Shankar
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1856
1857
h3. 2022 Jan 03
1858
1859
https://pulpito.ceph.com/vshankar-2021-12-22_07:37:44-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/
1860
https://pulpito.ceph.com/vshankar-2022-01-03_12:27:45-fs-wip-vshankar-testing-20220103-142738-testing-default-smithi/ (rerun)
1861
1862
* https://tracker.ceph.com/issues/50223
1863
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1864
* https://tracker.ceph.com/issues/51964
1865
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1866
* https://tracker.ceph.com/issues/51267
1867
    CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:...
1868
* https://tracker.ceph.com/issues/51282
1869
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
1870
* https://tracker.ceph.com/issues/50821
1871
    qa: untar_snap_rm failure during mds thrashing
1872 35 Ramana Raja
* https://tracker.ceph.com/issues/51278
1873
    mds: "FAILED ceph_assert(!segments.empty())"
1874
* https://tracker.ceph.com/issues/52279
1875 34 Venky Shankar
    cephadm tests fail due to: error adding seccomp filter rule for syscall bdflush: requested action matches default action of filter
1876 33 Patrick Donnelly
1877
1878
h3. 2021 Dec 22
1879
1880
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211222.014316
1881
1882
* https://tracker.ceph.com/issues/52624
1883
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1884
* https://tracker.ceph.com/issues/50223
1885
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1886
* https://tracker.ceph.com/issues/52279
1887
    cephadm tests fail due to: error adding seccomp filter rule for syscall bdflush: requested action matches default action of filter
1888
* https://tracker.ceph.com/issues/50224
1889
    qa: test_mirroring_init_failure_with_recovery failure
1890
* https://tracker.ceph.com/issues/48773
1891
    qa: scrub does not complete
1892 32 Venky Shankar
1893
1894
h3. 2021 Nov 30
1895
1896
https://pulpito.ceph.com/vshankar-2021-11-24_07:14:27-fs-wip-vshankar-testing-20211124-094330-testing-default-smithi/
1897
https://pulpito.ceph.com/vshankar-2021-11-30_06:23:32-fs-wip-vshankar-testing-20211124-094330-distro-default-smithi/ (rerun w/ QA fixes)
1898
1899
* https://tracker.ceph.com/issues/53436
1900
    mds, mon: mds beacon messages get dropped? (mds never reaches up:active state)
1901
* https://tracker.ceph.com/issues/51964
1902
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1903
* https://tracker.ceph.com/issues/48812
1904
    qa: test_scrub_pause_and_resume_with_abort failure
1905
* https://tracker.ceph.com/issues/51076
1906
    "wait_for_recovery: failed before timeout expired" during thrashosd test with EC backend.
1907
* https://tracker.ceph.com/issues/50223
1908
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1909
* https://tracker.ceph.com/issues/52624
1910
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1911
* https://tracker.ceph.com/issues/50250
1912
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
1913 31 Patrick Donnelly
1914
1915
h3. 2021 November 9
1916
1917
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211109.180315
1918
1919
* https://tracker.ceph.com/issues/53214
1920
    qa: "dd: error reading '/sys/kernel/debug/ceph/2a934501-6731-4052-a836-f42229a869be.client4874/metrics': Is a directory"
1921
* https://tracker.ceph.com/issues/48773
1922
    qa: scrub does not complete
1923
* https://tracker.ceph.com/issues/50223
1924
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1925
* https://tracker.ceph.com/issues/51282
1926
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
1927
* https://tracker.ceph.com/issues/52624
1928
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1929
* https://tracker.ceph.com/issues/53216
1930
    qa: "RuntimeError: value of attributes should be either str or None. client_id"
1931
* https://tracker.ceph.com/issues/50250
1932
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
1933
1934 30 Patrick Donnelly
1935
1936
h3. 2021 November 03
1937
1938
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211103.023355
1939
1940
* https://tracker.ceph.com/issues/51964
1941
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1942
* https://tracker.ceph.com/issues/51282
1943
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
1944
* https://tracker.ceph.com/issues/52436
1945
    fs/ceph: "corrupt mdsmap"
1946
* https://tracker.ceph.com/issues/53074
1947
    pybind/mgr/cephadm: upgrade sequence does not continue if no MDS are active
1948
* https://tracker.ceph.com/issues/53150
1949
    pybind/mgr/cephadm/upgrade: tolerate MDS failures during upgrade straddling v16.2.5
1950
* https://tracker.ceph.com/issues/53155
1951
    MDSMonitor: assertion during upgrade to v16.2.5+
1952 29 Patrick Donnelly
1953
1954
h3. 2021 October 26
1955
1956
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211025.000447
1957
1958
* https://tracker.ceph.com/issues/53074
1959
    pybind/mgr/cephadm: upgrade sequence does not continue if no MDS are active
1960
* https://tracker.ceph.com/issues/52997
1961
    testing: hang ing umount
1962
* https://tracker.ceph.com/issues/50824
1963
    qa: snaptest-git-ceph bus error
1964
* https://tracker.ceph.com/issues/52436
1965
    fs/ceph: "corrupt mdsmap"
1966
* https://tracker.ceph.com/issues/48773
1967
    qa: scrub does not complete
1968
* https://tracker.ceph.com/issues/53082
1969
    ceph-fuse: segmenetation fault in Client::handle_mds_map
1970
* https://tracker.ceph.com/issues/50223
1971
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
1972
* https://tracker.ceph.com/issues/52624
1973
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1974
* https://tracker.ceph.com/issues/50224
1975
    qa: test_mirroring_init_failure_with_recovery failure
1976
* https://tracker.ceph.com/issues/50821
1977
    qa: untar_snap_rm failure during mds thrashing
1978
* https://tracker.ceph.com/issues/50250
1979
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
1980
1981 27 Patrick Donnelly
1982
1983 28 Patrick Donnelly
h3. 2021 October 19
1984 27 Patrick Donnelly
1985
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211019.013028
1986
1987
* https://tracker.ceph.com/issues/52995
1988
    qa: test_standby_count_wanted failure
1989
* https://tracker.ceph.com/issues/52948
1990
    osd: fails to come up: "teuthology.misc:7 of 8 OSDs are up"
1991
* https://tracker.ceph.com/issues/52996
1992
    qa: test_perf_counters via test_openfiletable
1993
* https://tracker.ceph.com/issues/48772
1994
    qa: pjd: not ok 9, 44, 80
1995
* https://tracker.ceph.com/issues/52997
1996
    testing: hang ing umount
1997
* https://tracker.ceph.com/issues/50250
1998
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
1999
* https://tracker.ceph.com/issues/52624
2000
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2001
* https://tracker.ceph.com/issues/50223
2002
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2003
* https://tracker.ceph.com/issues/50821
2004
    qa: untar_snap_rm failure during mds thrashing
2005
* https://tracker.ceph.com/issues/48773
2006
    qa: scrub does not complete
2007 26 Patrick Donnelly
2008
2009
h3. 2021 October 12
2010
2011
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211012.192211
2012
2013
Some failures caused by teuthology bug: https://tracker.ceph.com/issues/52944
2014
2015
New test caused failure: https://github.com/ceph/ceph/pull/43297#discussion_r729883167
2016
2017
2018
* https://tracker.ceph.com/issues/51282
2019
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2020
* https://tracker.ceph.com/issues/52948
2021
    osd: fails to come up: "teuthology.misc:7 of 8 OSDs are up"
2022
* https://tracker.ceph.com/issues/48773
2023
    qa: scrub does not complete
2024
* https://tracker.ceph.com/issues/50224
2025
    qa: test_mirroring_init_failure_with_recovery failure
2026
* https://tracker.ceph.com/issues/52949
2027
    RuntimeError: The following counters failed to be set on mds daemons: {'mds.dir_split'}
2028 25 Patrick Donnelly
2029 23 Patrick Donnelly
2030 24 Patrick Donnelly
h3. 2021 October 02
2031
2032
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211002.163337
2033
2034
Some failures caused by cephadm upgrade test. Fixed in follow-up qa commit.
2035
2036
test_simple failures caused by PR in this set.
2037
2038
A few reruns because of QA infra noise.
2039
2040
* https://tracker.ceph.com/issues/52822
2041
    qa: failed pacific install on fs:upgrade
2042
* https://tracker.ceph.com/issues/52624
2043
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2044
* https://tracker.ceph.com/issues/50223
2045
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2046
* https://tracker.ceph.com/issues/48773
2047
    qa: scrub does not complete
2048
2049
2050 23 Patrick Donnelly
h3. 2021 September 20
2051
2052
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210917.174826
2053
2054
* https://tracker.ceph.com/issues/52677
2055
    qa: test_simple failure
2056
* https://tracker.ceph.com/issues/51279
2057
    kclient hangs on umount (testing branch)
2058
* https://tracker.ceph.com/issues/50223
2059
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2060
* https://tracker.ceph.com/issues/50250
2061
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2062
* https://tracker.ceph.com/issues/52624
2063
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2064
* https://tracker.ceph.com/issues/52438
2065
    qa: ffsb timeout
2066 22 Patrick Donnelly
2067
2068
h3. 2021 September 10
2069
2070
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210910.181451
2071
2072
* https://tracker.ceph.com/issues/50223
2073
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2074
* https://tracker.ceph.com/issues/50250
2075
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2076
* https://tracker.ceph.com/issues/52624
2077
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2078
* https://tracker.ceph.com/issues/52625
2079
    qa: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots)
2080
* https://tracker.ceph.com/issues/52439
2081
    qa: acls does not compile on centos stream
2082
* https://tracker.ceph.com/issues/50821
2083
    qa: untar_snap_rm failure during mds thrashing
2084
* https://tracker.ceph.com/issues/48773
2085
    qa: scrub does not complete
2086
* https://tracker.ceph.com/issues/52626
2087
    mds: ScrubStack.cc: 831: FAILED ceph_assert(diri)
2088
* https://tracker.ceph.com/issues/51279
2089
    kclient hangs on umount (testing branch)
2090 21 Patrick Donnelly
2091
2092
h3. 2021 August 27
2093
2094
Several jobs died because of device failures.
2095
2096
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210827.024746
2097
2098
* https://tracker.ceph.com/issues/52430
2099
    mds: fast async create client mount breaks racy test
2100
* https://tracker.ceph.com/issues/52436
2101
    fs/ceph: "corrupt mdsmap"
2102
* https://tracker.ceph.com/issues/52437
2103
    mds: InoTable::replay_release_ids abort via test_inotable_sync
2104
* https://tracker.ceph.com/issues/51282
2105
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2106
* https://tracker.ceph.com/issues/52438
2107
    qa: ffsb timeout
2108
* https://tracker.ceph.com/issues/52439
2109
    qa: acls does not compile on centos stream
2110 20 Patrick Donnelly
2111
2112
h3. 2021 July 30
2113
2114
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210729.214022
2115
2116
* https://tracker.ceph.com/issues/50250
2117
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2118
* https://tracker.ceph.com/issues/51282
2119
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2120
* https://tracker.ceph.com/issues/48773
2121
    qa: scrub does not complete
2122
* https://tracker.ceph.com/issues/51975
2123
    pybind/mgr/stats: KeyError
2124 19 Patrick Donnelly
2125
2126
h3. 2021 July 28
2127
2128
https://pulpito.ceph.com/pdonnell-2021-07-28_00:39:45-fs-wip-pdonnell-testing-20210727.213757-distro-basic-smithi/
2129
2130
with qa fix: https://pulpito.ceph.com/pdonnell-2021-07-28_16:20:28-fs-wip-pdonnell-testing-20210728.141004-distro-basic-smithi/
2131
2132
* https://tracker.ceph.com/issues/51905
2133
    qa: "error reading sessionmap 'mds1_sessionmap'"
2134
* https://tracker.ceph.com/issues/48773
2135
    qa: scrub does not complete
2136
* https://tracker.ceph.com/issues/50250
2137
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2138
* https://tracker.ceph.com/issues/51267
2139
    CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:...
2140
* https://tracker.ceph.com/issues/51279
2141
    kclient hangs on umount (testing branch)
2142 18 Patrick Donnelly
2143
2144
h3. 2021 July 16
2145
2146
https://pulpito.ceph.com/pdonnell-2021-07-16_05:50:11-fs-wip-pdonnell-testing-20210716.022804-distro-basic-smithi/
2147
2148
* https://tracker.ceph.com/issues/48773
2149
    qa: scrub does not complete
2150
* https://tracker.ceph.com/issues/48772
2151
    qa: pjd: not ok 9, 44, 80
2152
* https://tracker.ceph.com/issues/45434
2153
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2154
* https://tracker.ceph.com/issues/51279
2155
    kclient hangs on umount (testing branch)
2156
* https://tracker.ceph.com/issues/50824
2157
    qa: snaptest-git-ceph bus error
2158 17 Patrick Donnelly
2159
2160
h3. 2021 July 04
2161
2162
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210703.052904
2163
2164
* https://tracker.ceph.com/issues/48773
2165
    qa: scrub does not complete
2166
* https://tracker.ceph.com/issues/39150
2167
    mon: "FAILED ceph_assert(session_map.sessions.empty())" when out of quorum
2168
* https://tracker.ceph.com/issues/45434
2169
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2170
* https://tracker.ceph.com/issues/51282
2171
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2172
* https://tracker.ceph.com/issues/48771
2173
    qa: iogen: workload fails to cause balancing
2174
* https://tracker.ceph.com/issues/51279
2175
    kclient hangs on umount (testing branch)
2176
* https://tracker.ceph.com/issues/50250
2177
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2178 16 Patrick Donnelly
2179
2180
h3. 2021 July 01
2181
2182
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210701.192056
2183
2184
* https://tracker.ceph.com/issues/51197
2185
    qa: [WRN] Scrub error on inode 0x10000001520 (/client.0/tmp/t/linux-5.4/Documentation/driver-api) see mds.f log and `damage ls` output for details
2186
* https://tracker.ceph.com/issues/50866
2187
    osd: stat mismatch on objects
2188
* https://tracker.ceph.com/issues/48773
2189
    qa: scrub does not complete
2190 15 Patrick Donnelly
2191
2192
h3. 2021 June 26
2193
2194
https://pulpito.ceph.com/pdonnell-2021-06-26_00:57:00-fs-wip-pdonnell-testing-20210625.225421-distro-basic-smithi/
2195
2196
* https://tracker.ceph.com/issues/51183
2197
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2198
* https://tracker.ceph.com/issues/51410
2199
    kclient: fails to finish reconnect during MDS thrashing (testing branch)
2200
* https://tracker.ceph.com/issues/48773
2201
    qa: scrub does not complete
2202
* https://tracker.ceph.com/issues/51282
2203
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2204
* https://tracker.ceph.com/issues/51169
2205
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2206
* https://tracker.ceph.com/issues/48772
2207
    qa: pjd: not ok 9, 44, 80
2208 14 Patrick Donnelly
2209
2210
h3. 2021 June 21
2211
2212
https://pulpito.ceph.com/pdonnell-2021-06-22_00:27:21-fs-wip-pdonnell-testing-20210621.231646-distro-basic-smithi/
2213
2214
One failure caused by PR: https://github.com/ceph/ceph/pull/41935#issuecomment-866472599
2215
2216
* https://tracker.ceph.com/issues/51282
2217
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2218
* https://tracker.ceph.com/issues/51183
2219
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2220
* https://tracker.ceph.com/issues/48773
2221
    qa: scrub does not complete
2222
* https://tracker.ceph.com/issues/48771
2223
    qa: iogen: workload fails to cause balancing
2224
* https://tracker.ceph.com/issues/51169
2225
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2226
* https://tracker.ceph.com/issues/50495
2227
    libcephfs: shutdown race fails with status 141
2228
* https://tracker.ceph.com/issues/45434
2229
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2230
* https://tracker.ceph.com/issues/50824
2231
    qa: snaptest-git-ceph bus error
2232
* https://tracker.ceph.com/issues/50223
2233
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2234 13 Patrick Donnelly
2235
2236
h3. 2021 June 16
2237
2238
https://pulpito.ceph.com/pdonnell-2021-06-16_21:26:55-fs-wip-pdonnell-testing-20210616.191804-distro-basic-smithi/
2239
2240
MDS abort class of failures caused by PR: https://github.com/ceph/ceph/pull/41667
2241
2242
* https://tracker.ceph.com/issues/45434
2243
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2244
* https://tracker.ceph.com/issues/51169
2245
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2246
* https://tracker.ceph.com/issues/43216
2247
    MDSMonitor: removes MDS coming out of quorum election
2248
* https://tracker.ceph.com/issues/51278
2249
    mds: "FAILED ceph_assert(!segments.empty())"
2250
* https://tracker.ceph.com/issues/51279
2251
    kclient hangs on umount (testing branch)
2252
* https://tracker.ceph.com/issues/51280
2253
    mds: "FAILED ceph_assert(r == 0 || r == -2)"
2254
* https://tracker.ceph.com/issues/51183
2255
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2256
* https://tracker.ceph.com/issues/51281
2257
    qa: snaptest-snap-rm-cmp.sh: "echo 'FAIL: bad match, /tmp/a 4637e766853d1ad16a7b17079e2c6f03 != real c3883760b18d50e8d78819c54d579b00'"
2258
* https://tracker.ceph.com/issues/48773
2259
    qa: scrub does not complete
2260
* https://tracker.ceph.com/issues/51076
2261
    "wait_for_recovery: failed before timeout expired" during thrashosd test with EC backend.
2262
* https://tracker.ceph.com/issues/51228
2263
    qa: rmdir: failed to remove 'a/.snap/*': No such file or directory
2264
* https://tracker.ceph.com/issues/51282
2265
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2266 12 Patrick Donnelly
2267
2268
h3. 2021 June 14
2269
2270
https://pulpito.ceph.com/pdonnell-2021-06-14_20:53:05-fs-wip-pdonnell-testing-20210614.173325-distro-basic-smithi/
2271
2272
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
2273
2274
* https://tracker.ceph.com/issues/51169
2275
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2276
* https://tracker.ceph.com/issues/51228
2277
    qa: rmdir: failed to remove 'a/.snap/*': No such file or directory
2278
* https://tracker.ceph.com/issues/48773
2279
    qa: scrub does not complete
2280
* https://tracker.ceph.com/issues/51183
2281
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2282
* https://tracker.ceph.com/issues/45434
2283
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2284
* https://tracker.ceph.com/issues/51182
2285
    pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs'
2286
* https://tracker.ceph.com/issues/51229
2287
    qa: test_multi_snap_schedule list difference failure
2288
* https://tracker.ceph.com/issues/50821
2289
    qa: untar_snap_rm failure during mds thrashing
2290 11 Patrick Donnelly
2291
2292
h3. 2021 June 13
2293
2294
https://pulpito.ceph.com/pdonnell-2021-06-12_02:45:35-fs-wip-pdonnell-testing-20210612.002809-distro-basic-smithi/
2295
2296
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
2297
2298
* https://tracker.ceph.com/issues/51169
2299
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2300
* https://tracker.ceph.com/issues/48773
2301
    qa: scrub does not complete
2302
* https://tracker.ceph.com/issues/51182
2303
    pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs'
2304
* https://tracker.ceph.com/issues/51183
2305
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2306
* https://tracker.ceph.com/issues/51197
2307
    qa: [WRN] Scrub error on inode 0x10000001520 (/client.0/tmp/t/linux-5.4/Documentation/driver-api) see mds.f log and `damage ls` output for details
2308
* https://tracker.ceph.com/issues/45434
2309 10 Patrick Donnelly
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2310
2311
h3. 2021 June 11
2312
2313
https://pulpito.ceph.com/pdonnell-2021-06-11_18:02:10-fs-wip-pdonnell-testing-20210611.162716-distro-basic-smithi/
2314
2315
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
2316
2317
* https://tracker.ceph.com/issues/51169
2318
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2319
* https://tracker.ceph.com/issues/45434
2320
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2321
* https://tracker.ceph.com/issues/48771
2322
    qa: iogen: workload fails to cause balancing
2323
* https://tracker.ceph.com/issues/43216
2324
    MDSMonitor: removes MDS coming out of quorum election
2325
* https://tracker.ceph.com/issues/51182
2326
    pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs'
2327
* https://tracker.ceph.com/issues/50223
2328
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2329
* https://tracker.ceph.com/issues/48773
2330
    qa: scrub does not complete
2331
* https://tracker.ceph.com/issues/51183
2332
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2333
* https://tracker.ceph.com/issues/51184
2334
    qa: fs:bugs does not specify distro
2335 9 Patrick Donnelly
2336
2337
h3. 2021 June 03
2338
2339
https://pulpito.ceph.com/pdonnell-2021-06-03_03:40:33-fs-wip-pdonnell-testing-20210603.020013-distro-basic-smithi/
2340
2341
* https://tracker.ceph.com/issues/45434
2342
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2343
* https://tracker.ceph.com/issues/50016
2344
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2345
* https://tracker.ceph.com/issues/50821
2346
    qa: untar_snap_rm failure during mds thrashing
2347
* https://tracker.ceph.com/issues/50622 (regression)
2348
    msg: active_connections regression
2349
* https://tracker.ceph.com/issues/49845#note-2 (regression)
2350
    qa: failed umount in test_volumes
2351
* https://tracker.ceph.com/issues/48773
2352
    qa: scrub does not complete
2353
* https://tracker.ceph.com/issues/43216
2354
    MDSMonitor: removes MDS coming out of quorum election
2355 7 Patrick Donnelly
2356
2357 8 Patrick Donnelly
h3. 2021 May 18
2358
2359
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210518.214114
2360
2361
Regression in testing kernel caused some failures. Ilya fixed those and rerun
2362
looked better. Some odd new noise in the rerun relating to packaging and "No
2363
module named 'tasks.ceph'".
2364
2365
* https://tracker.ceph.com/issues/50824
2366
    qa: snaptest-git-ceph bus error
2367
* https://tracker.ceph.com/issues/50622 (regression)
2368
    msg: active_connections regression
2369
* https://tracker.ceph.com/issues/49845#note-2 (regression)
2370
    qa: failed umount in test_volumes
2371
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
2372
    qa: quota failure
2373
2374
2375 7 Patrick Donnelly
h3. 2021 May 18
2376
2377
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210518.025642
2378
2379
* https://tracker.ceph.com/issues/50821
2380
    qa: untar_snap_rm failure during mds thrashing
2381
* https://tracker.ceph.com/issues/48773
2382
    qa: scrub does not complete
2383
* https://tracker.ceph.com/issues/45591
2384
    mgr: FAILED ceph_assert(daemon != nullptr)
2385
* https://tracker.ceph.com/issues/50866
2386
    osd: stat mismatch on objects
2387
* https://tracker.ceph.com/issues/50016
2388
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2389
* https://tracker.ceph.com/issues/50867
2390
    qa: fs:mirror: reduced data availability
2391
* https://tracker.ceph.com/issues/50821
2392
    qa: untar_snap_rm failure during mds thrashing
2393
* https://tracker.ceph.com/issues/50622 (regression)
2394
    msg: active_connections regression
2395
* https://tracker.ceph.com/issues/50223
2396
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2397
* https://tracker.ceph.com/issues/50868
2398
    qa: "kern.log.gz already exists; not overwritten"
2399
* https://tracker.ceph.com/issues/50870
2400
    qa: test_full: "rm: cannot remove 'large_file_a': Permission denied"
2401 6 Patrick Donnelly
2402
2403
h3. 2021 May 11
2404
2405
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210511.232042
2406
2407
* one class of failures caused by PR
2408
* https://tracker.ceph.com/issues/48812
2409
    qa: test_scrub_pause_and_resume_with_abort failure
2410
* https://tracker.ceph.com/issues/50390
2411
    mds: monclient: wait_auth_rotating timed out after 30
2412
* https://tracker.ceph.com/issues/48773
2413
    qa: scrub does not complete
2414
* https://tracker.ceph.com/issues/50821
2415
    qa: untar_snap_rm failure during mds thrashing
2416
* https://tracker.ceph.com/issues/50224
2417
    qa: test_mirroring_init_failure_with_recovery failure
2418
* https://tracker.ceph.com/issues/50622 (regression)
2419
    msg: active_connections regression
2420
* https://tracker.ceph.com/issues/50825
2421
    qa: snaptest-git-ceph hang during mon thrashing v2
2422
* https://tracker.ceph.com/issues/50821
2423
    qa: untar_snap_rm failure during mds thrashing
2424
* https://tracker.ceph.com/issues/50823
2425
    qa: RuntimeError: timeout waiting for cluster to stabilize
2426 5 Patrick Donnelly
2427
2428
h3. 2021 May 14
2429
2430
https://pulpito.ceph.com/pdonnell-2021-05-14_21:45:42-fs-master-distro-basic-smithi/
2431
2432
* https://tracker.ceph.com/issues/48812
2433
    qa: test_scrub_pause_and_resume_with_abort failure
2434
* https://tracker.ceph.com/issues/50821
2435
    qa: untar_snap_rm failure during mds thrashing
2436
* https://tracker.ceph.com/issues/50622 (regression)
2437
    msg: active_connections regression
2438
* https://tracker.ceph.com/issues/50822
2439
    qa: testing kernel patch for client metrics causes mds abort
2440
* https://tracker.ceph.com/issues/48773
2441
    qa: scrub does not complete
2442
* https://tracker.ceph.com/issues/50823
2443
    qa: RuntimeError: timeout waiting for cluster to stabilize
2444
* https://tracker.ceph.com/issues/50824
2445
    qa: snaptest-git-ceph bus error
2446
* https://tracker.ceph.com/issues/50825
2447
    qa: snaptest-git-ceph hang during mon thrashing v2
2448
* https://tracker.ceph.com/issues/50826
2449
    kceph: stock RHEL kernel hangs on snaptests with mon|osd thrashers
2450 4 Patrick Donnelly
2451
2452
h3. 2021 May 01
2453
2454
https://pulpito.ceph.com/pdonnell-2021-05-01_09:07:09-fs-wip-pdonnell-testing-20210501.040415-distro-basic-smithi/
2455
2456
* https://tracker.ceph.com/issues/45434
2457
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2458
* https://tracker.ceph.com/issues/50281
2459
    qa: untar_snap_rm timeout
2460
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
2461
    qa: quota failure
2462
* https://tracker.ceph.com/issues/48773
2463
    qa: scrub does not complete
2464
* https://tracker.ceph.com/issues/50390
2465
    mds: monclient: wait_auth_rotating timed out after 30
2466
* https://tracker.ceph.com/issues/50250
2467
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details"
2468
* https://tracker.ceph.com/issues/50622 (regression)
2469
    msg: active_connections regression
2470
* https://tracker.ceph.com/issues/45591
2471
    mgr: FAILED ceph_assert(daemon != nullptr)
2472
* https://tracker.ceph.com/issues/50221
2473
    qa: snaptest-git-ceph failure in git diff
2474
* https://tracker.ceph.com/issues/50016
2475
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2476 3 Patrick Donnelly
2477
2478
h3. 2021 Apr 15
2479
2480
https://pulpito.ceph.com/pdonnell-2021-04-15_01:35:57-fs-wip-pdonnell-testing-20210414.230315-distro-basic-smithi/
2481
2482
* https://tracker.ceph.com/issues/50281
2483
    qa: untar_snap_rm timeout
2484
* https://tracker.ceph.com/issues/50220
2485
    qa: dbench workload timeout
2486
* https://tracker.ceph.com/issues/50246
2487
    mds: failure replaying journal (EMetaBlob)
2488
* https://tracker.ceph.com/issues/50250
2489
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details"
2490
* https://tracker.ceph.com/issues/50016
2491
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2492
* https://tracker.ceph.com/issues/50222
2493
    osd: 5.2s0 deep-scrub : stat mismatch
2494
* https://tracker.ceph.com/issues/45434
2495
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2496
* https://tracker.ceph.com/issues/49845
2497
    qa: failed umount in test_volumes
2498
* https://tracker.ceph.com/issues/37808
2499
    osd: osdmap cache weak_refs assert during shutdown
2500
* https://tracker.ceph.com/issues/50387
2501
    client: fs/snaps failure
2502
* https://tracker.ceph.com/issues/50389
2503
    mds: "cluster [ERR] Error recovering journal 0x203: (2) No such file or directory" in cluster log"
2504
* https://tracker.ceph.com/issues/50216
2505
    qa: "ls: cannot access 'lost+found': No such file or directory"
2506
* https://tracker.ceph.com/issues/50390
2507
    mds: monclient: wait_auth_rotating timed out after 30
2508
2509 1 Patrick Donnelly
2510
2511 2 Patrick Donnelly
h3. 2021 Apr 08
2512
2513
https://pulpito.ceph.com/pdonnell-2021-04-08_22:42:24-fs-wip-pdonnell-testing-20210408.192301-distro-basic-smithi/
2514
2515
* https://tracker.ceph.com/issues/45434
2516
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2517
* https://tracker.ceph.com/issues/50016
2518
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2519
* https://tracker.ceph.com/issues/48773
2520
    qa: scrub does not complete
2521
* https://tracker.ceph.com/issues/50279
2522
    qa: "Replacing daemon mds.b as rank 0 with standby daemon mds.c"
2523
* https://tracker.ceph.com/issues/50246
2524
    mds: failure replaying journal (EMetaBlob)
2525
* https://tracker.ceph.com/issues/48365
2526
    qa: ffsb build failure on CentOS 8.2
2527
* https://tracker.ceph.com/issues/50216
2528
    qa: "ls: cannot access 'lost+found': No such file or directory"
2529
* https://tracker.ceph.com/issues/50223
2530
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2531
* https://tracker.ceph.com/issues/50280
2532
    cephadm: RuntimeError: uid/gid not found
2533
* https://tracker.ceph.com/issues/50281
2534
    qa: untar_snap_rm timeout
2535
2536 1 Patrick Donnelly
h3. 2021 Apr 08
2537
2538
https://pulpito.ceph.com/pdonnell-2021-04-08_04:31:36-fs-wip-pdonnell-testing-20210408.024225-distro-basic-smithi/
2539
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210408.142238 (with logic inversion / QA fix)
2540
2541
* https://tracker.ceph.com/issues/50246
2542
    mds: failure replaying journal (EMetaBlob)
2543
* https://tracker.ceph.com/issues/50250
2544
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details"
2545
2546
2547
h3. 2021 Apr 07
2548
2549
https://pulpito.ceph.com/pdonnell-2021-04-07_02:12:41-fs-wip-pdonnell-testing-20210406.213012-distro-basic-smithi/
2550
2551
* https://tracker.ceph.com/issues/50215
2552
    qa: "log [ERR] : error reading sessionmap 'mds2_sessionmap'"
2553
* https://tracker.ceph.com/issues/49466
2554
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
2555
* https://tracker.ceph.com/issues/50216
2556
    qa: "ls: cannot access 'lost+found': No such file or directory"
2557
* https://tracker.ceph.com/issues/48773
2558
    qa: scrub does not complete
2559
* https://tracker.ceph.com/issues/49845
2560
    qa: failed umount in test_volumes
2561
* https://tracker.ceph.com/issues/50220
2562
    qa: dbench workload timeout
2563
* https://tracker.ceph.com/issues/50221
2564
    qa: snaptest-git-ceph failure in git diff
2565
* https://tracker.ceph.com/issues/50222
2566
    osd: 5.2s0 deep-scrub : stat mismatch
2567
* https://tracker.ceph.com/issues/50223
2568
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2569
* https://tracker.ceph.com/issues/50224
2570
    qa: test_mirroring_init_failure_with_recovery failure
2571
2572
h3. 2021 Apr 01
2573
2574
https://pulpito.ceph.com/pdonnell-2021-04-01_00:45:34-fs-wip-pdonnell-testing-20210331.222326-distro-basic-smithi/
2575
2576
* https://tracker.ceph.com/issues/48772
2577
    qa: pjd: not ok 9, 44, 80
2578
* https://tracker.ceph.com/issues/50177
2579
    osd: "stalled aio... buggy kernel or bad device?"
2580
* https://tracker.ceph.com/issues/48771
2581
    qa: iogen: workload fails to cause balancing
2582
* https://tracker.ceph.com/issues/49845
2583
    qa: failed umount in test_volumes
2584
* https://tracker.ceph.com/issues/48773
2585
    qa: scrub does not complete
2586
* https://tracker.ceph.com/issues/48805
2587
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
2588
* https://tracker.ceph.com/issues/50178
2589
    qa: "TypeError: run() got an unexpected keyword argument 'shell'"
2590
* https://tracker.ceph.com/issues/45434
2591
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2592
2593
h3. 2021 Mar 24
2594
2595
https://pulpito.ceph.com/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/
2596
2597
* https://tracker.ceph.com/issues/49500
2598
    qa: "Assertion `cb_done' failed."
2599
* https://tracker.ceph.com/issues/50019
2600
    qa: mount failure with cephadm "probably no MDS server is up?"
2601
* https://tracker.ceph.com/issues/50020
2602
    qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)"
2603
* https://tracker.ceph.com/issues/48773
2604
    qa: scrub does not complete
2605
* https://tracker.ceph.com/issues/45434
2606
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2607
* https://tracker.ceph.com/issues/48805
2608
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
2609
* https://tracker.ceph.com/issues/48772
2610
    qa: pjd: not ok 9, 44, 80
2611
* https://tracker.ceph.com/issues/50021
2612
    qa: snaptest-git-ceph failure during mon thrashing
2613
* https://tracker.ceph.com/issues/48771
2614
    qa: iogen: workload fails to cause balancing
2615
* https://tracker.ceph.com/issues/50016
2616
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2617
* https://tracker.ceph.com/issues/49466
2618
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
2619
2620
2621
h3. 2021 Mar 18
2622
2623
https://pulpito.ceph.com/pdonnell-2021-03-18_13:46:31-fs-wip-pdonnell-testing-20210318.024145-distro-basic-smithi/
2624
2625
* https://tracker.ceph.com/issues/49466
2626
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
2627
* https://tracker.ceph.com/issues/48773
2628
    qa: scrub does not complete
2629
* https://tracker.ceph.com/issues/48805
2630
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
2631
* https://tracker.ceph.com/issues/45434
2632
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2633
* https://tracker.ceph.com/issues/49845
2634
    qa: failed umount in test_volumes
2635
* https://tracker.ceph.com/issues/49605
2636
    mgr: drops command on the floor
2637
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
2638
    qa: quota failure
2639
* https://tracker.ceph.com/issues/49928
2640
    client: items pinned in cache preventing unmount x2
2641
2642
h3. 2021 Mar 15
2643
2644
https://pulpito.ceph.com/pdonnell-2021-03-15_22:16:56-fs-wip-pdonnell-testing-20210315.182203-distro-basic-smithi/
2645
2646
* https://tracker.ceph.com/issues/49842
2647
    qa: stuck pkg install
2648
* https://tracker.ceph.com/issues/49466
2649
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
2650
* https://tracker.ceph.com/issues/49822
2651
    test: test_mirroring_command_idempotency (tasks.cephfs.test_admin.TestMirroringCommands) failure
2652
* https://tracker.ceph.com/issues/49240
2653
    terminate called after throwing an instance of 'std::bad_alloc'
2654
* https://tracker.ceph.com/issues/48773
2655
    qa: scrub does not complete
2656
* https://tracker.ceph.com/issues/45434
2657
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2658
* https://tracker.ceph.com/issues/49500
2659
    qa: "Assertion `cb_done' failed."
2660
* https://tracker.ceph.com/issues/49843
2661
    qa: fs/snaps/snaptest-upchildrealms.sh failure
2662
* https://tracker.ceph.com/issues/49845
2663
    qa: failed umount in test_volumes
2664
* https://tracker.ceph.com/issues/48805
2665
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
2666
* https://tracker.ceph.com/issues/49605
2667
    mgr: drops command on the floor
2668
2669
and failure caused by PR: https://github.com/ceph/ceph/pull/39969
2670
2671
2672
h3. 2021 Mar 09
2673
2674
https://pulpito.ceph.com/pdonnell-2021-03-09_03:27:39-fs-wip-pdonnell-testing-20210308.214827-distro-basic-smithi/
2675
2676
* https://tracker.ceph.com/issues/49500
2677
    qa: "Assertion `cb_done' failed."
2678
* https://tracker.ceph.com/issues/48805
2679
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
2680
* https://tracker.ceph.com/issues/48773
2681
    qa: scrub does not complete
2682
* https://tracker.ceph.com/issues/45434
2683
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2684
* https://tracker.ceph.com/issues/49240
2685
    terminate called after throwing an instance of 'std::bad_alloc'
2686
* https://tracker.ceph.com/issues/49466
2687
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
2688
* https://tracker.ceph.com/issues/49684
2689
    qa: fs:cephadm mount does not wait for mds to be created
2690
* https://tracker.ceph.com/issues/48771
2691
    qa: iogen: workload fails to cause balancing