Project

General

Profile

Main » History » Version 256

Venky Shankar, 04/12/2024 05:14 AM

1 221 Patrick Donnelly
h1. <code>main</code> branch
2 1 Patrick Donnelly
3 256 Venky Shankar
h3. 2024-04-12
4
5
https://tracker.ceph.com/issues/65324
6
7
(Lot many `sudo systemctl stop ceph-ba42f8d0-efae-11ee-b647-cb9ed24678a4@mon.a` and infra issues failures in this run)
8
9
* "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711
10
* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502
11
* "qa: ceph tell 4.3a deep-scrub command not found":https://tracker.ceph.com/issues/64972
12
* "qa: scrub - object missing on disk; some files may be lost":https://tracker.ceph.com/issues/48562
13
* "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699
14 247 Rishabh Dave
15 253 Venky Shankar
h3. 2024-04-04
16
17
https://tracker.ceph.com/issues/65300
18
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240330.172700
19
20
(Lot many `sudo systemctl stop ceph-ba42f8d0-efae-11ee-b647-cb9ed24678a4@mon.a` failures in this run)
21
22
* "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711
23
* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502
24
* "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699
25
* "centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons":https://tracker.ceph.com/issues/61774
26 254 Venky Shankar
* "qa: scrub - object missing on disk; some files may be lost":https://tracker.ceph.com/issues/48562
27
* "upgrade stalls after upgrading one ceph-mgr daemon":https://tracker.ceph.com/issues/65263
28 253 Venky Shankar
* "qa: test_max_items_per_obj open procs not fully cleaned up":https://tracker.ceph.com/issues/65022
29
* "QA failure: test_fscrypt_dummy_encryption_with_quick_group":https://tracker.ceph.com/issues/65136
30 254 Venky Shankar
* "qa/cephfs: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)":https://tracker.ceph.com/issues/65246
31
* "qa: test_cd_with_args failure":https://tracker.ceph.com/issues/63700
32
* "valgrind error: Leak_PossiblyLost posix_memalign UnknownInlinedFun ceph::buffer::v15_2_0::list::refill_append_space(unsigned int)":https://tracker.ceph.com/issues/65314
33 253 Venky Shankar
34 249 Rishabh Dave
h3. 4 Apr 2024
35 246 Rishabh Dave
36
https://pulpito.ceph.com/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/
37
38
* https://tracker.ceph.com/issues/64927
39
  qa/cephfs: test_cephfs_mirror_blocklist raises "KeyError: 'rados_inst'"
40
* https://tracker.ceph.com/issues/65022
41
  qa: test_max_items_per_obj open procs not fully cleaned up
42
* https://tracker.ceph.com/issues/63699
43
  qa: failed cephfs-shell test_reading_conf
44
* https://tracker.ceph.com/issues/63700
45
  qa: test_cd_with_args failure
46
* https://tracker.ceph.com/issues/65136
47
  QA failure: test_fscrypt_dummy_encryption_with_quick_group
48
* https://tracker.ceph.com/issues/65246
49
  qa/cephfs: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)
50
51 248 Rishabh Dave
52 246 Rishabh Dave
* https://tracker.ceph.com/issues/58945
53 1 Patrick Donnelly
  qa: xfstests-dev's generic test suite has failures with fuse client
54
* https://tracker.ceph.com/issues/57656
55 251 Rishabh Dave
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
56 1 Patrick Donnelly
* https://tracker.ceph.com/issues/63265
57
  qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'
58 246 Rishabh Dave
* https://tracker.ceph.com/issues/62067
59 251 Rishabh Dave
  ffsb.sh failure "Resource temporarily unavailable"
60 246 Rishabh Dave
* https://tracker.ceph.com/issues/63949
61
  leak in mds.c detected by valgrind during CephFS QA run
62
* https://tracker.ceph.com/issues/48562
63
  qa: scrub - object missing on disk; some files may be lost
64
* https://tracker.ceph.com/issues/65020
65
  qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log
66
* https://tracker.ceph.com/issues/64572
67
  workunits/fsx.sh failure
68
* https://tracker.ceph.com/issues/57676
69
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
70 1 Patrick Donnelly
* https://tracker.ceph.com/issues/64502
71 246 Rishabh Dave
  client: ceph-fuse fails to unmount after upgrade to main
72 1 Patrick Donnelly
* https://tracker.ceph.com/issues/54741
73
  crash: MDSTableClient::got_journaled_ack(unsigned long)
74 250 Rishabh Dave
75 248 Rishabh Dave
* https://tracker.ceph.com/issues/65265
76
  qa: health warning "no active mgr (MGR_DOWN)" occurs before and after test_nfs runs
77 1 Patrick Donnelly
* https://tracker.ceph.com/issues/65308
78
  qa: fs was offline but also unexpectedly degraded
79
* https://tracker.ceph.com/issues/65309
80
  qa: dbench.sh failed with "ERROR: handle 10318 was not found"
81 250 Rishabh Dave
82
* https://tracker.ceph.com/issues/65018
83 251 Rishabh Dave
  PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)"
84 250 Rishabh Dave
* https://tracker.ceph.com/issues/52624
85
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
86 245 Rishabh Dave
87 240 Patrick Donnelly
h3. 2024-04-02
88
89
https://tracker.ceph.com/issues/65215
90
91
* "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676
92
* "qa: ceph tell 4.3a deep-scrub command not found":https://tracker.ceph.com/issues/64972
93
* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502
94
* "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711
95
* "workunits/fsx.sh failure":https://tracker.ceph.com/issues/64572
96
* "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699
97
* "centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons":https://tracker.ceph.com/issues/61774
98
* "qa: test_max_items_per_obj open procs not fully cleaned up":https://tracker.ceph.com/issues/65022
99
* "qa: dbench workload timeout":https://tracker.ceph.com/issues/50220
100
* "suites/fsstress.sh hangs on one client - test times out":https://tracker.ceph.com/issues/64707
101 255 Patrick Donnelly
* "qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) in cluster log":https://tracker.ceph.com/issues/65021
102 241 Patrick Donnelly
* "qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log":https://tracker.ceph.com/issues/65020
103
* "qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}":https://tracker.ceph.com/issues/54108
104
* "ffsb.sh failure Resource temporarily unavailable":https://tracker.ceph.com/issues/62067
105
* "QA failure: test_fscrypt_dummy_encryption_with_quick_group":https://tracker.ceph.com/issues/65136
106 244 Patrick Donnelly
* "qa: cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled in cluster log":https://tracker.ceph.com/issues/65271
107 241 Patrick Donnelly
* "qa: test_cephfs_mirror_cancel_sync fails in a 100 jobs run of fs:mirror suite":https://tracker.ceph.com/issues/64534
108 240 Patrick Donnelly
109 236 Patrick Donnelly
h3. 2024-03-28
110
111
https://tracker.ceph.com/issues/65213
112
113 237 Patrick Donnelly
* "qa: error during scrub thrashing: rank damage found: {'backtrace'}":https://tracker.ceph.com/issues/57676
114
* "workunits/fsx.sh failure":https://tracker.ceph.com/issues/64572
115
* "PG_DEGRADED warnings during cluster creation via cephadm: Health check failed: Degraded data":https://tracker.ceph.com/issues/65018
116 238 Patrick Donnelly
* "suites/fsstress.sh hangs on one client - test times out":https://tracker.ceph.com/issues/64707
117
* "qa: ceph tell 4.3a deep-scrub command not found":https://tracker.ceph.com/issues/64972
118
* "qa: iogen workunit: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}":https://tracker.ceph.com/issues/54108
119 239 Patrick Donnelly
* "qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details in cluster log":https://tracker.ceph.com/issues/65020
120
* "qa: failed cephfs-shell test_reading_conf":https://tracker.ceph.com/issues/63699
121
* "Test failure: test_cephfs_mirror_cancel_mirroring_and_readd":https://tracker.ceph.com/issues/64711
122
* "qa: test_max_items_per_obj open procs not fully cleaned up":https://tracker.ceph.com/issues/65022
123
* "pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main":https://tracker.ceph.com/issues/64502
124
* "centos 9 testing reveals rocksdb Leak_StillReachable memory leak in mons":https://tracker.ceph.com/issues/61774
125
* "qa: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)":https://tracker.ceph.com/issues/52624
126
* "qa: dbench workload timeout":https://tracker.ceph.com/issues/50220
127
128
129 236 Patrick Donnelly
130 235 Milind Changire
h3. 2024-03-25
131
132
https://pulpito.ceph.com/mchangir-2024-03-22_09:46:06-fs:upgrade-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/
133
* https://tracker.ceph.com/issues/64502
134
  fusermount -u fails with: teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds
135
136
https://pulpito.ceph.com/mchangir-2024-03-22_09:48:09-fs:libcephfs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/
137
138
* https://tracker.ceph.com/issues/62245
139
  libcephfs/test.sh failed - https://tracker.ceph.com/issues/62245#note-3
140
141
142 228 Patrick Donnelly
h3. 2024-03-20
143
144 234 Patrick Donnelly
https://pulpito.ceph.com/?branch=wip-batrick-testing-20240320.145742
145 228 Patrick Donnelly
146 233 Patrick Donnelly
https://github.com/batrick/ceph/commit/360516069d9393362c4cc6eb9371680fe16d66ab
147
148 229 Patrick Donnelly
Ubuntu jobs filtered out because builds were skipped by jenkins/shaman.
149 1 Patrick Donnelly
150 229 Patrick Donnelly
This run has a lot more failures because https://github.com/ceph/ceph/pull/55455 fixed log WRN/ERR checks.
151 228 Patrick Donnelly
152 229 Patrick Donnelly
* https://tracker.ceph.com/issues/57676
153
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
154
* https://tracker.ceph.com/issues/64572
155
    workunits/fsx.sh failure
156
* https://tracker.ceph.com/issues/65018
157
    PG_DEGRADED warnings during cluster creation via cephadm: "Health check failed: Degraded data redundancy: 2/192 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED)"
158
* https://tracker.ceph.com/issues/64707 (new issue)
159
    suites/fsstress.sh hangs on one client - test times out
160 1 Patrick Donnelly
* https://tracker.ceph.com/issues/64988
161
    qa: fs:workloads mgr client evicted indicated by "cluster [WRN] evicting unresponsive client smithi042:x (15288), after 303.306 seconds"
162
* https://tracker.ceph.com/issues/59684
163
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
164 230 Patrick Donnelly
* https://tracker.ceph.com/issues/64972
165
    qa: "ceph tell 4.3a deep-scrub" command not found
166
* https://tracker.ceph.com/issues/54108
167
    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"
168
* https://tracker.ceph.com/issues/65019
169
    qa/suites/fs/top: [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 
170
* https://tracker.ceph.com/issues/65020
171
    qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-3ea9-47a0-bd79-a0f50dc149d5/client.0/tmp/clients/client7/~dmtmp/PARADOX) see mds.b log and `damage ls` output for details" in cluster log
172
* https://tracker.ceph.com/issues/65021
173
    qa/suites/fs/nfs: cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log
174
* https://tracker.ceph.com/issues/63699
175
    qa: failed cephfs-shell test_reading_conf
176 231 Patrick Donnelly
* https://tracker.ceph.com/issues/64711
177
    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)
178
* https://tracker.ceph.com/issues/50821
179
    qa: untar_snap_rm failure during mds thrashing
180 232 Patrick Donnelly
* https://tracker.ceph.com/issues/65022
181
    qa: test_max_items_per_obj open procs not fully cleaned up
182 228 Patrick Donnelly
183 226 Venky Shankar
h3.  14th March 2024
184
185
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240307.013758
186
187 227 Venky Shankar
(pjd.sh failures are related to a bug in the testing kernel. See - https://tracker.ceph.com/issues/64679#note-4)
188 226 Venky Shankar
189
* https://tracker.ceph.com/issues/62067
190
    ffsb.sh failure "Resource temporarily unavailable"
191
* https://tracker.ceph.com/issues/57676
192
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
193
* https://tracker.ceph.com/issues/64502
194
    pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main
195
* https://tracker.ceph.com/issues/64572
196
    workunits/fsx.sh failure
197
* https://tracker.ceph.com/issues/63700
198
    qa: test_cd_with_args failure
199
* https://tracker.ceph.com/issues/59684
200
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
201
* https://tracker.ceph.com/issues/61243
202
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
203
204 225 Venky Shankar
h3. 5th March 2024
205
206
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240304.042522
207
208
* https://tracker.ceph.com/issues/57676
209
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
210
* https://tracker.ceph.com/issues/64502
211
    pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgrade to main
212
* https://tracker.ceph.com/issues/63949
213
    leak in mds.c detected by valgrind during CephFS QA run
214
* https://tracker.ceph.com/issues/57656
215
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
216
* https://tracker.ceph.com/issues/63699
217
    qa: failed cephfs-shell test_reading_conf
218
* https://tracker.ceph.com/issues/64572
219
    workunits/fsx.sh failure
220
* https://tracker.ceph.com/issues/64707 (new issue)
221
    suites/fsstress.sh hangs on one client - test times out
222
* https://tracker.ceph.com/issues/59684
223
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
224
* https://tracker.ceph.com/issues/63700
225
    qa: test_cd_with_args failure
226
* https://tracker.ceph.com/issues/64711
227
    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)
228
* https://tracker.ceph.com/issues/64729 (new issue)
229
    mon.a (mon.0) 1281 : cluster 3 [WRN] MDS_SLOW_METADATA_IO: 3 MDSs report slow metadata IOs" in cluster log
230
* https://tracker.ceph.com/issues/64730
231
    fs/misc/multiple_rsync.sh workunit times out
232
233 224 Venky Shankar
h3. 26th Feb 2024
234
235
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240216.060239
236
237
(This run is a bit messy due to
238
239
  a) OCI runtime issues in the testing kernel with centos9
240
  b) SELinux denials related failures
241
  c) Unrelated MON_DOWN warnings)
242
243
* https://tracker.ceph.com/issues/57676
244
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
245
* https://tracker.ceph.com/issues/63700
246
    qa: test_cd_with_args failure
247
* https://tracker.ceph.com/issues/63949
248
    leak in mds.c detected by valgrind during CephFS QA run
249
* https://tracker.ceph.com/issues/59684
250
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
251
* https://tracker.ceph.com/issues/61243
252
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
253
* https://tracker.ceph.com/issues/63699
254
    qa: failed cephfs-shell test_reading_conf
255
* https://tracker.ceph.com/issues/64172
256
    Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize)
257
* https://tracker.ceph.com/issues/57656
258
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
259
* https://tracker.ceph.com/issues/64572
260
    workunits/fsx.sh failure
261
262 222 Patrick Donnelly
h3. 20th Feb 2024
263
264
https://github.com/ceph/ceph/pull/55601
265
https://github.com/ceph/ceph/pull/55659
266
267
https://pulpito.ceph.com/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/
268
269
* https://tracker.ceph.com/issues/64502
270
    client: quincy ceph-fuse fails to unmount after upgrade to main
271
272 223 Patrick Donnelly
This run has numerous problems. #55601 introduces testing for the upgrade sequence from </code>reef/{v18.2.0,v18.2.1,reef}</code> as well as an extra dimension for the ceph-fuse client. The main "big" issue is i64502: the ceph-fuse client is not being unmounted when <code>fusermount -u</code> is called. Instead, the client begins to unmount only after daemons are shut down during test cleanup.
273 218 Venky Shankar
274
h3. 19th Feb 2024
275
276 220 Venky Shankar
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240217.015652
277
278 218 Venky Shankar
* https://tracker.ceph.com/issues/61243
279
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
280
* https://tracker.ceph.com/issues/63700
281
    qa: test_cd_with_args failure
282
* https://tracker.ceph.com/issues/63141
283
    qa/cephfs: test_idem_unaffected_root_squash fails
284
* https://tracker.ceph.com/issues/59684
285
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
286
* https://tracker.ceph.com/issues/63949
287
    leak in mds.c detected by valgrind during CephFS QA run
288
* https://tracker.ceph.com/issues/63764
289
    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)
290
* https://tracker.ceph.com/issues/63699
291
    qa: failed cephfs-shell test_reading_conf
292 219 Venky Shankar
* https://tracker.ceph.com/issues/64482
293
    ceph: stderr Error: OCI runtime error: crun: bpf create ``: Function not implemented
294 201 Rishabh Dave
295 217 Venky Shankar
h3. 29 Jan 2024
296
297
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240119.075157-1
298
299
* https://tracker.ceph.com/issues/57676
300
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
301
* https://tracker.ceph.com/issues/63949
302
    leak in mds.c detected by valgrind during CephFS QA run
303
* https://tracker.ceph.com/issues/62067
304
    ffsb.sh failure "Resource temporarily unavailable"
305
* https://tracker.ceph.com/issues/64172
306
    Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize)
307
* https://tracker.ceph.com/issues/63265
308
    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'
309
* https://tracker.ceph.com/issues/61243
310
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
311
* https://tracker.ceph.com/issues/59684
312
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
313
* https://tracker.ceph.com/issues/57656
314
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
315
* https://tracker.ceph.com/issues/64209
316
    snaptest-multiple-capsnaps.sh fails with "got remote process result: 1"
317
318 216 Venky Shankar
h3. 17th Jan 2024
319
320
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240103.072409-1
321
322
* https://tracker.ceph.com/issues/63764
323
    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)
324
* https://tracker.ceph.com/issues/57676
325
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
326
* https://tracker.ceph.com/issues/51964
327
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
328
* https://tracker.ceph.com/issues/63949
329
    leak in mds.c detected by valgrind during CephFS QA run
330
* https://tracker.ceph.com/issues/62067
331
    ffsb.sh failure "Resource temporarily unavailable"
332
* https://tracker.ceph.com/issues/61243
333
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
334
* https://tracker.ceph.com/issues/63259
335
    mds: failed to store backtrace and force file system read-only
336
* https://tracker.ceph.com/issues/63265
337
    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'
338
339
h3. 16 Jan 2024
340 215 Rishabh Dave
341 214 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-12-11_15:37:57-fs-rishabh-2023dec11-testing-default-smithi/
342
https://pulpito.ceph.com/rishabh-2023-12-17_11:19:43-fs-rishabh-2023dec11-testing-default-smithi/
343
https://pulpito.ceph.com/rishabh-2024-01-04_18:43:16-fs-rishabh-2024jan4-testing-default-smithi
344
345
* https://tracker.ceph.com/issues/63764
346
  Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)
347
* https://tracker.ceph.com/issues/63141
348
  qa/cephfs: test_idem_unaffected_root_squash fails
349
* https://tracker.ceph.com/issues/62067
350
  ffsb.sh failure "Resource temporarily unavailable" 
351
* https://tracker.ceph.com/issues/51964
352
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
353
* https://tracker.ceph.com/issues/54462 
354
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
355
* https://tracker.ceph.com/issues/57676
356
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
357
358
* https://tracker.ceph.com/issues/63949
359
  valgrind leak in MDS
360
* https://tracker.ceph.com/issues/64041
361
  qa/cephfs: fs/upgrade/nofs suite attempts to jump more than 2 releases
362
* fsstress failure in last run was due a kernel MM layer failure, unrelated to CephFS
363
* from last run, job #7507400 failed due to MGR; FS wasn't degraded, so it's unrelated to CephFS
364
365 213 Venky Shankar
h3. 06 Dec 2023
366
367
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818
368
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231206.125818-x (rerun w/ squid kickoff changes)
369
370
* https://tracker.ceph.com/issues/63764
371
    Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)
372
* https://tracker.ceph.com/issues/63233
373
    mon|client|mds: valgrind reports possible leaks in the MDS
374
* https://tracker.ceph.com/issues/57676
375
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
376
* https://tracker.ceph.com/issues/62580
377
    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
378
* https://tracker.ceph.com/issues/62067
379
    ffsb.sh failure "Resource temporarily unavailable"
380
* https://tracker.ceph.com/issues/61243
381
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
382
* https://tracker.ceph.com/issues/62081
383
    tasks/fscrypt-common does not finish, timesout
384
* https://tracker.ceph.com/issues/63265
385
    qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'
386
* https://tracker.ceph.com/issues/63806
387
    ffsb.sh workunit failure (MDS: std::out_of_range, damaged)
388
389 211 Patrick Donnelly
h3. 30 Nov 2023
390
391
https://pulpito.ceph.com/pdonnell-2023-11-30_08:05:19-fs:shell-wip-batrick-testing-20231130.014408-distro-default-smithi/
392
393
* https://tracker.ceph.com/issues/63699
394 212 Patrick Donnelly
    qa: failed cephfs-shell test_reading_conf
395
* https://tracker.ceph.com/issues/63700
396
    qa: test_cd_with_args failure
397 211 Patrick Donnelly
398 210 Venky Shankar
h3. 29 Nov 2023
399
400
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231107.042705
401
402
* https://tracker.ceph.com/issues/63233
403
    mon|client|mds: valgrind reports possible leaks in the MDS
404
* https://tracker.ceph.com/issues/63141
405
    qa/cephfs: test_idem_unaffected_root_squash fails
406
* https://tracker.ceph.com/issues/57676
407
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
408
* https://tracker.ceph.com/issues/57655
409
    qa: fs:mixed-clients kernel_untar_build failure
410
* https://tracker.ceph.com/issues/62067
411
    ffsb.sh failure "Resource temporarily unavailable"
412
* https://tracker.ceph.com/issues/61243
413
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
414
* https://tracker.ceph.com/issues/62510 (pending RHEL back port)
    snaptest-git-ceph.sh failure with fs/thrash
415
* https://tracker.ceph.com/issues/62810
416
    Failure in snaptest-git-ceph.sh (it's an async unlink/create bug) -- Need to fix again
417
418 206 Venky Shankar
h3. 14 Nov 2023
419 207 Milind Changire
(Milind)
420
421
https://pulpito.ceph.com/mchangir-2023-11-13_10:27:15-fs-wip-mchangir-testing-20231110.052303-testing-default-smithi/
422
423
* https://tracker.ceph.com/issues/53859
424
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
425
* https://tracker.ceph.com/issues/63233
426
  mon|client|mds: valgrind reports possible leaks in the MDS
427
* https://tracker.ceph.com/issues/63521
428
  qa: Test failure: test_scrub_merge_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks)
429
* https://tracker.ceph.com/issues/57655
430
  qa: fs:mixed-clients kernel_untar_build failure
431
* https://tracker.ceph.com/issues/62580
432
    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
433
* https://tracker.ceph.com/issues/57676
434
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
435
* https://tracker.ceph.com/issues/61243
436
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
437
* https://tracker.ceph.com/issues/63141
438
    qa/cephfs: test_idem_unaffected_root_squash fails
439
* https://tracker.ceph.com/issues/51964
440
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
441
* https://tracker.ceph.com/issues/63522
442
    No module named 'tasks.ceph_fuse'
443
    No module named 'tasks.kclient'
444
    No module named 'tasks.cephfs.fuse_mount'
445
    No module named 'tasks.ceph'
446
* https://tracker.ceph.com/issues/63523
447
    Command failed - qa/workunits/fs/misc/general_vxattrs.sh
448
449
450
h3. 14 Nov 2023
451 206 Venky Shankar
452
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231106.073650
453
454
(nvm the fs:upgrade test failure - the PR is excluded from merge)
455
456
* https://tracker.ceph.com/issues/57676
457
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
458
* https://tracker.ceph.com/issues/63233
459
    mon|client|mds: valgrind reports possible leaks in the MDS
460
* https://tracker.ceph.com/issues/63141
461
    qa/cephfs: test_idem_unaffected_root_squash fails
462
* https://tracker.ceph.com/issues/62580
463
    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
464
* https://tracker.ceph.com/issues/57655
465
    qa: fs:mixed-clients kernel_untar_build failure
466
* https://tracker.ceph.com/issues/51964
467
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
468
* https://tracker.ceph.com/issues/63519
469
    ceph-fuse: reef ceph-fuse crashes with main branch ceph-mds
470
* https://tracker.ceph.com/issues/57087
471
    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure
472
* https://tracker.ceph.com/issues/58945
473
    qa: xfstests-dev's generic test suite has 20 failures with fuse client
474
475 204 Rishabh Dave
h3. 7 Nov 2023
476
477 205 Rishabh Dave
fs: https://pulpito.ceph.com/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/
478
re-run: https://pulpito.ceph.com/rishabh-2023-11-05_14:10:09-fs-rishabh-2023nov3-testing-default-smithi/
479
smoke: https://pulpito.ceph.com/rishabh-2023-11-08_08:39:05-smoke-rishabh-2023nov3-testing-default-smithi/
480 204 Rishabh Dave
481
* https://tracker.ceph.com/issues/53859
482
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
483
* https://tracker.ceph.com/issues/63233
484
  mon|client|mds: valgrind reports possible leaks in the MDS
485
* https://tracker.ceph.com/issues/57655
486
  qa: fs:mixed-clients kernel_untar_build failure
487
* https://tracker.ceph.com/issues/57676
488
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
489
490
* https://tracker.ceph.com/issues/63473
491
  fsstress.sh failed with errno 124
492
493 202 Rishabh Dave
h3. 3 Nov 2023
494 203 Rishabh Dave
495 202 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-10-27_06:26:52-fs-rishabh-2023oct26-testing-default-smithi/
496
497
* https://tracker.ceph.com/issues/63141
498
  qa/cephfs: test_idem_unaffected_root_squash fails
499
* https://tracker.ceph.com/issues/63233
500
  mon|client|mds: valgrind reports possible leaks in the MDS
501
* https://tracker.ceph.com/issues/57656
502
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
503
* https://tracker.ceph.com/issues/57655
504
  qa: fs:mixed-clients kernel_untar_build failure
505
* https://tracker.ceph.com/issues/57676
506
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
507
508
* https://tracker.ceph.com/issues/59531
509
  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
510
* https://tracker.ceph.com/issues/52624
511
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
512
513 198 Patrick Donnelly
h3. 24 October 2023
514
515
https://pulpito.ceph.com/?branch=wip-batrick-testing-20231024.144545
516
517 200 Patrick Donnelly
Two failures:
518
519
https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438459/
520
https://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438468/
521
522
probably related to https://github.com/ceph/ceph/pull/53255. Killing the mount as part of the test did not complete. Will research more.
523
524 198 Patrick Donnelly
* https://tracker.ceph.com/issues/52624
525
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
526
* https://tracker.ceph.com/issues/57676
527 199 Patrick Donnelly
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
528
* https://tracker.ceph.com/issues/63233
529
    mon|client|mds: valgrind reports possible leaks in the MDS
530
* https://tracker.ceph.com/issues/59531
531
    "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS 
532
* https://tracker.ceph.com/issues/57655
533
    qa: fs:mixed-clients kernel_untar_build failure
534 200 Patrick Donnelly
* https://tracker.ceph.com/issues/62067
535
    ffsb.sh failure "Resource temporarily unavailable"
536
* https://tracker.ceph.com/issues/63411
537
    qa: flush journal may cause timeouts of `scrub status`
538
* https://tracker.ceph.com/issues/61243
539
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
540
* https://tracker.ceph.com/issues/63141
541 198 Patrick Donnelly
    test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails
542 148 Rishabh Dave
543 195 Venky Shankar
h3. 18 Oct 2023
544
545
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231018.065603
546
547
* https://tracker.ceph.com/issues/52624
548
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
549
* https://tracker.ceph.com/issues/57676
550
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
551
* https://tracker.ceph.com/issues/63233
552
    mon|client|mds: valgrind reports possible leaks in the MDS
553
* https://tracker.ceph.com/issues/63141
554
    qa/cephfs: test_idem_unaffected_root_squash fails
555
* https://tracker.ceph.com/issues/59531
556
  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 
557
* https://tracker.ceph.com/issues/62658
558
    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
559
* https://tracker.ceph.com/issues/62580
560
    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
561
* https://tracker.ceph.com/issues/62067
562
    ffsb.sh failure "Resource temporarily unavailable"
563
* https://tracker.ceph.com/issues/57655
564
    qa: fs:mixed-clients kernel_untar_build failure
565
* https://tracker.ceph.com/issues/62036
566
    src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty())
567
* https://tracker.ceph.com/issues/58945
568
    qa: xfstests-dev's generic test suite has 20 failures with fuse client
569
* https://tracker.ceph.com/issues/62847
570
    mds: blogbench requests stuck (5mds+scrub+snaps-flush)
571
572 193 Venky Shankar
h3. 13 Oct 2023
573
574
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20231013.093215
575
576
* https://tracker.ceph.com/issues/52624
577
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
578
* https://tracker.ceph.com/issues/62936
579
    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
580
* https://tracker.ceph.com/issues/47292
581
    cephfs-shell: test_df_for_valid_file failure
582
* https://tracker.ceph.com/issues/63141
583
    qa/cephfs: test_idem_unaffected_root_squash fails
584
* https://tracker.ceph.com/issues/62081
585
    tasks/fscrypt-common does not finish, timesout
586 1 Patrick Donnelly
* https://tracker.ceph.com/issues/58945
587
    qa: xfstests-dev's generic test suite has 20 failures with fuse client
588 194 Venky Shankar
* https://tracker.ceph.com/issues/63233
589
    mon|client|mds: valgrind reports possible leaks in the MDS
590 193 Venky Shankar
591 190 Patrick Donnelly
h3. 16 Oct 2023
592
593
https://pulpito.ceph.com/?branch=wip-batrick-testing-20231016.203825
594
595 192 Patrick Donnelly
Infrastructure issues:
596
* /teuthology/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432286/teuthology.log
597
    Host lost.
598
599 196 Patrick Donnelly
One followup fix:
600
* https://pulpito.ceph.com/pdonnell-2023-10-20_00:33:29-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/
601
602 192 Patrick Donnelly
Failures:
603
604
* https://tracker.ceph.com/issues/56694
605
    qa: avoid blocking forever on hung umount
606
* https://tracker.ceph.com/issues/63089
607
    qa: tasks/mirror times out
608
* https://tracker.ceph.com/issues/52624
609
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
610
* https://tracker.ceph.com/issues/59531
611
  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 
612
* https://tracker.ceph.com/issues/57676
613
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
614
* https://tracker.ceph.com/issues/62658 
615
    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
616
* https://tracker.ceph.com/issues/61243
617
  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
618
* https://tracker.ceph.com/issues/57656
619
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
620
* https://tracker.ceph.com/issues/63233
621
  mon|client|mds: valgrind reports possible leaks in the MDS
622 197 Patrick Donnelly
* https://tracker.ceph.com/issues/63278
623
  kclient: may wrongly decode session messages and believe it is blocklisted (dead jobs)
624 192 Patrick Donnelly
625 189 Rishabh Dave
h3. 9 Oct 2023
626
627
https://pulpito.ceph.com/rishabh-2023-10-06_11:56:52-fs-rishabh-cephfs-mon-testing-default-smithi/
628
629
* https://tracker.ceph.com/issues/54460
630
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
631
* https://tracker.ceph.com/issues/63141
632
  test_idem_unaffected_root_squash (test_admin.TestFsAuthorizeUpdate) fails
633
* https://tracker.ceph.com/issues/62937
634
  logrotate doesn't support parallel execution on same set of logfiles
635
* https://tracker.ceph.com/issues/61400
636
  valgrind+ceph-mon issues
637
* https://tracker.ceph.com/issues/57676
638
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
639
* https://tracker.ceph.com/issues/55805
640
  error during scrub thrashing reached max tries in 900 secs
641
642 188 Venky Shankar
h3. 26 Sep 2023
643
644
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230926.081818
645
646
* https://tracker.ceph.com/issues/52624
647
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
648
* https://tracker.ceph.com/issues/62873
649
    qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
650
* https://tracker.ceph.com/issues/61400
651
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
652
* https://tracker.ceph.com/issues/57676
653
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
654
* https://tracker.ceph.com/issues/62682
655
    mon: no mdsmap broadcast after "fs set joinable" is set to true
656
* https://tracker.ceph.com/issues/63089
657
    qa: tasks/mirror times out
658
659 185 Rishabh Dave
h3. 22 Sep 2023
660
661
https://pulpito.ceph.com/rishabh-2023-09-12_12:12:15-fs-wip-rishabh-2023sep12-b2-testing-default-smithi/
662
663
* https://tracker.ceph.com/issues/59348
664
  qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota
665
* https://tracker.ceph.com/issues/59344
666
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
667
* https://tracker.ceph.com/issues/59531
668
  "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)" 
669
* https://tracker.ceph.com/issues/61574
670
  build failure for mdtest project
671
* https://tracker.ceph.com/issues/62702
672
  fsstress.sh: MDS slow requests for the internal 'rename' requests
673
* https://tracker.ceph.com/issues/57676
674
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
675
676
* https://tracker.ceph.com/issues/62863 
677
  deadlock in ceph-fuse causes teuthology job to hang and fail
678
* https://tracker.ceph.com/issues/62870
679
  test_cluster_info (tasks.cephfs.test_nfs.TestNFS)
680
* https://tracker.ceph.com/issues/62873
681
  test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
682
683 186 Venky Shankar
h3. 20 Sep 2023
684
685
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230920.072635
686
687
* https://tracker.ceph.com/issues/52624
688
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
689
* https://tracker.ceph.com/issues/61400
690
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
691
* https://tracker.ceph.com/issues/61399
692
    libmpich: undefined references to fi_strerror
693
* https://tracker.ceph.com/issues/62081
694
    tasks/fscrypt-common does not finish, timesout
695
* https://tracker.ceph.com/issues/62658 
696
    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
697
* https://tracker.ceph.com/issues/62915
698
    qa/suites/fs/nfs: No orchestrator configured (try `ceph orch set backend`) while running test cases
699
* https://tracker.ceph.com/issues/59531
700
    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
701
* https://tracker.ceph.com/issues/62873
702
  qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
703
* https://tracker.ceph.com/issues/62936
704
    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
705
* https://tracker.ceph.com/issues/62937
706
    Command failed on smithi027 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
707
* https://tracker.ceph.com/issues/62510
708
    snaptest-git-ceph.sh failure with fs/thrash
709
* https://tracker.ceph.com/issues/62081
710
    tasks/fscrypt-common does not finish, timesout
711
* https://tracker.ceph.com/issues/62126
712
    test failure: suites/blogbench.sh stops running
713 187 Venky Shankar
* https://tracker.ceph.com/issues/62682
714
    mon: no mdsmap broadcast after "fs set joinable" is set to true
715 186 Venky Shankar
716 184 Milind Changire
h3. 19 Sep 2023
717
718
http://pulpito.front.sepia.ceph.com/mchangir-2023-09-12_05:40:22-fs-wip-mchangir-testing-20230908.140927-testing-default-smithi/
719
720
* https://tracker.ceph.com/issues/58220#note-9
721
  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure
722
* https://tracker.ceph.com/issues/62702
723
  Command failed (workunit test suites/fsstress.sh) on smithi124 with status 124
724
* https://tracker.ceph.com/issues/57676
725
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
726
* https://tracker.ceph.com/issues/59348
727
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
728
* https://tracker.ceph.com/issues/52624
729
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
730
* https://tracker.ceph.com/issues/51964
731
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
732
* https://tracker.ceph.com/issues/61243
733
  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
734
* https://tracker.ceph.com/issues/59344
735
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
736
* https://tracker.ceph.com/issues/62873
737
  qa: FAIL: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
738
* https://tracker.ceph.com/issues/59413
739
  cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"
740
* https://tracker.ceph.com/issues/53859
741
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
742
* https://tracker.ceph.com/issues/62482
743
  qa: cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)
744
745 178 Patrick Donnelly
746 177 Venky Shankar
h3. 13 Sep 2023
747
748
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230908.065909
749
750
* https://tracker.ceph.com/issues/52624
751
      qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
752
* https://tracker.ceph.com/issues/57655
753
    qa: fs:mixed-clients kernel_untar_build failure
754
* https://tracker.ceph.com/issues/57676
755
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
756
* https://tracker.ceph.com/issues/61243
757
    qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed
758
* https://tracker.ceph.com/issues/62567
759
    postgres workunit times out - MDS_SLOW_REQUEST in logs
760
* https://tracker.ceph.com/issues/61400
761
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
762
* https://tracker.ceph.com/issues/61399
763
    libmpich: undefined references to fi_strerror
764
* https://tracker.ceph.com/issues/57655
765
    qa: fs:mixed-clients kernel_untar_build failure
766
* https://tracker.ceph.com/issues/57676
767
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
768
* https://tracker.ceph.com/issues/51964
769
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
770
* https://tracker.ceph.com/issues/62081
771
    tasks/fscrypt-common does not finish, timesout
772 178 Patrick Donnelly
773 179 Patrick Donnelly
h3. 2023 Sep 12
774 178 Patrick Donnelly
775
https://pulpito.ceph.com/pdonnell-2023-09-12_14:07:50-fs-wip-batrick-testing-20230912.122437-distro-default-smithi/
776 1 Patrick Donnelly
777 181 Patrick Donnelly
A few failures caused by qa refactoring in https://github.com/ceph/ceph/pull/48130 ; notably:
778
779 182 Patrick Donnelly
* Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) caused by fragmentation from config changes.
780 181 Patrick Donnelly
781
Failures:
782
783 179 Patrick Donnelly
* https://tracker.ceph.com/issues/59348
784
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
785
* https://tracker.ceph.com/issues/57656
786
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
787
* https://tracker.ceph.com/issues/55805
788
  error scrub thrashing reached max tries in 900 secs
789
* https://tracker.ceph.com/issues/62067
790
    ffsb.sh failure "Resource temporarily unavailable"
791
* https://tracker.ceph.com/issues/59344
792
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
793
* https://tracker.ceph.com/issues/61399
794 180 Patrick Donnelly
  libmpich: undefined references to fi_strerror
795
* https://tracker.ceph.com/issues/62832
796
  common: config_proxy deadlock during shutdown (and possibly other times)
797
* https://tracker.ceph.com/issues/59413
798 1 Patrick Donnelly
  cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"
799 181 Patrick Donnelly
* https://tracker.ceph.com/issues/57676
800
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
801
* https://tracker.ceph.com/issues/62567
802
  Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"
803
* https://tracker.ceph.com/issues/54460
804
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
805
* https://tracker.ceph.com/issues/58220#note-9
806
  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure
807
* https://tracker.ceph.com/issues/59348
808
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
809 183 Patrick Donnelly
* https://tracker.ceph.com/issues/62847
810
    mds: blogbench requests stuck (5mds+scrub+snaps-flush)
811
* https://tracker.ceph.com/issues/62848
812
    qa: fail_fs upgrade scenario hanging
813
* https://tracker.ceph.com/issues/62081
814
    tasks/fscrypt-common does not finish, timesout
815 177 Venky Shankar
816 176 Venky Shankar
h3. 11 Sep 2023
817 175 Venky Shankar
818
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230830.153114
819
820
* https://tracker.ceph.com/issues/52624
821
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
822
* https://tracker.ceph.com/issues/61399
823
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
824
* https://tracker.ceph.com/issues/57655
825
    qa: fs:mixed-clients kernel_untar_build failure
826
* https://tracker.ceph.com/issues/61399
827
    ior build failure
828
* https://tracker.ceph.com/issues/59531
829
    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
830
* https://tracker.ceph.com/issues/59344
831
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
832
* https://tracker.ceph.com/issues/59346
833
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
834
* https://tracker.ceph.com/issues/59348
835
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
836
* https://tracker.ceph.com/issues/57676
837
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
838
* https://tracker.ceph.com/issues/61243
839
  qa: tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev - 17 tests failed
840
* https://tracker.ceph.com/issues/62567
841
  postgres workunit times out - MDS_SLOW_REQUEST in logs
842
843
844 174 Rishabh Dave
h3. 6 Sep 2023 Run 2
845
846
https://pulpito.ceph.com/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/ 
847
848
* https://tracker.ceph.com/issues/51964
849
  test_cephfs_mirror_restart_sync_on_blocklist failure
850
* https://tracker.ceph.com/issues/59348
851
  test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
852
* https://tracker.ceph.com/issues/53859
853
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
854
* https://tracker.ceph.com/issues/61892
855
  test_strays.TestStrays.test_snapshot_remove failed
856
* https://tracker.ceph.com/issues/54460
857
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
858
* https://tracker.ceph.com/issues/59346
859
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
860
* https://tracker.ceph.com/issues/59344
861
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
862
* https://tracker.ceph.com/issues/62484
863
  qa: ffsb.sh test failure
864
* https://tracker.ceph.com/issues/62567
865
  Command failed with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"
866
  
867
* https://tracker.ceph.com/issues/61399
868
  ior build failure
869
* https://tracker.ceph.com/issues/57676
870
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
871
* https://tracker.ceph.com/issues/55805
872
  error scrub thrashing reached max tries in 900 secs
873
874 172 Rishabh Dave
h3. 6 Sep 2023
875 171 Rishabh Dave
876 173 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-08-10_20:16:46-fs-wip-rishabh-2023Aug1-b4-testing-default-smithi/
877 171 Rishabh Dave
878 1 Patrick Donnelly
* https://tracker.ceph.com/issues/53859
879
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
880 173 Rishabh Dave
* https://tracker.ceph.com/issues/51964
881
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
882 1 Patrick Donnelly
* https://tracker.ceph.com/issues/61892
883 173 Rishabh Dave
  test_snapshot_remove (test_strays.TestStrays) failed
884
* https://tracker.ceph.com/issues/59348
885
  qa: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota
886
* https://tracker.ceph.com/issues/54462
887
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
888
* https://tracker.ceph.com/issues/62556
889
  test_acls: xfstests_dev: python2 is missing
890
* https://tracker.ceph.com/issues/62067
891
  ffsb.sh failure "Resource temporarily unavailable"
892
* https://tracker.ceph.com/issues/57656
893
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
894 1 Patrick Donnelly
* https://tracker.ceph.com/issues/59346
895
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
896 171 Rishabh Dave
* https://tracker.ceph.com/issues/59344
897 173 Rishabh Dave
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
898
899 171 Rishabh Dave
* https://tracker.ceph.com/issues/61399
900
  ior build failure
901
* https://tracker.ceph.com/issues/57676
902
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
903
* https://tracker.ceph.com/issues/55805
904
  error scrub thrashing reached max tries in 900 secs
905 173 Rishabh Dave
906
* https://tracker.ceph.com/issues/62567
907
  Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'"
908
* https://tracker.ceph.com/issues/62702
909
  workunit test suites/fsstress.sh on smithi066 with status 124
910 170 Rishabh Dave
911
h3. 5 Sep 2023
912
913
https://pulpito.ceph.com/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/
914
orch:cephadm suite run: http://pulpito.front.sepia.ceph.com/rishabh-2023-09-05_12:16:09-orch:cephadm-wip-rishabh-2023aug3-b5-testing-default-smithi/
915
  this run has failures but acc to Adam King these are not relevant and should be ignored
916
917
* https://tracker.ceph.com/issues/61892
918
  test_snapshot_remove (test_strays.TestStrays) failed
919
* https://tracker.ceph.com/issues/59348
920
  test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota
921
* https://tracker.ceph.com/issues/54462
922
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
923
* https://tracker.ceph.com/issues/62067
924
  ffsb.sh failure "Resource temporarily unavailable"
925
* https://tracker.ceph.com/issues/57656 
926
  dbench: write failed on handle 10010 (Resource temporarily unavailable)
927
* https://tracker.ceph.com/issues/59346
928
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
929
* https://tracker.ceph.com/issues/59344
930
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
931
* https://tracker.ceph.com/issues/50223
932
  client.xxxx isn't responding to mclientcaps(revoke)
933
* https://tracker.ceph.com/issues/57655
934
  qa: fs:mixed-clients kernel_untar_build failure
935
* https://tracker.ceph.com/issues/62187
936
  iozone.sh: line 5: iozone: command not found
937
 
938
* https://tracker.ceph.com/issues/61399
939
  ior build failure
940
* https://tracker.ceph.com/issues/57676
941
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
942
* https://tracker.ceph.com/issues/55805
943
  error scrub thrashing reached max tries in 900 secs
944 169 Venky Shankar
945
946
h3. 31 Aug 2023
947
948
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230824.045828
949
950
* https://tracker.ceph.com/issues/52624
951
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
952
* https://tracker.ceph.com/issues/62187
953
    iozone: command not found
954
* https://tracker.ceph.com/issues/61399
955
    ior build failure
956
* https://tracker.ceph.com/issues/59531
957
    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
958
* https://tracker.ceph.com/issues/61399
959
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
960
* https://tracker.ceph.com/issues/57655
961
    qa: fs:mixed-clients kernel_untar_build failure
962
* https://tracker.ceph.com/issues/59344
963
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
964
* https://tracker.ceph.com/issues/59346
965
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
966
* https://tracker.ceph.com/issues/59348
967
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
968
* https://tracker.ceph.com/issues/59413
969
    cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"
970
* https://tracker.ceph.com/issues/62653
971
    qa: unimplemented fcntl command: 1036 with fsstress
972
* https://tracker.ceph.com/issues/61400
973
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
974
* https://tracker.ceph.com/issues/62658
975
    error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
976
* https://tracker.ceph.com/issues/62188
977
    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
978 168 Venky Shankar
979
980
h3. 25 Aug 2023
981
982
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.064807
983
984
* https://tracker.ceph.com/issues/59344
985
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
986
* https://tracker.ceph.com/issues/59346
987
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
988
* https://tracker.ceph.com/issues/59348
989
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
990
* https://tracker.ceph.com/issues/57655
991
    qa: fs:mixed-clients kernel_untar_build failure
992
* https://tracker.ceph.com/issues/61243
993
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
994
* https://tracker.ceph.com/issues/61399
995
    ior build failure
996
* https://tracker.ceph.com/issues/61399
997
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
998
* https://tracker.ceph.com/issues/62484
999
    qa: ffsb.sh test failure
1000
* https://tracker.ceph.com/issues/59531
1001
    quincy: "OSD bench result of 228617.361065 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)"
1002
* https://tracker.ceph.com/issues/62510
1003
    snaptest-git-ceph.sh failure with fs/thrash
1004 167 Venky Shankar
1005
1006
h3. 24 Aug 2023
1007
1008
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230822.060131
1009
1010
* https://tracker.ceph.com/issues/57676
1011
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1012
* https://tracker.ceph.com/issues/51964
1013
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1014
* https://tracker.ceph.com/issues/59344
1015
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
1016
* https://tracker.ceph.com/issues/59346
1017
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
1018
* https://tracker.ceph.com/issues/59348
1019
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1020
* https://tracker.ceph.com/issues/61399
1021
    ior build failure
1022
* https://tracker.ceph.com/issues/61399
1023
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
1024
* https://tracker.ceph.com/issues/62510
1025
    snaptest-git-ceph.sh failure with fs/thrash
1026
* https://tracker.ceph.com/issues/62484
1027
    qa: ffsb.sh test failure
1028
* https://tracker.ceph.com/issues/57087
1029
    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure
1030
* https://tracker.ceph.com/issues/57656
1031
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1032
* https://tracker.ceph.com/issues/62187
1033
    iozone: command not found
1034
* https://tracker.ceph.com/issues/62188
1035
    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
1036
* https://tracker.ceph.com/issues/62567
1037
    postgres workunit times out - MDS_SLOW_REQUEST in logs
1038 166 Venky Shankar
1039
1040
h3. 22 Aug 2023
1041
1042
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230809.035933
1043
1044
* https://tracker.ceph.com/issues/57676
1045
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1046
* https://tracker.ceph.com/issues/51964
1047
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1048
* https://tracker.ceph.com/issues/59344
1049
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
1050
* https://tracker.ceph.com/issues/59346
1051
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
1052
* https://tracker.ceph.com/issues/59348
1053
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1054
* https://tracker.ceph.com/issues/61399
1055
    ior build failure
1056
* https://tracker.ceph.com/issues/61399
1057
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
1058
* https://tracker.ceph.com/issues/57655
1059
    qa: fs:mixed-clients kernel_untar_build failure
1060
* https://tracker.ceph.com/issues/61243
1061
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
1062
* https://tracker.ceph.com/issues/62188
1063
    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
1064
* https://tracker.ceph.com/issues/62510
1065
    snaptest-git-ceph.sh failure with fs/thrash
1066
* https://tracker.ceph.com/issues/62511
1067
    src/mds/MDLog.cc: 299: FAILED ceph_assert(!mds_is_shutting_down)
1068 165 Venky Shankar
1069
1070
h3. 14 Aug 2023
1071
1072
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230808.093601
1073
1074
* https://tracker.ceph.com/issues/51964
1075
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1076
* https://tracker.ceph.com/issues/61400
1077
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
1078
* https://tracker.ceph.com/issues/61399
1079
    ior build failure
1080
* https://tracker.ceph.com/issues/59348
1081
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1082
* https://tracker.ceph.com/issues/59531
1083
    cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold
1084
* https://tracker.ceph.com/issues/59344
1085
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
1086
* https://tracker.ceph.com/issues/59346
1087
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
1088
* https://tracker.ceph.com/issues/61399
1089
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
1090
* https://tracker.ceph.com/issues/59684 [kclient bug]
1091
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
1092
* https://tracker.ceph.com/issues/61243 (NEW)
1093
    test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
1094
* https://tracker.ceph.com/issues/57655
1095
    qa: fs:mixed-clients kernel_untar_build failure
1096
* https://tracker.ceph.com/issues/57656
1097
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1098 163 Venky Shankar
1099
1100
h3. 28 JULY 2023
1101
1102
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230725.053049
1103
1104
* https://tracker.ceph.com/issues/51964
1105
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1106
* https://tracker.ceph.com/issues/61400
1107
    valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
1108
* https://tracker.ceph.com/issues/61399
1109
    ior build failure
1110
* https://tracker.ceph.com/issues/57676
1111
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1112
* https://tracker.ceph.com/issues/59348
1113
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1114
* https://tracker.ceph.com/issues/59531
1115
    cluster [WRN] OSD bench result of 137706.272521 IOPS exceeded the threshold
1116
* https://tracker.ceph.com/issues/59344
1117
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
1118
* https://tracker.ceph.com/issues/59346
1119
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
1120
* https://github.com/ceph/ceph/pull/52556
1121
    task/test_nfs: AttributeError: 'TestNFS' object has no attribute 'run_ceph_cmd' (see note #4)
1122
* https://tracker.ceph.com/issues/62187
1123
    iozone: command not found
1124
* https://tracker.ceph.com/issues/61399
1125
    qa: build failure for ior (the failed instance is when compiling `mdtest`)
1126
* https://tracker.ceph.com/issues/62188
1127 164 Rishabh Dave
    AttributeError: 'RemoteProcess' object has no attribute 'read' (happens only with multis-auth test)
1128 158 Rishabh Dave
1129
h3. 24 Jul 2023
1130
1131
https://pulpito.ceph.com/rishabh-2023-07-13_21:35:13-fs-wip-rishabh-2023Jul13-testing-default-smithi/
1132
https://pulpito.ceph.com/rishabh-2023-07-14_10:26:42-fs-wip-rishabh-2023Jul13-testing-default-smithi/
1133
There were few failure from one of the PRs under testing. Following run confirms that removing this PR fixes these failures -
1134
https://pulpito.ceph.com/rishabh-2023-07-18_02:11:50-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/
1135
One more extra run to check if blogbench.sh fail every time:
1136
https://pulpito.ceph.com/rishabh-2023-07-21_17:58:19-fs-wip-rishabh-2023Jul13-m-quota-testing-default-smithi/
1137
blogbench.sh failure were seen on above runs for first time, following run with main branch that confirms that "blogbench.sh" was not related to any of the PRs that are under testing -
1138 161 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-07-21_21:30:53-fs-wip-rishabh-2023Jul13-base-2-testing-default-smithi/
1139
1140
* https://tracker.ceph.com/issues/61892
1141
  test_snapshot_remove (test_strays.TestStrays) failed
1142
* https://tracker.ceph.com/issues/53859
1143
  test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1144
* https://tracker.ceph.com/issues/61982
1145
  test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
1146
* https://tracker.ceph.com/issues/52438
1147
  qa: ffsb timeout
1148
* https://tracker.ceph.com/issues/54460
1149
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1150
* https://tracker.ceph.com/issues/57655
1151
  qa: fs:mixed-clients kernel_untar_build failure
1152
* https://tracker.ceph.com/issues/48773
1153
  reached max tries: scrub does not complete
1154
* https://tracker.ceph.com/issues/58340
1155
  mds: fsstress.sh hangs with multimds
1156
* https://tracker.ceph.com/issues/61400
1157
  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
1158
* https://tracker.ceph.com/issues/57206
1159
  libcephfs/test.sh: ceph_test_libcephfs_reclaim
1160
  
1161
* https://tracker.ceph.com/issues/57656
1162
  [testing] dbench: write failed on handle 10010 (Resource temporarily unavailable)
1163
* https://tracker.ceph.com/issues/61399
1164
  ior build failure
1165
* https://tracker.ceph.com/issues/57676
1166
  error during scrub thrashing: backtrace
1167
  
1168
* https://tracker.ceph.com/issues/38452
1169
  'sudo -u postgres -- pgbench -s 500 -i' failed
1170 158 Rishabh Dave
* https://tracker.ceph.com/issues/62126
1171 157 Venky Shankar
  blogbench.sh failure
1172
1173
h3. 18 July 2023
1174
1175
* https://tracker.ceph.com/issues/52624
1176
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1177
* https://tracker.ceph.com/issues/57676
1178
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1179
* https://tracker.ceph.com/issues/54460
1180
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1181
* https://tracker.ceph.com/issues/57655
1182
    qa: fs:mixed-clients kernel_untar_build failure
1183
* https://tracker.ceph.com/issues/51964
1184
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1185
* https://tracker.ceph.com/issues/59344
1186
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
1187
* https://tracker.ceph.com/issues/61182
1188
    cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds
1189
* https://tracker.ceph.com/issues/61957
1190
    test_client_limits.TestClientLimits.test_client_release_bug
1191
* https://tracker.ceph.com/issues/59348
1192
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1193
* https://tracker.ceph.com/issues/61892
1194
    test_strays.TestStrays.test_snapshot_remove failed
1195
* https://tracker.ceph.com/issues/59346
1196
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
1197
* https://tracker.ceph.com/issues/44565
1198
    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())
1199
* https://tracker.ceph.com/issues/62067
1200
    ffsb.sh failure "Resource temporarily unavailable"
1201 156 Venky Shankar
1202
1203
h3. 17 July 2023
1204
1205
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230704.040136
1206
1207
* https://tracker.ceph.com/issues/61982
1208
    Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
1209
* https://tracker.ceph.com/issues/59344
1210
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
1211
* https://tracker.ceph.com/issues/61182
1212
    cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds
1213
* https://tracker.ceph.com/issues/61957
1214
    test_client_limits.TestClientLimits.test_client_release_bug
1215
* https://tracker.ceph.com/issues/61400
1216
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
1217
* https://tracker.ceph.com/issues/59348
1218
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1219
* https://tracker.ceph.com/issues/61892
1220
    test_strays.TestStrays.test_snapshot_remove failed
1221
* https://tracker.ceph.com/issues/59346
1222
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
1223
* https://tracker.ceph.com/issues/62036
1224
    src/mds/MDCache.cc: 5131: FAILED ceph_assert(isolated_inodes.empty())
1225
* https://tracker.ceph.com/issues/61737
1226
    coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific'
1227
* https://tracker.ceph.com/issues/44565
1228
    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())
1229 155 Rishabh Dave
1230 1 Patrick Donnelly
1231 153 Rishabh Dave
h3. 13 July 2023 Run 2
1232 152 Rishabh Dave
1233
1234
https://pulpito.ceph.com/rishabh-2023-07-08_23:33:40-fs-wip-rishabh-2023Jul9-testing-default-smithi/
1235
https://pulpito.ceph.com/rishabh-2023-07-09_20:19:09-fs-wip-rishabh-2023Jul9-testing-default-smithi/
1236
1237
* https://tracker.ceph.com/issues/61957
1238
  test_client_limits.TestClientLimits.test_client_release_bug
1239
* https://tracker.ceph.com/issues/61982
1240
  Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
1241
* https://tracker.ceph.com/issues/59348
1242
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1243
* https://tracker.ceph.com/issues/59344
1244
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
1245
* https://tracker.ceph.com/issues/54460
1246
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1247
* https://tracker.ceph.com/issues/57655
1248
  qa: fs:mixed-clients kernel_untar_build failure
1249
* https://tracker.ceph.com/issues/61400
1250
  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
1251
* https://tracker.ceph.com/issues/61399
1252
  ior build failure
1253
1254 151 Venky Shankar
h3. 13 July 2023
1255
1256
https://pulpito.ceph.com/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/
1257
1258
* https://tracker.ceph.com/issues/54460
1259
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1260
* https://tracker.ceph.com/issues/61400
1261
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
1262
* https://tracker.ceph.com/issues/57655
1263
    qa: fs:mixed-clients kernel_untar_build failure
1264
* https://tracker.ceph.com/issues/61945
1265
    LibCephFS.DelegTimeout failure
1266
* https://tracker.ceph.com/issues/52624
1267
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1268
* https://tracker.ceph.com/issues/57676
1269
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1270
* https://tracker.ceph.com/issues/59348
1271
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1272
* https://tracker.ceph.com/issues/59344
1273
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
1274
* https://tracker.ceph.com/issues/51964
1275
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1276
* https://tracker.ceph.com/issues/59346
1277
    fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
1278
* https://tracker.ceph.com/issues/61982
1279
    Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
1280 150 Rishabh Dave
1281
1282
h3. 13 Jul 2023
1283
1284
https://pulpito.ceph.com/rishabh-2023-07-05_22:21:20-fs-wip-rishabh-2023Jul5-testing-default-smithi/
1285
https://pulpito.ceph.com/rishabh-2023-07-06_19:33:28-fs-wip-rishabh-2023Jul5-testing-default-smithi/
1286
1287
* https://tracker.ceph.com/issues/61957
1288
  test_client_limits.TestClientLimits.test_client_release_bug
1289
* https://tracker.ceph.com/issues/59348
1290
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1291
* https://tracker.ceph.com/issues/59346
1292
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write" 
1293
* https://tracker.ceph.com/issues/48773
1294
  scrub does not complete: reached max tries
1295
* https://tracker.ceph.com/issues/59344
1296
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument" 
1297
* https://tracker.ceph.com/issues/52438
1298
  qa: ffsb timeout
1299
* https://tracker.ceph.com/issues/57656
1300
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1301
* https://tracker.ceph.com/issues/58742
1302
  xfstests-dev: kcephfs: generic
1303
* https://tracker.ceph.com/issues/61399
1304 148 Rishabh Dave
  libmpich: undefined references to fi_strerror
1305 149 Rishabh Dave
1306 148 Rishabh Dave
h3. 12 July 2023
1307
1308
https://pulpito.ceph.com/rishabh-2023-07-05_18:32:52-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/
1309
https://pulpito.ceph.com/rishabh-2023-07-06_19:46:43-fs-wip-rishabh-CephManager-in-CephFSTestCase-testing-default-smithi/
1310
1311
* https://tracker.ceph.com/issues/61892
1312
  test_strays.TestStrays.test_snapshot_remove failed
1313
* https://tracker.ceph.com/issues/59348
1314
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1315
* https://tracker.ceph.com/issues/53859
1316
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1317
* https://tracker.ceph.com/issues/59346
1318
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
1319
* https://tracker.ceph.com/issues/58742
1320
  xfstests-dev: kcephfs: generic
1321
* https://tracker.ceph.com/issues/59344
1322
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
1323
* https://tracker.ceph.com/issues/52438
1324
  qa: ffsb timeout
1325
* https://tracker.ceph.com/issues/57656
1326
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1327
* https://tracker.ceph.com/issues/54460
1328
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1329
* https://tracker.ceph.com/issues/57655
1330
  qa: fs:mixed-clients kernel_untar_build failure
1331
* https://tracker.ceph.com/issues/61182
1332
  cephfs-mirror-ha-workunit: reached maximum tries (50) after waiting for 300 seconds
1333
* https://tracker.ceph.com/issues/61400
1334
  valgrind+ceph-mon issues: sudo ceph --cluster ceph osd crush tunables default
1335 147 Rishabh Dave
* https://tracker.ceph.com/issues/48773
1336 146 Patrick Donnelly
  reached max tries: scrub does not complete
1337
1338
h3. 05 July 2023
1339
1340
https://pulpito.ceph.com/pdonnell-2023-07-05_03:38:33-fs:libcephfs-wip-pdonnell-testing-20230705.003205-distro-default-smithi/
1341
1342 137 Rishabh Dave
* https://tracker.ceph.com/issues/59346
1343 143 Rishabh Dave
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
1344
1345
h3. 27 Jun 2023
1346
1347
https://pulpito.ceph.com/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/
1348 144 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/
1349
1350
* https://tracker.ceph.com/issues/59348
1351
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1352
* https://tracker.ceph.com/issues/54460
1353
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1354
* https://tracker.ceph.com/issues/59346
1355
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
1356
* https://tracker.ceph.com/issues/59344
1357
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
1358
* https://tracker.ceph.com/issues/61399
1359
  libmpich: undefined references to fi_strerror
1360
* https://tracker.ceph.com/issues/50223
1361
  client.xxxx isn't responding to mclientcaps(revoke)
1362 143 Rishabh Dave
* https://tracker.ceph.com/issues/61831
1363
  Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
1364 142 Venky Shankar
1365
1366
h3. 22 June 2023
1367
1368
* https://tracker.ceph.com/issues/57676
1369
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1370
* https://tracker.ceph.com/issues/54460
1371
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1372
* https://tracker.ceph.com/issues/59344
1373
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
1374
* https://tracker.ceph.com/issues/59348
1375
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1376
* https://tracker.ceph.com/issues/61400
1377
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
1378
* https://tracker.ceph.com/issues/57655
1379
    qa: fs:mixed-clients kernel_untar_build failure
1380
* https://tracker.ceph.com/issues/61394
1381
    qa/quincy: cluster [WRN] evicting unresponsive client smithi152 (4298), after 303.726 seconds" in cluster log
1382
* https://tracker.ceph.com/issues/61762
1383
    qa: wait_for_clean: failed before timeout expired
1384
* https://tracker.ceph.com/issues/61775
1385
    cephfs-mirror: mirror daemon does not shutdown (in mirror ha tests)
1386
* https://tracker.ceph.com/issues/44565
1387
    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())
1388
* https://tracker.ceph.com/issues/61790
1389
    cephfs client to mds comms remain silent after reconnect
1390
* https://tracker.ceph.com/issues/61791
1391
    snaptest-git-ceph.sh test timed out (job dead)
1392 139 Venky Shankar
1393
1394
h3. 20 June 2023
1395
1396
https://pulpito.ceph.com/vshankar-2023-06-15_04:58:28-fs-wip-vshankar-testing-20230614.124123-testing-default-smithi/
1397
1398
* https://tracker.ceph.com/issues/57676
1399
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1400
* https://tracker.ceph.com/issues/54460
1401
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1402 140 Venky Shankar
* https://tracker.ceph.com/issues/54462
1403 1 Patrick Donnelly
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
1404 141 Venky Shankar
* https://tracker.ceph.com/issues/58340
1405 139 Venky Shankar
  mds: fsstress.sh hangs with multimds
1406
* https://tracker.ceph.com/issues/59344
1407
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
1408
* https://tracker.ceph.com/issues/59348
1409
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1410
* https://tracker.ceph.com/issues/57656
1411
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1412
* https://tracker.ceph.com/issues/61400
1413
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
1414
* https://tracker.ceph.com/issues/57655
1415
    qa: fs:mixed-clients kernel_untar_build failure
1416
* https://tracker.ceph.com/issues/44565
1417
    src/mds/SimpleLock.h: 528: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE || state == LOCK_XLOCKSNAP || state == LOCK_LOCK_XLOCK || state == LOCK_LOCK || is_locallock())
1418
* https://tracker.ceph.com/issues/61737
1419 138 Rishabh Dave
    coredump from '/bin/podman pull quay.ceph.io/ceph-ci/ceph:pacific'
1420
1421
h3. 16 June 2023
1422
1423 1 Patrick Donnelly
https://pulpito.ceph.com/rishabh-2023-05-16_10:39:13-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
1424 145 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-05-17_11:09:48-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
1425 138 Rishabh Dave
https://pulpito.ceph.com/rishabh-2023-05-18_10:01:53-fs-wip-rishabh-2023May15-1524-testing-default-smithi/
1426 1 Patrick Donnelly
(bins were rebuilt with a subset of orig PRs) https://pulpito.ceph.com/rishabh-2023-06-09_10:19:22-fs-wip-rishabh-2023Jun9-1308-testing-default-smithi/
1427
1428
1429
* https://tracker.ceph.com/issues/59344
1430
  qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
1431 138 Rishabh Dave
* https://tracker.ceph.com/issues/59348
1432
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1433 145 Rishabh Dave
* https://tracker.ceph.com/issues/59346
1434
  fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
1435
* https://tracker.ceph.com/issues/57656
1436
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1437
* https://tracker.ceph.com/issues/54460
1438
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1439 138 Rishabh Dave
* https://tracker.ceph.com/issues/54462
1440
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
1441 145 Rishabh Dave
* https://tracker.ceph.com/issues/61399
1442
  libmpich: undefined references to fi_strerror
1443
* https://tracker.ceph.com/issues/58945
1444
  xfstests-dev: ceph-fuse: generic 
1445 138 Rishabh Dave
* https://tracker.ceph.com/issues/58742
1446 136 Patrick Donnelly
  xfstests-dev: kcephfs: generic
1447
1448
h3. 24 May 2023
1449
1450
https://pulpito.ceph.com/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/
1451
1452
* https://tracker.ceph.com/issues/57676
1453
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1454
* https://tracker.ceph.com/issues/59683
1455
    Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests
1456
* https://tracker.ceph.com/issues/61399
1457
    qa: "[Makefile:299: ior] Error 1"
1458
* https://tracker.ceph.com/issues/61265
1459
    qa: tasks.cephfs.fuse_mount:process failed to terminate after unmount
1460
* https://tracker.ceph.com/issues/59348
1461
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1462
* https://tracker.ceph.com/issues/59346
1463
    qa/workunits/fs/test_python.sh failed with "AssertionError: DiskQuotaExceeded not raised by write"
1464
* https://tracker.ceph.com/issues/61400
1465
    valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
1466
* https://tracker.ceph.com/issues/54460
1467
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1468
* https://tracker.ceph.com/issues/51964
1469
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1470
* https://tracker.ceph.com/issues/59344
1471
    qa: workunit test fs/quota/quota.sh failed with "setfattr: .: Invalid argument"
1472
* https://tracker.ceph.com/issues/61407
1473
    mds: abort on CInode::verify_dirfrags
1474
* https://tracker.ceph.com/issues/48773
1475
    qa: scrub does not complete
1476
* https://tracker.ceph.com/issues/57655
1477
    qa: fs:mixed-clients kernel_untar_build failure
1478
* https://tracker.ceph.com/issues/61409
1479 128 Venky Shankar
    qa: _test_stale_caps does not wait for file flush before stat
1480
1481
h3. 15 May 2023
1482 130 Venky Shankar
1483 128 Venky Shankar
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.090020
1484
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.090020-6
1485
1486
* https://tracker.ceph.com/issues/52624
1487
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1488
* https://tracker.ceph.com/issues/54460
1489
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1490
* https://tracker.ceph.com/issues/57676
1491
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1492
* https://tracker.ceph.com/issues/59684 [kclient bug]
1493
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
1494
* https://tracker.ceph.com/issues/59348
1495
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1496 131 Venky Shankar
* https://tracker.ceph.com/issues/61148
1497
    dbench test results in call trace in dmesg [kclient bug]
1498 133 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/58340
1499 134 Kotresh Hiremath Ravishankar
    mds: fsstress.sh hangs with multimds
1500 125 Venky Shankar
1501
 
1502 129 Rishabh Dave
h3. 11 May 2023
1503
1504
https://pulpito.ceph.com/yuriw-2023-05-10_18:21:40-fs-wip-yuri7-testing-2023-05-10-0742-distro-default-smithi/
1505
1506
* https://tracker.ceph.com/issues/59684 [kclient bug]
1507
  Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
1508
* https://tracker.ceph.com/issues/59348
1509
  qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1510
* https://tracker.ceph.com/issues/57655
1511
  qa: fs:mixed-clients kernel_untar_build failure
1512
* https://tracker.ceph.com/issues/57676
1513
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
1514
* https://tracker.ceph.com/issues/55805
1515
  error during scrub thrashing reached max tries in 900 secs
1516
* https://tracker.ceph.com/issues/54460
1517
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1518
* https://tracker.ceph.com/issues/57656
1519
  [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1520
* https://tracker.ceph.com/issues/58220
1521
  Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:
1522 1 Patrick Donnelly
* https://tracker.ceph.com/issues/58220#note-9
1523
  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure
1524 134 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/59342
1525
  qa/workunits/kernel_untar_build.sh failed when compiling the Linux source
1526 135 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/58949
1527
    test_cephfs.test_disk_quota_exceeeded_error - AssertionError: DiskQuotaExceeded not raised by write
1528 129 Rishabh Dave
* https://tracker.ceph.com/issues/61243 (NEW)
1529
  test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) failed
1530
1531 125 Venky Shankar
h3. 11 May 2023
1532 127 Venky Shankar
1533
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230509.054005
1534 126 Venky Shankar
1535 125 Venky Shankar
(no fsstress job failure [https://tracker.ceph.com/issues/58340] since https://github.com/ceph/ceph/pull/49553
1536
 was included in the branch, however, the PR got updated and needs retest).
1537
1538
* https://tracker.ceph.com/issues/52624
1539
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1540
* https://tracker.ceph.com/issues/54460
1541
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1542
* https://tracker.ceph.com/issues/57676
1543
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1544
* https://tracker.ceph.com/issues/59683
1545
    Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests
1546
* https://tracker.ceph.com/issues/59684 [kclient bug]
1547
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
1548
* https://tracker.ceph.com/issues/59348
1549 124 Venky Shankar
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1550
1551
h3. 09 May 2023
1552
1553
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230506.143554
1554
1555
* https://tracker.ceph.com/issues/52624
1556
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1557
* https://tracker.ceph.com/issues/58340
1558
    mds: fsstress.sh hangs with multimds
1559
* https://tracker.ceph.com/issues/54460
1560
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1561
* https://tracker.ceph.com/issues/57676
1562
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1563
* https://tracker.ceph.com/issues/51964
1564
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1565
* https://tracker.ceph.com/issues/59350
1566
    qa: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ERROR
1567
* https://tracker.ceph.com/issues/59683
1568
    Error: Unable to find a match: userspace-rcu-devel libedit-devel device-mapper-devel with fscrypt tests
1569
* https://tracker.ceph.com/issues/59684 [kclient bug]
1570
    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
1571
* https://tracker.ceph.com/issues/59348
1572 123 Venky Shankar
    qa: ERROR: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
1573
1574
h3. 10 Apr 2023
1575
1576
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230330.105356
1577
1578
* https://tracker.ceph.com/issues/52624
1579
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1580
* https://tracker.ceph.com/issues/58340
1581
    mds: fsstress.sh hangs with multimds
1582
* https://tracker.ceph.com/issues/54460
1583
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1584
* https://tracker.ceph.com/issues/57676
1585
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1586 119 Rishabh Dave
* https://tracker.ceph.com/issues/51964
1587 120 Rishabh Dave
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1588 121 Rishabh Dave
1589 120 Rishabh Dave
h3. 31 Mar 2023
1590 122 Rishabh Dave
1591
run: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/
1592 120 Rishabh Dave
re-run (some PRs removed from batch): http://pulpito.front.sepia.ceph.com/rishabh-2023-03-11_05:54:03-fs-wip-rishabh-2023Mar10-1727-testing-default-smithi/
1593
re-run (some PRs removed from batch): http://pulpito.front.sepia.ceph.com/rishabh-2023-03-23_08:27:28-fs-wip-rishabh-2023Mar20-2250-testing-default-smithi/
1594
1595
There were many more re-runs for "failed+dead" jobs as well as for individual jobs. half of the PRs from the batch were removed (gradually over subsequent re-runs).
1596
1597
* https://tracker.ceph.com/issues/57676
1598
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
1599
* https://tracker.ceph.com/issues/54460
1600
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1601
* https://tracker.ceph.com/issues/58220
1602
  Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
1603
* https://tracker.ceph.com/issues/58220#note-9
1604
  workunit fs/test_python.sh: test_disk_quota_exceeeded_error failure
1605
* https://tracker.ceph.com/issues/56695
1606
  Command failed (workunit test suites/pjd.sh)
1607
* https://tracker.ceph.com/issues/58564 
1608
  workuit dbench failed with error code 1
1609
* https://tracker.ceph.com/issues/57206
1610
  libcephfs/test.sh: ceph_test_libcephfs_reclaim
1611
* https://tracker.ceph.com/issues/57580
1612
  Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
1613
* https://tracker.ceph.com/issues/58940
1614
  ceph osd hit ceph_abort
1615
* https://tracker.ceph.com/issues/55805
1616 118 Venky Shankar
  error scrub thrashing reached max tries in 900 secs
1617
1618
h3. 30 March 2023
1619
1620
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230315.085747
1621
1622
* https://tracker.ceph.com/issues/58938
1623
    qa: xfstests-dev's generic test suite has 7 failures with kclient
1624
* https://tracker.ceph.com/issues/51964
1625
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1626
* https://tracker.ceph.com/issues/58340
1627 114 Venky Shankar
    mds: fsstress.sh hangs with multimds
1628
1629 115 Venky Shankar
h3. 29 March 2023
1630 114 Venky Shankar
1631
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230317.095222
1632
1633
* https://tracker.ceph.com/issues/56695
1634
    [RHEL stock] pjd test failures
1635
* https://tracker.ceph.com/issues/57676
1636
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1637
* https://tracker.ceph.com/issues/57087
1638
    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure
1639 116 Venky Shankar
* https://tracker.ceph.com/issues/58340
1640
    mds: fsstress.sh hangs with multimds
1641 114 Venky Shankar
* https://tracker.ceph.com/issues/57655
1642
    qa: fs:mixed-clients kernel_untar_build failure
1643 117 Venky Shankar
* https://tracker.ceph.com/issues/59230
1644
    Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage)
1645 114 Venky Shankar
* https://tracker.ceph.com/issues/58938
1646 113 Venky Shankar
    qa: xfstests-dev's generic test suite has 7 failures with kclient
1647
1648
h3. 13 Mar 2023
1649
1650
* https://tracker.ceph.com/issues/56695
1651
    [RHEL stock] pjd test failures
1652
* https://tracker.ceph.com/issues/57676
1653
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1654
* https://tracker.ceph.com/issues/51964
1655
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1656
* https://tracker.ceph.com/issues/54460
1657
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1658
* https://tracker.ceph.com/issues/57656
1659 112 Venky Shankar
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1660
1661
h3. 09 Mar 2023
1662
1663
https://pulpito.ceph.com/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/
1664
https://pulpito.ceph.com/vshankar-2023-03-08_15:12:36-fs-wip-vshankar-testing-20230308.112059-testing-default-smithi/
1665
1666
* https://tracker.ceph.com/issues/56695
1667
    [RHEL stock] pjd test failures
1668
* https://tracker.ceph.com/issues/57676
1669
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1670
* https://tracker.ceph.com/issues/51964
1671
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1672
* https://tracker.ceph.com/issues/54460
1673
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1674
* https://tracker.ceph.com/issues/58340
1675
    mds: fsstress.sh hangs with multimds
1676
* https://tracker.ceph.com/issues/57087
1677 111 Venky Shankar
    qa: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) failure
1678
1679
h3. 07 Mar 2023
1680
1681
https://pulpito.ceph.com/vshankar-2023-03-02_09:21:58-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/
1682
https://pulpito.ceph.com/vshankar-2023-03-07_05:15:12-fs-wip-vshankar-testing-20230307.030510-testing-default-smithi/
1683
1684
* https://tracker.ceph.com/issues/56695
1685
    [RHEL stock] pjd test failures
1686
* https://tracker.ceph.com/issues/57676
1687
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1688
* https://tracker.ceph.com/issues/51964
1689
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1690
* https://tracker.ceph.com/issues/57656
1691
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1692
* https://tracker.ceph.com/issues/57655
1693
    qa: fs:mixed-clients kernel_untar_build failure
1694
* https://tracker.ceph.com/issues/58220
1695
    Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:
1696
* https://tracker.ceph.com/issues/54460
1697
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1698
* https://tracker.ceph.com/issues/58934
1699 109 Venky Shankar
    snaptest-git-ceph.sh failure with ceph-fuse
1700
1701
h3. 28 Feb 2023
1702
1703
https://pulpito.ceph.com/vshankar-2023-02-24_02:11:45-fs-wip-vshankar-testing-20230222.025426-testing-default-smithi/
1704
1705
* https://tracker.ceph.com/issues/56695
1706
    [RHEL stock] pjd test failures
1707
* https://tracker.ceph.com/issues/57676
1708
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1709 110 Venky Shankar
* https://tracker.ceph.com/issues/56446
1710 109 Venky Shankar
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1711
1712 107 Venky Shankar
(teuthology infra issues causing testing delays - merging PRs which have tests passing)
1713
1714
h3. 25 Jan 2023
1715
1716
https://pulpito.ceph.com/vshankar-2023-01-25_07:57:32-fs-wip-vshankar-testing-20230125.055346-testing-default-smithi/
1717
1718
* https://tracker.ceph.com/issues/52624
1719
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1720
* https://tracker.ceph.com/issues/56695
1721
    [RHEL stock] pjd test failures
1722
* https://tracker.ceph.com/issues/57676
1723
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1724
* https://tracker.ceph.com/issues/56446
1725
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1726
* https://tracker.ceph.com/issues/57206
1727
    libcephfs/test.sh: ceph_test_libcephfs_reclaim
1728
* https://tracker.ceph.com/issues/58220
1729
    Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:
1730
* https://tracker.ceph.com/issues/58340
1731
  mds: fsstress.sh hangs with multimds
1732
* https://tracker.ceph.com/issues/56011
1733
    fs/thrash: snaptest-snap-rm-cmp.sh fails in mds5sum comparison
1734
* https://tracker.ceph.com/issues/54460
1735 101 Rishabh Dave
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1736
1737
h3. 30 JAN 2023
1738
1739
run: http://pulpito.front.sepia.ceph.com/rishabh-2022-11-28_08:04:11-fs-wip-rishabh-testing-2022Nov24-1818-testing-default-smithi/
1740
re-run: http://pulpito.front.sepia.ceph.com/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/
1741 105 Rishabh Dave
re-run of re-run: http://pulpito.front.sepia.ceph.com/rishabh-2023-01-23_18:53:32-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/
1742
1743 101 Rishabh Dave
* https://tracker.ceph.com/issues/52624
1744
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" 
1745
* https://tracker.ceph.com/issues/56695
1746
  [RHEL stock] pjd test failures
1747
* https://tracker.ceph.com/issues/57676
1748
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
1749
* https://tracker.ceph.com/issues/55332
1750
  Failure in snaptest-git-ceph.sh
1751
* https://tracker.ceph.com/issues/51964
1752
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1753
* https://tracker.ceph.com/issues/56446
1754
  Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1755
* https://tracker.ceph.com/issues/57655 
1756
  qa: fs:mixed-clients kernel_untar_build failure
1757
* https://tracker.ceph.com/issues/54460
1758
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1759 103 Rishabh Dave
* https://tracker.ceph.com/issues/58340
1760
  mds: fsstress.sh hangs with multimds
1761 101 Rishabh Dave
* https://tracker.ceph.com/issues/58219
1762 102 Rishabh Dave
  Command crashed: 'ceph-dencoder type inode_backtrace_t import - decode dump_json'
1763
1764
* "Failed to load ceph-mgr modules: prometheus" in cluster log"
1765 106 Rishabh Dave
  http://pulpito.front.sepia.ceph.com/rishabh-2023-01-23_18:53:32-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7134086
1766
  Acc to Venky this was fixed in https://github.com/ceph/ceph/commit/cf6089200d96fc56b08ee17a4e31f19823370dc8
1767 102 Rishabh Dave
* Created https://tracker.ceph.com/issues/58564
1768 100 Venky Shankar
  workunit test suites/dbench.sh failed error code 1
1769
1770
h3. 15 Dec 2022
1771
1772
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221215.112736
1773
1774
* https://tracker.ceph.com/issues/52624
1775
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1776
* https://tracker.ceph.com/issues/56695
1777
    [RHEL stock] pjd test failures
1778
* https://tracker.ceph.com/issues/58219
1779
* https://tracker.ceph.com/issues/57655
1780
* qa: fs:mixed-clients kernel_untar_build failure
1781
    Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
1782
* https://tracker.ceph.com/issues/57676
1783
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1784
* https://tracker.ceph.com/issues/58340
1785 96 Venky Shankar
    mds: fsstress.sh hangs with multimds
1786
1787
h3. 08 Dec 2022
1788 99 Venky Shankar
1789 96 Venky Shankar
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221130.043104
1790
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20221209.043803
1791
1792
(lots of transient git.ceph.com failures)
1793
1794
* https://tracker.ceph.com/issues/52624
1795
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1796
* https://tracker.ceph.com/issues/56695
1797
    [RHEL stock] pjd test failures
1798
* https://tracker.ceph.com/issues/57655
1799
    qa: fs:mixed-clients kernel_untar_build failure
1800
* https://tracker.ceph.com/issues/58219
1801
    Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
1802
* https://tracker.ceph.com/issues/58220
1803
    Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:
1804 97 Venky Shankar
* https://tracker.ceph.com/issues/57676
1805
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1806 98 Venky Shankar
* https://tracker.ceph.com/issues/53859
1807
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1808
* https://tracker.ceph.com/issues/54460
1809
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1810 96 Venky Shankar
* https://tracker.ceph.com/issues/58244
1811 95 Venky Shankar
    Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan)
1812
1813
h3. 14 Oct 2022
1814
1815
https://pulpito.ceph.com/vshankar-2022-10-12_04:56:59-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/
1816
https://pulpito.ceph.com/vshankar-2022-10-14_04:04:57-fs-wip-vshankar-testing-20221014-072608-testing-default-smithi/
1817
1818
* https://tracker.ceph.com/issues/52624
1819
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1820
* https://tracker.ceph.com/issues/55804
1821
    Command failed (workunit test suites/pjd.sh)
1822
* https://tracker.ceph.com/issues/51964
1823
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1824
* https://tracker.ceph.com/issues/57682
1825
    client: ERROR: test_reconnect_after_blocklisted
1826 90 Rishabh Dave
* https://tracker.ceph.com/issues/54460
1827 91 Rishabh Dave
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1828
1829
h3. 10 Oct 2022
1830 92 Rishabh Dave
1831 91 Rishabh Dave
http://pulpito.front.sepia.ceph.com/rishabh-2022-09-30_19:45:21-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
1832
1833
reruns
1834
* fs-thrash, passed: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-04_13:19:47-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
1835 94 Rishabh Dave
* fs-verify, passed: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-05_12:25:37-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
1836 91 Rishabh Dave
* cephadm failures also passed after many re-runs: http://pulpito.front.sepia.ceph.com/rishabh-2022-10-06_13:50:51-fs-wip-rishabh-testing-30Sep2022-2-testing-default-smithi/
1837 93 Rishabh Dave
    ** needed this PR to be merged in ceph-ci branch - https://github.com/ceph/ceph/pull/47458
1838 91 Rishabh Dave
1839
known bugs
1840
* https://tracker.ceph.com/issues/52624
1841
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1842
* https://tracker.ceph.com/issues/50223
1843
  client.xxxx isn't responding to mclientcaps(revoke
1844
* https://tracker.ceph.com/issues/57299
1845
  qa: test_dump_loads fails with JSONDecodeError
1846
* https://tracker.ceph.com/issues/57655 [Exist in main as well]
1847
  qa: fs:mixed-clients kernel_untar_build failure
1848
* https://tracker.ceph.com/issues/57206
1849 90 Rishabh Dave
  libcephfs/test.sh: ceph_test_libcephfs_reclaim
1850
1851
h3. 2022 Sep 29
1852
1853
http://pulpito.front.sepia.ceph.com/rishabh-2022-09-14_12:48:43-fs-wip-rishabh-testing-2022Sep9-1708-testing-default-smithi/
1854
1855
* https://tracker.ceph.com/issues/55804
1856
  Command failed (workunit test suites/pjd.sh)
1857
* https://tracker.ceph.com/issues/36593
1858
  Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
1859
* https://tracker.ceph.com/issues/52624
1860
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1861
* https://tracker.ceph.com/issues/51964
1862
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1863
* https://tracker.ceph.com/issues/56632
1864
  Test failure: test_subvolume_snapshot_clone_quota_exceeded
1865
* https://tracker.ceph.com/issues/50821
1866 88 Patrick Donnelly
  qa: untar_snap_rm failure during mds thrashing
1867
1868
h3. 2022 Sep 26
1869
1870
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220923.171109
1871
1872
* https://tracker.ceph.com/issues/55804
1873
    qa failure: pjd link tests failed
1874
* https://tracker.ceph.com/issues/57676
1875
    qa: error during scrub thrashing: rank damage found: {'backtrace'}
1876
* https://tracker.ceph.com/issues/52624
1877
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1878
* https://tracker.ceph.com/issues/57580
1879
    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
1880
* https://tracker.ceph.com/issues/48773
1881
    qa: scrub does not complete
1882
* https://tracker.ceph.com/issues/57299
1883
    qa: test_dump_loads fails with JSONDecodeError
1884
* https://tracker.ceph.com/issues/57280
1885
    qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman
1886
* https://tracker.ceph.com/issues/57205
1887
    Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
1888
* https://tracker.ceph.com/issues/57656
1889
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1890
* https://tracker.ceph.com/issues/57677
1891
    qa: "1 MDSs behind on trimming (MDS_TRIM)"
1892
* https://tracker.ceph.com/issues/57206
1893
    libcephfs/test.sh: ceph_test_libcephfs_reclaim
1894
* https://tracker.ceph.com/issues/57446
1895
    qa: test_subvolume_snapshot_info_if_orphan_clone fails
1896 89 Patrick Donnelly
* https://tracker.ceph.com/issues/57655 [Exist in main as well]
1897
    qa: fs:mixed-clients kernel_untar_build failure
1898 88 Patrick Donnelly
* https://tracker.ceph.com/issues/57682
1899
    client: ERROR: test_reconnect_after_blocklisted
1900 87 Patrick Donnelly
1901
1902
h3. 2022 Sep 22
1903
1904
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220920.234701
1905
1906
* https://tracker.ceph.com/issues/57299
1907
    qa: test_dump_loads fails with JSONDecodeError
1908
* https://tracker.ceph.com/issues/57205
1909
    Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
1910
* https://tracker.ceph.com/issues/52624
1911
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1912
* https://tracker.ceph.com/issues/57580
1913
    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
1914
* https://tracker.ceph.com/issues/57280
1915
    qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman
1916
* https://tracker.ceph.com/issues/48773
1917
    qa: scrub does not complete
1918
* https://tracker.ceph.com/issues/56446
1919
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1920
* https://tracker.ceph.com/issues/57206
1921
    libcephfs/test.sh: ceph_test_libcephfs_reclaim
1922
* https://tracker.ceph.com/issues/51267
1923
    CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:...
1924
1925
NEW:
1926
1927
* https://tracker.ceph.com/issues/57656
1928
    [testing] dbench: write failed on handle 10009 (Resource temporarily unavailable)
1929
* https://tracker.ceph.com/issues/57655 [Exist in main as well]
1930
    qa: fs:mixed-clients kernel_untar_build failure
1931
* https://tracker.ceph.com/issues/57657
1932
    mds: scrub locates mismatch between child accounted_rstats and self rstats
1933
1934
Segfault probably caused by: https://github.com/ceph/ceph/pull/47795#issuecomment-1255724799
1935 80 Venky Shankar
1936 79 Venky Shankar
1937
h3. 2022 Sep 16
1938
1939
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220905-132828
1940
1941
* https://tracker.ceph.com/issues/57446
1942
    qa: test_subvolume_snapshot_info_if_orphan_clone fails
1943
* https://tracker.ceph.com/issues/57299
1944
    qa: test_dump_loads fails with JSONDecodeError
1945
* https://tracker.ceph.com/issues/50223
1946
    client.xxxx isn't responding to mclientcaps(revoke)
1947
* https://tracker.ceph.com/issues/52624
1948
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1949
* https://tracker.ceph.com/issues/57205
1950
    Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
1951
* https://tracker.ceph.com/issues/57280
1952
    qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman
1953
* https://tracker.ceph.com/issues/51282
1954
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
1955
* https://tracker.ceph.com/issues/48203
1956
  https://tracker.ceph.com/issues/36593
1957
    qa: quota failure
1958
    qa: quota failure caused by clients stepping on each other
1959
* https://tracker.ceph.com/issues/57580
1960 77 Rishabh Dave
    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
1961
1962 76 Rishabh Dave
1963
h3. 2022 Aug 26
1964
1965
http://pulpito.front.sepia.ceph.com/rishabh-2022-08-22_17:49:59-fs-wip-rishabh-testing-2022Aug19-testing-default-smithi/
1966
http://pulpito.front.sepia.ceph.com/rishabh-2022-08-24_11:56:51-fs-wip-rishabh-testing-2022Aug19-testing-default-smithi/
1967
1968
* https://tracker.ceph.com/issues/57206
1969
  libcephfs/test.sh: ceph_test_libcephfs_reclaim
1970
* https://tracker.ceph.com/issues/56632
1971
  Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
1972
* https://tracker.ceph.com/issues/56446
1973
  Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
1974
* https://tracker.ceph.com/issues/51964
1975
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
1976
* https://tracker.ceph.com/issues/53859
1977
  qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1978
1979
* https://tracker.ceph.com/issues/54460
1980
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1981
* https://tracker.ceph.com/issues/54462
1982
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
1983
* https://tracker.ceph.com/issues/54460
1984
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
1985
* https://tracker.ceph.com/issues/36593
1986
  Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
1987
1988
* https://tracker.ceph.com/issues/52624
1989
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
1990
* https://tracker.ceph.com/issues/55804
1991
  Command failed (workunit test suites/pjd.sh)
1992
* https://tracker.ceph.com/issues/50223
1993
  client.xxxx isn't responding to mclientcaps(revoke)
1994 75 Venky Shankar
1995
1996
h3. 2022 Aug 22
1997
1998
https://pulpito.ceph.com/vshankar-2022-08-12_09:34:24-fs-wip-vshankar-testing1-20220812-072441-testing-default-smithi/
1999
https://pulpito.ceph.com/vshankar-2022-08-18_04:30:42-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/ (drop problematic PR and re-run)
2000
2001
* https://tracker.ceph.com/issues/52624
2002
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2003
* https://tracker.ceph.com/issues/56446
2004
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2005
* https://tracker.ceph.com/issues/55804
2006
    Command failed (workunit test suites/pjd.sh)
2007
* https://tracker.ceph.com/issues/51278
2008
    mds: "FAILED ceph_assert(!segments.empty())"
2009
* https://tracker.ceph.com/issues/54460
2010
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
2011
* https://tracker.ceph.com/issues/57205
2012
    Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
2013
* https://tracker.ceph.com/issues/57206
2014
    ceph_test_libcephfs_reclaim crashes during test
2015
* https://tracker.ceph.com/issues/53859
2016
  Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2017
* https://tracker.ceph.com/issues/50223
2018 72 Venky Shankar
    client.xxxx isn't responding to mclientcaps(revoke)
2019
2020
h3. 2022 Aug 12
2021
2022
https://pulpito.ceph.com/vshankar-2022-08-10_04:06:00-fs-wip-vshankar-testing-20220805-190751-testing-default-smithi/
2023
https://pulpito.ceph.com/vshankar-2022-08-11_12:16:58-fs-wip-vshankar-testing-20220811-145809-testing-default-smithi/ (drop problematic PR and re-run)
2024
2025
* https://tracker.ceph.com/issues/52624
2026
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2027
* https://tracker.ceph.com/issues/56446
2028
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2029
* https://tracker.ceph.com/issues/51964
2030
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
2031
* https://tracker.ceph.com/issues/55804
2032
    Command failed (workunit test suites/pjd.sh)
2033
* https://tracker.ceph.com/issues/50223
2034
    client.xxxx isn't responding to mclientcaps(revoke)
2035
* https://tracker.ceph.com/issues/50821
2036 73 Venky Shankar
    qa: untar_snap_rm failure during mds thrashing
2037 72 Venky Shankar
* https://tracker.ceph.com/issues/54460
2038 71 Venky Shankar
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
2039
2040
h3. 2022 Aug 04
2041
2042
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220804-123835 (only mgr/volumes, mgr/stats)
2043
2044 69 Rishabh Dave
Unrealted teuthology failure on rhel
2045 68 Rishabh Dave
2046
h3. 2022 Jul 25
2047
2048
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-22_11:34:20-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/
2049
2050 74 Rishabh Dave
1st re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_03:51:19-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi
2051
2nd re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_08:53:36-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/
2052 68 Rishabh Dave
3rd re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-24_08:53:36-fs-wip-rishabh-testing-2022Jul22-1400-testing-default-smithi/
2053
4th (final) re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-28_03:59:01-fs-wip-rishabh-testing-2022Jul28-0143-testing-default-smithi/
2054
2055
* https://tracker.ceph.com/issues/55804
2056
  Command failed (workunit test suites/pjd.sh)
2057
* https://tracker.ceph.com/issues/50223
2058
  client.xxxx isn't responding to mclientcaps(revoke)
2059
2060
* https://tracker.ceph.com/issues/54460
2061
  Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithixxx with status 1
2062 1 Patrick Donnelly
* https://tracker.ceph.com/issues/36593
2063 74 Rishabh Dave
  Command failed (workunit test fs/quota/quota.sh) on smithixxx with status 1
2064 68 Rishabh Dave
* https://tracker.ceph.com/issues/54462
2065 67 Patrick Donnelly
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128~
2066
2067
h3. 2022 July 22
2068
2069
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220721.235756
2070
2071
MDS_HEALTH_DUMMY error in log fixed by followup commit.
2072
transient selinux ping failure
2073
2074
* https://tracker.ceph.com/issues/56694
2075
    qa: avoid blocking forever on hung umount
2076
* https://tracker.ceph.com/issues/56695
2077
    [RHEL stock] pjd test failures
2078
* https://tracker.ceph.com/issues/56696
2079
    admin keyring disappears during qa run
2080
* https://tracker.ceph.com/issues/56697
2081
    qa: fs/snaps fails for fuse
2082
* https://tracker.ceph.com/issues/50222
2083
    osd: 5.2s0 deep-scrub : stat mismatch
2084
* https://tracker.ceph.com/issues/56698
2085
    client: FAILED ceph_assert(_size == 0)
2086
* https://tracker.ceph.com/issues/50223
2087
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2088 66 Rishabh Dave
2089 65 Rishabh Dave
2090
h3. 2022 Jul 15
2091
2092
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-08_23:53:34-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/
2093
2094
re-run: http://pulpito.front.sepia.ceph.com/rishabh-2022-07-15_06:42:04-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/
2095
2096
* https://tracker.ceph.com/issues/53859
2097
  Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2098
* https://tracker.ceph.com/issues/55804
2099
  Command failed (workunit test suites/pjd.sh)
2100
* https://tracker.ceph.com/issues/50223
2101
  client.xxxx isn't responding to mclientcaps(revoke)
2102
* https://tracker.ceph.com/issues/50222
2103
  osd: deep-scrub : stat mismatch
2104
2105
* https://tracker.ceph.com/issues/56632
2106
  Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2107
* https://tracker.ceph.com/issues/56634
2108
  workunit test fs/snaps/snaptest-intodir.sh
2109
* https://tracker.ceph.com/issues/56644
2110
  Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2111
2112 61 Rishabh Dave
2113
2114
h3. 2022 July 05
2115 62 Rishabh Dave
2116 64 Rishabh Dave
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/
2117
2118
On 1st re-run some jobs passed - http://pulpito.front.sepia.ceph.com/rishabh-2022-07-03_15:10:28-fs-wip-rishabh-testing-20220702-1631-distro-default-smithi/
2119
2120
On 2nd re-run only few jobs failed -
2121 62 Rishabh Dave
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-06_05:24:29-fs-wip-rishabh-testing-20220705-2132-distro-default-smithi/
2122
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-06_05:24:29-fs-wip-rishabh-testing-20220705-2132-distro-default-smithi/
2123
2124
* https://tracker.ceph.com/issues/56446
2125
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2126
* https://tracker.ceph.com/issues/55804
2127
    Command failed (workunit test suites/pjd.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/
2128
2129
* https://tracker.ceph.com/issues/56445
2130 63 Rishabh Dave
    Command failed on smithi080 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"
2131
* https://tracker.ceph.com/issues/51267
2132
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
2133 62 Rishabh Dave
* https://tracker.ceph.com/issues/50224
2134
    Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
2135 61 Rishabh Dave
2136 58 Venky Shankar
2137
2138
h3. 2022 July 04
2139
2140
https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/
2141
(rhel runs were borked due to: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/JSZQFUKVLDND4W33PXDGCABPHNSPT6SS/, tests ran with --filter-out=rhel)
2142
2143
* https://tracker.ceph.com/issues/56445
2144 59 Rishabh Dave
    Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"
2145
* https://tracker.ceph.com/issues/56446
2146
    Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2147
* https://tracker.ceph.com/issues/51964
2148 60 Rishabh Dave
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
2149 59 Rishabh Dave
* https://tracker.ceph.com/issues/52624
2150 57 Venky Shankar
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2151
2152
h3. 2022 June 20
2153
2154
https://pulpito.ceph.com/vshankar-2022-06-15_04:03:39-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/
2155
https://pulpito.ceph.com/vshankar-2022-06-19_08:22:46-fs-wip-vshankar-testing1-20220619-102531-testing-default-smithi/
2156
2157
* https://tracker.ceph.com/issues/52624
2158
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2159
* https://tracker.ceph.com/issues/55804
2160
    qa failure: pjd link tests failed
2161
* https://tracker.ceph.com/issues/54108
2162
    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"
2163
* https://tracker.ceph.com/issues/55332
2164 56 Patrick Donnelly
    Failure in snaptest-git-ceph.sh (it's an async unlink/create bug)
2165
2166
h3. 2022 June 13
2167
2168
https://pulpito.ceph.com/pdonnell-2022-06-12_05:08:12-fs:workload-wip-pdonnell-testing-20220612.004943-distro-default-smithi/
2169
2170
* https://tracker.ceph.com/issues/56024
2171
    cephadm: removes ceph.conf during qa run causing command failure
2172
* https://tracker.ceph.com/issues/48773
2173
    qa: scrub does not complete
2174
* https://tracker.ceph.com/issues/56012
2175
    mds: src/mds/MDLog.cc: 283: FAILED ceph_assert(!mds->is_ any_replay())
2176 55 Venky Shankar
2177 54 Venky Shankar
2178
h3. 2022 Jun 13
2179
2180
https://pulpito.ceph.com/vshankar-2022-06-07_00:25:50-fs-wip-vshankar-testing-20220606-223254-testing-default-smithi/
2181
https://pulpito.ceph.com/vshankar-2022-06-10_01:04:46-fs-wip-vshankar-testing-20220609-175550-testing-default-smithi/
2182
2183
* https://tracker.ceph.com/issues/52624
2184
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2185
* https://tracker.ceph.com/issues/51964
2186
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
2187
* https://tracker.ceph.com/issues/53859
2188
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2189
* https://tracker.ceph.com/issues/55804
2190
    qa failure: pjd link tests failed
2191
* https://tracker.ceph.com/issues/56003
2192
    client: src/include/xlist.h: 81: FAILED ceph_assert(_size == 0)
2193
* https://tracker.ceph.com/issues/56011
2194
    fs/thrash: snaptest-snap-rm-cmp.sh fails in mds5sum comparison
2195
* https://tracker.ceph.com/issues/56012
2196 53 Venky Shankar
    mds: src/mds/MDLog.cc: 283: FAILED ceph_assert(!mds->is_ any_replay())
2197
2198
h3. 2022 Jun 07
2199
2200
https://pulpito.ceph.com/vshankar-2022-06-06_21:25:41-fs-wip-vshankar-testing1-20220606-230129-testing-default-smithi/
2201
https://pulpito.ceph.com/vshankar-2022-06-07_10:53:31-fs-wip-vshankar-testing1-20220607-104134-testing-default-smithi/ (rerun after dropping a problematic PR)
2202
2203
* https://tracker.ceph.com/issues/52624
2204
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2205
* https://tracker.ceph.com/issues/50223
2206
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2207
* https://tracker.ceph.com/issues/50224
2208 51 Venky Shankar
    qa: test_mirroring_init_failure_with_recovery failure
2209
2210
h3. 2022 May 12
2211 52 Venky Shankar
2212 51 Venky Shankar
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20220509-125847
2213
https://pulpito.ceph.com/vshankar-2022-05-13_17:09:16-fs-wip-vshankar-testing-20220513-120051-testing-default-smithi/ (drop prs + rerun)
2214
2215
* https://tracker.ceph.com/issues/52624
2216
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2217
* https://tracker.ceph.com/issues/50223
2218
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2219
* https://tracker.ceph.com/issues/55332
2220
    Failure in snaptest-git-ceph.sh
2221
* https://tracker.ceph.com/issues/53859
2222 1 Patrick Donnelly
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2223 52 Venky Shankar
* https://tracker.ceph.com/issues/55538
2224
    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
2225 51 Venky Shankar
* https://tracker.ceph.com/issues/55258
2226 49 Venky Shankar
    lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs (cropss up again, though very infrequent)
2227
2228 50 Venky Shankar
h3. 2022 May 04
2229
2230
https://pulpito.ceph.com/vshankar-2022-05-01_13:18:44-fs-wip-vshankar-testing1-20220428-204527-testing-default-smithi/
2231 49 Venky Shankar
https://pulpito.ceph.com/vshankar-2022-05-02_16:58:59-fs-wip-vshankar-testing1-20220502-201957-testing-default-smithi/ (after dropping PRs)
2232
2233
* https://tracker.ceph.com/issues/52624
2234
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2235
* https://tracker.ceph.com/issues/50223
2236
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2237
* https://tracker.ceph.com/issues/55332
2238
    Failure in snaptest-git-ceph.sh
2239
* https://tracker.ceph.com/issues/53859
2240
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2241
* https://tracker.ceph.com/issues/55516
2242
    qa: fs suite tests failing with "json.decoder.JSONDecodeError: Extra data: line 2 column 82 (char 82)"
2243
* https://tracker.ceph.com/issues/55537
2244
    mds: crash during fs:upgrade test
2245
* https://tracker.ceph.com/issues/55538
2246 48 Venky Shankar
    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
2247
2248
h3. 2022 Apr 25
2249
2250
https://pulpito.ceph.com/?branch=wip-vshankar-testing-20220420-113951 (owner vshankar)
2251
2252
* https://tracker.ceph.com/issues/52624
2253
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2254
* https://tracker.ceph.com/issues/50223
2255
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2256
* https://tracker.ceph.com/issues/55258
2257
    lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs
2258
* https://tracker.ceph.com/issues/55377
2259 47 Venky Shankar
    kclient: mds revoke Fwb caps stuck after the kclient tries writebcak once
2260
2261
h3. 2022 Apr 14
2262
2263
https://pulpito.ceph.com/?branch=wip-vshankar-testing1-20220411-144044
2264
2265
* https://tracker.ceph.com/issues/52624
2266
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2267
* https://tracker.ceph.com/issues/50223
2268
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2269
* https://tracker.ceph.com/issues/52438
2270
    qa: ffsb timeout
2271
* https://tracker.ceph.com/issues/55170
2272
    mds: crash during rejoin (CDir::fetch_keys)
2273
* https://tracker.ceph.com/issues/55331
2274
    pjd failure
2275
* https://tracker.ceph.com/issues/48773
2276
    qa: scrub does not complete
2277
* https://tracker.ceph.com/issues/55332
2278
    Failure in snaptest-git-ceph.sh
2279
* https://tracker.ceph.com/issues/55258
2280 45 Venky Shankar
    lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs
2281
2282 46 Venky Shankar
h3. 2022 Apr 11
2283 45 Venky Shankar
2284
https://pulpito.ceph.com/?branch=wip-vshankar-testing-55110-20220408-203242
2285
2286
* https://tracker.ceph.com/issues/48773
2287
    qa: scrub does not complete
2288
* https://tracker.ceph.com/issues/52624
2289
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2290
* https://tracker.ceph.com/issues/52438
2291
    qa: ffsb timeout
2292
* https://tracker.ceph.com/issues/48680
2293
    mds: scrubbing stuck "scrub active (0 inodes in the stack)"
2294
* https://tracker.ceph.com/issues/55236
2295
    qa: fs/snaps tests fails with "hit max job timeout"
2296
* https://tracker.ceph.com/issues/54108
2297
    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"
2298
* https://tracker.ceph.com/issues/54971
2299
    Test failure: test_perf_stats_stale_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2300
* https://tracker.ceph.com/issues/50223
2301
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2302
* https://tracker.ceph.com/issues/55258
2303 44 Venky Shankar
    lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs
2304 42 Venky Shankar
2305 43 Venky Shankar
h3. 2022 Mar 21
2306
2307
https://pulpito.ceph.com/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/
2308
2309
Run didn't go well, lots of failures - debugging by dropping PRs and running against master branch. Only merging unrelated PRs that pass tests.
2310
2311
2312 42 Venky Shankar
h3. 2022 Mar 08
2313
2314
https://pulpito.ceph.com/vshankar-2022-02-28_04:32:15-fs-wip-vshankar-testing-20220226-211550-testing-default-smithi/
2315
2316
rerun with
2317
- (drop) https://github.com/ceph/ceph/pull/44679
2318
- (drop) https://github.com/ceph/ceph/pull/44958
2319
https://pulpito.ceph.com/vshankar-2022-03-06_14:47:51-fs-wip-vshankar-testing-20220304-132102-testing-default-smithi/
2320
2321
* https://tracker.ceph.com/issues/54419 (new)
2322
    `ceph orch upgrade start` seems to never reach completion
2323
* https://tracker.ceph.com/issues/51964
2324
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
2325
* https://tracker.ceph.com/issues/52624
2326
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2327
* https://tracker.ceph.com/issues/50223
2328
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2329
* https://tracker.ceph.com/issues/52438
2330
    qa: ffsb timeout
2331
* https://tracker.ceph.com/issues/50821
2332
    qa: untar_snap_rm failure during mds thrashing
2333 41 Venky Shankar
2334
2335
h3. 2022 Feb 09
2336
2337
https://pulpito.ceph.com/vshankar-2022-02-05_17:27:49-fs-wip-vshankar-testing-20220201-113815-testing-default-smithi/
2338
2339
rerun with
2340
- (drop) https://github.com/ceph/ceph/pull/37938
2341
- (drop) https://github.com/ceph/ceph/pull/44335
2342
- (drop) https://github.com/ceph/ceph/pull/44491
2343
- (drop) https://github.com/ceph/ceph/pull/44501
2344
https://pulpito.ceph.com/vshankar-2022-02-08_14:27:29-fs-wip-vshankar-testing-20220208-181241-testing-default-smithi/
2345
2346
* https://tracker.ceph.com/issues/51964
2347
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
2348
* https://tracker.ceph.com/issues/54066
2349
    test_subvolume_no_upgrade_v1_sanity fails with `AssertionError: 1000 != 0`
2350
* https://tracker.ceph.com/issues/48773
2351
    qa: scrub does not complete
2352
* https://tracker.ceph.com/issues/52624
2353
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2354
* https://tracker.ceph.com/issues/50223
2355
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2356
* https://tracker.ceph.com/issues/52438
2357 40 Patrick Donnelly
    qa: ffsb timeout
2358
2359
h3. 2022 Feb 01
2360
2361
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20220127.171526
2362
2363
* https://tracker.ceph.com/issues/54107
2364
    kclient: hang during umount
2365
* https://tracker.ceph.com/issues/54106
2366
    kclient: hang during workunit cleanup
2367
* https://tracker.ceph.com/issues/54108
2368
    qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}"
2369
* https://tracker.ceph.com/issues/48773
2370
    qa: scrub does not complete
2371
* https://tracker.ceph.com/issues/52438
2372
    qa: ffsb timeout
2373 36 Venky Shankar
2374
2375
h3. 2022 Jan 13
2376 39 Venky Shankar
2377 36 Venky Shankar
https://pulpito.ceph.com/vshankar-2022-01-06_13:18:41-fs-wip-vshankar-testing-20220106-145819-testing-default-smithi/
2378 38 Venky Shankar
2379
rerun with:
2380 36 Venky Shankar
- (add) https://github.com/ceph/ceph/pull/44570
2381
- (drop) https://github.com/ceph/ceph/pull/43184
2382
https://pulpito.ceph.com/vshankar-2022-01-13_04:42:40-fs-wip-vshankar-testing-20220106-145819-testing-default-smithi/
2383
2384
* https://tracker.ceph.com/issues/50223
2385
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2386
* https://tracker.ceph.com/issues/51282
2387
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2388
* https://tracker.ceph.com/issues/48773
2389
    qa: scrub does not complete
2390
* https://tracker.ceph.com/issues/52624
2391
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2392
* https://tracker.ceph.com/issues/53859
2393 34 Venky Shankar
    qa: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2394
2395
h3. 2022 Jan 03
2396
2397
https://pulpito.ceph.com/vshankar-2021-12-22_07:37:44-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/
2398
https://pulpito.ceph.com/vshankar-2022-01-03_12:27:45-fs-wip-vshankar-testing-20220103-142738-testing-default-smithi/ (rerun)
2399
2400
* https://tracker.ceph.com/issues/50223
2401
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2402
* https://tracker.ceph.com/issues/51964
2403
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
2404
* https://tracker.ceph.com/issues/51267
2405
    CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:...
2406
* https://tracker.ceph.com/issues/51282
2407
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2408
* https://tracker.ceph.com/issues/50821
2409
    qa: untar_snap_rm failure during mds thrashing
2410 35 Ramana Raja
* https://tracker.ceph.com/issues/51278
2411
    mds: "FAILED ceph_assert(!segments.empty())"
2412
* https://tracker.ceph.com/issues/52279
2413 34 Venky Shankar
    cephadm tests fail due to: error adding seccomp filter rule for syscall bdflush: requested action matches default action of filter
2414 33 Patrick Donnelly
2415
2416
h3. 2021 Dec 22
2417
2418
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211222.014316
2419
2420
* https://tracker.ceph.com/issues/52624
2421
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2422
* https://tracker.ceph.com/issues/50223
2423
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2424
* https://tracker.ceph.com/issues/52279
2425
    cephadm tests fail due to: error adding seccomp filter rule for syscall bdflush: requested action matches default action of filter
2426
* https://tracker.ceph.com/issues/50224
2427
    qa: test_mirroring_init_failure_with_recovery failure
2428
* https://tracker.ceph.com/issues/48773
2429
    qa: scrub does not complete
2430 32 Venky Shankar
2431
2432
h3. 2021 Nov 30
2433
2434
https://pulpito.ceph.com/vshankar-2021-11-24_07:14:27-fs-wip-vshankar-testing-20211124-094330-testing-default-smithi/
2435
https://pulpito.ceph.com/vshankar-2021-11-30_06:23:32-fs-wip-vshankar-testing-20211124-094330-distro-default-smithi/ (rerun w/ QA fixes)
2436
2437
* https://tracker.ceph.com/issues/53436
2438
    mds, mon: mds beacon messages get dropped? (mds never reaches up:active state)
2439
* https://tracker.ceph.com/issues/51964
2440
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
2441
* https://tracker.ceph.com/issues/48812
2442
    qa: test_scrub_pause_and_resume_with_abort failure
2443
* https://tracker.ceph.com/issues/51076
2444
    "wait_for_recovery: failed before timeout expired" during thrashosd test with EC backend.
2445
* https://tracker.ceph.com/issues/50223
2446
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2447
* https://tracker.ceph.com/issues/52624
2448
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2449
* https://tracker.ceph.com/issues/50250
2450
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2451 31 Patrick Donnelly
2452
2453
h3. 2021 November 9
2454
2455
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211109.180315
2456
2457
* https://tracker.ceph.com/issues/53214
2458
    qa: "dd: error reading '/sys/kernel/debug/ceph/2a934501-6731-4052-a836-f42229a869be.client4874/metrics': Is a directory"
2459
* https://tracker.ceph.com/issues/48773
2460
    qa: scrub does not complete
2461
* https://tracker.ceph.com/issues/50223
2462
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2463
* https://tracker.ceph.com/issues/51282
2464
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2465
* https://tracker.ceph.com/issues/52624
2466
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2467
* https://tracker.ceph.com/issues/53216
2468
    qa: "RuntimeError: value of attributes should be either str or None. client_id"
2469
* https://tracker.ceph.com/issues/50250
2470
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2471
2472 30 Patrick Donnelly
2473
2474
h3. 2021 November 03
2475
2476
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211103.023355
2477
2478
* https://tracker.ceph.com/issues/51964
2479
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
2480
* https://tracker.ceph.com/issues/51282
2481
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2482
* https://tracker.ceph.com/issues/52436
2483
    fs/ceph: "corrupt mdsmap"
2484
* https://tracker.ceph.com/issues/53074
2485
    pybind/mgr/cephadm: upgrade sequence does not continue if no MDS are active
2486
* https://tracker.ceph.com/issues/53150
2487
    pybind/mgr/cephadm/upgrade: tolerate MDS failures during upgrade straddling v16.2.5
2488
* https://tracker.ceph.com/issues/53155
2489
    MDSMonitor: assertion during upgrade to v16.2.5+
2490 29 Patrick Donnelly
2491
2492
h3. 2021 October 26
2493
2494
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211025.000447
2495
2496
* https://tracker.ceph.com/issues/53074
2497
    pybind/mgr/cephadm: upgrade sequence does not continue if no MDS are active
2498
* https://tracker.ceph.com/issues/52997
2499
    testing: hang ing umount
2500
* https://tracker.ceph.com/issues/50824
2501
    qa: snaptest-git-ceph bus error
2502
* https://tracker.ceph.com/issues/52436
2503
    fs/ceph: "corrupt mdsmap"
2504
* https://tracker.ceph.com/issues/48773
2505
    qa: scrub does not complete
2506
* https://tracker.ceph.com/issues/53082
2507
    ceph-fuse: segmenetation fault in Client::handle_mds_map
2508
* https://tracker.ceph.com/issues/50223
2509
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2510
* https://tracker.ceph.com/issues/52624
2511
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2512
* https://tracker.ceph.com/issues/50224
2513
    qa: test_mirroring_init_failure_with_recovery failure
2514
* https://tracker.ceph.com/issues/50821
2515
    qa: untar_snap_rm failure during mds thrashing
2516
* https://tracker.ceph.com/issues/50250
2517
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2518
2519 27 Patrick Donnelly
2520
2521 28 Patrick Donnelly
h3. 2021 October 19
2522 27 Patrick Donnelly
2523
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211019.013028
2524
2525
* https://tracker.ceph.com/issues/52995
2526
    qa: test_standby_count_wanted failure
2527
* https://tracker.ceph.com/issues/52948
2528
    osd: fails to come up: "teuthology.misc:7 of 8 OSDs are up"
2529
* https://tracker.ceph.com/issues/52996
2530
    qa: test_perf_counters via test_openfiletable
2531
* https://tracker.ceph.com/issues/48772
2532
    qa: pjd: not ok 9, 44, 80
2533
* https://tracker.ceph.com/issues/52997
2534
    testing: hang ing umount
2535
* https://tracker.ceph.com/issues/50250
2536
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2537
* https://tracker.ceph.com/issues/52624
2538
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2539
* https://tracker.ceph.com/issues/50223
2540
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2541
* https://tracker.ceph.com/issues/50821
2542
    qa: untar_snap_rm failure during mds thrashing
2543
* https://tracker.ceph.com/issues/48773
2544
    qa: scrub does not complete
2545 26 Patrick Donnelly
2546
2547
h3. 2021 October 12
2548
2549
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211012.192211
2550
2551
Some failures caused by teuthology bug: https://tracker.ceph.com/issues/52944
2552
2553
New test caused failure: https://github.com/ceph/ceph/pull/43297#discussion_r729883167
2554
2555
2556
* https://tracker.ceph.com/issues/51282
2557
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2558
* https://tracker.ceph.com/issues/52948
2559
    osd: fails to come up: "teuthology.misc:7 of 8 OSDs are up"
2560
* https://tracker.ceph.com/issues/48773
2561
    qa: scrub does not complete
2562
* https://tracker.ceph.com/issues/50224
2563
    qa: test_mirroring_init_failure_with_recovery failure
2564
* https://tracker.ceph.com/issues/52949
2565
    RuntimeError: The following counters failed to be set on mds daemons: {'mds.dir_split'}
2566 25 Patrick Donnelly
2567 23 Patrick Donnelly
2568 24 Patrick Donnelly
h3. 2021 October 02
2569
2570
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211002.163337
2571
2572
Some failures caused by cephadm upgrade test. Fixed in follow-up qa commit.
2573
2574
test_simple failures caused by PR in this set.
2575
2576
A few reruns because of QA infra noise.
2577
2578
* https://tracker.ceph.com/issues/52822
2579
    qa: failed pacific install on fs:upgrade
2580
* https://tracker.ceph.com/issues/52624
2581
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2582
* https://tracker.ceph.com/issues/50223
2583
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2584
* https://tracker.ceph.com/issues/48773
2585
    qa: scrub does not complete
2586
2587
2588 23 Patrick Donnelly
h3. 2021 September 20
2589
2590
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210917.174826
2591
2592
* https://tracker.ceph.com/issues/52677
2593
    qa: test_simple failure
2594
* https://tracker.ceph.com/issues/51279
2595
    kclient hangs on umount (testing branch)
2596
* https://tracker.ceph.com/issues/50223
2597
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2598
* https://tracker.ceph.com/issues/50250
2599
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2600
* https://tracker.ceph.com/issues/52624
2601
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2602
* https://tracker.ceph.com/issues/52438
2603
    qa: ffsb timeout
2604 22 Patrick Donnelly
2605
2606
h3. 2021 September 10
2607
2608
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210910.181451
2609
2610
* https://tracker.ceph.com/issues/50223
2611
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2612
* https://tracker.ceph.com/issues/50250
2613
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2614
* https://tracker.ceph.com/issues/52624
2615
    qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
2616
* https://tracker.ceph.com/issues/52625
2617
    qa: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots)
2618
* https://tracker.ceph.com/issues/52439
2619
    qa: acls does not compile on centos stream
2620
* https://tracker.ceph.com/issues/50821
2621
    qa: untar_snap_rm failure during mds thrashing
2622
* https://tracker.ceph.com/issues/48773
2623
    qa: scrub does not complete
2624
* https://tracker.ceph.com/issues/52626
2625
    mds: ScrubStack.cc: 831: FAILED ceph_assert(diri)
2626
* https://tracker.ceph.com/issues/51279
2627
    kclient hangs on umount (testing branch)
2628 21 Patrick Donnelly
2629
2630
h3. 2021 August 27
2631
2632
Several jobs died because of device failures.
2633
2634
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210827.024746
2635
2636
* https://tracker.ceph.com/issues/52430
2637
    mds: fast async create client mount breaks racy test
2638
* https://tracker.ceph.com/issues/52436
2639
    fs/ceph: "corrupt mdsmap"
2640
* https://tracker.ceph.com/issues/52437
2641
    mds: InoTable::replay_release_ids abort via test_inotable_sync
2642
* https://tracker.ceph.com/issues/51282
2643
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2644
* https://tracker.ceph.com/issues/52438
2645
    qa: ffsb timeout
2646
* https://tracker.ceph.com/issues/52439
2647
    qa: acls does not compile on centos stream
2648 20 Patrick Donnelly
2649
2650
h3. 2021 July 30
2651
2652
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210729.214022
2653
2654
* https://tracker.ceph.com/issues/50250
2655
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2656
* https://tracker.ceph.com/issues/51282
2657
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2658
* https://tracker.ceph.com/issues/48773
2659
    qa: scrub does not complete
2660
* https://tracker.ceph.com/issues/51975
2661
    pybind/mgr/stats: KeyError
2662 19 Patrick Donnelly
2663
2664
h3. 2021 July 28
2665
2666
https://pulpito.ceph.com/pdonnell-2021-07-28_00:39:45-fs-wip-pdonnell-testing-20210727.213757-distro-basic-smithi/
2667
2668
with qa fix: https://pulpito.ceph.com/pdonnell-2021-07-28_16:20:28-fs-wip-pdonnell-testing-20210728.141004-distro-basic-smithi/
2669
2670
* https://tracker.ceph.com/issues/51905
2671
    qa: "error reading sessionmap 'mds1_sessionmap'"
2672
* https://tracker.ceph.com/issues/48773
2673
    qa: scrub does not complete
2674
* https://tracker.ceph.com/issues/50250
2675
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2676
* https://tracker.ceph.com/issues/51267
2677
    CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:...
2678
* https://tracker.ceph.com/issues/51279
2679
    kclient hangs on umount (testing branch)
2680 18 Patrick Donnelly
2681
2682
h3. 2021 July 16
2683
2684
https://pulpito.ceph.com/pdonnell-2021-07-16_05:50:11-fs-wip-pdonnell-testing-20210716.022804-distro-basic-smithi/
2685
2686
* https://tracker.ceph.com/issues/48773
2687
    qa: scrub does not complete
2688
* https://tracker.ceph.com/issues/48772
2689
    qa: pjd: not ok 9, 44, 80
2690
* https://tracker.ceph.com/issues/45434
2691
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2692
* https://tracker.ceph.com/issues/51279
2693
    kclient hangs on umount (testing branch)
2694
* https://tracker.ceph.com/issues/50824
2695
    qa: snaptest-git-ceph bus error
2696 17 Patrick Donnelly
2697
2698
h3. 2021 July 04
2699
2700
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210703.052904
2701
2702
* https://tracker.ceph.com/issues/48773
2703
    qa: scrub does not complete
2704
* https://tracker.ceph.com/issues/39150
2705
    mon: "FAILED ceph_assert(session_map.sessions.empty())" when out of quorum
2706
* https://tracker.ceph.com/issues/45434
2707
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2708
* https://tracker.ceph.com/issues/51282
2709
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2710
* https://tracker.ceph.com/issues/48771
2711
    qa: iogen: workload fails to cause balancing
2712
* https://tracker.ceph.com/issues/51279
2713
    kclient hangs on umount (testing branch)
2714
* https://tracker.ceph.com/issues/50250
2715
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
2716 16 Patrick Donnelly
2717
2718
h3. 2021 July 01
2719
2720
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210701.192056
2721
2722
* https://tracker.ceph.com/issues/51197
2723
    qa: [WRN] Scrub error on inode 0x10000001520 (/client.0/tmp/t/linux-5.4/Documentation/driver-api) see mds.f log and `damage ls` output for details
2724
* https://tracker.ceph.com/issues/50866
2725
    osd: stat mismatch on objects
2726
* https://tracker.ceph.com/issues/48773
2727
    qa: scrub does not complete
2728 15 Patrick Donnelly
2729
2730
h3. 2021 June 26
2731
2732
https://pulpito.ceph.com/pdonnell-2021-06-26_00:57:00-fs-wip-pdonnell-testing-20210625.225421-distro-basic-smithi/
2733
2734
* https://tracker.ceph.com/issues/51183
2735
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2736
* https://tracker.ceph.com/issues/51410
2737
    kclient: fails to finish reconnect during MDS thrashing (testing branch)
2738
* https://tracker.ceph.com/issues/48773
2739
    qa: scrub does not complete
2740
* https://tracker.ceph.com/issues/51282
2741
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2742
* https://tracker.ceph.com/issues/51169
2743
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2744
* https://tracker.ceph.com/issues/48772
2745
    qa: pjd: not ok 9, 44, 80
2746 14 Patrick Donnelly
2747
2748
h3. 2021 June 21
2749
2750
https://pulpito.ceph.com/pdonnell-2021-06-22_00:27:21-fs-wip-pdonnell-testing-20210621.231646-distro-basic-smithi/
2751
2752
One failure caused by PR: https://github.com/ceph/ceph/pull/41935#issuecomment-866472599
2753
2754
* https://tracker.ceph.com/issues/51282
2755
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2756
* https://tracker.ceph.com/issues/51183
2757
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2758
* https://tracker.ceph.com/issues/48773
2759
    qa: scrub does not complete
2760
* https://tracker.ceph.com/issues/48771
2761
    qa: iogen: workload fails to cause balancing
2762
* https://tracker.ceph.com/issues/51169
2763
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2764
* https://tracker.ceph.com/issues/50495
2765
    libcephfs: shutdown race fails with status 141
2766
* https://tracker.ceph.com/issues/45434
2767
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2768
* https://tracker.ceph.com/issues/50824
2769
    qa: snaptest-git-ceph bus error
2770
* https://tracker.ceph.com/issues/50223
2771
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2772 13 Patrick Donnelly
2773
2774
h3. 2021 June 16
2775
2776
https://pulpito.ceph.com/pdonnell-2021-06-16_21:26:55-fs-wip-pdonnell-testing-20210616.191804-distro-basic-smithi/
2777
2778
MDS abort class of failures caused by PR: https://github.com/ceph/ceph/pull/41667
2779
2780
* https://tracker.ceph.com/issues/45434
2781
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2782
* https://tracker.ceph.com/issues/51169
2783
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2784
* https://tracker.ceph.com/issues/43216
2785
    MDSMonitor: removes MDS coming out of quorum election
2786
* https://tracker.ceph.com/issues/51278
2787
    mds: "FAILED ceph_assert(!segments.empty())"
2788
* https://tracker.ceph.com/issues/51279
2789
    kclient hangs on umount (testing branch)
2790
* https://tracker.ceph.com/issues/51280
2791
    mds: "FAILED ceph_assert(r == 0 || r == -2)"
2792
* https://tracker.ceph.com/issues/51183
2793
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2794
* https://tracker.ceph.com/issues/51281
2795
    qa: snaptest-snap-rm-cmp.sh: "echo 'FAIL: bad match, /tmp/a 4637e766853d1ad16a7b17079e2c6f03 != real c3883760b18d50e8d78819c54d579b00'"
2796
* https://tracker.ceph.com/issues/48773
2797
    qa: scrub does not complete
2798
* https://tracker.ceph.com/issues/51076
2799
    "wait_for_recovery: failed before timeout expired" during thrashosd test with EC backend.
2800
* https://tracker.ceph.com/issues/51228
2801
    qa: rmdir: failed to remove 'a/.snap/*': No such file or directory
2802
* https://tracker.ceph.com/issues/51282
2803
    pybind/mgr/mgr_util: .mgr pool may be created to early causing spurious PG_DEGRADED warnings
2804 12 Patrick Donnelly
2805
2806
h3. 2021 June 14
2807
2808
https://pulpito.ceph.com/pdonnell-2021-06-14_20:53:05-fs-wip-pdonnell-testing-20210614.173325-distro-basic-smithi/
2809
2810
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
2811
2812
* https://tracker.ceph.com/issues/51169
2813
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2814
* https://tracker.ceph.com/issues/51228
2815
    qa: rmdir: failed to remove 'a/.snap/*': No such file or directory
2816
* https://tracker.ceph.com/issues/48773
2817
    qa: scrub does not complete
2818
* https://tracker.ceph.com/issues/51183
2819
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2820
* https://tracker.ceph.com/issues/45434
2821
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2822
* https://tracker.ceph.com/issues/51182
2823
    pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs'
2824
* https://tracker.ceph.com/issues/51229
2825
    qa: test_multi_snap_schedule list difference failure
2826
* https://tracker.ceph.com/issues/50821
2827
    qa: untar_snap_rm failure during mds thrashing
2828 11 Patrick Donnelly
2829
2830
h3. 2021 June 13
2831
2832
https://pulpito.ceph.com/pdonnell-2021-06-12_02:45:35-fs-wip-pdonnell-testing-20210612.002809-distro-basic-smithi/
2833
2834
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
2835
2836
* https://tracker.ceph.com/issues/51169
2837
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2838
* https://tracker.ceph.com/issues/48773
2839
    qa: scrub does not complete
2840
* https://tracker.ceph.com/issues/51182
2841
    pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs'
2842
* https://tracker.ceph.com/issues/51183
2843
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2844
* https://tracker.ceph.com/issues/51197
2845
    qa: [WRN] Scrub error on inode 0x10000001520 (/client.0/tmp/t/linux-5.4/Documentation/driver-api) see mds.f log and `damage ls` output for details
2846
* https://tracker.ceph.com/issues/45434
2847 10 Patrick Donnelly
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2848
2849
h3. 2021 June 11
2850
2851
https://pulpito.ceph.com/pdonnell-2021-06-11_18:02:10-fs-wip-pdonnell-testing-20210611.162716-distro-basic-smithi/
2852
2853
Some Ubuntu 20.04 upgrade fallout. In particular, upgrade tests are failing due to missing packages for 18.04 Pacific.
2854
2855
* https://tracker.ceph.com/issues/51169
2856
    qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp
2857
* https://tracker.ceph.com/issues/45434
2858
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2859
* https://tracker.ceph.com/issues/48771
2860
    qa: iogen: workload fails to cause balancing
2861
* https://tracker.ceph.com/issues/43216
2862
    MDSMonitor: removes MDS coming out of quorum election
2863
* https://tracker.ceph.com/issues/51182
2864
    pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs'
2865
* https://tracker.ceph.com/issues/50223
2866
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2867
* https://tracker.ceph.com/issues/48773
2868
    qa: scrub does not complete
2869
* https://tracker.ceph.com/issues/51183
2870
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
2871
* https://tracker.ceph.com/issues/51184
2872
    qa: fs:bugs does not specify distro
2873 9 Patrick Donnelly
2874
2875
h3. 2021 June 03
2876
2877
https://pulpito.ceph.com/pdonnell-2021-06-03_03:40:33-fs-wip-pdonnell-testing-20210603.020013-distro-basic-smithi/
2878
2879
* https://tracker.ceph.com/issues/45434
2880
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2881
* https://tracker.ceph.com/issues/50016
2882
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2883
* https://tracker.ceph.com/issues/50821
2884
    qa: untar_snap_rm failure during mds thrashing
2885
* https://tracker.ceph.com/issues/50622 (regression)
2886
    msg: active_connections regression
2887
* https://tracker.ceph.com/issues/49845#note-2 (regression)
2888
    qa: failed umount in test_volumes
2889
* https://tracker.ceph.com/issues/48773
2890
    qa: scrub does not complete
2891
* https://tracker.ceph.com/issues/43216
2892
    MDSMonitor: removes MDS coming out of quorum election
2893 7 Patrick Donnelly
2894
2895 8 Patrick Donnelly
h3. 2021 May 18
2896
2897
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210518.214114
2898
2899
Regression in testing kernel caused some failures. Ilya fixed those and rerun
2900
looked better. Some odd new noise in the rerun relating to packaging and "No
2901
module named 'tasks.ceph'".
2902
2903
* https://tracker.ceph.com/issues/50824
2904
    qa: snaptest-git-ceph bus error
2905
* https://tracker.ceph.com/issues/50622 (regression)
2906
    msg: active_connections regression
2907
* https://tracker.ceph.com/issues/49845#note-2 (regression)
2908
    qa: failed umount in test_volumes
2909
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
2910
    qa: quota failure
2911
2912
2913 7 Patrick Donnelly
h3. 2021 May 18
2914
2915
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210518.025642
2916
2917
* https://tracker.ceph.com/issues/50821
2918
    qa: untar_snap_rm failure during mds thrashing
2919
* https://tracker.ceph.com/issues/48773
2920
    qa: scrub does not complete
2921
* https://tracker.ceph.com/issues/45591
2922
    mgr: FAILED ceph_assert(daemon != nullptr)
2923
* https://tracker.ceph.com/issues/50866
2924
    osd: stat mismatch on objects
2925
* https://tracker.ceph.com/issues/50016
2926
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
2927
* https://tracker.ceph.com/issues/50867
2928
    qa: fs:mirror: reduced data availability
2929
* https://tracker.ceph.com/issues/50821
2930
    qa: untar_snap_rm failure during mds thrashing
2931
* https://tracker.ceph.com/issues/50622 (regression)
2932
    msg: active_connections regression
2933
* https://tracker.ceph.com/issues/50223
2934
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
2935
* https://tracker.ceph.com/issues/50868
2936
    qa: "kern.log.gz already exists; not overwritten"
2937
* https://tracker.ceph.com/issues/50870
2938
    qa: test_full: "rm: cannot remove 'large_file_a': Permission denied"
2939 6 Patrick Donnelly
2940
2941
h3. 2021 May 11
2942
2943
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210511.232042
2944
2945
* one class of failures caused by PR
2946
* https://tracker.ceph.com/issues/48812
2947
    qa: test_scrub_pause_and_resume_with_abort failure
2948
* https://tracker.ceph.com/issues/50390
2949
    mds: monclient: wait_auth_rotating timed out after 30
2950
* https://tracker.ceph.com/issues/48773
2951
    qa: scrub does not complete
2952
* https://tracker.ceph.com/issues/50821
2953
    qa: untar_snap_rm failure during mds thrashing
2954
* https://tracker.ceph.com/issues/50224
2955
    qa: test_mirroring_init_failure_with_recovery failure
2956
* https://tracker.ceph.com/issues/50622 (regression)
2957
    msg: active_connections regression
2958
* https://tracker.ceph.com/issues/50825
2959
    qa: snaptest-git-ceph hang during mon thrashing v2
2960
* https://tracker.ceph.com/issues/50821
2961
    qa: untar_snap_rm failure during mds thrashing
2962
* https://tracker.ceph.com/issues/50823
2963
    qa: RuntimeError: timeout waiting for cluster to stabilize
2964 5 Patrick Donnelly
2965
2966
h3. 2021 May 14
2967
2968
https://pulpito.ceph.com/pdonnell-2021-05-14_21:45:42-fs-master-distro-basic-smithi/
2969
2970
* https://tracker.ceph.com/issues/48812
2971
    qa: test_scrub_pause_and_resume_with_abort failure
2972
* https://tracker.ceph.com/issues/50821
2973
    qa: untar_snap_rm failure during mds thrashing
2974
* https://tracker.ceph.com/issues/50622 (regression)
2975
    msg: active_connections regression
2976
* https://tracker.ceph.com/issues/50822
2977
    qa: testing kernel patch for client metrics causes mds abort
2978
* https://tracker.ceph.com/issues/48773
2979
    qa: scrub does not complete
2980
* https://tracker.ceph.com/issues/50823
2981
    qa: RuntimeError: timeout waiting for cluster to stabilize
2982
* https://tracker.ceph.com/issues/50824
2983
    qa: snaptest-git-ceph bus error
2984
* https://tracker.ceph.com/issues/50825
2985
    qa: snaptest-git-ceph hang during mon thrashing v2
2986
* https://tracker.ceph.com/issues/50826
2987
    kceph: stock RHEL kernel hangs on snaptests with mon|osd thrashers
2988 4 Patrick Donnelly
2989
2990
h3. 2021 May 01
2991
2992
https://pulpito.ceph.com/pdonnell-2021-05-01_09:07:09-fs-wip-pdonnell-testing-20210501.040415-distro-basic-smithi/
2993
2994
* https://tracker.ceph.com/issues/45434
2995
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
2996
* https://tracker.ceph.com/issues/50281
2997
    qa: untar_snap_rm timeout
2998
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
2999
    qa: quota failure
3000
* https://tracker.ceph.com/issues/48773
3001
    qa: scrub does not complete
3002
* https://tracker.ceph.com/issues/50390
3003
    mds: monclient: wait_auth_rotating timed out after 30
3004
* https://tracker.ceph.com/issues/50250
3005
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details"
3006
* https://tracker.ceph.com/issues/50622 (regression)
3007
    msg: active_connections regression
3008
* https://tracker.ceph.com/issues/45591
3009
    mgr: FAILED ceph_assert(daemon != nullptr)
3010
* https://tracker.ceph.com/issues/50221
3011
    qa: snaptest-git-ceph failure in git diff
3012
* https://tracker.ceph.com/issues/50016
3013
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
3014 3 Patrick Donnelly
3015
3016
h3. 2021 Apr 15
3017
3018
https://pulpito.ceph.com/pdonnell-2021-04-15_01:35:57-fs-wip-pdonnell-testing-20210414.230315-distro-basic-smithi/
3019
3020
* https://tracker.ceph.com/issues/50281
3021
    qa: untar_snap_rm timeout
3022
* https://tracker.ceph.com/issues/50220
3023
    qa: dbench workload timeout
3024
* https://tracker.ceph.com/issues/50246
3025
    mds: failure replaying journal (EMetaBlob)
3026
* https://tracker.ceph.com/issues/50250
3027
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details"
3028
* https://tracker.ceph.com/issues/50016
3029
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
3030
* https://tracker.ceph.com/issues/50222
3031
    osd: 5.2s0 deep-scrub : stat mismatch
3032
* https://tracker.ceph.com/issues/45434
3033
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
3034
* https://tracker.ceph.com/issues/49845
3035
    qa: failed umount in test_volumes
3036
* https://tracker.ceph.com/issues/37808
3037
    osd: osdmap cache weak_refs assert during shutdown
3038
* https://tracker.ceph.com/issues/50387
3039
    client: fs/snaps failure
3040
* https://tracker.ceph.com/issues/50389
3041
    mds: "cluster [ERR] Error recovering journal 0x203: (2) No such file or directory" in cluster log"
3042
* https://tracker.ceph.com/issues/50216
3043
    qa: "ls: cannot access 'lost+found': No such file or directory"
3044
* https://tracker.ceph.com/issues/50390
3045
    mds: monclient: wait_auth_rotating timed out after 30
3046
3047 1 Patrick Donnelly
3048
3049 2 Patrick Donnelly
h3. 2021 Apr 08
3050
3051
https://pulpito.ceph.com/pdonnell-2021-04-08_22:42:24-fs-wip-pdonnell-testing-20210408.192301-distro-basic-smithi/
3052
3053
* https://tracker.ceph.com/issues/45434
3054
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
3055
* https://tracker.ceph.com/issues/50016
3056
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
3057
* https://tracker.ceph.com/issues/48773
3058
    qa: scrub does not complete
3059
* https://tracker.ceph.com/issues/50279
3060
    qa: "Replacing daemon mds.b as rank 0 with standby daemon mds.c"
3061
* https://tracker.ceph.com/issues/50246
3062
    mds: failure replaying journal (EMetaBlob)
3063
* https://tracker.ceph.com/issues/48365
3064
    qa: ffsb build failure on CentOS 8.2
3065
* https://tracker.ceph.com/issues/50216
3066
    qa: "ls: cannot access 'lost+found': No such file or directory"
3067
* https://tracker.ceph.com/issues/50223
3068
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
3069
* https://tracker.ceph.com/issues/50280
3070
    cephadm: RuntimeError: uid/gid not found
3071
* https://tracker.ceph.com/issues/50281
3072
    qa: untar_snap_rm timeout
3073
3074 1 Patrick Donnelly
h3. 2021 Apr 08
3075
3076
https://pulpito.ceph.com/pdonnell-2021-04-08_04:31:36-fs-wip-pdonnell-testing-20210408.024225-distro-basic-smithi/
3077
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210408.142238 (with logic inversion / QA fix)
3078
3079
* https://tracker.ceph.com/issues/50246
3080
    mds: failure replaying journal (EMetaBlob)
3081
* https://tracker.ceph.com/issues/50250
3082
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details"
3083
3084
3085
h3. 2021 Apr 07
3086
3087
https://pulpito.ceph.com/pdonnell-2021-04-07_02:12:41-fs-wip-pdonnell-testing-20210406.213012-distro-basic-smithi/
3088
3089
* https://tracker.ceph.com/issues/50215
3090
    qa: "log [ERR] : error reading sessionmap 'mds2_sessionmap'"
3091
* https://tracker.ceph.com/issues/49466
3092
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
3093
* https://tracker.ceph.com/issues/50216
3094
    qa: "ls: cannot access 'lost+found': No such file or directory"
3095
* https://tracker.ceph.com/issues/48773
3096
    qa: scrub does not complete
3097
* https://tracker.ceph.com/issues/49845
3098
    qa: failed umount in test_volumes
3099
* https://tracker.ceph.com/issues/50220
3100
    qa: dbench workload timeout
3101
* https://tracker.ceph.com/issues/50221
3102
    qa: snaptest-git-ceph failure in git diff
3103
* https://tracker.ceph.com/issues/50222
3104
    osd: 5.2s0 deep-scrub : stat mismatch
3105
* https://tracker.ceph.com/issues/50223
3106
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
3107
* https://tracker.ceph.com/issues/50224
3108
    qa: test_mirroring_init_failure_with_recovery failure
3109
3110
h3. 2021 Apr 01
3111
3112
https://pulpito.ceph.com/pdonnell-2021-04-01_00:45:34-fs-wip-pdonnell-testing-20210331.222326-distro-basic-smithi/
3113
3114
* https://tracker.ceph.com/issues/48772
3115
    qa: pjd: not ok 9, 44, 80
3116
* https://tracker.ceph.com/issues/50177
3117
    osd: "stalled aio... buggy kernel or bad device?"
3118
* https://tracker.ceph.com/issues/48771
3119
    qa: iogen: workload fails to cause balancing
3120
* https://tracker.ceph.com/issues/49845
3121
    qa: failed umount in test_volumes
3122
* https://tracker.ceph.com/issues/48773
3123
    qa: scrub does not complete
3124
* https://tracker.ceph.com/issues/48805
3125
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
3126
* https://tracker.ceph.com/issues/50178
3127
    qa: "TypeError: run() got an unexpected keyword argument 'shell'"
3128
* https://tracker.ceph.com/issues/45434
3129
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
3130
3131
h3. 2021 Mar 24
3132
3133
https://pulpito.ceph.com/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/
3134
3135
* https://tracker.ceph.com/issues/49500
3136
    qa: "Assertion `cb_done' failed."
3137
* https://tracker.ceph.com/issues/50019
3138
    qa: mount failure with cephadm "probably no MDS server is up?"
3139
* https://tracker.ceph.com/issues/50020
3140
    qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)"
3141
* https://tracker.ceph.com/issues/48773
3142
    qa: scrub does not complete
3143
* https://tracker.ceph.com/issues/45434
3144
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
3145
* https://tracker.ceph.com/issues/48805
3146
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
3147
* https://tracker.ceph.com/issues/48772
3148
    qa: pjd: not ok 9, 44, 80
3149
* https://tracker.ceph.com/issues/50021
3150
    qa: snaptest-git-ceph failure during mon thrashing
3151
* https://tracker.ceph.com/issues/48771
3152
    qa: iogen: workload fails to cause balancing
3153
* https://tracker.ceph.com/issues/50016
3154
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
3155
* https://tracker.ceph.com/issues/49466
3156
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
3157
3158
3159
h3. 2021 Mar 18
3160
3161
https://pulpito.ceph.com/pdonnell-2021-03-18_13:46:31-fs-wip-pdonnell-testing-20210318.024145-distro-basic-smithi/
3162
3163
* https://tracker.ceph.com/issues/49466
3164
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
3165
* https://tracker.ceph.com/issues/48773
3166
    qa: scrub does not complete
3167
* https://tracker.ceph.com/issues/48805
3168
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
3169
* https://tracker.ceph.com/issues/45434
3170
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
3171
* https://tracker.ceph.com/issues/49845
3172
    qa: failed umount in test_volumes
3173
* https://tracker.ceph.com/issues/49605
3174
    mgr: drops command on the floor
3175
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
3176
    qa: quota failure
3177
* https://tracker.ceph.com/issues/49928
3178
    client: items pinned in cache preventing unmount x2
3179
3180
h3. 2021 Mar 15
3181
3182
https://pulpito.ceph.com/pdonnell-2021-03-15_22:16:56-fs-wip-pdonnell-testing-20210315.182203-distro-basic-smithi/
3183
3184
* https://tracker.ceph.com/issues/49842
3185
    qa: stuck pkg install
3186
* https://tracker.ceph.com/issues/49466
3187
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
3188
* https://tracker.ceph.com/issues/49822
3189
    test: test_mirroring_command_idempotency (tasks.cephfs.test_admin.TestMirroringCommands) failure
3190
* https://tracker.ceph.com/issues/49240
3191
    terminate called after throwing an instance of 'std::bad_alloc'
3192
* https://tracker.ceph.com/issues/48773
3193
    qa: scrub does not complete
3194
* https://tracker.ceph.com/issues/45434
3195
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
3196
* https://tracker.ceph.com/issues/49500
3197
    qa: "Assertion `cb_done' failed."
3198
* https://tracker.ceph.com/issues/49843
3199
    qa: fs/snaps/snaptest-upchildrealms.sh failure
3200
* https://tracker.ceph.com/issues/49845
3201
    qa: failed umount in test_volumes
3202
* https://tracker.ceph.com/issues/48805
3203
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
3204
* https://tracker.ceph.com/issues/49605
3205
    mgr: drops command on the floor
3206
3207
and failure caused by PR: https://github.com/ceph/ceph/pull/39969
3208
3209
3210
h3. 2021 Mar 09
3211
3212
https://pulpito.ceph.com/pdonnell-2021-03-09_03:27:39-fs-wip-pdonnell-testing-20210308.214827-distro-basic-smithi/
3213
3214
* https://tracker.ceph.com/issues/49500
3215
    qa: "Assertion `cb_done' failed."
3216
* https://tracker.ceph.com/issues/48805
3217
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
3218
* https://tracker.ceph.com/issues/48773
3219
    qa: scrub does not complete
3220
* https://tracker.ceph.com/issues/45434
3221
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
3222
* https://tracker.ceph.com/issues/49240
3223
    terminate called after throwing an instance of 'std::bad_alloc'
3224
* https://tracker.ceph.com/issues/49466
3225
    qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"
3226
* https://tracker.ceph.com/issues/49684
3227
    qa: fs:cephadm mount does not wait for mds to be created
3228
* https://tracker.ceph.com/issues/48771
3229
    qa: iogen: workload fails to cause balancing