Project

General

Profile

Pacific » History » Version 132

Milind Changire, 01/24/2024 07:20 AM

1 1 Patrick Donnelly
h1. Pacific
2
3 4 Patrick Donnelly
h2. On-call Schedule
4
5 98 Venky Shankar
* Jul: Venky
6
* Aug: Patrick
7 118 Jos Collin
* Sep: Jos Collin
8 98 Venky Shankar
* Oct: Xiubo
9
* Nov: Rishabh
10
* Dec: Kotresh
11
* Jan: Milind
12 81 Venky Shankar
13 1 Patrick Donnelly
h2. Reviews
14 85 Rishabh Dave
15 119 Rishabh Dave
h3. ADD NEW ENTRY BELOW
16
17 132 Milind Changire
18
h3. 24 Jan 2024
19
20
https://pulpito.ceph.com/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/
21
22
* https://tracker.ceph.com/issues/64059
23
  https://download.ceph.com/qa/ior.tbz2 - ERROR 404: Not Found
24
* https://tracker.ceph.com/issues/62501
25
  pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC (fs/full/subvolume_snapshot_rm.sh)
26
* https://tracker.ceph.com/issues/51964
27
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
28
* https://tracker.ceph.com/issues/58992
29
  test_acls (tasks.cephfs.test_acls.TestACLs) 
30
* https://tracker.ceph.com/issues/50224
31
  test_mirroring_init_failure_with_recovery - (tasks.cephfs.test_mirroring.TestMirroring) 
32
33
34 131 Kotresh Hiremath Ravishankar
h3. 29 Dec 2023
35
36
https://pulpito.ceph.com/yuriw-2023-12-28_15:45:00-fs-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/
37
38
* https://tracker.ceph.com/issues/63212
39
  qa: failed to download ior.tbz2
40
* https://tracker.ceph.com/issues/52624
41
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
42
* https://tracker.ceph.com/issues/62501
43
  pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC (fs/full/subvolume_snapshot_rm.sh)
44
* https://tracker.ceph.com/issues/50224
45
  test_mirroring_init_failure_with_recovery - (tasks.cephfs.test_mirroring.TestMirroring) 
46
* https://tracker.ceph.com/issues/58992
47
  test_acls (tasks.cephfs.test_acls.TestACLs) 
48
* https://tracker.ceph.com/issues/50223
49
  qa: "client.4737 isn't responding to mclientcaps(revoke)"
50
* https://tracker.ceph.com/issues/63907
51
  cephfs-mirror: Mirror::update_fs_mirrors crashes while taking lock (coredump)
52
53 128 Venky Shankar
h3. 21 Dec 2023
54
55
https://pulpito.ceph.com/?branch=wip-yuri5-testing-2023-12-14-1107-pacific
56
57
* https://tracker.ceph.com/issues/52624
58
    cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
59
* https://tracker.ceph.com/issues/63539
60
    fs/full/subvolume_clone.sh: Health check failed: 1 full osd(s) (OSD_FULL)
61
* test_acls failed because known distro wasn't detected
62 129 Venky Shankar
* https://tracker.ceph.com/issues/50223
63
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
64 128 Venky Shankar
65 127 Rishabh Dave
h3. 4 Dec 2023 2
66 126 Rishabh Dave
67
https://pulpito.ceph.com/yuriw-2023-11-27_22:36:25-fs-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/
68
69
* ior/mdtest failures because packages were missing from download.ceph.com
70
* test_acls failed because known distro wasn't detected
71
* https://tracker.ceph.com/issues/63539
72
  fs/full/subvolume_clone.sh: Health check failed: 1 full osd(s) (OSD_FULL)
73
* https://tracker.ceph.com/issues/52624
74
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
75
* https://tracker.ceph.com/issues/37808
76
  osd: osdmap cache weak_refs assert during shutdown
77 125 Kotresh Hiremath Ravishankar
78
h3. 04 Dec 2023
79
80
https://pulpito.ceph.com/yuriw-2023-11-30_15:46:13-fs-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/
81
82
* ior package was missing because it was deleted from download.ceph.com
83
* Error re-imaging machines:
84
  https://pulpito.ceph.com/yuriw-2023-11-30_15:46:13-fs-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472083
85
  https://pulpito.ceph.com/yuriw-2023-11-30_15:46:13-fs-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472084
86
  https://pulpito.ceph.com/yuriw-2023-11-30_15:46:13-fs-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472104
87
* https://tracker.ceph.com/issues/52624
88
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
89
* https://tracker.ceph.com/issues/51282
90
  cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
91
* https://tracker.ceph.com/issues/62501
92
  pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC (fs/full/subvolume_snapshot_rm.sh)
93
* https://tracker.ceph.com/issues/58992
94
  test_acls (tasks.cephfs.test_acls.TestACLs)
95
* https://tracker.ceph.com/issues/54462
96
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
97
98
99 123 Rishabh Dave
h3. 22 Nov 2023
100
101
https://pulpito.ceph.com/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/
102 124 Rishabh Dave
https://pulpito.ceph.com/yuriw-2023-11-21_15:54:56-smoke-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/
103 123 Rishabh Dave
104
* https://tracker.ceph.com/issues/62501
105
  pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC (fs/full/subvolume_snapshot_rm.sh)
106
* https://tracker.ceph.com/issues/50223
107
  qa: "client.4737 isn't responding to mclientcaps(revoke)"
108
* ior/mdtest failures because packages were missing from download.ceph.com
109
* test_acls failed because known distro wasn't detected
110
* https://tracker.ceph.com/issues/52624
111
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
112
113 122 Rishabh Dave
h3. 15 Nov 2023
114
115
https://pulpito.ceph.com/yuriw-2023-10-24_00:02:48-fs-wip-yuri7-testing-2023-10-23-1230-pacific-distro-default-smithi/
116
117
* https://tracker.ceph.com/issues/63539
118
  fs/full/subvolume_clone.sh: Health check failed: 1 full osd(s) (OSD_FULL)
119
* test_acls: distro name for RHEL 8.4 wasn't recognized one by xfstests_dev.py
120
* ior package was missing because it was deleted from download.ceph.com
121
* https://tracker.ceph.com/issues/52624
122
  qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
123
124 120 Rishabh Dave
h3. 8 Nov 2023
125 121 Rishabh Dave
126 120 Rishabh Dave
fs: https://pulpito.ceph.com/vshankar-2023-11-06_07:50:57-fs-wip-vshankar-testing1-2023-11-02-1726-pacific-testing-default-smithi/
127 121 Rishabh Dave
smoke: https://pulpito.ceph.com/vshankar-2023-11-06_07:53:57-smoke-wip-vshankar-testing1-2023-11-02-1726-pacific-testing-default-smithi/
128 120 Rishabh Dave
129
* https://tracker.ceph.com/issues/51964
130
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
131
* https://tracker.ceph.com/issues/62501
132
  pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC (fs/full/subvolume_snapshot_rm.sh)
133
* https://tracker.ceph.com/issues/52624
134
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
135
136 114 Jos Collin
h3. 2023 September 12
137
138 117 Jos Collin
https://pulpito.ceph.com/yuriw-2023-08-21_21:30:15-fs-wip-yuri2-testing-2023-08-21-0910-pacific-distro-default-smithi
139 115 Jos Collin
140 117 Jos Collin
  * https://tracker.ceph.com/issues/58992
141
    test_acls (tasks.cephfs.test_acls.TestACLs)
142
  * https://tracker.ceph.com/issues/62580
143
    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
144 114 Jos Collin
145 111 Patrick Donnelly
h3. 2023 August 31
146
147 113 Patrick Donnelly
https://github.com/ceph/ceph/pull/53189
148
https://github.com/ceph/ceph/pull/53243
149
https://github.com/ceph/ceph/pull/53185
150
https://github.com/ceph/ceph/pull/52744
151
https://github.com/ceph/ceph/pull/51045
152
153 111 Patrick Donnelly
https://pulpito.ceph.com/pdonnell-2023-08-31_15:31:51-fs-wip-batrick-testing-20230831.124848-pacific-distro-default-smithi/
154
155 112 Patrick Donnelly
* https://tracker.ceph.com/issues/62501
156
    pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC (fs/full/subvolume_snapshot_rm.sh)
157
* https://tracker.ceph.com/issues/52624
158
    cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
159
* https://tracker.ceph.com/issues/54462
160
    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
161
* https://tracker.ceph.com/issues/50222
162
    osd: 5.2s0 deep-scrub : stat mismatch
163
* https://tracker.ceph.com/issues/50250
164
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details" ("freshly-calculated rstats don't match existing ones")
165
166
Some spurious infrastructure / valgrind noise during cleanup.
167 111 Patrick Donnelly
168 107 Patrick Donnelly
h3. 2023 August 22
169
170
Pacific v16.2.14 QA
171
172
https://pulpito.ceph.com/yuriw-2023-08-22_14:48:56-fs-pacific-release-distro-default-smithi/
173
https://pulpito.ceph.com/yuriw-2023-08-22_23:49:19-fs-pacific-release-distro-default-smithi/
174
175
* https://tracker.ceph.com/issues/62578
176
    mon: osd pg-upmap-items command causes PG_DEGRADED warnings
177 109 Patrick Donnelly
* https://tracker.ceph.com/issues/52624
178
    cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
179 108 Patrick Donnelly
* https://tracker.ceph.com/issues/62501
180
    pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC (fs/full/subvolume_snapshot_rm.sh)
181 107 Patrick Donnelly
* https://tracker.ceph.com/issues/58992
182
    test_acls (tasks.cephfs.test_acls.TestACLs)
183 108 Patrick Donnelly
* https://tracker.ceph.com/issues/62579
184
    client: evicted warning because client completes unmount before thrashed MDS comes back
185 110 Patrick Donnelly
* https://tracker.ceph.com/issues/62580
186
    testing: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
187 107 Patrick Donnelly
188 106 Patrick Donnelly
h3. 2023 August 16-2
189
190
https://trello.com/c/qRJogcXu/1827-wip-yuri2-testing-2023-08-16-1142-pacific
191
https://pulpito.ceph.com/?branch=wip-yuri2-testing-2023-08-16-1142-pacific
192
193
* https://tracker.ceph.com/issues/52624
194
    cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
195
* https://tracker.ceph.com/issues/58992
196
    test_acls (tasks.cephfs.test_acls.TestACLs)
197
* https://tracker.ceph.com/issues/62501
198
    pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC
199
200 105 Patrick Donnelly
h3. 2023 August 16
201
202
https://trello.com/c/4TiiTR0k/1821-wip-yuri7-testing-2023-08-16-1309-pacific-old-wip-yuri7-testing-2023-08-16-0933-pacific-old-wip-yuri7-testing-2023-08-15-0741-pa
203
https://pulpito.ceph.com/?branch=wip-yuri7-testing-2023-08-16-1309-pacific
204
205
* https://tracker.ceph.com/issues/62499
206
    testing (?): deadlock ffsb task
207
* https://tracker.ceph.com/issues/62501
208
    pacific(?): qa: mgr-osd-full causes OSD aborts due to ENOSPC
209
210 103 Patrick Donnelly
h3. 2023 August 11
211
212
https://trello.com/c/ONHeA3yz/1823-wip-yuri8-testing-2023-08-11-0834-pacific
213
https://pulpito.ceph.com/yuriw-2023-08-15_01:22:18-fs-wip-yuri8-testing-2023-08-11-0834-pacific-distro-default-smithi/
214
215
Some infra noise causes dead job.
216
217
* https://tracker.ceph.com/issues/52624
218
    cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
219
* https://tracker.ceph.com/issues/58992
220
    test_acls (tasks.cephfs.test_acls.TestACLs)
221 1 Patrick Donnelly
* https://tracker.ceph.com/issues/58340
222 107 Patrick Donnelly
    fsstress.sh failed with errno 124 
223 103 Patrick Donnelly
* https://tracker.ceph.com/issues/48773
224 107 Patrick Donnelly
    Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p...' - qa: scrub does not complete
225 103 Patrick Donnelly
* https://tracker.ceph.com/issues/50527
226
    pacific: qa: untar_snap_rm timeout during osd thrashing (ceph-fuse)
227
228 100 Venky Shankar
h3. 2023 August 8
229
230
https://pulpito.ceph.com/?branch=wip-yuri10-testing-2023-08-01-0753-pacific
231
232
* https://tracker.ceph.com/issues/52624
233
    cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
234 1 Patrick Donnelly
* https://tracker.ceph.com/issues/50223
235
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
236 101 Patrick Donnelly
* https://tracker.ceph.com/issues/62164
237
    qa: "cluster [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13..."
238
* https://tracker.ceph.com/issues/51964
239
    qa: test_cephfs_mirror_restart_sync_on_blocklist failure
240
* https://tracker.ceph.com/issues/58992
241
    test_acls (tasks.cephfs.test_acls.TestACLs)
242
* https://tracker.ceph.com/issues/62465
243
    pacific (?): LibCephFS.ShutdownRace segmentation fault
244 102 Patrick Donnelly
* "fs/full/subvolume_snapshot_rm.sh" failure caused by bluestore crash.
245 100 Venky Shankar
246 104 Patrick Donnelly
h3. 2023 August 03
247
248
https://trello.com/c/8HALLv9T/1813-wip-yuri6-testing-2023-08-03-0807-pacific-old-wip-yuri6-testing-2023-07-24-0819-pacific2-old-wip-yuri6-testing-2023-07-24-0819-p
249
https://pulpito.ceph.com/?branch=wip-yuri6-testing-2023-08-03-0807-pacific
250
251
* https://tracker.ceph.com/issues/52624
252
    cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
253
* https://tracker.ceph.com/issues/58992
254
    test_acls (tasks.cephfs.test_acls.TestACLs)
255
256 99 Venky Shankar
h3. 2023 July 25
257
258
https://pulpito.ceph.com/?branch=wip-yuri-testing-2023-07-19-1340-pacific
259
260
* https://tracker.ceph.com/issues/52624
261
    cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
262
* https://tracker.ceph.com/issues/58992
263
    test_acls (tasks.cephfs.test_acls.TestACLs)
264
* https://tracker.ceph.com/issues/50223
265
    qa: "client.4737 isn't responding to mclientcaps(revoke)"
266
* https://tracker.ceph.com/issues/62160
267
    mds: MDS abort because newly corrupt dentry to be committed
268
* https://tracker.ceph.com/issues/61201
269
    qa: test_rebuild_moved_file (tasks/data-scan) fails because mds crashes in pacific (recovery thread crash)
270
271
272 94 Kotresh Hiremath Ravishankar
h3. 2023 May 17
273
274
https://pulpito.ceph.com/yuriw-2023-05-15_21:56:33-fs-wip-yuri2-testing-2023-05-15-0810-pacific_2-distro-default-smithi/
275 97 Kotresh Hiremath Ravishankar
https://pulpito.ceph.com/yuriw-2023-05-17_14:20:33-fs-wip-yuri2-testing-2023-05-15-0810-pacific_2-distro-default-smithi/ (fresh pkg and re-run because of package/installation issues)
276 94 Kotresh Hiremath Ravishankar
277
* https://tracker.ceph.com/issues/52624
278
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
279 96 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/61201 (NEW)
280 94 Kotresh Hiremath Ravishankar
  Test failure: test_rebuild_moved_file (tasks.cephfs.test_data_scan.TestDataScan)
281
* https://tracker.ceph.com/issues/58340
282
  fsstress.sh failed with errno 124 
283 1 Patrick Donnelly
* https://tracker.ceph.com/issues/52624
284 94 Kotresh Hiremath Ravishankar
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
285
* https://tracker.ceph.com/issues/58992
286
  test_acls (tasks.cephfs.test_acls.TestACLs)
287
* https://tracker.ceph.com/issues/54462
288 95 Kotresh Hiremath Ravishankar
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128 
289 94 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/58674
290 95 Kotresh Hiremath Ravishankar
  teuthology.exceptions.MaxWhileTries: reached maximum tries (180) after waiting for 180 seconds 
291 94 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/55446
292 95 Kotresh Hiremath Ravishankar
  fs/upgrade/mds_upgrade_sequence - hit max job timeout
293
 
294 94 Kotresh Hiremath Ravishankar
295
296 91 Kotresh Hiremath Ravishankar
h3. 2023 May 11
297
298
https://pulpito.ceph.com/yuriw-2023-05-09_19:39:46-fs-wip-yuri4-testing-2023-05-08-0846-pacific-distro-default-smithi
299
300
* https://tracker.ceph.com/issues/52624
301
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
302
* https://tracker.ceph.com/issues/51964
303
  qa: test_cephfs_mirror_restart_sync_on_blocklist failure
304
* https://tracker.ceph.com/issues/48773
305
  Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p...' - qa: scrub does not complete
306
* https://tracker.ceph.com/issues/50223
307
  qa: "client.4737 isn't responding to mclientcaps(revoke)"
308 1 Patrick Donnelly
* https://tracker.ceph.com/issues/58992
309
  test_acls (tasks.cephfs.test_acls.TestACLs)
310 91 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/58340
311
  fsstress.sh failed with errno 124
312 93 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/51964
313
  Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
314
* https://tracker.ceph.com/issues/55446
315
  fs/upgrade/mds_upgrade_sequence - hit max job timeout
316 91 Kotresh Hiremath Ravishankar
317 90 Rishabh Dave
h3. 2023 May 4
318
319
https://pulpito.ceph.com/yuriw-2023-04-25_19:03:49-fs-wip-yuri5-testing-2023-04-25-0837-pacific-distro-default-smithi/
320
321
* https://tracker.ceph.com/issues/59560
322
  qa: RuntimeError: more than one file system available
323
* https://tracker.ceph.com/issues/59626
324
  FSMissing: File system xxxx does not exist in the map
325
* https://tracker.ceph.com/issues/58340
326
  fsstress.sh failed with errno 124
327
* https://tracker.ceph.com/issues/58992
328
  test_acls
329
* https://tracker.ceph.com/issues/48773
330
  Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p...' - qa: scrub does not complete
331
* https://tracker.ceph.com/issues/57676
332
  qa: error during scrub thrashing: rank damage found: {'backtrace'}
333
334
335 86 Rishabh Dave
h3. 2023 Apr 13
336
337
https://pulpito.ceph.com/yuriw-2023-04-04_15:06:57-fs-wip-yuri8-testing-2023-03-31-0812-pacific-distro-default-smithi/
338
339
https://tracker.ceph.com/issues/52624
340
cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
341
* https://tracker.ceph.com/issues/57594
342
  Test failure: test_rebuild_moved_dir (tasks.cephfs.test_data_scan.TestDataScan)
343
* https://tracker.ceph.com/issues/54108
344
  qa: iogen workunit: "The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}" 
345
* https://tracker.ceph.com/issues/58340
346
  fsstress.sh failed with errno 125
347
* https://tracker.ceph.com/issues/54462
348
  Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi055 with status 128
349
* https://tracker.ceph.com/issues/49287     
350
  cephadm: podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found
351
* https://tracker.ceph.com/issues/58726
352
  test_acls: expected a yum based or a apt based system
353 40 Ramana Raja
354 84 Venky Shankar
h3. 2022 Dec 07
355
356
https://pulpito.ceph.com/yuriw-2022-12-07_15:45:30-fs-wip-yuri4-testing-2022-12-05-0921-pacific-distro-default-smithi/
357
358
many transient git.ceph.com related timeouts
359
360
* https://tracker.ceph.com/issues/52624
361
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
362
* https://tracker.ceph.com/issues/50224
363
    test_mirroring_init_failure_with_recovery - (tasks.cephfs.test_mirroring.TestMirroring)
364
* https://tracker.ceph.com/issues/56644
365
    qa: test_rapid_creation fails with "No space left on device"
366
* https://tracker.ceph.com/issues/58221
367
    pacific: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
368
369 83 Venky Shankar
h3. 2022 Dec 02
370
371
many transient git.ceph.com related timeouts
372
many transient 'Failed to connect to the host via ssh' failures
373
374
* https://tracker.ceph.com/issues/57723
375
  pacific: qa: test_subvolume_snapshot_info_if_orphan_clone fails
376
* https://tracker.ceph.com/issues/52624
377
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
378
379 82 Venky Shankar
h3. 2022 Dec 01
380
381
many transient git.ceph.com related timeouts
382
383
* https://tracker.ceph.com/issues/57723
384
  pacific: qa: test_subvolume_snapshot_info_if_orphan_clone fails
385
* https://tracker.ceph.com/issues/52624
386
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
387
388 80 Patrick Donnelly
h3. 2022 Nov 18
389
390
https://trello.com/c/ecysAxl6/1656-wip-yuri-testing-2022-11-18-1500-pacific-old-wip-yuri-testing-2022-10-23-0729-pacific-old-wip-yuri-testing-2022-10-21-0927-pacif
391
https://pulpito.ceph.com/yuriw-2022-11-28_16:28:56-fs-wip-yuri-testing-2022-11-18-1500-pacific-distro-default-smithi/
392
393
2 ansible dead failures.
394
12 transient git.ceph.com related timeouts
395
396
* https://tracker.ceph.com/issues/57723
397
  pacific: qa: test_subvolume_snapshot_info_if_orphan_clone fails
398
* https://tracker.ceph.com/issues/52624
399
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
400
401 78 Milind Changire
h3. 2022 Oct 19
402
403
http://pulpito.front.sepia.ceph.com/yuriw-2022-10-10_17:18:40-fs-wip-yuri5-testing-2022-10-10-0837-pacific-distro-default-smithi/
404
405
* https://tracker.ceph.com/issues/57723
406
  pacific: qa: test_subvolume_snapshot_info_if_orphan_clone fails
407
* https://tracker.ceph.com/issues/52624
408
  cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
409
* https://tracker.ceph.com/issues/56644
410
  qa: test_rapid_creation fails with "No space left on device"
411
* https://tracker.ceph.com/issues/54460
412
  snaptest-multiple-capsnaps.sh test failure
413
* https://tracker.ceph.com/issues/57892
414
  sudo dd of=/home/ubuntu/cephtest/valgrind.supp failure during node setup phase
415
416
417
418 76 Venky Shankar
h3. 2022 Oct 06
419
420
http://pulpito.front.sepia.ceph.com/yuriw-2022-09-21_13:07:25-rados-wip-yuri5-testing-2022-09-20-1347-pacific-distro-default-smithi/
421
422
* https://tracker.ceph.com/issues/52624
423
    cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
424
* https://tracker.ceph.com/issues/50223
425
	qa: "client.4737 isn't responding to mclientcaps(revoke)"
426 77 Milind Changire
* https://tracker.ceph.com/issues/56507
427
  Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
428 76 Venky Shankar
429 75 Venky Shankar
h3. 2022 Sep 27
430
431
http://pulpito.front.sepia.ceph.com/yuriw-2022-09-22_22:35:04-fs-wip-yuri2-testing-2022-09-22-1400-pacific-distro-default-smithi/
432
433
* https://tracker.ceph.com/issues/52624
434
    cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
435
* https://tracker.ceph.com/issues/48773
436
    Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p...' - qa: scrub does not complete
437
* https://tracker.ceph.com/issues/50224
438
    test_mirroring_init_failure_with_recovery - (tasks.cephfs.test_mirroring.TestMirroring)
439
* https://tracker.ceph.com/issues/56507
440
    Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
441
* https://tracker.ceph.com/issues/50223
442
	qa: "client.4737 isn't responding to mclientcaps(revoke)"
443
444 74 Venky Shankar
h3. 2022 Sep 22
445
446
http://pulpito.front.sepia.ceph.com/?branch=wip-yuri2-testing-2022-09-06-1007-pacific
447
448
* https://tracker.ceph.com/issues/52624
449
    cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
450
* https://tracker.ceph.com/issues/51282
451
    cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
452
* https://tracker.ceph.com/issues/53360
453
    pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
454
455 73 Venky Shankar
h3. 2022 Sep 19
456
457
http://pulpito.front.sepia.ceph.com/yuriw-2022-09-16_19:05:25-fs-wip-yuri11-testing-2022-09-16-0958-pacific-distro-default-smithi/
458
459
* https://tracker.ceph.com/issues/52624
460
    cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
461
* https://tracker.ceph.com/issues/57594
462
    pacific: Test failure: test_rebuild_moved_dir (tasks.cephfs.test_data_scan.TestDataScan)
463
464 71 Venky Shankar
h3. 2022 Sep 15
465
466
http://pulpito.front.sepia.ceph.com/yuriw-2022-09-13_14:22:44-fs-wip-yuri5-testing-2022-09-09-1109-pacific-distro-default-smithi/
467
468
* https://tracker.ceph.com/issues/51282
469
    cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
470
* https://tracker.ceph.com/issues/52624
471
    cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
472
* https://tracker.ceph.com/issues/53360
473
    pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
474 72 Venky Shankar
* https://tracker.ceph.com/issues/48773
475
    Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p...' - qa: scrub does not complete
476 71 Venky Shankar
477 70 Kotresh Hiremath Ravishankar
h3. 2022 AUG 18
478
479
https://pulpito.ceph.com/yuriw-2022-08-18_23:16:33-fs-wip-yuri10-testing-2022-08-18-1400-pacific-distro-default-smithi/
480
* https://tracker.ceph.com/issues/52624 - cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
481
* https://tracker.ceph.com/issues/51267 - tasks/workunit/snaps failure - Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status
482
* https://tracker.ceph.com/issues/48773 - Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p...' - qa: scrub does not complete
483
* https://tracker.ceph.com/issues/51964 - qa: test_cephfs_mirror_restart_sync_on_blocklist failure
484
* https://tracker.ceph.com/issues/57218 - qa: tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} fails
485
* https://tracker.ceph.com/issues/56507 - Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
486
487
488
Re-run1: https://pulpito.ceph.com/yuriw-2022-08-19_21:01:11-fs-wip-yuri10-testing-2022-08-18-1400-pacific-distro-default-smithi/
489
* https://tracker.ceph.com/issues/52624 - cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
490
* https://tracker.ceph.com/issues/57219 - mds crashed while running workunit test fs/misc/dirfrag.sh
491
492
493 69 Kotresh Hiremath Ravishankar
h3. 2022 AUG 11
494
495
https://pulpito.ceph.com/yuriw-2022-08-11_16:57:01-fs-wip-yuri3-testing-2022-08-11-0809-pacific-distro-default-smithi/
496
* Most of the failures are passed in re-run. Please check rerun failures below.
497
  - https://tracker.ceph.com/issues/57147 - test_full_fsync (tasks.cephfs.test_full.TestClusterFull) - mds stuck in up:creating state - osd crash
498
  - https://tracker.ceph.com/issues/52624 - cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
499
  - https://tracker.ceph.com/issues/51282 - cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
500
  - https://tracker.ceph.com/issues/50224 - test_mirroring_init_failure_with_recovery - (tasks.cephfs.test_mirroring.TestMirroring)
501
  - https://tracker.ceph.com/issues/57083 - https://tracker.ceph.com/issues/53360 - tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04} 
502
  - https://tracker.ceph.com/issues/51183 - Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
503
  - https://tracker.ceph.com/issues/56507 - Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)   
504
505
Re-run-1: https://pulpito.ceph.com/yuriw-2022-08-15_13:45:54-fs-wip-yuri3-testing-2022-08-11-0809-pacific-distro-default-smithi/
506
* https://tracker.ceph.com/issues/52624
507
  cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
508
* tasks/mirror - Command failed on smithi078 with status 123: "sudo find /var/log/ceph -name '*.log' -print0 ..." - Asked for re-run.
509
* https://tracker.ceph.com/issues/57083
510
* https://tracker.ceph.com/issues/53360
511
  tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04} - tasks.cephfs.fuse_mount:mount command failed
512
  client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
513
* https://tracker.ceph.com/issues/51183
514
  Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
515
* https://tracker.ceph.com/issues/56507
516
  Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
517
518 68 Kotresh Hiremath Ravishankar
519
h3. 2022 AUG 04
520
521
https://pulpito.ceph.com/yuriw-2022-08-04_20:54:08-fs-wip-yuri6-testing-2022-08-04-0617-pacific-distro-default-smithi/
522
523
* https://tracker.ceph.com/issues/57087
524
  test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
525
* https://tracker.ceph.com/issues/52624
526
  cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
527
* https://tracker.ceph.com/issues/51267
528
  tasks/workunit/snaps failure - Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status      
529
* https://tracker.ceph.com/issues/53360
530
* https://tracker.ceph.com/issues/57083
531
  qa/import-legacy tasks.cephfs.fuse_mount:mount command failed - :ModuleNotFoundError: No module named 'ceph_volume_client'
532
* https://tracker.ceph.com/issues/56507
533
  Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
534
535
536 67 Kotresh Hiremath Ravishankar
h3. 2022 July 15
537
538
https://pulpito.ceph.com/yuriw-2022-07-21_22:57:30-fs-wip-yuri2-testing-2022-07-15-0755-pacific-distro-default-smithi
539
Re-run: https://pulpito.ceph.com/yuriw-2022-07-24_15:34:38-fs-wip-yuri2-testing-2022-07-15-0755-pacific-distro-default-smithi
540
541
* https://tracker.ceph.com/issues/52624
542
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
543
* https://tracker.ceph.com/issues/57083
544
* https://tracker.ceph.com/issues/53360
545
        tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04} - tasks.cephfs.fuse_mount:mount command failed
546
        client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]" 
547
* https://tracker.ceph.com/issues/51183
548
        Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
549
* https://tracker.ceph.com/issues/56507
550
        pacific: Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
551
552
553 66 Venky Shankar
h3. 2022 July 08
554
555
* https://tracker.ceph.com/issues/52624
556
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
557
* https://tracker.ceph.com/issues/53360
558
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
559
* https://tracker.ceph.com/issues/51183
560
        Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
561
* https://tracker.ceph.com/issues/56506
562
        pacific: Test failure: test_rebuild_backtraceless (tasks.cephfs.test_data_scan.TestDataScan)
563
* https://tracker.ceph.com/issues/56507
564
        pacific: Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
565
566 65 Venky Shankar
h3. 2022 Jun 28
567
568
https://pulpito.ceph.com/?branch=wip-yuri3-testing-2022-06-22-1121-pacific
569
570
* https://tracker.ceph.com/issues/52624
571
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
572
* https://tracker.ceph.com/issues/53360
573
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
574
* https://tracker.ceph.com/issues/51183
575
        Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
576
577 64 Venky Shankar
h3. 2022 Jun 22
578
579
https://pulpito.ceph.com/?branch=wip-yuri4-testing-2022-06-21-0704-pacific
580
581
* https://tracker.ceph.com/issues/52624
582
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
583
* https://tracker.ceph.com/issues/53360
584
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
585
* https://tracker.ceph.com/issues/51183
586
        Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
587
588 63 Venky Shankar
h3. 2022 Jun 17
589
590
https://pulpito.ceph.com/yuriw-2022-06-15_17:11:32-fs-wip-yuri10-testing-2022-06-15-0732-pacific-distro-default-smithi/
591
592
* https://tracker.ceph.com/issues/52624
593
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
594
* https://tracker.ceph.com/issues/53360
595
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
596
* https://tracker.ceph.com/issues/51183
597
        Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
598
599 62 Venky Shankar
h3. 2022 Jun 16
600
601
https://pulpito.ceph.com/yuriw-2022-06-15_17:08:32-fs-wip-yuri-testing-2022-06-14-0744-pacific-distro-default-smithi/
602
603
* https://tracker.ceph.com/issues/52624
604
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
605
* https://tracker.ceph.com/issues/53360
606
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
607
* https://tracker.ceph.com/issues/55449
608
        pacific: mds_upgrade_sequence: "overall HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available; 33 daemons have recently crashed" during suites/fsstress.sh
609
* https://tracker.ceph.com/issues/51267
610
        CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1:...
611
* https://tracker.ceph.com/issues/55332
612
        Failure in snaptest-git-ceph.sh (it's an async unlink/create bug)
613
614 61 Venky Shankar
h3. 2022 Jun 15
615
616
http://pulpito.front.sepia.ceph.com/yuriw-2022-06-09_13:32:29-fs-wip-yuri10-testing-2022-06-08-0730-pacific-distro-default-smithi/
617
618
* https://tracker.ceph.com/issues/52624
619
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
620
* https://tracker.ceph.com/issues/53360
621
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
622
* https://tracker.ceph.com/issues/55449
623
        pacific: mds_upgrade_sequence: "overall HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available; 33 daemons have recently crashed" during suites/fsstress.sh
624
625
626 60 Venky Shankar
h3. 2022 Jun 10
627
628
https://pulpito.ceph.com/?branch=wip-yuri5-testing-2022-06-07-1529-pacific
629
630
* https://tracker.ceph.com/issues/52624
631
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
632
* https://tracker.ceph.com/issues/53360
633
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
634
* https://tracker.ceph.com/issues/55449
635
        pacific: mds_upgrade_sequence: "overall HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available; 33 daemons have recently crashed" during suites/fsstress.sh
636
637 59 Venky Shankar
h3. 2022 Jun 09
638
639
https://pulpito.ceph.com/yuriw-2022-06-07_16:05:25-fs-wip-yuri4-testing-2022-06-01-1350-pacific-distro-default-smithi/
640
641
* https://tracker.ceph.com/issues/52624
642
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
643
* https://tracker.ceph.com/issues/53360
644
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
645
* https://tracker.ceph.com/issues/55449
646
        pacific: mds_upgrade_sequence: "overall HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available; 33 daemons have recently crashed" during suites/fsstress.sh
647
648 58 Venky Shankar
h3. 2022 May 06
649
650
https://pulpito.ceph.com/?sha1=73636a1b00037ff974bcdc969b009c5ecec626cc
651
652
* https://tracker.ceph.com/issues/52624
653
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
654
* https://tracker.ceph.com/issues/53360
655
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
656
657 57 Venky Shankar
h3. 2022 April 18
658
659
http://pulpito.front.sepia.ceph.com/?branch=wip-yuri4-testing-2022-04-18-0609-pacific
660
(only mgr/snap_schedule backport pr)
661
662
* https://tracker.ceph.com/issues/52624
663
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
664 56 Venky Shankar
665
h3. 2022 March 28
666
667
http://pulpito.front.sepia.ceph.com/yuriw-2022-03-25_20:52:40-fs-wip-yuri4-testing-2022-03-25-1203-pacific-distro-default-smithi/
668
http://pulpito.front.sepia.ceph.com/yuriw-2022-03-26_19:52:48-fs-wip-yuri4-testing-2022-03-25-1203-pacific-distro-default-smithi/
669
670
* https://tracker.ceph.com/issues/52624
671
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
672
* https://tracker.ceph.com/issues/53360
673
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
674
* https://tracker.ceph.com/issues/54411
675
	mds_upgrade_sequence: "overall HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available;
676
	33 daemons have recently crashed" during suites/fsstress.sh
677
678 55 Venky Shankar
h3. 2022 March 25
679
680
https://pulpito.ceph.com/yuriw-2022-03-22_18:42:28-fs-wip-yuri8-testing-2022-03-22-0910-pacific-distro-default-smithi/
681
https://pulpito.ceph.com/yuriw-2022-03-23_14:04:56-fs-wip-yuri8-testing-2022-03-22-0910-pacific-distro-default-smithi/
682
683
* https://tracker.ceph.com/issues/52624
684
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
685
* https://tracker.ceph.com/issues/52606
686
        qa: test_dirfrag_limit
687
* https://tracker.ceph.com/issues/51905
688
        qa: "error reading sessionmap 'mds1_sessionmap'"
689
* https://tracker.ceph.com/issues/53360
690
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
691
* https://tracker.ceph.com/issues/51183
692
        Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
693 47 Patrick Donnelly
694 54 Venky Shankar
h3. 2022 March 22
695
696
https://pulpito.ceph.com/yuriw-2022-03-17_15:03:06-fs-wip-yuri10-testing-2022-03-16-1432-pacific-distro-default-smithi/
697
698
* https://tracker.ceph.com/issues/52624
699
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
700
* https://tracker.ceph.com/issues/52606
701
        qa: test_dirfrag_limit
702
* https://tracker.ceph.com/issues/51183
703
        Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
704
* https://tracker.ceph.com/issues/51905
705
        qa: "error reading sessionmap 'mds1_sessionmap'"
706
* https://tracker.ceph.com/issues/53360
707
        pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
708
* https://tracker.ceph.com/issues/54411
709
	mds_upgrade_sequence: "overall HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available;
710
	33 daemons have recently crashed" during suites/fsstress.sh
711 47 Patrick Donnelly
712 46 Kotresh Hiremath Ravishankar
h3. 2021 November 22
713
714
http://pulpito.front.sepia.ceph.com/yuriw-2021-11-12_00:33:28-fs-wip-yuri7-testing-2021-11-11-1339-pacific-distro-basic-smithi
715
http://pulpito.front.sepia.ceph.com/yuriw-2021-11-08_20:21:11-fs-wip-yuri5-testing-2021-11-08-1003-pacific-distro-basic-smithi
716
http://pulpito.front.sepia.ceph.com/yuriw-2021-11-08_15:19:37-fs-wip-yuri2-testing-2021-11-06-1322-pacific-distro-basic-smithi
717
718
719
* https://tracker.ceph.com/issues/53300
720
	qa: cluster [WRN] Scrub error on inode
721
* https://tracker.ceph.com/issues/53302
722
	qa: sudo logrotate /etc/logrotate.d/ceph-test.conf failed with status 1
723
* https://tracker.ceph.com/issues/53314
724
	qa: fs/upgrade/mds_upgrade_sequence test timeout
725
* https://tracker.ceph.com/issues/53316
726
	qa: (smithi150) slow request osd_op, currently waiting for sub ops warning
727
* https://tracker.ceph.com/issues/52624
728
	qa: "Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)"
729
* https://tracker.ceph.com/issues/52396
730
	pacific: qa: ERROR: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)
731
* https://tracker.ceph.com/issues/52875
732
	pacific: qa: test_dirfrag_limit
733
* https://tracker.ceph.com/issues/51705
734 1 Patrick Donnelly
	pacific: qa: tasks.cephfs.fuse_mount:mount command failed
735
* https://tracker.ceph.com/issues/39634
736
	qa: test_full_same_file timeout
737 46 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/49748
738
	gibba: Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds
739 48 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/51964
740
	qa: test_cephfs_mirror_restart_sync_on_blocklist failure
741 49 Kotresh Hiremath Ravishankar
* https://tracker.ceph.com/issues/50223
742
	qa: "client.4737 isn't responding to mclientcaps(revoke)"
743 48 Kotresh Hiremath Ravishankar
744 47 Patrick Donnelly
745
h3. 2021 November 20
746
747
https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20211120.024903-pacific
748
749
* https://tracker.ceph.com/issues/53360
750
    pacific: client: "handle_auth_bad_method server allowed_methods [2] but i only support [2]"
751 46 Kotresh Hiremath Ravishankar
752 41 Patrick Donnelly
h3. 2021 September 14 (QE)
753
754
https://pulpito.ceph.com/yuriw-2021-09-10_15:01:12-fs-pacific-distro-basic-smithi/
755 1 Patrick Donnelly
http://pulpito.front.sepia.ceph.com/yuriw-2021-09-13_14:09:59-fs-pacific-distro-basic-smithi/
756 43 Ramana Raja
https://pulpito.ceph.com/yuriw-2021-09-14_15:09:45-fs-pacific-distro-basic-smithi/
757 41 Patrick Donnelly
758
* https://tracker.ceph.com/issues/52606
759
    qa: test_dirfrag_limit
760
* https://tracker.ceph.com/issues/52607
761
    qa: "mon.a (mon.0) 1022 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY)"
762 44 Ramana Raja
* https://tracker.ceph.com/issues/51705
763
    qa: tasks.cephfs.fuse_mount:mount command failed
764 41 Patrick Donnelly
765 40 Ramana Raja
h3. 2021 Sep 7
766
767
https://pulpito.ceph.com/yuriw-2021-09-07_17:38:38-fs-wip-yuri-testing-2021-09-07-0905-pacific-distro-basic-smithi/
768
https://pulpito.ceph.com/yuriw-2021-09-07_23:57:28-fs-wip-yuri-testing-2021-09-07-0905-pacific-distro-basic-smithi/
769
770
771
* https://tracker.ceph.com/issues/52396
772
    qa: ERROR: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)
773
* https://tracker.ceph.com/issues/51705
774
    qa: tasks.cephfs.fuse_mount:mount command failed
775 4 Patrick Donnelly
776 34 Ramana Raja
h3. 2021 Aug 30
777
778
https://pulpito.ceph.com/?branch=wip-yuri8-testing-2021-08-30-0930-pacific
779
780
* https://tracker.ceph.com/issues/45434
781
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
782
* https://tracker.ceph.com/issues/51705
783
    qa: tasks.cephfs.fuse_mount:mount command failed
784
* https://tracker.ceph.com/issues/52396
785
    qa: ERROR: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)
786 35 Ramana Raja
* https://tracker.ceph.com/issues/52487
787 36 Ramana Raja
    qa: Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation)
788
* https://tracker.ceph.com/issues/51267
789 37 Ramana Raja
   Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh)
790
* https://tracker.ceph.com/issues/48772
791
   qa: pjd: not ok 9, 44, 80
792 35 Ramana Raja
793 34 Ramana Raja
794 31 Ramana Raja
h3. 2021 Aug 23
795
796
https://pulpito.ceph.com/yuriw-2021-08-23_19:33:26-fs-wip-yuri4-testing-2021-08-23-0812-pacific-distro-basic-smithi/
797
798
* https://tracker.ceph.com/issues/45434
799
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
800
* https://tracker.ceph.com/issues/51705
801
    qa: tasks.cephfs.fuse_mount:mount command failed
802 32 Ramana Raja
* https://tracker.ceph.com/issues/52396
803 33 Ramana Raja
    qa: ERROR: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)
804
* https://tracker.ceph.com/issues/52397
805
    qa: test_acls (tasks.cephfs.test_acls.TestACLs) failed
806
807 31 Ramana Raja
808 29 Ramana Raja
h3. 2021 Aug 11
809
810
https://pulpito.ceph.com/yuriw-2021-08-11_14:28:23-fs-wip-yuri8-testing-2021-08-09-0844-pacific-distro-basic-smithi/
811
https://pulpito.ceph.com/yuriw-2021-08-12_15:32:35-fs-wip-yuri8-testing-2021-08-09-0844-pacific-distro-basic-smithi/
812
813
* https://tracker.ceph.com/issues/45434
814
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
815
* https://tracker.ceph.com/issues/51705
816
    qa: tasks.cephfs.fuse_mount:mount command failed
817 30 Ramana Raja
* https://tracker.ceph.com/issues/50222
818
    osd: 5.2s0 deep-scrub : stat mismatch
819 29 Ramana Raja
820 19 Jos Collin
h3. 2021 July 15
821 20 Jos Collin
822 19 Jos Collin
https://pulpito.ceph.com/yuriw-2021-07-13_17:37:59-fs-wip-yuri-testing-2021-07-13-0812-pacific-distro-basic-smithi/
823
824 26 Jos Collin
* https://tracker.ceph.com/issues/45434
825
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
826
* https://tracker.ceph.com/issues/51705
827
    qa: tasks.cephfs.fuse_mount:mount command failed
828 27 Jos Collin
* https://tracker.ceph.com/issues/51183
829
    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
830 28 Jos Collin
* https://tracker.ceph.com/issues/50528
831
    qa: fs:thrash: pjd suite not ok 80
832
* https://tracker.ceph.com/issues/51706
833
    qa: osd deep-scrub stat mismatch
834 26 Jos Collin
835 19 Jos Collin
h3. 2021 July 13
836 20 Jos Collin
837 19 Jos Collin
https://pulpito.ceph.com/yuriw-2021-07-08_23:33:26-fs-wip-yuri2-testing-2021-07-08-1142-pacific-distro-basic-smithi/
838
839 21 Jos Collin
* https://tracker.ceph.com/issues/51704
840
    Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)
841 23 Jos Collin
* https://tracker.ceph.com/issues/45434
842
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
843 24 Jos Collin
* https://tracker.ceph.com/issues/51705
844 25 Jos Collin
    qa: tasks.cephfs.fuse_mount:mount command failed
845
* https://tracker.ceph.com/issues/48640
846 24 Jos Collin
    qa: snapshot mismatch during mds thrashing
847 21 Jos Collin
848 18 Patrick Donnelly
h3. 2021 June 29 (Integration Branch)
849
850
https://pulpito.ceph.com/yuriw-2021-06-29_00:15:33-fs-wip-yuri3-testing-2021-06-28-1259-pacific-distro-basic-smithi/
851
852
Some failures caused by incomplete backup of: https://github.com/ceph/ceph/pull/42065
853
Some package failures caused by nautilus missing package, like: https://pulpito.ceph.com/yuriw-2021-06-29_00:15:33-fs-wip-yuri3-testing-2021-06-28-1259-pacific-distro-basic-smithi/6241300/
854
855
* https://tracker.ceph.com/issues/45434
856
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
857
* https://tracker.ceph.com/issues/50260
858
    pacific: qa: "rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty"
859
* https://tracker.ceph.com/issues/51183
860
    qa: FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/ceph/3fab6bea-f243-47a4-a956-8c03a62b61b5.client4721/mds_sessions'
861
862
863 16 Jeff Layton
h3. 2021 June 28
864
865
https://pulpito.ceph.com/yuriw-2021-06-29_00:14:14-fs-wip-yuri-testing-2021-06-28-1259-pacific-distro-basic-smithi/
866
867
* https://tracker.ceph.com/issues/45434
868
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
869
* https://tracker.ceph.com/issues/51440
870 17 Jeff Layton
    fallocate fails with EACCES
871 16 Jeff Layton
* https://tracker.ceph.com/issues/51264
872
    TestVolumeClient failure
873
* https://tracker.ceph.com/issues/51266
874
    test cleanup failure
875
* https://tracker.ceph.com/issues/51183
876
    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) 
877
878 12 Jeff Layton
h3. 2021 June 14
879
880
https://pulpito.ceph.com/yuriw-2021-06-14_16:21:07-fs-wip-yuri2-testing-2021-06-14-0717-pacific-distro-basic-smithi/
881
 
882
* https://tracker.ceph.com/issues/45434
883
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
884
* https://bugzilla.redhat.com/show_bug.cgi?id=1973276
885
    Could not reconnect to ubuntu@smithi076.front.sepia.ceph.com
886
* https://tracker.ceph.com/issues/51263
887
    pjdfstest rename test 10.t failed with EACCES
888
* https://tracker.ceph.com/issues/51264
889
    TestVolumeClient failure
890 13 Jeff Layton
* https://tracker.ceph.com/issues/51266
891
    Command failed on smithi204 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 
892 14 Jeff Layton
* https://tracker.ceph.com/issues/50279
893
    qa: "Replacing daemon mds.b as rank 0 with standby daemon mds.c"
894 15 Jeff Layton
* https://tracker.ceph.com/issues/51267
895
    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi096 with status 1
896 11 Patrick Donnelly
897
h3. 2021 June 07 (Integration Branch)
898
899
http://pulpito.front.sepia.ceph.com/?branch=wip-yuri4-testing-2021-06-07-0955-pacific
900
901
* https://tracker.ceph.com/issues/45434
902
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
903
* https://tracker.ceph.com/issues/50279
904
    qa: "Replacing daemon mds.b as rank 0 with standby daemon mds.c"
905
* https://tracker.ceph.com/issues/48773
906
    qa: scrub does not complete
907
* https://tracker.ceph.com/issues/51170
908
    pacific: qa: AttributeError: 'RemoteProcess' object has no attribute 'split'
909
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
910
    qa: quota failure
911
912
913 6 Patrick Donnelly
h3. 2021 Apr 28 (QE pre-release)
914
915
https://pulpito.ceph.com/yuriw-2021-04-28_19:27:30-fs-pacific-distro-basic-smithi/
916
https://pulpito.ceph.com/yuriw-2021-04-30_13:46:58-fs-pacific-distro-basic-smithi/
917
918
* https://tracker.ceph.com/issues/45434
919
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
920
* https://tracker.ceph.com/issues/50258
921
    pacific: qa: "run() got an unexpected keyword argument 'stdin_data'"
922
* https://tracker.ceph.com/issues/50260
923
    pacific: qa: "rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty"
924
* https://tracker.ceph.com/issues/49962
925
    'sudo ceph --cluster ceph osd crush tunables default' fails due to valgrind: Unknown option: --exit-on-first-error=yes
926
* https://tracker.ceph.com/issues/50016
927
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
928
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
929
    qa: quota failure
930
* https://tracker.ceph.com/issues/50528
931
    pacific: qa: fs:thrash: pjd suite not ok 20
932
933
934 3 Patrick Donnelly
h3. 2021 Apr 22 (Integration Branch)
935
936
https://pulpito.ceph.com/yuriw-2021-04-22_16:46:11-fs-wip-yuri5-testing-2021-04-20-0819-pacific-distro-basic-smithi/
937
938
* https://tracker.ceph.com/issues/50527
939
    pacific: qa: untar_snap_rm timeout during osd thrashing (ceph-fuse)
940
* https://tracker.ceph.com/issues/50528
941
    pacific: qa: fs:thrash: pjd suite not ok 20
942
* https://tracker.ceph.com/issues/49500 (fixed in another integration run)
943
    qa: "Assertion `cb_done' failed."
944
* https://tracker.ceph.com/issues/49500
945
    qa: "Assertion `cb_done' failed."
946
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
947
    qa: quota failure
948
* https://tracker.ceph.com/issues/45434
949
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
950
* https://tracker.ceph.com/issues/50279
951
    qa: "Replacing daemon mds.b as rank 0 with standby daemon mds.c"
952
* https://tracker.ceph.com/issues/50258
953
    pacific: qa: "run() got an unexpected keyword argument 'stdin_data'"
954
* https://tracker.ceph.com/issues/49962
955
    'sudo ceph --cluster ceph osd crush tunables default' fails due to valgrind: Unknown option: --exit-on-first-error=yes
956
* https://tracker.ceph.com/issues/49962
957
    'sudo ceph --cluster ceph osd crush tunables default' fails due to valgrind: Unknown option: --exit-on-first-error=yes
958
* https://tracker.ceph.com/issues/50530
959
    pacific: client: abort after MDS blocklist
960
961 2 Patrick Donnelly
962
h3. 2021 Apr 21 (Integration Branch)
963
964
https://pulpito.ceph.com/yuriw-2021-04-21_19:25:00-fs-wip-yuri3-testing-2021-04-21-0937-pacific-distro-basic-smithi/
965
966
* https://tracker.ceph.com/issues/45434
967
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
968
* https://tracker.ceph.com/issues/50250
969
    mds: "log [WRN] : Scrub error on inode 0x10000004506 (/client.0/tmp/clients/client3/~dmtmp/COREL) see mds.a log and `damage ls` output for details"
970
* https://tracker.ceph.com/issues/50258
971
    pacific: qa: "run() got an unexpected keyword argument 'stdin_data'"
972
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
973
    qa: quota failure
974
* https://tracker.ceph.com/issues/50016
975
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
976
* https://tracker.ceph.com/issues/50495
977
    pacific: client: shutdown race fails with status 141
978
979
980 1 Patrick Donnelly
h3. 2021 Apr 07 (Integration Branch)
981
982
https://pulpito.ceph.com/yuriw-2021-04-07_17:37:43-fs-wip-yuri-testing-2021-04-07-0905-pacific-distro-basic-smithi/
983
984
* https://tracker.ceph.com/issues/45434
985
    qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
986
* https://tracker.ceph.com/issues/48805
987
    mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and `damage ls` output for details"
988
* https://tracker.ceph.com/issues/49500
989
    qa: "Assertion `cb_done' failed."
990
* https://tracker.ceph.com/issues/50258 (new)
991
    pacific: qa: "run() got an unexpected keyword argument 'stdin_data'"
992
* https://tracker.ceph.com/issues/49962
993
    'sudo ceph --cluster ceph osd crush tunables default' fails due to valgrind: Unknown option: --exit-on-first-error=yes
994
* https://tracker.ceph.com/issues/48203 (stock kernel update required)
995
    qa: quota failure
996
* https://tracker.ceph.com/issues/50260
997
    pacific: qa: "rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty"
998
* https://tracker.ceph.com/issues/50016
999
    qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"