This issue occurred due to greping through teuthology.log for `</kind> in `
Once it finds the the line that has this grep, it will try to searche for a match for a regular expression at the beginning of a string.
In this case it searches for "<kind>(.+)</kind> in .+/(.+)"
These are all the lines in the run that contains `</kind> in `
There are 4 distinct jobs that grep was able to catch
ksirivad@teuthology:/a/yuriw-2023-08-17_21:18:20-rados-wip-yuri11-testing-2023-08-17-0823-distro-default-smithi$ grep -nr "</kind> in "
7372158/teuthology.log:3164:2023-08-17T23:09:30.783 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
7372158/teuthology.log:3166:2023-08-17T23:09:30.784 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.b.log
7372158/teuthology.log:3168:2023-08-17T23:09:30.784 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
7372092/teuthology.log:33841:2023-08-17T23:09:36.407 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
7372092/teuthology.log:33843:2023-08-17T23:09:36.407 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
7372092/teuthology.log:33846:2023-08-17T23:09:36.432 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.b.log
7372159/teuthology.log:7464:2023-08-17T23:30:35.303 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
7372159/teuthology.log:7466:2023-08-17T23:30:35.303 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
7372159/teuthology.log:7469:2023-08-17T23:30:35.334 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.b.log
7372228/teuthology.log:19527:2023-08-18T01:07:25.312 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
7372228/teuthology.log:19529:2023-08-18T01:07:25.312 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
7372228/teuthology.log:19532:2023-08-18T01:07:25.338 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.b.log
Scrape correctly identifies 4 Jobs with valgrind error:
Found 16 distinct failure reasons
Valgrind: mon (Leak_StillReachable)
4 jobs: ['7372158', '7372092', '7372159', '7372228']
suites intersection: []
suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}']
Real question is how did empty string gets picked up by the initial grep of `</kind> in ` and obviously gets rejected during regex matching against "<kind>(.+)</kind> in .+/(.+)"
Since:
if not match:
log.warning("Misunderstood line: {0}".format(line))
continue
We are printing blank, meaning we caught empty string during the grep process
Misunderstood line: