Bug #47830
closed
crimson unit tests never end
Added by Kefu Chai over 3 years ago.
Updated about 3 years ago.
Description
$ ps aux|grep unit
jenkins+ 7533 0.0 0.0 14728 1092 pts/0 S+ 04:40 0:00 grep unit
jenkins+ 25479 4.2 0.6 21477427816 1624836 ? Sl Oct07 258:46 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_transaction_manager
jenkins+ 25481 4.7 0.5 21477507372 1504816 ? Sl Oct07 291:15 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_btree_lba_manager
jenkins+ 25486 4.3 0.5 21477512300 1510480 ? Sl Oct07 268:50 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_extmap_manager
jenkins+ 27148 1.3 0.2 21475418104 541396 ? Sl Oct07 87:45 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_seastar_messenger --memory 256M --smp 1
jenkins+ 27218 4.3 0.5 21477401008 1382504 ? Sl Oct07 279:10 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_transaction_manager
jenkins+ 27220 4.2 0.5 21477512360 1376040 ? Sl Oct07 271:01 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_btree_lba_manager
jenkins+ 27279 4.2 0.5 21477400788 1376724 ? Sl Oct07 271:37 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_extmap_manager
- Has duplicate Bug #49426: /src/seastar/include/seastar/core/sharded.hh:seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed. added
i think this issue could be contributed by the jenkins' job termination.
see https://gist.github.com/datagrok/dfe9604cb907523f4a2f
sometimes, the "make check" job is canceled by jenkins when we push a new change to a PR or when we manually cancel a jenkins job.
according to the article above and http://jenkins-ci.361315.n4.nabble.com/JIRA-JENKINS-17116-gracefull-job-termination-td4658448.html , this could lead to never-ending tests left in the system.
according to https://www.jenkins.io/doc/book/using/aborting-a-build/, the executor cleans up the workspace by sending SIGTERM to all the descendant processes.
but when looking at the zombie process of unittest-seastar-messenger, it could not be killed by SIGTERM
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root 17071 0.0 0.0 14728 1012 pts/0 S+ 06:43 0:00 grep --color=auto unit
jenkins+ 29795 1.8 0.8 21475448192 582620 ? Sl 03:40 3:24 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger --memory 256M --smp 1
root@irvingi03:/home/jenkins-build# ^C aux|grep unit
root@irvingi03:/home/jenkins-build# ^Cll 29795
root@irvingi03:/home/jenkins-build# man ^C
root@irvingi03:/home/jenkins-build# kill -SIGTERM 29795
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root 26511 0.0 0.0 14728 1116 pts/0 S+ 06:44 0:00 grep --color=auto unit
jenkins+ 29795 1.8 0.8 21475448192 582620 ? Sl 03:40 3:25 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger --memory 256M --smp 1
root@irvingi03:/home/jenkins-build# kill -SIGTERM 29795
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root 27790 0.0 0.0 14728 1008 pts/0 S+ 06:44 0:00 grep --color=auto unit
jenkins+ 29795 1.8 0.8 21475448192 582620 ? Sl 03:40 3:25 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger --memory 256M --smp 1
root@irvingi03:/home/jenkins-build# kill -SIGTERM 29795
root@irvingi03:/home/jenkins-build# ps aux|grep unit^C
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root 4079 0.0 0.0 14728 1092 pts/0 S+ 06:44 0:00 grep --color=auto unit
jenkins+ 29795 1.8 0.8 21475448192 582620 ? Sl 03:40 3:26 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger --memory 256M --smp 1
root@irvingi03:/home/jenkins-build# kill -9 29795
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root 6114 0.0 0.0 14728 1008 pts/0 S+ 06:45 0:00 grep --color=auto unit
root@irvingi03:/home/jenkins-build# exit
- Status changed from New to Fix Under Review
- Status changed from Fix Under Review to Resolved
- Assignee set to Kefu Chai
Also available in: Atom
PDF