Bug #47830: crimson unit tests never end - crimson - Ceph

Actions

Copy link

Bug #47830

closed

crimson unit tests never end

Added by Kefu Chai over 3 years ago. Updated about 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Kefu Chai

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

$ ps aux|grep unit
jenkins+  7533  0.0  0.0  14728  1092 pts/0    S+   04:40   0:00 grep unit
jenkins+ 25479  4.2  0.6 21477427816 1624836 ? Sl   Oct07 258:46 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_transaction_manager
jenkins+ 25481  4.7  0.5 21477507372 1504816 ? Sl   Oct07 291:15 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_btree_lba_manager
jenkins+ 25486  4.3  0.5 21477512300 1510480 ? Sl   Oct07 268:50 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_extmap_manager
jenkins+ 27148  1.3  0.2 21475418104 541396 ?  Sl   Oct07  87:45 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_seastar_messenger --memory 256M --smp 1
jenkins+ 27218  4.3  0.5 21477401008 1382504 ? Sl   Oct07 279:10 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_transaction_manager
jenkins+ 27220  4.2  0.5 21477512360 1376040 ? Sl   Oct07 271:01 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_btree_lba_manager
jenkins+ 27279  4.2  0.5 21477400788 1376724 ? Sl   Oct07 271:37 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_extmap_manager

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Kefu Chai about 3 years ago

Has duplicate Bug #49426: /src/seastar/include/seastar/core/sharded.hh:seastar::sharded<T>::~sharded() [with Service = crimson::common::ConfigProxy]: Assertion `_instances.empty()' failed. added

Actions

Copy link

Updated by Kefu Chai about 3 years ago

i think this issue could be contributed by the jenkins' job termination.

see https://gist.github.com/datagrok/dfe9604cb907523f4a2f

sometimes, the "make check" job is canceled by jenkins when we push a new change to a PR or when we manually cancel a jenkins job.

according to the article above and http://jenkins-ci.361315.n4.nabble.com/JIRA-JENKINS-17116-gracefull-job-termination-td4658448.html , this could lead to never-ending tests left in the system.

according to https://www.jenkins.io/doc/book/using/aborting-a-build/, the executor cleans up the workspace by sending SIGTERM to all the descendant processes.

but when looking at the zombie process of unittest-seastar-messenger, it could not be killed by SIGTERM

root@irvingi03:/home/jenkins-build# ps aux|grep unit
root     17071  0.0  0.0  14728  1012 pts/0    S+   06:43   0:00 grep --color=auto unit
jenkins+ 29795  1.8  0.8 21475448192 582620 ?  Sl   03:40   3:24 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger --memory 256M --smp 1
root@irvingi03:/home/jenkins-build# ^C aux|grep unit
root@irvingi03:/home/jenkins-build# ^Cll 29795
root@irvingi03:/home/jenkins-build# man ^C
root@irvingi03:/home/jenkins-build# kill -SIGTERM 29795
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root     26511  0.0  0.0  14728  1116 pts/0    S+   06:44   0:00 grep --color=auto unit
jenkins+ 29795  1.8  0.8 21475448192 582620 ?  Sl   03:40   3:25 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger --memory 256M --smp 1
root@irvingi03:/home/jenkins-build# kill -SIGTERM 29795
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root     27790  0.0  0.0  14728  1008 pts/0    S+   06:44   0:00 grep --color=auto unit
jenkins+ 29795  1.8  0.8 21475448192 582620 ?  Sl   03:40   3:25 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger --memory 256M --smp 1
root@irvingi03:/home/jenkins-build# kill -SIGTERM 29795
root@irvingi03:/home/jenkins-build# ps aux|grep unit^C
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root      4079  0.0  0.0  14728  1092 pts/0    S+   06:44   0:00 grep --color=auto unit
jenkins+ 29795  1.8  0.8 21475448192 582620 ?  Sl   03:40   3:26 /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest-seastar-messenger --memory 256M --smp 1
root@irvingi03:/home/jenkins-build# kill -9 29795
root@irvingi03:/home/jenkins-build# ps aux|grep unit
root      6114  0.0  0.0  14728  1008 pts/0    S+   06:45   0:00 grep --color=auto unit
root@irvingi03:/home/jenkins-build# exit

Actions

Copy link