Project

General

Profile

Bug #36405

unittest_seastar_messenger failure on ARM

Added by John Spray over 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Correctness/Safety
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We often ignore these failures, but when I looked at the log I realised it's actually a recently added test that's failing, so probably a genuine bug?

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/24113/console

The following tests FAILED:
    157 - unittest_seastar_messenger (SEGFAULT)

History

#1 Updated by Neha Ojha over 5 years ago

Kefu, could you please take a look.

#2 Updated by Kefu Chai over 5 years ago

  • Assignee set to Kefu Chai

#3 Updated by Kefu Chai over 5 years ago

from dmesg:

[19321510.804024] unittest_seasta[46470]: unhandled level 3 permission fault (11) at 0x3284d0e0, esr 0x8300000f
[19321510.804032] pgd = ffff80048b1df000
[19321510.811872] [3284d0e0] *pgd=00000002e4566003, *pud=00000006b3335003, *pmd=000000038da8b003, *pte=00e8000de49c5f53

[19321510.826681] CPU: 0 PID: 46470 Comm: unittest_seasta Not tainted 4.4.0-121-generic #145-Ubuntu
[19321510.826684] Hardware name: GIGABYTE R120-T32-00/MT30-GS1-00, BIOS T32 03/03/2017
[19321510.826688] task: ffff800f8315ce00 ti: ffff8001d8a74000 task.ti: ffff8001d8a74000
[19321510.826693] PC is at 0x3284d0e0
[19321510.826696] LR is at 0x988588
[19321510.826699] pc : [<000000003284d0e0>] lr : [<0000000000988588>] pstate: 20000000
[19321510.826702] sp : 0000ffffdb7a2f40
[19321510.826704] x29: 0000ffffdb7a2f40 x28: 0000000000000000
[19321510.826710] x27: 0000000000000000 x26: 0000000000000000
[19321510.826714] x25: 0000000000000000 x24: 0000000000000000
[19321510.826718] x23: 0000000000000000 x22: 0000000000000000
[19321510.826723] x21: 0000ffffdb7a3068 x20: 000000003284d0e0
[19321510.826727] x19: 000000003284d0e0 x18: 0000000000000000
[19321510.826732] x17: 0000ffff7c1aab38 x16: 0000000000fe8148
[19321510.826736] x15: 003b9aca00000000 x14: 0000000000000000
[19321510.826741] x13: 0000000000000000 x12: 0000ffff7c041258
[19321510.826745] x11: 00000000004001c8 x10: 0000ffff7c041218
[19321510.826750] x9 : 0000000000000000 x8 : 0000ffffdb7a3068
[19321510.826754] x7 : 7f7f7f7fff7fff7f x6 : 0000000000000004
[19321510.826759] x5 : 00000000009c9e5c x4 : 0000000000fe8d28
[19321510.826763] x3 : 37fae0bca59bd300 x2 : 37fae0bca59bd300
[19321510.826767] x1 : 0000ffffdb7a2f80 x0 : 000000003284d0e0

#4 Updated by Sebastian Wagner almost 5 years ago

Another one:

162/172 Test #169: unittest_seastar_messenger ..............***Exception: SegFault 18.11 sec
WARNING: debug mode. Not for benchmarking or production
WARN  2019-05-21 09:42:22,426 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,441 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,441 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,441 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,442 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,443 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,443 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,445 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,445 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,445 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,445 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,445 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,454 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,453 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,469 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:22,469 seastar - Seastar compiled with default allocator, heap profiler not supported
WARN  2019-05-21 09:42:23,405 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
WARN  2019-05-21 09:42:23,692 [shard 0] seastar - membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED) is not available, reactor will not sleep when idle. Upgrade to Linux 4.14 or later
INFO  2019-05-21 09:42:23,715 [shard 0] ms - test_echo(rounds=512, keepalive_ratio=0.1, v2=0):
INFO  2019-05-21 09:42:23,819 [shard 0] ms - listening on 127.0.0.1:9010
INFO  2019-05-21 09:42:23,820 [shard 0] ms - listening on 127.0.0.1:9011
INFO  2019-05-21 09:42:23,971 [shard 2] ms - [osd.3(client2) v1:127.0.0.1:0/4@59330 >> v1:127.0.0.1:9010/1]: connected to v1:127.0.0.1:9010/1
INFO  2019-05-21 09:42:23,974 [shard 2] ms - [osd.2(client1) v1:127.0.0.1:0/3@62450 >> v1:127.0.0.1:9011/2]: connected to v1:127.0.0.1:9011/2
INFO  2019-05-21 09:42:23,984 [shard 2] ms - [osd.2(client1) v1:127.0.0.1:0/3@59618 >> v1:127.0.0.1:9010/1]: connected to v1:127.0.0.1:9010/1
INFO  2019-05-21 09:42:24,010 [shard 2] ms - [osd.3(client2) v1:127.0.0.1:0/4@55202 >> v1:127.0.0.1:9011/2]: connected to v1:127.0.0.1:9011/2
INFO  2019-05-21 09:42:25,950 [shard 0] ms - [osd.3(client2) v1:127.0.0.1:0/4@55202 >> v1:127.0.0.1:9011/2]: finished sending 512 pings with 59 keepalives
INFO  2019-05-21 09:42:26,003 [shard 0] ms - [osd.2(client1) v1:127.0.0.1:0/3@59618 >> v1:127.0.0.1:9010/1]: finished sending 512 pings with 62 keepalives
INFO  2019-05-21 09:42:29,077 [shard 2] ms - [osd.3(client2) v1:127.0.0.1:0/4@59330 >> v1:127.0.0.1:9010/1]: finished sending 512 pings with 51 keepalives
INFO  2019-05-21 09:42:29,220 [shard 2] ms - [osd.2(client1) v1:127.0.0.1:0/3@62450 >> v1:127.0.0.1:9011/2]: finished sending 512 pings with 52 keepalives
INFO  2019-05-21 09:42:36,687 [shard 2] ms - [osd.3(client2) v1:127.0.0.1:0/4@59330 >> v1:127.0.0.1:9010/1]: finished receiving 512 pongs
INFO  2019-05-21 09:42:36,690 [shard 2] ms - [osd.3(client2) v1:127.0.0.1:0/4@59330 >> v1:127.0.0.1:9010/1]: handshake 0.104079, pingpong 12.7155
INFO  2019-05-21 09:42:36,693 [shard 2] ms - [osd.2(client1) v1:127.0.0.1:0/3@59618 >> v1:127.0.0.1:9010/1]: finished receiving 512 pongs
INFO  2019-05-21 09:42:36,694 [shard 2] ms - [osd.2(client1) v1:127.0.0.1:0/3@59618 >> v1:127.0.0.1:9010/1]: handshake 0.116535, pingpong 12.7091
INFO  2019-05-21 09:42:36,716 [shard 2] ms - [osd.2(client1) v1:127.0.0.1:0/3@62450 >> v1:127.0.0.1:9011/2]: finished receiving 512 pongs
INFO  2019-05-21 09:42:36,720 [shard 2] ms - [osd.2(client1) v1:127.0.0.1:0/3@62450 >> v1:127.0.0.1:9011/2]: handshake 0.107043, pingpong 12.7415
INFO  2019-05-21 09:42:36,727 [shard 2] ms - [osd.3(client2) v1:127.0.0.1:0/4@55202 >> v1:127.0.0.1:9011/2]: finished receiving 512 pongs
INFO  2019-05-21 09:42:36,728 [shard 2] ms - [osd.3(client2) v1:127.0.0.1:0/4@55202 >> v1:127.0.0.1:9011/2]: handshake 0.142769, pingpong 12.7171
ISegmentation fault on shard 2.
Backtrace:
  0x0000000001e4aa6b
  0x0000000001e3ff6f
  0x0000000001db90bf
  0x0000000001db9117
  0x0000000001dd6fef
  0x0000000001dea667
  0x0000000001dea6cf
  0x000000000000050f
  0x0000ffffb2395bb7

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/267/console

#5 Updated by Kefu Chai almost 5 years ago

  • Status changed from New to Resolved
  • Assignee deleted (Kefu Chai)
  • Pull request ID set to 28362

Also available in: Atom PDF