Ceph

/a/sage-2016-12-03_19:34:03-rados-wip-sage-testing---basic-smithi/599254

       EAGAIN A system-imposed limit on the number of threads was
              encountered.  There are a number of limits that may trigger
              this error: the RLIMIT_NPROC soft resource limit (set via
              setrlimit(2)), which limits the number of processes and
              threads for a real user ID, was reached; the kernel's system-
              wide limit on the number of processes and threads,
              /proc/sys/kernel/threads-max, was reached (see proc(5)); or
              the maximum number of PIDs, /proc/sys/kernel/pid_max, was
              reached (see proc(5)).

2016-12-04T05:06:43.303 INFO:tasks.ceph.osd.2.smithi112.stderr:2016-12-04 05:06:43.304099 7fd777f6a700 -1 received  signal: Hangup from  PID: 32602 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  UID: 0
2016-12-04T05:06:43.323 INFO:teuthology.orchestra.run.smithi109:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops'
2016-12-04T05:06:43.332 INFO:tasks.ceph.osd.0.smithi112.stderr:Thread::try_create(): pthread_create failed with error 11Thread::try_create(): pthread_create failed with error 11Thread::try_create(): pthread_create failed with error 11Thread::try_create(): pthread_create failed with error 11Thread::try_create(): pth
read_create failed with error 11Thread::try_create(): pthread_create failed with error 11/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.2-2202-g0ba4bc1/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7fe6f2142700 time 2016-12-04 05:06:43.329976
2016-12-04T05:06:43.332 INFO:tasks.ceph.osd.0.smithi112.stderr:/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.2-2202-g0ba4bc1/src/common/Thread.cc: 162: FAILED assert(ret == 0)
2016-12-04T05:06:43.332 INFO:tasks.ceph.osd.0.smithi112.stderr: ceph version 11.0.2-2202-g0ba4bc1 (0ba4bc1734bd3746c1811d0d5980c47ee623ebdd)
2016-12-04T05:06:43.333 INFO:tasks.ceph.osd.0.smithi112.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55a08a42ef30]
2016-12-04T05:06:43.333 INFO:tasks.ceph.osd.0.smithi112.stderr: 2: (Thread::create(char const*, unsigned long)+0xba) [0x55a08a5056aa]
2016-12-04T05:06:43.333 INFO:tasks.ceph.osd.0.smithi112.stderr: 3: (Pipe::connect()+0x3317) [0x55a08a6044a7]
2016-12-04T05:06:43.333 INFO:tasks.ceph.osd.0.smithi112.stderr: 4: (Pipe::writer()+0xe84) [0x55a08a6067a4]

Not clear why this would end up with too many threads..

Distro is xenial; maybe the ulimits are different?