Project

General

Profile

Bug #19874

std::length_error in OSDMonitor::prime_pg_temp()

Added by Sage Weil 18 days ago. Updated 18 days ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Category:
-
Target version:
-
Start date:
05/05/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

2017-05-04T21:47:58.351 INFO:tasks.ceph.mon.a.smithi069.stderr:tcmalloc: large alloc 1879318528 bytes == 0x7f9b2671c000 @  0x7f9aa3443c4c 0x7f9aa3464219 0x7f9aa4e987c3 0x7f9aa4ef7e53 0x7f9aa50f9e74 0x7f9aa501b815 0x7f9aa501c7e0 0x7f9aa3e6a184 0x7f9aa272d37d (nil)
2017-05-04T21:47:58.454 INFO:tasks.ceph.osd.3.smithi058.stderr:2017-05-04 21:47:58.445906 7fac94ff5700 -1 received  signal: Hangup from  PID: 26218 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3  UID: 0
2017-05-04T21:47:58.548 INFO:tasks.ceph.osd.0.smithi069.stderr:2017-05-04 21:47:58.548500 7f42d4e4d700 -1 received  signal: Hangup from  PID: 66456 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0  UID: 0
2017-05-04T21:47:58.648 INFO:tasks.ceph.osd.4.smithi058.stderr:2017-05-04 21:47:58.646860 7faf90f35700 -1 received  signal: Hangup from  PID: 30429 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4  UID: 0
2017-05-04T21:47:58.749 INFO:tasks.ceph.osd.2.smithi069.stderr:2017-05-04 21:47:58.749801 7fb53e620700 -1 received  signal: Hangup from  PID: 66461 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  UID: 0
2017-05-04T21:47:58.764 INFO:tasks.ceph.mon.a.smithi069.stderr:terminate called after throwing an instance of 'std::length_error'
2017-05-04T21:47:58.764 INFO:tasks.ceph.mon.a.smithi069.stderr:  what():  vector::_M_default_append
2017-05-04T21:47:58.764 INFO:tasks.ceph.mon.a.smithi069.stderr:*** Caught signal (Aborted) **
2017-05-04T21:47:58.764 INFO:tasks.ceph.mon.a.smithi069.stderr: in thread 7f9a9ca11700 thread_name:cpu_tp
2017-05-04T21:47:58.765 INFO:tasks.ceph.mon.a.smithi069.stderr: ceph version 12.0.1-2012-g796d7bf (796d7bf7996170cb9d8f0ec48af6459b6300b693)
2017-05-04T21:47:58.765 INFO:tasks.ceph.mon.a.smithi069.stderr: 1: (()+0x7949a7) [0x7f9aa526e9a7]
2017-05-04T21:47:58.765 INFO:tasks.ceph.mon.a.smithi069.stderr: 2: (()+0x10330) [0x7f9aa3e72330]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 3: (gsignal()+0x37) [0x7f9aa2669c37]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 4: (abort()+0x148) [0x7f9aa266d028]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9aa2f74535]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 6: (()+0x5e6d6) [0x7f9aa2f726d6]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 7: (()+0x5e703) [0x7f9aa2f72703]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 8: (()+0x5e922) [0x7f9aa2f72922]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 9: (std::__throw_length_error(char const*)+0x67) [0x7f9aa2fc43a7]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 10: (std::vector<int, std::allocator<int> >::_M_default_append(unsigned long)+0x138) [0x7f9aa4eee978]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 11: (OSDMonitor::prime_pg_temp(OSDMap const&, pg_t)+0x3f0) [0x7f9aa4e984b0]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 12: (OSDMonitor::PrimeTempJob::process(long, unsigned int, unsigned int)+0x43) [0x7f9aa4ef7e53]
2017-05-04T21:47:58.766 INFO:tasks.ceph.mon.a.smithi069.stderr: 13: (ParallelPGMapper::WQ::_process(ParallelPGMapper::Item*, ThreadPool::TPHandle&)+0x64) [0x7f9aa50f9e74]
2017-05-04T21:47:58.767 INFO:tasks.ceph.mon.a.smithi069.stderr: 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xb65) [0x7f9aa501b815]
2017-05-04T21:47:58.767 INFO:tasks.ceph.mon.a.smithi069.stderr: 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f9aa501c7e0]
2017-05-04T21:47:58.767 INFO:tasks.ceph.mon.a.smithi069.stderr: 16: (()+0x8184) [0x7f9aa3e6a184]
2017-05-04T21:47:58.767 INFO:tasks.ceph.mon.a.smithi069.stderr: 17: (clone()+0x6d) [0x7f9aa272d37d]
2017-05-04T21:47:58.851 INFO:tasks.ceph.osd.4.smithi058.stderr:2017-05-04 21:47:58.849351 7faf90f35700 -1 received  signal: Hangup from  PID: 30429 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4  UID: 0
2017-05-04T21:47:58.951 INFO:tasks.ceph.osd.2.smithi069.stderr:2017-05-04 21:47:58.952187 7fb53e620700 -1 received  signal: Hangup from  PID: 66461 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  UID: 0
2017-05-04T21:47:59.052 INFO:tasks.ceph.osd.5.smithi058.stderr:2017-05-04 21:47:59.050514 7fef385ec700 -1 received  signal: Hangup from  PID: 17638 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5  UID: 0
2017-05-04T21:47:59.152 INFO:tasks.ceph.osd.1.smithi069.stderr:2017-05-04 21:47:59.153067 7f0b453c7700 -1 received  signal: Hangup from  PID: 66459 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1  UID: 0
2017-05-04T21:47:59.400 INFO:tasks.ceph.mon.a.smithi069.stderr:2017-05-04 21:47:58.766191 7f9a9ca11700 -1 *** Caught signal (Aborted) **

/a/sage-2017-05-04_19:31:08-rados-wip-sage-testing---basic-smithi/1102497

Related issues

Duplicated by Bug #19852: mon: huge acting set returned by mapping.get() Resolved 05/04/2017

History

#1 Updated by Sage Weil 18 days ago

    -1> 2017-05-04 21:47:38.559978 7f9a9ca11700 20 mon.a@1(leader).osd e447 prime_pg_temp 0.19 [33554432,-1400497664,4525845,0,-1107296256,2136542112,1762726151,1,16777216,32256,1869768192,1663516013,1852139884,541011572,775042865,825110834,959852085,875507770,909193265,842216498,1696606004,1953068142,1663516025,18
52139884,1684090484,661547373,1684890400,578509629,1717924464,975337577,1936663072,1853169764,578053491,1797398572,975337829,1869488672,1885693284,1919120173,2099405429,1679833693,1634759529,90727284,1627389952,1953064053,18743811,65536,0,16777216,1835265,65536,0,1048576,131072,363627802,14863,0,-1598160896,-194207
1029,181998417,0,65536,209,1634559856,863379568,540686133,1881159221,540701543,1919098932,1769234789,1764452206,1836016494,1952803952,824192101,1853169713,1936876900,1684372073,1734698027,1701077362,1701849956,1684370021,540287020,1769235297,1663788406,1851876716,1634628395,1769108592,1635213165,539784297,162950123
5,1986622563,1818438501,997089637,808464672,1646276920,1936028793,1952539680,924855393,540225846,1965048397,744777075,825570592,541214496,892674095,1111957560,1635148064,540765289,858732853,1864382514,1667590754,1679848308,1634887525,543450468,775369000,624441401,1833,1970037504,1919251571,11862531,65536,0,16777216
,1835265,65536,0,1048576,131072,363627802,14863,0,-1598160896,537680139,182064635,0,65536,106,1836020326,1818437437,1953391977,959722542,757086521,1852121127,2037672308,1818437437,1953391977,1835295022,539455081,1029991779,1881307995,1768318322,540680824,1685286690,1936618784,740455525,1701519904,540680825,16850242
90,762340709,1970430835,1568481890,1768169530,1952542835,354403,1969291264,7629156,0,0,-291176453,268296184,-1082130432,-1082130432,-1082130432,-1643663131,2107379180,0,0,0,0,0,-1337185280,32666,684,0,-1,40,44435208,17039360,427,-433416246,1599074911,1582270352,-126312163,202,1493934150,576379356,137,0,497664,0,0,-
1,1,137,0,14222617,33619968,524800,524288,0,0,13238272,0,0,0,0,0,0,0,65536,0,0,131072,0,0,0,0,-16777216,-1,16777215,-1,-1,-1,-1,0,65793,0,0,0,0,0,0,0,-2147483648,1562,3125,0,0,0,0,0,0,-1073741824,2343,16777216,0,0,67174656,0,0,16777216,-1996488704,0,553648128,1946157056,762606437,1868849522,1885416819,1836264809,17
68453225,758724144,825833270,875900211,0,1,3,469827841,16777216,1862270976,268435558,33554432,-1398990336,3804949,0,0,0,0,0,0,0,0,0,0,0,0,67108864,59137,256,768,16843008,28,1,26223,16,-1625686014,974067116,0,0,0,1,3,5,178,0,0,0,1,3,469827841,16777216,1862270976,268435558,33554432,-1398924800,3804949,0,0,0,16777216,
50331648,50331648,7169,195047168,2144803929,24,0,-1526727936,-35653394,16777231,256,768,16843008,28,1,26223,16,-1608908798,974067116,0,0,-291176453,268296184,-1082130432,-1082130432,-1082130432,291409122,-49350716,0,0,0,0,-291176453,268296184,-1082130432,-1082130432,-1082130432,-1918332912,-1479717745,696414464,-48
1912835,0,0,-1337186816,32666,736,0,-1,40,1812206592,7169903,1819047270,20001631,2101,0,307338,1869350420,1711304039,1600941173,-2080294861,11,-1332805632,76594180,1812337664,7169903,1819047270,20198495,3564,-805306368,-1182267200,101908484,1835495276,1819633152,20275052,375,-587202560,-1870656263,101974020,1835495
276,1819633152,909467500,1147393,0,-1307473438,335545510,1735355398,1969619053,912223340,325910835,0,363895040,311286,1869350420,1711304039,1600941173,805384247,21,-258736128,78368025,1812337664,7169903,1819047270,20395871,5976,-436207616,-1846141285,101974020,1835495276,1819633152,876109676,1649665,0,-1440567839,3
35545538,1735355398,1969619053,962554988,449052976,0,653299712,295919,1869350420,1711304039,1600941173,1124153401,29,-192675840,79429162,1862668800,1634559091,892403824,650051889,0,800098048,296364,1936655890,1885433188,926167552,3783681,0,-869026376,268436608,1685286662,7364973,18743605,0,934709504,295088,19366558
95,1885433188,1819633152,808542060,471269684,0,1001831936,302306,1936655895,1885433188,1819633152,825319276,526057783,0,1070834944,299706,1936655895,1885433188,1819633152,858873708,572195120,0,1139139584,302577,1936655895,1885433188,1819633152,875650924,625344820,0,1216600576,303850,1936655895,1885433188,1819633152
,909205356,684196145,0,1285998848,299938,1936655895,1885433188,1819633152,925982572,743506233,0,1354362880,299471,1936655895,1885433188,1819633152,959537004,798228789,0,1422640128,301799,1936655894,1885433188,1819633152,825384812,329729,0,-1588009296,385877133,1685286662,7364973,1819047270,875704927,3480577,0,-1705
...

#2 Updated by Sage Weil 18 days ago

  • Status changed from New to Need Review

#3 Updated by Kefu Chai 18 days ago

  • Duplicated by Bug #19852: mon: huge acting set returned by mapping.get() added

#4 Updated by Sage Weil 18 days ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF