Project

General

Profile

Bug #57296

Internal compiler errors and unmet dependencies on some sepia nodes

Added by Laura Flores 3 months ago. Updated 3 months ago.

Status:
New
Priority:
Urgent
Assignee:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/65020

make[4]: Leaving directory '/build/ceph-16.2.10-677-g5c73933c/obj-x86_64-linux-gnu'
[100%] Built target ceph-dencoder
make[4]: Leaving directory '/build/ceph-16.2.10-677-g5c73933c/obj-x86_64-linux-gnu'
make[3]: *** [CMakeFiles/Makefile2:16816: src/test/librbd/CMakeFiles/unittest_librbd.dir/all] Error 2
make[3]: Leaving directory '/build/ceph-16.2.10-677-g5c73933c/obj-x86_64-linux-gnu'
make[2]: *** [Makefile:144: all] Error 2
make[2]: Leaving directory '/build/ceph-16.2.10-677-g5c73933c/obj-x86_64-linux-gnu'
dh_auto_build: error: cd obj-x86_64-linux-gnu && make -j90 returned exit code 2
make[1]: *** [debian/rules:47: override_dh_auto_build] Error 25
make[1]: Leaving directory '/build/ceph-16.2.10-677-g5c73933c'
make: *** [debian/rules:40: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
E: Failed autobuilding of package

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=bionic,DIST=bionic,MACHINE_SIZE=gigantic/65031
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
src/tools/ceph-dencoder/CMakeFiles/ceph-dencoder.dir/build.make:162: recipe for target 'src/tools/ceph-dencoder/CMakeFiles/ceph-dencoder.dir/osd_types.cc.o' failed
make[4]: *** [src/tools/ceph-dencoder/CMakeFiles/ceph-dencoder.dir/osd_types.cc.o] Error 4
make[4]: Leaving directory '/build/ceph-16.2.10-677-g44f3f2ab/obj-x86_64-linux-gnu'
CMakeFiles/Makefile2:9982: recipe for target 'src/tools/ceph-dencoder/CMakeFiles/ceph-dencoder.dir/all' failed

...

make[4]: Leaving directory '/build/ceph-16.2.10-677-g44f3f2ab/obj-x86_64-linux-gnu'
CMakeFiles/Makefile2:18643: recipe for target 'src/test/librbd/CMakeFiles/unittest_librbd.dir/all' failed
make[3]: *** [src/test/librbd/CMakeFiles/unittest_librbd.dir/all] Error 2
Makefile:143: recipe for target 'all' failed
make[2]: *** [all] Error 2
dh_auto_build: cd obj-x86_64-linux-gnu && make -j90 -O returned exit code 2
debian/rules:47: recipe for target 'override_dh_auto_build' failed
make[1]: *** [override_dh_auto_build] Error 25
make[1]: Leaving directory '/build/ceph-16.2.10-677-g44f3f2ab'
debian/rules:40: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
E: Failed autobuilding of package

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=bionic,DIST=bionic,MACHINE_SIZE=gigantic/65039

In file included from /build/ceph-16.2.10-677-gc035170d/src/librados/AioCompletionImpl.h:21:0,
                 from /build/ceph-16.2.10-677-gc035170d/src/rgw/cls_fifo_legacy.h:41,
                 from /build/ceph-16.2.10-677-gc035170d/src/rgw/rgw_log_backing.h:35,
                 from /build/ceph-16.2.10-677-gc035170d/src/rgw/rgw_datalog.h:39,
                 from /build/ceph-16.2.10-677-gc035170d/src/rgw/services/svc_bi_rados.h:19,
                 from /build/ceph-16.2.10-677-gc035170d/src/rgw/rgw_rados.h:34,
                 from /build/ceph-16.2.10-677-gc035170d/src/rgw/rgw_admin.cc:39:
/build/ceph-16.2.10-677-gc035170d/src/osd/osd_types.h: In member function 'void object_ref_delta_t::mut_ref(const hobject_t&, int)':
/build/ceph-16.2.10-677-gc035170d/src/osd/osd_types.h:5551:35: warning: unused variable '_' [-Wunused-variable]
     [[maybe_unused]] auto [iter, _] = ref_delta.try_emplace(hoid, 0);
                                   ^
/build/ceph-16.2.10-677-gc035170d/src/rgw/rgw_admin.cc: In function 'int main(int, const char**)':
/build/ceph-16.2.10-677-gc035170d/src/rgw/rgw_admin.cc:4671:33: warning: unused variable 'name' [-Wunused-variable]
           for (auto& [name, zone] : zonegroup.zones) {
                                 ^
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
src/rgw/CMakeFiles/radosgw-admin.dir/build.make:65: recipe for target 'src/rgw/CMakeFiles/radosgw-admin.dir/rgw_admin.cc.o' failed
make[4]: *** [src/rgw/CMakeFiles/radosgw-admin.dir/rgw_admin.cc.o] Error 4
make[4]: Leaving directory '/build/ceph-16.2.10-677-gc035170d/obj-x86_64-linux-gnu'
CMakeFiles/Makefile2:28779: recipe for target 'src/rgw/CMakeFiles/radosgw-admin.dir/all' failed
make[3]: *** [src/rgw/CMakeFiles/radosgw-admin.dir/all] Error 2

...

ake[4]: Leaving directory '/build/ceph-16.2.10-677-gc035170d/obj-x86_64-linux-gnu'
CMakeFiles/Makefile2:18643: recipe for target 'src/test/librbd/CMakeFiles/unittest_librbd.dir/all' failed
make[3]: *** [src/test/librbd/CMakeFiles/unittest_librbd.dir/all] Error 2
Makefile:143: recipe for target 'all' failed
make[2]: *** [all] Error 2
dh_auto_build: cd obj-x86_64-linux-gnu && make -j90 -O returned exit code 2
debian/rules:47: recipe for target 'override_dh_auto_build' failed
make[1]: *** [override_dh_auto_build] Error 25
make[1]: Leaving directory '/build/ceph-16.2.10-677-gc035170d'
debian/rules:40: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

Some more examples:

Related issues

Related to Ceph - Bug #38629: ceph_dencoder.cc: c++: internal compiler error: Killed (program cc1plus) Closed

History

#1 Updated by Laura Flores 3 months ago

  • Related to Bug #38629: ceph_dencoder.cc: c++: internal compiler error: Killed (program cc1plus) added

#2 Updated by Laura Flores 3 months ago

  • Priority changed from Normal to Urgent

This is a big problem in terms of getting things approved in the teuthology testing suite.

#3 Updated by Laura Flores 3 months ago

  • Subject changed from Internal compiler errors on some sepia nodes to Internal compiler errors and unmet dependencies on some sepia nodes

This is an "unmet dependencies" example. (Actually the first one linked in the description is unmet dependencies as well).

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/65086

c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[4]: *** [src/test/librbd/CMakeFiles/unittest_librbd.dir/build.make:599: src/test/librbd/CMakeFiles/unittest_librbd.dir/io/test_mock_SimpleSchedulerObjectDispatch.cc.o] Error 1
make[4]: Leaving directory '/build/ceph-16.2.10-685-g33392c83/obj-x86_64-linux-gnu'
[100%] Built target ceph-dencoder
make[4]: Leaving directory '/build/ceph-16.2.10-685-g33392c83/obj-x86_64-linux-gnu'
make[3]: *** [CMakeFiles/Makefile2:16816: src/test/librbd/CMakeFiles/unittest_librbd.dir/all] Error 2
make[3]: Leaving directory '/build/ceph-16.2.10-685-g33392c83/obj-x86_64-linux-gnu'
make[2]: *** [Makefile:144: all] Error 2
make[2]: Leaving directory '/build/ceph-16.2.10-685-g33392c83/obj-x86_64-linux-gnu'
dh_auto_build: error: cd obj-x86_64-linux-gnu && make -j90 returned exit code 2
make[1]: *** [debian/rules:47: override_dh_auto_build] Error 25
make[1]: Leaving directory '/build/ceph-16.2.10-685-g33392c83'
make: *** [debian/rules:40: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

...

Need to get 0 B/486 MB of archives. After unpacking 1672 MB will be used.
The following packages have unmet dependencies:
 libc6-dev : Depends: libc6 (= 2.31-0ubuntu9) but 2.31-0ubuntu9.9 is to be installed
 curl : Depends: libcurl4 (= 7.68.0-1ubuntu2) but 7.68.0-1ubuntu2.12 is to be installed
The following actions will resolve these dependencies:

     Upgrade the following packages:                                                            
1)     curl [7.68.0-1ubuntu2 (focal, now) -> 7.68.0-1ubuntu2.12 (focal-security, focal-updates)]
2)     libc-dev-bin [2.31-0ubuntu9 (focal, now) -> 2.31-0ubuntu9.9 (focal-updates)]             
3)     libc6-dev [2.31-0ubuntu9 (focal, now) -> 2.31-0ubuntu9.9 (focal-updates)]                

#4 Updated by Brad Hubbard 3 months ago

I tried to reproduce this locally with the latest focal container but so far no joy so this may be something specific to the environment that is failing.

[ 92%] Building CXX object src/test/librbd/CMakeFiles/unittest_librbd.dir/io/test_mock_ImageRequest.cc.o              
[ 92%] Building CXX object src/test/librbd/CMakeFiles/unittest_librbd.dir/io/test_mock_ObjectRequest.cc.o             
[ 92%] Building CXX object src/test/librbd/CMakeFiles/unittest_librbd.dir/io/test_mock_SimpleSchedulerObjectDispatch.cc.o                                                                                                                   
[ 92%] Building CXX object src/test/librbd/CMakeFiles/unittest_librbd.dir/journal/test_mock_OpenRequest.cc.o          
[ 92%] Building CXX object src/test/librbd/CMakeFiles/unittest_librbd.dir/journal/test_mock_PromoteRequest.cc.o
...
[100%] Linking CXX executable ../../../bin/unittest_librbd                                                            
[100%] Built target unittest_librbd 

I'll try a 'make all' but I'd say that won't reproduce either based on historical experience.

Suggest we start working out which build machines specifically are failing, what (if anything) is common about those machines, and whether we can reproduce this on one of those machines so we can run the compiler command manually (VERBOSE=1 make unittest_librbd) and see what we can glean from that. I would do this but I'm not sure how safe this is going to be on a machine that is running other build jobs nor how to remove a machine temporarily from the pool of machines that jobs can run on. Let me know how to do that if anyone knows and I'll take a crack at it.

Also available in: Atom PDF