Bug #5405
closedceph-deploy: transient pushy exception on install
0%
Description
flab:ceph-deploy 09:22 AM $ ./ceph-deploy install mira09{4,5,6} OK Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 504, in run self.__target(*self.__args, **self.__kwargs) File "/home/sage/src/ceph-deploy/virtualenv/local/lib/python2.7/site-packages/pushy-0.5.1-py2.7.egg/pushy/protocol/baseconnection.py", line 268, in serve_forever self.__handle(m) File "/home/sage/src/ceph-deploy/virtualenv/local/lib/python2.7/site-packages/pushy-0.5.1-py2.7.egg/pushy/protocol/baseconnection.py", line 653, in __handle self.__send_message(MessageType.exception, e) File "/home/sage/src/ceph-deploy/virtualenv/local/lib/python2.7/site-packages/pushy-0.5.1-py2.7.egg/pushy/protocol/baseconnection.py", line 560, in __send_message self.__ostream.send_message(m) File "/home/sage/src/ceph-deploy/virtualenv/local/lib/python2.7/site-packages/pushy-0.5.1-py2.7.egg/pushy/protocol/baseconnection.py", line 97, in send_message self.__file.write(bytes_) ValueError: I/O operation on closed file flab:ceph-deploy 09:22 AM $ ./ceph-deploy install mira09{4,5,6}
Updated by Tamilarasi muthamizhan almost 11 years ago
on centos 6.4,
[ubuntu@burnupi63 ceph-deploy]$ ./ceph-deploy install burnupi63 ################################################## Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/home/ubuntu/cdep/ceph-deploy/virtualenv/lib/python2.6/site-packages/pushy-0.5.1-py2.6.egg/pushy/protocol/baseconnection.py", line 268, in serve_forever self.__handle(m) File "/home/ubuntu/cdep/ceph-deploy/virtualenv/lib/python2.6/site-packages/pushy-0.5.1-py2.6.egg/pushy/protocol/baseconnection.py", line 653, in __handle self.__send_message(MessageType.exception, e) File "/home/ubuntu/cdep/ceph-deploy/virtualenv/lib/python2.6/site-packages/pushy-0.5.1-py2.6.egg/pushy/protocol/baseconnection.py", line 560, in __send_message self.__ostream.send_message(m) File "/home/ubuntu/cdep/ceph-deploy/virtualenv/lib/python2.6/site-packages/pushy-0.5.1-py2.6.egg/pushy/protocol/baseconnection.py", line 97, in send_message self.__file.write(bytes_) ValueError: I/O operation on closed file
Updated by Alfredo Deza almost 11 years ago
This is very hard to replicate, but after adding a bit of verbosity in pushy, I managed to narrow it down to the PushyClient performing a connection close before the thread finished.
The `Duplicate source.list` warnings may (or may not) be of substance, but it is clear that as the `close()` method is called the thread attempts to keep writing.
$ ceph-deploy -v install node{1,2} Installing stable version cuttlefish on cluster ceph hosts node1 node2 args.host: ['node1', 'node2'] hostname is: node1 Detecting platform for host node1 ... compiling source compiling source Distro Ubuntu release 12.04 codename precise compiling source Installing on host node1 ... W: Duplicate sources.list entry http://ceph.com/debian-cuttlefish/ precise/main amd64 Packages (/var/lib/apt/lists/ceph.com_debian-cuttlefish_dists_precise_main_binary-amd64_Packages) W: Duplicate sources.list entry http://ceph.com/debian-cuttlefish/ precise/main i386 Packages (/var/lib/apt/lists/ceph.com_debian-cuttlefish_dists_precise_main_binary-i386_Packages) W: You may want to run apt-get update to correct these problems closing the connection Exception in thread Thread-1: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/Users/alfredo/.virtualenvs/ceph-deploy/lib/python2.7/site-packages/pushy-0.5.1-py2.7.egg/pushy/protocol/baseconnection.py", line 268, in serve_forever self.__handle(m) File "/Users/alfredo/.virtualenvs/ceph-deploy/lib/python2.7/site-packages/pushy-0.5.1-py2.7.egg/pushy/protocol/baseconnection.py", line 653, in __handle self.__send_message(MessageType.exception, e) File "/Users/alfredo/.virtualenvs/ceph-deploy/lib/python2.7/site-packages/pushy-0.5.1-py2.7.egg/pushy/protocol/baseconnection.py", line 560, in __send_message self.__ostream.send_message(m) File "/Users/alfredo/.virtualenvs/ceph-deploy/lib/python2.7/site-packages/pushy-0.5.1-py2.7.egg/pushy/protocol/baseconnection.py", line 97, in send_message self.__file.write(bytes_) ValueError: I/O operation on closed file
Updated by Alfredo Deza almost 11 years ago
I've opened a pull request for pushy (https://github.com/axw/pushy/pull/42) that should solve this problem.
What happens (as detailed in the pull request) is that the `close()` method will close the connection first and then join the threads, but sometimes a thread will want to write to the file descriptor but it will not be able to do so because the closing of the connection closes the file as well.
The BaseConnection class already catches IOError but does not account for ValueError that is related (as in this case).
In the meantime, we can sub-class and fix this in ceph-deploy until the pull request gets merged and a release gets cut.
Updated by Sage Weil almost 11 years ago
- Status changed from New to 15
once this his an upstream we need to build a package.
Updated by Alfredo Deza almost 11 years ago
- Assignee changed from Anonymous to Alfredo Deza
Updated by Alfredo Deza almost 11 years ago
- Assignee changed from Alfredo Deza to Anonymous