Project

General

Profile

Bug #18872 » ceph-client.admin.log

Client logs with 'debug client = 20 debug ms = 1' - Jan Fajerski, 02/09/2017 03:26 PM

 
2017-02-09 15:47:04.005845 3fff87c39bb0 0 ceph version 10.2.5-239-g3a6a822 (3a6a822c8125858afaeac7a1ee0d121d063660f0), process ceph-fuse, pid 32306
2017-02-09 15:47:04.008738 3fff87c39bb0 -1 init, newargv = 0x10000f6edd0 newargc=11
2017-02-09 15:47:04.008849 3fff87c39bb0 1 -- :/0 messenger.start
2017-02-09 15:47:04.009467 3fff87c39bb0 1 -- :/2037822863 --> 10.162.2.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x10000f73620 con 0x10000f725c0
2017-02-09 15:47:04.009493 3fff87c39bb0 10 client.-1 Subscribing to map 'mdsmap'
2017-02-09 15:47:04.009561 3fff87c39bb0 20 client.-1 populate_metadata read hostname 'pinotage-2'
2017-02-09 15:47:04.010488 3fff847dea60 1 -- 10.161.26.49:0/2037822863 learned my addr 10.161.26.49:0/2037822863
2017-02-09 15:47:04.010831 3fff7d7fea60 10 client.-1 ms_handle_connect on 10.162.2.78:6789/0
2017-02-09 15:47:04.011484 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mon.0 10.162.2.78:6789/0 1 ==== mon_map magic: 0 v1 ==== 500+0+0 (2749416025 0 0) 0x3fff70000b90 con 0x10000f725c0
2017-02-09 15:47:04.011537 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mon.0 10.162.2.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3641171469 0 0) 0x3fff70001010 con 0x10000f725c0
2017-02-09 15:47:04.011606 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x3fff6c001950 con 0x10000f725c0
2017-02-09 15:47:04.012483 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mon.0 10.162.2.78:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3109663248 0 0) 0x3fff70001010 con 0x10000f725c0
2017-02-09 15:47:04.012548 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x3fff6c003460 con 0x10000f725c0
2017-02-09 15:47:04.013488 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mon.0 10.162.2.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (475164506 0 0) 0x3fff700008c0 con 0x10000f725c0
2017-02-09 15:47:04.013560 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x10000f773a0 con 0x10000f725c0
2017-02-09 15:47:04.013574 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6789/0 -- mon_subscribe({mdsmap=0+}) v2 -- ?+0 0x10000f777b0 con 0x10000f725c0
2017-02-09 15:47:04.013626 3fff87c39bb0 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:04.013648 3fff87c39bb0 2 client.94152 mounted: have mdsmap 0
2017-02-09 15:47:04.014348 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mon.0 10.162.2.78:6789/0 5 ==== osd_map(99..99 src has 1..99) v3 ==== 5393+0+0 (2761367640 0 0) 0x3fff700008c0 con 0x10000f725c0
2017-02-09 15:47:04.014453 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mon.0 10.162.2.78:6789/0 6 ==== mdsmap(e 32) v1 ==== 684+0+0 (253898769 0 0) 0x3fff70002ac0 con 0x10000f725c0
2017-02-09 15:47:04.014461 3fff7d7fea60 1 client.94152 handle_mds_map epoch 32
2017-02-09 15:47:04.014495 3fff87c39bb0 10 client.94152 did not get mds through better means, so chose random mds 0
2017-02-09 15:47:04.014498 3fff87c39bb0 20 client.94152 mds is 0
2017-02-09 15:47:04.014503 3fff87c39bb0 10 client.94152 _open_mds_session mds.0
2017-02-09 15:47:04.014545 3fff87c39bb0 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_session(request_open) v2 -- ?+0 0x10000f6be90 con 0x10000f69740
2017-02-09 15:47:04.014557 3fff87c39bb0 10 client.94152 waiting for session to mds.0 to open
2017-02-09 15:47:04.015763 3fff7d7fea60 10 client.94152 ms_handle_connect on 10.162.2.78:6800/1968
2017-02-09 15:47:04.031942 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 1 ==== client_session(open) v1 ==== 28+0+0 (2719320884 0 0) 0x3fff680009c0 con 0x10000f69740
2017-02-09 15:47:04.031953 3fff7d7fea60 10 client.94152 handle_client_session client_session(open) v1 from mds.0
2017-02-09 15:47:04.031957 3fff7d7fea60 10 client.94152 renew_caps mds.0
2017-02-09 15:47:04.031960 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_session(request_renewcaps seq 1) v2 -- ?+0 0x3fff6c00b4c0 con 0x10000f69740
2017-02-09 15:47:04.031970 3fff7d7fea60 10 client.94152 connect_mds_targets for mds.0
2017-02-09 15:47:04.031982 3fff87c39bb0 10 client.94152 did not get mds through better means, so chose random mds 0
2017-02-09 15:47:04.031985 3fff87c39bb0 20 client.94152 mds is 0
2017-02-09 15:47:04.031989 3fff87c39bb0 10 client.94152 send_request rebuilding request 1 for mds.0
2017-02-09 15:47:04.031994 3fff87c39bb0 20 client.94152 encode_cap_releases enter (req: 0x10000f76390, mds: 0)
2017-02-09 15:47:04.032013 3fff87c39bb0 20 client.94152 send_request set sent_stamp to 2017-02-09 15:47:04.032012
2017-02-09 15:47:04.032020 3fff87c39bb0 10 client.94152 send_request client_request(unknown.0:1 getattr pAsLsXsFs #1 2017-02-09 15:47:04.014491) v3 to mds.0
2017-02-09 15:47:04.032034 3fff87c39bb0 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_request(client.94152:1 getattr pAsLsXsFs #1 2017-02-09 15:47:04.014491) v3 -- ?+0 0x10000f76d10 con 0x10000f69740
2017-02-09 15:47:04.032048 3fff87c39bb0 20 client.94152 awaiting reply|forward|kick on 0x3fffde680b18
2017-02-09 15:47:04.032616 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 2 ==== client_session(renewcaps seq 1) v1 ==== 28+0+0 (2614397064 0 0) 0x3fff680009c0 con 0x10000f69740
2017-02-09 15:47:04.032625 3fff7d7fea60 10 client.94152 handle_client_session client_session(renewcaps seq 1) v1 from mds.0
2017-02-09 15:47:04.033288 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 3 ==== client_reply(???:1 = 0 (0) Success) v1 ==== 354+0+0 (3099556690 0 0) 0x3fff68000e10 con 0x10000f69740
2017-02-09 15:47:04.033302 3fff7d7fea60 20 client.94152 handle_client_reply got a reply. Safe:1 tid 1
2017-02-09 15:47:04.033306 3fff7d7fea60 10 client.94152 insert_trace from 2017-02-09 15:47:04.032012 mds.0 is_target=1 is_dentry=0
2017-02-09 15:47:04.033311 3fff7d7fea60 10 client.94152 features 0x7ffffffefdfbfff
2017-02-09 15:47:04.033313 3fff7d7fea60 10 client.94152 update_snap_trace len 48
2017-02-09 15:47:04.033319 3fff7d7fea60 20 client.94152 get_snap_realm 1 0x3fff6c00c040 0 -> 1
2017-02-09 15:47:04.033322 3fff7d7fea60 10 client.94152 update_snap_trace snaprealm(1 nref=1 c=0 seq=0 parent=0 my_snaps=[] cached_snapc=0=[]) seq 1 > 0
2017-02-09 15:47:04.033331 3fff7d7fea60 10 client.94152 invalidate_snaprealm_and_children snaprealm(1 nref=2 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=0=[])
2017-02-09 15:47:04.033335 3fff7d7fea60 15 client.94152 update_snap_trace snaprealm(1 nref=2 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=0=[]) self|parent updated
2017-02-09 15:47:04.033339 3fff7d7fea60 15 client.94152 snapc 1=[]
2017-02-09 15:47:04.033342 3fff7d7fea60 20 client.94152 put_snap_realm 1 0x3fff6c00c040 2 -> 1
2017-02-09 15:47:04.033345 3fff7d7fea60 10 client.94152 no new snap on snaprealm(1 nref=1 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=1=[])
2017-02-09 15:47:04.033348 3fff7d7fea60 20 client.94152 put_snap_realm 1 0x3fff6c00c040 1 -> 0
2017-02-09 15:47:04.033351 3fff7d7fea60 10 client.94152 hrm is_target=1 is_dentry=0
2017-02-09 15:47:04.033363 3fff7d7fea60 15 inode.get on 0x3fff6c00cb90 1.head now 1
2017-02-09 15:47:04.033365 3fff7d7fea60 12 client.94152 add_update_inode adding 1.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40000 size=0/0 mtime=0.000000 caps=- 0x3fff6c00cb90) caps pAsLsXsFs
2017-02-09 15:47:04.033378 3fff7d7fea60 20 client.94152 dir hash is 2
2017-02-09 15:47:04.033381 3fff7d7fea60 10 client.94152 update_inode_file_bits 1.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 mtime=0.000000 caps=- has_dir_layout 0x3fff6c00cb90) - mtime 2017-02-09 15:29:15.934503
2017-02-09 15:47:04.033390 3fff7d7fea60 20 client.94152 get_snap_realm 1 0x3fff6c00d5b0 0 -> 1
2017-02-09 15:47:04.033393 3fff7d7fea60 15 client.94152 add_update_cap first one, opened snaprealm 0x3fff6c00d5b0
2017-02-09 15:47:04.033397 3fff7d7fea60 10 client.94152 add_update_cap issued - -> pAsLsXsFs from mds.0 on 1.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:04.033413 3fff7d7fea60 15 inode.get on 0x3fff6c00cb90 1.head now 2
2017-02-09 15:47:04.033415 3fff7d7fea60 15 inode.get on 0x3fff6c00cb90 1.head now 3
2017-02-09 15:47:04.033418 3fff7d7fea60 20 client.94152 handle_client_reply signalling caller 0x3fffde680b18
2017-02-09 15:47:04.033422 3fff7d7fea60 20 client.94152 handle_client_reply awaiting kickback on tid 1 0x3fff7d7fd828
2017-02-09 15:47:04.033426 3fff87c39bb0 20 client.94152 sendrecv kickback on tid 1 0x3fff7d7fd828
2017-02-09 15:47:04.033430 3fff87c39bb0 20 client.94152 lat 0.001416
2017-02-09 15:47:04.033433 3fff87c39bb0 15 inode.get on 0x3fff6c00cb90 1.head now 4
2017-02-09 15:47:04.033435 3fff87c39bb0 20 client.94152 _ll_get 0x3fff6c00cb90 1 -> 1
2017-02-09 15:47:04.033443 3fff7d7fea60 10 client.94152 put_inode on 1.head(faked_ino=0 ref=4 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:04.033462 3fff7d7fea60 15 inode.put on 0x3fff6c00cb90 1.head now 3
2017-02-09 15:47:04.033466 3fff7d7fea60 10 client.94152 put_inode on 1.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:04.033481 3fff7d7fea60 15 inode.put on 0x3fff6c00cb90 1.head now 2
2017-02-09 15:47:04.033874 3fff87c39bb0 10 client.94152 ll_register_callbacks cb 0x10000f6ebe0 invalidate_ino_cb 1 invalidate_dentry_cb 1 getgroups_cb1 switch_interrupt_cb 1 remount_cb 1
2017-02-09 15:47:04.034255 3fff797fea60 1 client.94152 using remount_cb
2017-02-09 15:47:04.040383 3fff787fea60 3 client.94152 ll_getattr 1.head
2017-02-09 15:47:04.040454 3fff787fea60 10 client.94152 _getattr mask pAsLsXsFs issued=1
2017-02-09 15:47:04.040504 3fff787fea60 10 client.94152 fill_stat on 1 snap/devhead mode 040755 mtime 2017-02-09 15:29:15.934503 ctime 2017-02-09 15:29:15.934503
2017-02-09 15:47:04.040516 3fff787fea60 3 client.94152 ll_getattr 1.head = 0
2017-02-09 15:47:04.040538 3fff787fea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:04.040571 3fff78ffea60 3 client.94152 ll_getattr 1.head
2017-02-09 15:47:04.040574 3fff78ffea60 10 client.94152 _getattr mask pAsLsXsFs issued=1
2017-02-09 15:47:04.040579 3fff78ffea60 10 client.94152 fill_stat on 1 snap/devhead mode 040755 mtime 2017-02-09 15:29:15.934503 ctime 2017-02-09 15:29:15.934503
2017-02-09 15:47:04.040584 3fff78ffea60 3 client.94152 ll_getattr 1.head = 0
2017-02-09 15:47:04.040591 3fff78ffea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:04.040604 3fff787fea60 3 client.94152 ll_opendir 1.head
2017-02-09 15:47:04.040607 3fff787fea60 15 inode.get on 0x3fff6c00cb90 1.head now 3
2017-02-09 15:47:04.040612 3fff787fea60 3 client.94152 _opendir(1) = 0 (0x3fff54001000)
2017-02-09 15:47:04.040615 3fff787fea60 3 client.94152 ll_opendir 1.head = 0 (0x3fff54001000)
2017-02-09 15:47:04.040621 3fff787fea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:04.040643 3fff78ffea60 3 client.94152 ll_getattr 1.head
2017-02-09 15:47:04.040646 3fff78ffea60 10 client.94152 _getattr mask pAsLsXsFs issued=1
2017-02-09 15:47:04.040651 3fff78ffea60 10 client.94152 fill_stat on 1 snap/devhead mode 040755 mtime 2017-02-09 15:29:15.934503 ctime 2017-02-09 15:29:15.934503
2017-02-09 15:47:04.040656 3fff78ffea60 3 client.94152 ll_getattr 1.head = 0
2017-02-09 15:47:04.040662 3fff78ffea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:04.040675 3fff787fea60 3 client.94152 ll_releasedir 0x3fff54001000
2017-02-09 15:47:04.040679 3fff787fea60 10 client.94152 _closedir(0x3fff54001000)
2017-02-09 15:47:04.040682 3fff787fea60 10 client.94152 _closedir detaching inode 0x3fff6c00cb90
2017-02-09 15:47:04.040685 3fff787fea60 10 client.94152 put_inode on 1.head(faked_ino=0 ref=3 ll_ref=5 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:04.040709 3fff787fea60 15 inode.put on 0x3fff6c00cb90 1.head now 2
2017-02-09 15:47:04.040712 3fff787fea60 10 client.94152 _readdir_drop_dirp_buffer 0x3fff54001000
2017-02-09 15:47:04.040764 3fff78ffea60 3 client.94152 ll_getattr 1.head
2017-02-09 15:47:04.040768 3fff78ffea60 10 client.94152 _getattr mask pAsLsXsFs issued=1
2017-02-09 15:47:04.040774 3fff78ffea60 10 client.94152 fill_stat on 1 snap/devhead mode 040755 mtime 2017-02-09 15:29:15.934503 ctime 2017-02-09 15:29:15.934503
2017-02-09 15:47:04.040779 3fff78ffea60 3 client.94152 ll_getattr 1.head = 0
2017-02-09 15:47:04.040785 3fff78ffea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:04.041222 3fff5bffea60 3 client.94152 ll_getattr 1.head
2017-02-09 15:47:04.041229 3fff5bffea60 10 client.94152 _getattr mask pAsLsXsFs issued=1
2017-02-09 15:47:04.041237 3fff5bffea60 10 client.94152 fill_stat on 1 snap/devhead mode 040755 mtime 2017-02-09 15:29:15.934503 ctime 2017-02-09 15:29:15.934503
2017-02-09 15:47:04.041247 3fff5bffea60 3 client.94152 ll_getattr 1.head = 0
2017-02-09 15:47:04.041256 3fff5bffea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:05.013724 3fff7e7fea60 10 client.94152 renew_caps()
2017-02-09 15:47:05.013731 3fff7e7fea60 15 client.94152 renew_caps requesting from mds.0
2017-02-09 15:47:05.013734 3fff7e7fea60 10 client.94152 renew_caps mds.0
2017-02-09 15:47:05.013739 3fff7e7fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_session(request_renewcaps seq 2) v2 -- ?+0 0x3fff5c000d20 con 0x10000f69740
2017-02-09 15:47:05.013766 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:05.014690 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 4 ==== client_session(renewcaps seq 2) v1 ==== 28+0+0 (1357429227 0 0) 0x3fff68000e10 con 0x10000f69740
2017-02-09 15:47:05.014704 3fff7d7fea60 10 client.94152 handle_client_session client_session(renewcaps seq 2) v1 from mds.0
2017-02-09 15:47:06.013786 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:07.013850 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:08.013915 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:09.013990 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:10.014078 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:11.014169 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:12.014261 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:13.014365 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:14.014435 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:15.014540 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:16.014625 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:17.014713 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:18.014811 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:19.014875 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:20.014974 3fff7e7fea60 20 client.94152 trim_cache size 0 max 16384
2017-02-09 15:47:20.190262 3fff787fea60 3 client.94152 ll_getattr 1.head
2017-02-09 15:47:20.190277 3fff787fea60 10 client.94152 _getattr mask pAsLsXsFs issued=1
2017-02-09 15:47:20.190292 3fff787fea60 10 client.94152 fill_stat on 1 snap/devhead mode 040755 mtime 2017-02-09 15:29:15.934503 ctime 2017-02-09 15:29:15.934503
2017-02-09 15:47:20.190302 3fff787fea60 3 client.94152 ll_getattr 1.head = 0
2017-02-09 15:47:20.190340 3fff787fea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:20.190375 3fff787fea60 3 client.94152 ll_getattr 1.head
2017-02-09 15:47:20.190377 3fff787fea60 10 client.94152 _getattr mask pAsLsXsFs issued=1
2017-02-09 15:47:20.190383 3fff787fea60 10 client.94152 fill_stat on 1 snap/devhead mode 040755 mtime 2017-02-09 15:29:15.934503 ctime 2017-02-09 15:29:15.934503
2017-02-09 15:47:20.190389 3fff787fea60 3 client.94152 ll_getattr 1.head = 0
2017-02-09 15:47:20.190395 3fff787fea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:20.190433 3fff5bffea60 3 client.94152 ll_lookup 0x3fff6c00cb90 jf_2.img
2017-02-09 15:47:20.190468 3fff5bffea60 15 inode.get on 0x3fff6c00cb90 1.head now 3
2017-02-09 15:47:20.190471 3fff5bffea60 10 client.94152 _do_lookup on #1/jf_2.img
2017-02-09 15:47:20.190480 3fff5bffea60 20 client.94152 choose_target_mds starting with req->inode 1.head(faked_ino=0 ref=3 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.190502 3fff5bffea60 20 client.94152 choose_target_mds inode dir hash is 2 on jf_2.img => 4277851118
2017-02-09 15:47:20.190505 3fff5bffea60 20 client.94152 choose_target_mds 1.head(faked_ino=0 ref=3 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90) is_hash=1 hash=4277851118
2017-02-09 15:47:20.190561 3fff5bffea60 10 client.94152 choose_target_mds from caps on inode 1.head(faked_ino=0 ref=3 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.190577 3fff5bffea60 20 client.94152 mds is 0
2017-02-09 15:47:20.190582 3fff5bffea60 10 client.94152 send_request rebuilding request 2 for mds.0
2017-02-09 15:47:20.190589 3fff5bffea60 20 client.94152 encode_cap_releases enter (req: 0x3fff50000f90, mds: 0)
2017-02-09 15:47:20.190593 3fff5bffea60 20 client.94152 send_request set sent_stamp to 2017-02-09 15:47:20.190593
2017-02-09 15:47:20.190597 3fff5bffea60 10 client.94152 send_request client_request(unknown.0:2 lookup #1/jf_2.img 2017-02-09 15:47:20.190474) v3 to mds.0
2017-02-09 15:47:20.190626 3fff5bffea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_request(client.94152:2 lookup #1/jf_2.img 2017-02-09 15:47:20.190474) v3 -- ?+0 0x3fff50001d70 con 0x10000f69740
2017-02-09 15:47:20.190673 3fff5bffea60 20 client.94152 awaiting reply|forward|kick on 0x3fff5bffd748
2017-02-09 15:47:20.191758 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 5 ==== client_reply(???:2 = -2 (2) No such file or directory) v1 ==== 384+0+0 (1529693188 0 0) 0x3fff680012e0 con 0x10000f69740
2017-02-09 15:47:20.191850 3fff7d7fea60 20 client.94152 handle_client_reply got a reply. Safe:1 tid 2
2017-02-09 15:47:20.191856 3fff7d7fea60 10 client.94152 insert_trace from 2017-02-09 15:47:20.190593 mds.0 is_target=0 is_dentry=1
2017-02-09 15:47:20.191884 3fff7d7fea60 10 client.94152 features 0x7ffffffefdfbfff
2017-02-09 15:47:20.191888 3fff7d7fea60 10 client.94152 update_snap_trace len 48
2017-02-09 15:47:20.191894 3fff7d7fea60 20 client.94152 get_snap_realm 1 0x3fff6c00d5b0 1 -> 2
2017-02-09 15:47:20.191898 3fff7d7fea60 10 client.94152 update_snap_trace snaprealm(1 nref=2 c=0 seq=0 parent=0 my_snaps=[] cached_snapc=0=[]) seq 1 > 0
2017-02-09 15:47:20.191909 3fff7d7fea60 10 client.94152 invalidate_snaprealm_and_children snaprealm(1 nref=3 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=0=[])
2017-02-09 15:47:20.191913 3fff7d7fea60 15 client.94152 update_snap_trace snaprealm(1 nref=3 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=0=[]) self|parent updated
2017-02-09 15:47:20.191916 3fff7d7fea60 15 client.94152 snapc 1=[]
2017-02-09 15:47:20.191918 3fff7d7fea60 20 client.94152 put_snap_realm 1 0x3fff6c00d5b0 3 -> 2
2017-02-09 15:47:20.191921 3fff7d7fea60 10 client.94152 no new snap on snaprealm(1 nref=2 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=1=[])
2017-02-09 15:47:20.191923 3fff7d7fea60 20 client.94152 put_snap_realm 1 0x3fff6c00d5b0 2 -> 1
2017-02-09 15:47:20.191926 3fff7d7fea60 10 client.94152 hrm is_target=0 is_dentry=1
2017-02-09 15:47:20.191955 3fff7d7fea60 12 client.94152 add_update_inode had 1.head(faked_ino=0 ref=3 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90) caps pAsLsXsFs
2017-02-09 15:47:20.192001 3fff7d7fea60 10 client.94152 add_update_cap issued pAsLsXsFs -> pAsLsXsFs from mds.0 on 1.head(faked_ino=0 ref=3 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.192021 3fff7d7fea60 20 client.94152 got dirfrag map for 1 frag * to mds 0
2017-02-09 15:47:20.192029 3fff7d7fea60 20 client.94152 handle_client_reply signalling caller 0x3fff5bffd748
2017-02-09 15:47:20.192036 3fff7d7fea60 20 client.94152 handle_client_reply awaiting kickback on tid 2 0x3fff7d7fd828
2017-02-09 15:47:20.192063 3fff5bffea60 20 client.94152 sendrecv kickback on tid 2 0x3fff7d7fd828
2017-02-09 15:47:20.192071 3fff5bffea60 20 client.94152 lat 0.001476
2017-02-09 15:47:20.192090 3fff5bffea60 10 client.94152 _do_lookup res is -2
2017-02-09 15:47:20.192096 3fff5bffea60 10 client.94152 _lookup 1.head(faked_ino=0 ref=3 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90) jf_2.img = -2
2017-02-09 15:47:20.192114 3fff5bffea60 3 client.94152 ll_lookup 0x3fff6c00cb90 jf_2.img -> -2 (0)
2017-02-09 15:47:20.192126 3fff7d7fea60 10 client.94152 put_inode on 1.head(faked_ino=0 ref=3 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.192165 3fff7d7fea60 15 inode.put on 0x3fff6c00cb90 1.head now 2
2017-02-09 15:47:20.192181 3fff5bffea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:20.192213 3fff78ffea60 3 client.94152 ll_create 1.head jf_2.img 0100644 66113, uid 0, gid 0
2017-02-09 15:47:20.192226 3fff78ffea60 15 inode.get on 0x3fff6c00cb90 1.head now 3
2017-02-09 15:47:20.192228 3fff78ffea60 10 client.94152 _do_lookup on #1/jf_2.img
2017-02-09 15:47:20.192233 3fff78ffea60 20 client.94152 choose_target_mds starting with req->inode 1.head(faked_ino=0 ref=3 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.192246 3fff78ffea60 20 client.94152 choose_target_mds inode dir hash is 2 on jf_2.img => 4277851118
2017-02-09 15:47:20.192249 3fff78ffea60 20 client.94152 choose_target_mds 1.head(faked_ino=0 ref=3 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90) is_hash=1 hash=4277851118
2017-02-09 15:47:20.192261 3fff78ffea60 10 client.94152 choose_target_mds from caps on inode 1.head(faked_ino=0 ref=3 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.192292 3fff78ffea60 20 client.94152 mds is 0
2017-02-09 15:47:20.192296 3fff78ffea60 10 client.94152 send_request rebuilding request 3 for mds.0
2017-02-09 15:47:20.192317 3fff78ffea60 20 client.94152 encode_cap_releases enter (req: 0x3fff60022c90, mds: 0)
2017-02-09 15:47:20.192319 3fff78ffea60 20 client.94152 send_request set sent_stamp to 2017-02-09 15:47:20.192319
2017-02-09 15:47:20.192325 3fff78ffea60 10 client.94152 send_request client_request(unknown.0:3 lookup #1/jf_2.img 2017-02-09 15:47:20.192231) v3 to mds.0
2017-02-09 15:47:20.192332 3fff78ffea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_request(client.94152:3 lookup #1/jf_2.img 2017-02-09 15:47:20.192231) v3 -- ?+0 0x3fff60023920 con 0x10000f69740
2017-02-09 15:47:20.192348 3fff78ffea60 20 client.94152 awaiting reply|forward|kick on 0x3fff78ffd5f8
2017-02-09 15:47:20.193297 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 6 ==== client_reply(???:3 = -2 (2) No such file or directory) v1 ==== 384+0+0 (3011934671 0 0) 0x3fff680012e0 con 0x10000f69740
2017-02-09 15:47:20.193363 3fff7d7fea60 20 client.94152 handle_client_reply got a reply. Safe:1 tid 3
2017-02-09 15:47:20.193368 3fff7d7fea60 10 client.94152 insert_trace from 2017-02-09 15:47:20.192319 mds.0 is_target=0 is_dentry=1
2017-02-09 15:47:20.193373 3fff7d7fea60 10 client.94152 features 0x7ffffffefdfbfff
2017-02-09 15:47:20.193375 3fff7d7fea60 10 client.94152 update_snap_trace len 48
2017-02-09 15:47:20.193380 3fff7d7fea60 20 client.94152 get_snap_realm 1 0x3fff6c00d5b0 1 -> 2
2017-02-09 15:47:20.193406 3fff7d7fea60 10 client.94152 update_snap_trace snaprealm(1 nref=2 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=1=[]) seq 1 <= 1 and same parent, SKIPPING
2017-02-09 15:47:20.193431 3fff7d7fea60 20 client.94152 put_snap_realm 1 0x3fff6c00d5b0 2 -> 1
2017-02-09 15:47:20.193435 3fff7d7fea60 10 client.94152 hrm is_target=0 is_dentry=1
2017-02-09 15:47:20.193447 3fff7d7fea60 12 client.94152 add_update_inode had 1.head(faked_ino=0 ref=3 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90) caps pAsLsXsFs
2017-02-09 15:47:20.193475 3fff7d7fea60 10 client.94152 add_update_cap issued pAsLsXsFs -> pAsLsXsFs from mds.0 on 1.head(faked_ino=0 ref=3 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.193555 3fff7d7fea60 20 client.94152 got dirfrag map for 1 frag * to mds 0
2017-02-09 15:47:20.193562 3fff7d7fea60 20 client.94152 handle_client_reply signalling caller 0x3fff78ffd5f8
2017-02-09 15:47:20.193571 3fff7d7fea60 20 client.94152 handle_client_reply awaiting kickback on tid 3 0x3fff7d7fd828
2017-02-09 15:47:20.193581 3fff78ffea60 20 client.94152 sendrecv kickback on tid 3 0x3fff7d7fd828
2017-02-09 15:47:20.193586 3fff78ffea60 20 client.94152 lat 0.001264
2017-02-09 15:47:20.193622 3fff78ffea60 10 client.94152 _do_lookup res is -2
2017-02-09 15:47:20.193628 3fff78ffea60 10 client.94152 _lookup 1.head(faked_ino=0 ref=3 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90) jf_2.img = -2
2017-02-09 15:47:20.193667 3fff78ffea60 3 client.94152 _create(1 jf_2.img, 0100644)
2017-02-09 15:47:20.193691 3fff78ffea60 15 inode.get on 0x3fff6c00cb90 1.head now 4
2017-02-09 15:47:20.193728 3fff78ffea60 20 client.94152 get_or_create 1.head(faked_ino=0 ref=4 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90) name jf_2.img
2017-02-09 15:47:20.193775 3fff78ffea60 15 open_dir 0x3fff60024a80 on 0x3fff6c00cb90
2017-02-09 15:47:20.193779 3fff78ffea60 15 inode.get on 0x3fff6c00cb90 1.head now 5
2017-02-09 15:47:20.193788 3fff78ffea60 15 client.94152 link dir 0x3fff6c00cb90 'jf_2.img' to inode 0 dn 0x3fff60023b00 (new dn)
2017-02-09 15:47:20.193797 3fff78ffea60 20 client.94152 choose_target_mds starting with req->inode 1.head(faked_ino=0 ref=5 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.193868 3fff78ffea60 20 client.94152 choose_target_mds inode dir hash is 2 on jf_2.img => 4277851118
2017-02-09 15:47:20.193872 3fff78ffea60 20 client.94152 choose_target_mds 1.head(faked_ino=0 ref=5 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90) is_hash=1 hash=4277851118
2017-02-09 15:47:20.193894 3fff78ffea60 10 client.94152 choose_target_mds from caps on inode 1.head(faked_ino=0 ref=5 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.193978 3fff78ffea60 20 client.94152 mds is 0
2017-02-09 15:47:20.193984 3fff78ffea60 10 client.94152 send_request rebuilding request 4 for mds.0
2017-02-09 15:47:20.193990 3fff78ffea60 20 client.94152 encode_cap_releases enter (req: 0x3fff600244e0, mds: 0)
2017-02-09 15:47:20.193995 3fff78ffea60 20 client.94152 encode_dentry_release enter(dn:0x3fff60023b00)
2017-02-09 15:47:20.193998 3fff78ffea60 20 client.94152 encode_inode_release enter(in:1.head(faked_ino=0 ref=5 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x3fff6c00cb90), req:0x3fff600244e0 mds:0, drop:256, unless:512, have:, force:1)
2017-02-09 15:47:20.194023 3fff78ffea60 20 client.94152 send_request set sent_stamp to 2017-02-09 15:47:20.194023
2017-02-09 15:47:20.194027 3fff78ffea60 10 client.94152 send_request client_request(unknown.0:4 create #1/jf_2.img 2017-02-09 15:47:20.193795) v3 to mds.0
2017-02-09 15:47:20.194065 3fff78ffea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_request(client.94152:4 create #1/jf_2.img 2017-02-09 15:47:20.193795) v3 -- ?+0 0x3fff60025640 con 0x10000f69740
2017-02-09 15:47:20.194078 3fff78ffea60 20 client.94152 awaiting reply|forward|kick on 0x3fff78ffd6f8
2017-02-09 15:47:20.194089 3fff7d7fea60 10 client.94152 put_inode on 1.head(faked_ino=0 ref=5 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXs(0=pAsLsXs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.194109 3fff7d7fea60 15 inode.put on 0x3fff6c00cb90 1.head now 4
2017-02-09 15:47:20.203386 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 7 ==== client_reply(???:4 = 0 (0) Success safe) v1 ==== 667+0+0 (270369193 0 0) 0x3fff680012e0 con 0x10000f69740
2017-02-09 15:47:20.203400 3fff7d7fea60 20 client.94152 handle_client_reply got a reply. Safe:1 tid 4
2017-02-09 15:47:20.203402 3fff7d7fea60 10 client.94152 insert_trace from 2017-02-09 15:47:20.194023 mds.0 is_target=1 is_dentry=1
2017-02-09 15:47:20.203406 3fff7d7fea60 10 client.94152 features 0x7ffffffefdfbfff
2017-02-09 15:47:20.203408 3fff7d7fea60 10 client.94152 update_snap_trace len 48
2017-02-09 15:47:20.203410 3fff7d7fea60 20 client.94152 get_snap_realm 1 0x3fff6c00d5b0 1 -> 2
2017-02-09 15:47:20.203412 3fff7d7fea60 10 client.94152 update_snap_trace snaprealm(1 nref=2 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=1=[]) seq 1 <= 1 and same parent, SKIPPING
2017-02-09 15:47:20.203415 3fff7d7fea60 20 client.94152 put_snap_realm 1 0x3fff6c00d5b0 2 -> 1
2017-02-09 15:47:20.203418 3fff7d7fea60 10 client.94152 hrm is_target=1 is_dentry=1
2017-02-09 15:47:20.203427 3fff7d7fea60 12 client.94152 add_update_inode adding 10000000040.head(faked_ino=0 ref=0 ll_ref=0 cap_refs={} open={} mode=100000 size=0/0 mtime=0.000000 caps=- objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] 0x3fff6c011910) caps pLsFscr
2017-02-09 15:47:20.203454 3fff7d7fea60 10 client.94152 update_inode_file_bits 10000000040.head(faked_ino=0 ref=0 ll_ref=0 cap_refs={} open={} mode=100644 size=0/0 mtime=0.000000 caps=- objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] 0x3fff6c011910) - mtime 2017-02-09 15:47:20.193795
2017-02-09 15:47:20.203463 3fff7d7fea60 20 client.94152 get_snap_realm 1 0x3fff6c00d5b0 1 -> 2
2017-02-09 15:47:20.203466 3fff7d7fea60 15 client.94152 add_update_cap first one, opened snaprealm 0x3fff6c00d5b0
2017-02-09 15:47:20.203470 3fff7d7fea60 10 client.94152 add_update_cap issued - -> pLsFscr from mds.0 on 10000000040.head(faked_ino=0 ref=0 ll_ref=0 cap_refs={} open={} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] 0x3fff6c011910)
2017-02-09 15:47:20.203484 3fff7d7fea60 12 client.94152 add_update_inode had 1.head(faked_ino=0 ref=4 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXs(0=pAsLsXs) has_dir_layout 0x3fff6c00cb90) caps pAsLsXs
2017-02-09 15:47:20.203496 3fff7d7fea60 20 client.94152 dir hash is 2
2017-02-09 15:47:20.203498 3fff7d7fea60 10 client.94152 update_inode_file_bits 1.head(faked_ino=0 ref=4 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:29:15.934503 caps=pAsLsXs(0=pAsLsXs) has_dir_layout 0x3fff6c00cb90) pAsLsXs mtime 2017-02-09 15:47:20.193795
2017-02-09 15:47:20.203513 3fff7d7fea60 10 client.94152 add_update_cap issued pAsLsXs -> pAsLsXs from mds.0 on 1.head(faked_ino=0 ref=4 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pAsLsXs(0=pAsLsXs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.203529 3fff7d7fea60 20 client.94152 got dirfrag map for 1 frag * to mds 0
2017-02-09 15:47:20.203534 3fff7d7fea60 12 client.94152 insert_dentry_inode 'jf_2.img' vino 10000000040.head in dir 1.head dn 0x3fff60023b00
2017-02-09 15:47:20.203537 3fff7d7fea60 15 inode.get on 0x3fff6c011910 10000000040.head now 1
2017-02-09 15:47:20.203539 3fff7d7fea60 15 client.94152 link dir 0x3fff6c00cb90 'jf_2.img' to inode 0x3fff6c011910 dn 0x3fff60023b00 (old dn)
2017-02-09 15:47:20.203542 3fff7d7fea60 15 inode.get on 0x3fff6c011910 10000000040.head now 2
2017-02-09 15:47:20.203544 3fff7d7fea60 20 client.94152 link inode 0x3fff6c011910 parents now 0x3fff60023b00
2017-02-09 15:47:20.203546 3fff7d7fea60 10 client.94152 put_inode on 10000000040.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={} open={} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.203557 3fff7d7fea60 15 inode.put on 0x3fff6c011910 10000000040.head now 1
2017-02-09 15:47:20.203560 3fff7d7fea60 10 client.94152 got dentry lease on jf_2.img dur 30000ms ttl 2017-02-09 15:47:50.194023
2017-02-09 15:47:20.203565 3fff7d7fea60 15 inode.get on 0x3fff6c011910 10000000040.head now 2
2017-02-09 15:47:20.203566 3fff7d7fea60 20 client.94152 handle_client_reply signalling caller 0x3fff78ffd6f8
2017-02-09 15:47:20.203570 3fff7d7fea60 20 client.94152 handle_client_reply awaiting kickback on tid 4 0x3fff7d7fd828
2017-02-09 15:47:20.203579 3fff78ffea60 20 client.94152 sendrecv kickback on tid 4 0x3fff7d7fd828
2017-02-09 15:47:20.203583 3fff78ffea60 10 client.94152 make_request created ino 10000000040
2017-02-09 15:47:20.203585 3fff78ffea60 15 inode.get on 0x3fff6c011910 10000000040.head now 3
2017-02-09 15:47:20.203588 3fff78ffea60 20 client.94152 make_request target is 10000000040.head(faked_ino=0 ref=3 ll_ref=0 cap_refs={} open={} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.203601 3fff78ffea60 20 client.94152 lat 0.009577
2017-02-09 15:47:20.203610 3fff78ffea60 15 inode.get on 0x3fff6c011910 10000000040.head now 4
2017-02-09 15:47:20.203612 3fff78ffea60 10 client.94152 _create_fh 10000000040 mode 2
2017-02-09 15:47:20.203626 3fff78ffea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:20.203629 3fff78ffea60 3 client.94152 create(#1/jf_2.img, 0100644 layout 0 0 0) = 0
2017-02-09 15:47:20.203632 3fff78ffea60 10 client.94152 fill_stat on 10000000040 snap/devhead mode 0100644 mtime 2017-02-09 15:47:20.193795 ctime 2017-02-09 15:47:20.193795
2017-02-09 15:47:20.203638 3fff78ffea60 20 client.94152 ll_create created = 1
2017-02-09 15:47:20.203642 3fff78ffea60 3 client.94152 ll_create 0x3fff6c00cb90 jf_2.img 0100644 66113 = 0 (0x3fff60026530 10000000040)
2017-02-09 15:47:20.203645 3fff78ffea60 15 inode.get on 0x3fff6c011910 10000000040.head now 5
2017-02-09 15:47:20.203648 3fff78ffea60 20 client.94152 _ll_get 0x3fff6c011910 10000000040 -> 1
2017-02-09 15:47:20.203650 3fff78ffea60 10 client.94152 put_inode on 10000000040.head(faked_ino=0 ref=5 ll_ref=1 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.203661 3fff78ffea60 15 inode.put on 0x3fff6c011910 10000000040.head now 4
2017-02-09 15:47:20.203670 3fff7d7fea60 10 client.94152 put_inode on 10000000040.head(faked_ino=0 ref=4 ll_ref=1 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.203688 3fff7d7fea60 15 inode.put on 0x3fff6c011910 10000000040.head now 3
2017-02-09 15:47:20.203691 3fff7d7fea60 10 client.94152 put_inode on 1.head(faked_ino=0 ref=4 ll_ref=11 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pAsLsXs(0=pAsLsXs) has_dir_layout 0x3fff6c00cb90)
2017-02-09 15:47:20.203707 3fff7d7fea60 15 inode.put on 0x3fff6c00cb90 1.head now 3
2017-02-09 15:47:20.203715 3fff787fea60 20 client.94152 _ll_get 0x3fff6c011910 10000000040 -> 2
2017-02-09 15:47:20.203721 3fff787fea60 3 client.94152 ll_getattr 10000000040.head
2017-02-09 15:47:20.203724 3fff787fea60 10 client.94152 _getattr mask pAsLsXsFs issued=0
2017-02-09 15:47:20.203731 3fff787fea60 15 inode.get on 0x3fff6c011910 10000000040.head now 4
2017-02-09 15:47:20.203734 3fff787fea60 20 client.94152 choose_target_mds starting with req->inode 10000000040.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.203746 3fff787fea60 20 client.94152 choose_target_mds 10000000040.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910) is_hash=0 hash=0
2017-02-09 15:47:20.203757 3fff787fea60 10 client.94152 choose_target_mds from caps on inode 10000000040.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.203768 3fff787fea60 20 client.94152 mds is 0
2017-02-09 15:47:20.203771 3fff787fea60 10 client.94152 send_request rebuilding request 5 for mds.0
2017-02-09 15:47:20.203774 3fff787fea60 20 client.94152 encode_cap_releases enter (req: 0x3fff54002770, mds: 0)
2017-02-09 15:47:20.203776 3fff787fea60 20 client.94152 send_request set sent_stamp to 2017-02-09 15:47:20.203775
2017-02-09 15:47:20.203779 3fff787fea60 10 client.94152 send_request client_request(unknown.0:5 getattr pAsLsXsFs #10000000040 2017-02-09 15:47:20.203733) v3 to mds.0
2017-02-09 15:47:20.203787 3fff787fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_request(client.94152:5 getattr pAsLsXsFs #10000000040 2017-02-09 15:47:20.203733) v3 -- ?+0 0x3fff54003280 con 0x10000f69740
2017-02-09 15:47:20.203802 3fff787fea60 20 client.94152 awaiting reply|forward|kick on 0x3fff787fd8f8
2017-02-09 15:47:20.203810 3fff78ffea60 3 client.94152 ll_forget 1 1
2017-02-09 15:47:20.204857 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 8 ==== client_reply(???:5 = 0 (0) Success) v1 ==== 354+0+0 (3205946329 0 0) 0x3fff680012e0 con 0x10000f69740
2017-02-09 15:47:20.204870 3fff7d7fea60 20 client.94152 handle_client_reply got a reply. Safe:1 tid 5
2017-02-09 15:47:20.204872 3fff7d7fea60 10 client.94152 insert_trace from 2017-02-09 15:47:20.203775 mds.0 is_target=1 is_dentry=0
2017-02-09 15:47:20.204875 3fff7d7fea60 10 client.94152 features 0x7ffffffefdfbfff
2017-02-09 15:47:20.204877 3fff7d7fea60 10 client.94152 update_snap_trace len 48
2017-02-09 15:47:20.204880 3fff7d7fea60 20 client.94152 get_snap_realm 1 0x3fff6c00d5b0 2 -> 3
2017-02-09 15:47:20.204881 3fff7d7fea60 10 client.94152 update_snap_trace snaprealm(1 nref=3 c=0 seq=1 parent=0 my_snaps=[] cached_snapc=1=[]) seq 1 <= 1 and same parent, SKIPPING
2017-02-09 15:47:20.204885 3fff7d7fea60 20 client.94152 put_snap_realm 1 0x3fff6c00d5b0 3 -> 2
2017-02-09 15:47:20.204887 3fff7d7fea60 10 client.94152 hrm is_target=1 is_dentry=0
2017-02-09 15:47:20.204891 3fff7d7fea60 12 client.94152 add_update_inode had 10000000040.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pLsFscr(0=pLsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910) caps pAsLsXsFscr
2017-02-09 15:47:20.204907 3fff7d7fea60 10 client.94152 add_update_cap issued pLsFscr -> pAsLsXsFscr from mds.0 on 10000000040.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.204925 3fff7d7fea60 15 inode.get on 0x3fff6c011910 10000000040.head now 5
2017-02-09 15:47:20.204928 3fff7d7fea60 20 client.94152 handle_client_reply signalling caller 0x3fff787fd8f8
2017-02-09 15:47:20.204931 3fff7d7fea60 20 client.94152 handle_client_reply awaiting kickback on tid 5 0x3fff7d7fd828
2017-02-09 15:47:20.204935 3fff787fea60 20 client.94152 sendrecv kickback on tid 5 0x3fff7d7fd828
2017-02-09 15:47:20.204939 3fff787fea60 20 client.94152 lat 0.001162
2017-02-09 15:47:20.204941 3fff787fea60 10 client.94152 _getattr result=0
2017-02-09 15:47:20.204944 3fff787fea60 10 client.94152 fill_stat on 10000000040 snap/devhead mode 0100644 mtime 2017-02-09 15:47:20.193795 ctime 2017-02-09 15:47:20.193795
2017-02-09 15:47:20.204949 3fff787fea60 3 client.94152 ll_getattr 10000000040.head = 0
2017-02-09 15:47:20.204956 3fff7d7fea60 10 client.94152 put_inode on 10000000040.head(faked_ino=0 ref=5 ll_ref=2 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.204976 3fff7d7fea60 15 inode.put on 0x3fff6c011910 10000000040.head now 4
2017-02-09 15:47:20.204979 3fff7d7fea60 10 client.94152 put_inode on 10000000040.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910)
2017-02-09 15:47:20.204996 3fff7d7fea60 15 inode.put on 0x3fff6c011910 10000000040.head now 3
2017-02-09 15:47:20.205003 3fff787fea60 3 client.94152 ll_forget 10000000040 1
2017-02-09 15:47:20.205006 3fff787fea60 20 client.94152 _ll_put 0x3fff6c011910 10000000040 1 -> 1
2017-02-09 15:47:20.205014 3fff5bffea60 3 client.94152 ll_flush 0x3fff60026530 10000000040
2017-02-09 15:47:20.205018 3fff5bffea60 10 client.94152 _flush: 0x3fff60026530 on inode 10000000040.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} open={2=1} mode=100644 size=0/0 mtime=2017-02-09 15:47:20.193795 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[10000000040 ts 0/0 objects 0 dirty_or_tx 0] parents=0x3fff60023b00 0x3fff6c011910) no async_err state
2017-02-09 15:47:20.378688 3fff78ffea60 20 client.94152 _ll_get 0x3fff6c011910 10000000040 -> 2
2017-02-09 15:47:20.378711 3fff78ffea60 3 client.94152 ll_getxattr 10000000040.head security.capability size 0
2017-02-09 15:47:20.378720 3fff78ffea60 10 client.94152 _getattr mask Xs issued=1
2017-02-09 15:47:20.378729 3fff78ffea60 3 client.94152 _getxattr(10000000040, "security.capability", 0) = -61
2017-02-09 15:47:20.378744 3fff78ffea60 3 client.94152 ll_forget 10000000040 1
2017-02-09 15:47:20.378748 3fff78ffea60 20 client.94152 _ll_put 0x3fff6c011910 10000000040 1 -> 1
2017-02-09 15:47:20.378829 3fff787fea60 3 client.94152 ll_write 0x3fff60026530 10000000040 0~131072
2017-02-09 15:47:20.378847 3fff787fea60 10 client.94152 cur file size is 0
2017-02-09 15:47:20.379025 3fff787fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.81:6800/8250 -- osd_op(client.94152.0:1 1.91c50563 10000000040.00000000 [stat] snapc 0=[] ack+write+known_if_redirected e99) v7 -- ?+0 0x3fff54027220 con 0x3fff54025e40
2017-02-09 15:47:20.379083 3fff787fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.81:6800/8250 -- osd_op(client.94152.0:2 1.91c50563 10000000040.00000000 [create 0~0 [excl]] snapc 0=[] ack+write+known_if_redirected e99) v7 -- ?+0 0x3fff54027d00 con 0x3fff54025e40
2017-02-09 15:47:20.380701 3fff7d7fea60 10 client.94152 ms_handle_connect on 10.162.2.81:6800/8250
2017-02-09 15:47:20.381828 3fff8427ea60 1 -- 10.161.26.49:0/2037822863 <== osd.3 10.162.2.81:6800/8250 1 ==== osd_op_reply(1 10000000040.00000000 [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 140+0+0 (381197315 0 0) 0x3fff40000a30 con 0x3fff54025e40
2017-02-09 15:47:20.426543 3fff8427ea60 1 -- 10.161.26.49:0/2037822863 <== osd.3 10.162.2.81:6800/8250 2 ==== osd_op_reply(2 10000000040.00000000 [create 0~0 [excl]] v99'5437 uv5437 ack = 0) v7 ==== 140+0+0 (2687690268 0 0) 0x3fff40000a30 con 0x3fff54025e40
2017-02-09 15:47:20.426573 3fff787fea60 10 client.94152 waiting for caps need Fw want Fb
2017-02-09 15:47:21.015056 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:22.015129 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:23.015221 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:24.015286 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:25.015369 3fff7e7fea60 10 client.94152 renew_caps()
2017-02-09 15:47:25.015383 3fff7e7fea60 15 client.94152 renew_caps requesting from mds.0
2017-02-09 15:47:25.015387 3fff7e7fea60 10 client.94152 renew_caps mds.0
2017-02-09 15:47:25.015392 3fff7e7fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_session(request_renewcaps seq 3) v2 -- ?+0 0x3fff5c0023d0 con 0x10000f69740
2017-02-09 15:47:25.015415 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:25.016249 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 9 ==== client_session(renewcaps seq 3) v1 ==== 28+0+0 (3936844645 0 0) 0x3fff680012e0 con 0x10000f69740
2017-02-09 15:47:25.016264 3fff7d7fea60 10 client.94152 handle_client_session client_session(renewcaps seq 3) v1 from mds.0
2017-02-09 15:47:25.016284 3fff787fea60 10 client.94152 waiting for caps need Fw want Fb
2017-02-09 15:47:26.015443 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:27.015535 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:28.015612 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:29.015676 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:30.015758 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:31.015851 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:32.015934 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:33.016006 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:34.016072 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:35.016169 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:36.016245 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:37.016336 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:38.016429 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:39.016504 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:40.016577 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:41.016653 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:42.016744 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:43.016809 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:44.016886 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:45.016952 3fff7e7fea60 10 client.94152 renew_caps()
2017-02-09 15:47:45.016956 3fff7e7fea60 15 client.94152 renew_caps requesting from mds.0
2017-02-09 15:47:45.016960 3fff7e7fea60 10 client.94152 renew_caps mds.0
2017-02-09 15:47:45.016964 3fff7e7fea60 1 -- 10.161.26.49:0/2037822863 --> 10.162.2.78:6800/1968 -- client_session(request_renewcaps seq 4) v2 -- ?+0 0x3fff5c0038f0 con 0x10000f69740
2017-02-09 15:47:45.016986 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:45.017890 3fff7d7fea60 1 -- 10.161.26.49:0/2037822863 <== mds.0 10.162.2.78:6800/1968 10 ==== client_session(renewcaps seq 4) v1 ==== 28+0+0 (3279728092 0 0) 0x3fff680012e0 con 0x10000f69740
2017-02-09 15:47:45.017905 3fff7d7fea60 10 client.94152 handle_client_session client_session(renewcaps seq 4) v1 from mds.0
2017-02-09 15:47:45.017924 3fff787fea60 10 client.94152 waiting for caps need Fw want Fb
2017-02-09 15:47:46.017021 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:47.017086 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:48.017170 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:49.017245 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:50.017334 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:51.017426 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:52.017504 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:53.017598 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:54.017674 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:55.017764 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:56.017842 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:57.017934 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
2017-02-09 15:47:58.018027 3fff7e7fea60 20 client.94152 trim_cache size 1 max 16384
(1-1/2)