Project

General

Profile

Bug #20334

I/O become slowly when multi mds which subtree root has replica

Added by yanmei ding over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

History

#1 Updated by Greg Farnum over 4 years ago

  • Project changed from Ceph to CephFS

#2 Updated by John Spray over 4 years ago

Yanmei Ding: can you be more specific about how to reproduce this or what is going wrong internally?

#3 Updated by yanmei ding over 4 years ago

John Spray wrote:

Yanmei Ding: can you be more specific about how to reproduce this or what is going wrong internally?

you can pin dir to mds,it has replica,then,write or mkdir in this dir,you can see,it is slow,about 5s,I think it does not execute mdslog::flush() immediately.

#4 Updated by Zheng Yan over 4 years ago

yanmei ding wrote:

John Spray wrote:

Yanmei Ding: can you be more specific about how to reproduce this or what is going wrong internally?

you can pin dir to mds,it has replica,then,write or mkdir in this dir,you can see,it is slow,about 5s,I think it does not execute mdslog::flush() immediately.

I can't reproduce this for code in master branch. which version did you test?

#5 Updated by yanmei ding over 4 years ago

Zheng Yan wrote:

yanmei ding wrote:

John Spray wrote:

Yanmei Ding: can you be more specific about how to reproduce this or what is going wrong internally?

you can pin dir to mds,it has replica,then,write or mkdir in this dir,you can see,it is slow,about 5s,I think it does not execute mdslog::flush() immediately.

I can't reproduce this for code in master branch. which version did you test?

v12.0.3

#6 Updated by John Spray over 4 years ago

yanmei ding: there have been fixes on master since 12.0.3, please could you retest with the tip of master?

#7 Updated by yanmei ding over 4 years ago

John Spray wrote:

yanmei ding: there have been fixes on master since 12.0.3, please could you retest with the tip of master?

I can only use this version, can you tell me how to fix it?

#8 Updated by John Spray over 4 years ago

In that case I suggest you wait for 12.1.0 to see if the issue is fixed there.

#9 Updated by yanmei ding over 4 years ago

John Spray wrote:

In that case I suggest you wait for 12.1.0 to see if the issue is fixed there.

John Spray: I test the master,and I found that,problem ("I/O become slowly when multi mds which subtree root has replica") has been resolved ,but When a directory is fragmented, and the fragmented directory forms a subtree,it is still slow.
Here is a part of my log.

  • subtrees *******************************************
    2017-06-23 09:32:49.464943 7efe81bfb700 15 mds.1.cache show_subtrees
    2017-06-23 09:32:49.464947 7efe84b7c700 10 -- 192.168.87.83:6832/2030350718 >> 192.168.87.82:6828/3077785281 conn(0x5581e55d0800 :-1 s=STATE_OPEN pgs=38 cs=1 l=0).handle_write
    2017-06-23 09:32:49.464954 7efe84b7c700 20 -- 192.168.87.83:6832/2030350718 >> 192.168.87.82:6828/3077785281 conn(0x5581e55d0800 :-1 s=STATE_OPEN pgs=38 cs=1 l=0).prepare_send_message m HB magic: 0 v1
    2017-06-23 09:32:49.464958 7efe84b7c700 20 -- 192.168.87.83:6832/2030350718 >> 192.168.87.82:6828/3077785281 conn(0x5581e55d0800 :-1 s=STATE_OPEN pgs=38 cs=1 l=0).prepare_send_message encoding features 1152323339925389307 0x5581e4f83b80 HB magic: 0 v1
    2017-06-23 09:32:49.464980 7efe84b7c700 20 -- 192.168.87.83:6832/2030350718 >> 192.168.87.82:6828/3077785281 conn(0x5581e55d0800 :-1 s=STATE_OPEN pgs=38 cs=1 l=0).write_message signed m=0x5581e4f83b80): sig = 5890328181927825083
    2017-06-23 09:32:49.464973 7efe81bfb700 10 mds.1.cache |______ 1 auth [dir 101 ~mds1/ [2,head] auth v=7 cv=0/0 dir_auth=1 state=1610612738|complete f(v0 10=0+10) n(v1 rc2017-06-23 09:29:36.932918 b68 11=1+10) hs=10+0,ss=0+0 dirty=1 | child=1 subtree=1 subtreetemp=0 dirty=1 waiter=0 authpin=0 0x5581e4d376c0]
    2017-06-23 09:32:49.464988 7efe84b7c700 20 -- 192.168.87.83:6832/2030350718 >> 192.168.87.82:6828/3077785281 conn(0x5581e55d0800 :-1 s=STATE_OPEN pgs=38 cs=1 l=0).write_message sending message type=1280 src mds.1 front=366 data=0 off 0
    2017-06-23 09:32:49.464994 7efe84b7c700 20 -- 192.168.87.83:6832/2030350718 >> 192.168.87.82:6828/3077785281 conn(0x5581e55d0800 :-1 s=STATE_OPEN pgs=38 cs=1 l=0).write_message sending 3722 0x5581e4f83b80
    2017-06-23 09:32:49.464993 7efe81bfb700 10 mds.1.cache |_.____ 0 rep [dir 1 / [2,head] REP dir_auth=0 state=0 f(v0 m2017-06-22 18:11:58.880601 3=0+3) n(v153 rc2017-06-23 09:07:41.996982 b20450656318 4992864=4992851+13)/n(v153 rc2017-06-23 09:06:35.268985 b20444839998 4991446=4991433+13) hs=1+0,ss=0+0 | child=1 subtree=1 subtreetemp=0 0x5581e4d37d40]
    2017-06-23 09:32:49.465007 7efe84b7c700 10 -- 192.168.87.83:6832/2030350718 >> 192.168.87.82:6828/3077785281 conn(0x5581e55d0800 :-1 s=STATE_OPEN pgs=38 cs=1 l=0)._try_send sent bytes 441 remaining bytes 0
    2017-06-23 09:32:49.465011 7efe84b7c700 10 -- 192.168.87.83:6832/2030350718 >> 192.168.87.82:6828/3077785281 conn(0x5581e55d0800 :-1 s=STATE_OPEN pgs=38 cs=1 l=0).write_message sending 0x5581e4f83b80 done.
    2017-06-23 09:32:49.465008 7efe81bfb700 10 mds.1.cache |_.__ 1 auth [dir 100000003f1 /data/NFS1/test_agaim/ [2,head] auth{0=7} pv=29836176 v=29836174 cv=0/0 REP dir_auth=1 ap=1+3+4 state=1610612738|complete f(v11 m2017-06-23 09:30:56.632914 8=5+3) n(v10676 rc2017-06-23 09:32:15.772910 b20423757824 4986306=4986296+10) hs=8+2,ss=0+0 dirty=9 | child=1 subtree=1 subtreetemp=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5581e4d383c0]
    2017-06-23 09:32:49.465026 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.000001100111* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1257=1257+0) n(v4769 b5148672 1257=1257+0) hs=0+0,ss=0+0 | subtree=1 0x5581f7dab6c0]
    2017-06-23 09:32:49.465036 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001011011111* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1230=1230+0) n(v4769 b5038080 1230=1230+0) hs=0+0,ss=0+0 | subtree=1 0x5581f79ae700]
    2017-06-23 09:32:49.465042 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001011011110* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1223=1223+0) n(v4769 b5009408 1223=1223+0) hs=0+0,ss=0+0 | subtree=1 0x5581f4993a80]
    2017-06-23 09:32:49.465050 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001011011101* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1261=1261+0) n(v4769 b5165056 1261=1261+0) hs=0+0,ss=0+0 | subtree=1 0x5581f4993740]
    2017-06-23 09:32:49.465056 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001011011100* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1207=1207+0) n(v4769 b4943872 1207=1207+0) hs=0+0,ss=0+0 | subtree=1 0x5581f4993400]
    2017-06-23 09:32:49.465063 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001011011011* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1272=1272+0) n(v4769 b5210112 1272=1272+0) hs=0+0,ss=0+0 | subtree=1 0x5581f49930c0]
    2017-06-23 09:32:49.465069 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001011011010* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1232=1232+0) n(v4769 b5046272 1232=1232+0) hs=0+0,ss=0+0 | subtree=1 0x5581f4992d80]
    2017-06-23 09:32:49.465076 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001011011001* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1267=1267+0) n(v4769 b5189632 1267=1267+0) hs=0+0,ss=0+0 | subtree=1 0x5581f4992a40]
    2017-06-23 09:32:49.465083 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.000001100001* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1159=1159+0) n(v4769 b4747264 1159=1159+0) hs=0+0,ss=0+0 | subtree=1 0x5581f3ac7a80]
    2017-06-23 09:32:49.465089 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.000001100010* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1252=1252+0) n(v4769 b5128192 1252=1252+0) hs=0+0,ss=0+0 | subtree=1 0x5581f3ac7740]
    2017-06-23 09:32:49.465095 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.000001100011* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1190=1190+0) n(v4769 b4874240 1190=1190+0) hs=0+0,ss=0+0 | subtree=1 0x5581f3ac7400]
    2017-06-23 09:32:49.465101 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.000001100100* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1197=1197+0) n(v4769 b4902912 1197=1197+0) hs=0+0,ss=0+0 | subtree=1 0x5581f1dfda80]
    2017-06-23 09:32:49.465109 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.000001100101* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1177=1177+0) n(v4769 b4820992 1177=1177+0) hs=0+0,ss=0+0 | subtree=1 0x5581f1dfd740]
    2017-06-23 09:32:49.465115 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001111111010* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1184=1184+0) n(v4769 b4849664 1184=1184+0) hs=0+0,ss=0+0 | subtree=1 0x5581e6b85a80]
    2017-06-23 09:32:49.465122 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001111111011* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1309=1309+0) n(v4769 b5361664 1309=1309+0) hs=0+0,ss=0+0 | subtree=1 0x5581e6b85740]
    2017-06-23 09:32:49.465127 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001111111100* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1150=1150+0) n(v4769 b4710400 1150=1150+0) hs=0+0,ss=0+0 | subtree=1 0x5581e6b85400]
    2017-06-23 09:32:49.465135 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001111111101* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1234=1234+0) n(v4769 b5054464 1234=1234+0) hs=0+0,ss=0+0 | subtree=1 0x5581e6b850c0]
    2017-06-23 09:32:49.465141 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001111111110* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1164=1164+0) n(v4769 b4767744 1164=1164+0) hs=0+0,ss=0+0 | subtree=1 0x5581e6b84d80]
    2017-06-23 09:32:49.465148 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001111111111* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1279=1279+0) n(v4769 b5238784 1279=1279+0) hs=0+0,ss=0+0 | subtree=1 0x5581e6b84a40]
    2017-06-23 09:32:49.465153 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001111111001* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1222=1222+0) n(v4769 b5005312 1222=1222+0) hs=0+0,ss=0+0 | subtree=1 0x5581e6b2e000]
    2017-06-23 09:32:49.465159 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001111111000* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1195=1195+0) n(v4769 b4894720 1195=1195+0)/n(v4769 rc2017-06-23 09:16:12.996957 b4894720 1195=1195+0) hs=0+0,ss=0+0 | subtree=1 0x5581e52ab0c0]
    2017-06-23 09:32:49.465168 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.001011011000* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1200=1200+0) n(v4769 b4915200 1200=1200+0)/n(v4769 rc2017-06-23 09:07:20.332983 b4915200 1200=1200+0) hs=0+0,ss=0+0 | subtree=1 0x5581e51bb380]
    2017-06-23 09:32:49.465177 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.000001100110* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1237=1237+0) n(v4769 b5066752 1237=1237+0) hs=0+0,ss=0+0 | subtree=1 0x5581e512b6c0]
    2017-06-23 09:32:49.465183 7efe81bfb700 10 mds.1.cache |__ 0 rep [dir 100000003f2.000001100000* /data/NFS1/test_agaim/vdb.1_1.dir/ [2,head] dir_auth=0 state=0 f(v4302 1264=1264+0) n(v4769 b5177344 1264=1264+0)/n(v4769 rc2017-06-23 09:07:41.528982 b5177344 1264=1264+0) hs=0+0,ss=0+0 | subtree=1 0x5581e5119400]
  • subtrees *******************************************
  • create file under [/data/NFS1/test_agaim] fast **********************
    2017-06-23 09:30:56.635446 7efe81bfb700 4 mds.1.server handle_client_request client_request(client.224716:12337764 create #100000003f1/laura 2017-06-23 09:30:56.632914 caller_uid=0, caller_gid=0{}) v2
    2017-06-23 09:30:56.635456 7efe81bfb700 20 mds.1.server get_session have 0x5581e4d37380 client.224716 192.168.87.81:0/1136778258 state open
    2017-06-23 09:30:56.635461 7efe81bfb700 15 mds.1.server oldest_client_tid=12337764
    2017-06-23 09:30:56.635470 7efe81bfb700 7 mds.1.cache request_start request(client.224716:12337764 cr=0x5581e625c400)
    2017-06-23 09:30:56.635476 7efe81bfb700 7 mds.1.server dispatch_client_request client_request(client.224716:12337764 create #100000003f1/laura 2017-06-23 09:30:56.632914 caller_uid=0, caller_gid=0{}) v2
    2017-06-23 09:30:56.635482 7efe81bfb700 7 mds.1.server open w/ O_CREAT on #100000003f1/laura

2017-06-23 09:30:56.643636 7efe81bfb700 7 mds.1.locker handle_file_lock a=sync on (ifile mix w=1 dirty) from mds.0 [inode 100000003f1 [...2,head] /data/NFS1/test_agaim/ v25218 f(v10 m2017-06-23 09:30:18.808916 7=4+3) n(v10673 rc2017-06-23 09:30:18.808916 b20423757824 4986304=4986293+11)/n(v10514 rc2017-06-23 09:07:41.996982 b20421586944 4985754=4985750+4) (inest mix w=1 dirty) (ifile mix w=1 dirty) (iversion lock) caps={224716=pAsLsXs/p@7} | dirtyscattered=2 request=1 lock=2 dirfrag=1 caps=1 waiter=0 0x5581e4da9200]
2017-06-23 09:30:56.643666 7efe81bfb700 10 mds.1.locker eval_gather (ifile mix->sync w=1 dirty) on [inode 100000003f1 [...2,head] /data/NFS1/test_agaim/ v25218 f(v10 m2017-06-23 09:30:18.808916 7=4+3) n(v10673 rc2017-06-23 09:30:18.808916 b20423757824 4986304=4986293+11)/n(v10514 rc2017-06-23 09:07:41.996982 b20421586944 4985754=4985750+4) (inest mix w=1 dirty) (ifile mix->sync w=1 dirty) (iversion lock) caps={224716=pAsLsXs/p@7} | dirtyscattered=2 request=1 lock=2 dirfrag=1 caps=1 waiter=0 0x5581e4da9200]
2017-06-23 09:30:56.643683 7efe81bfb700 10 mds.1.locker next state is sync issued/allows loner /scr xlocker /scr other /scr
2017-06-23 09:30:56.643688 7efe81bfb700 10 mds.1.journaler.mdlog(rw) _do_flush flushing 38330390160~1658

2017-06-23 09:30:56.645511 7efe7bbef700 20 mds.1.bal hit_dir 1 pop is 9, frag * size 8
2017-06-23 09:30:56.645511 7efe83b7a700 20 -- 192.168.87.83:6832/2030350718 >> 192.168.87.81:0/1136778258 conn(0x5581e55dd000 :6832 s=STATE_OPEN pgs=676 cs=1 l=0).prepare_send_message m client_caps(grant ino 2000048a5dc 18352451 seq 2 caps=pAsxLsXsxFsxcrwb dirty=- wanted=pAsxXsxFxwb follows 0 size 0/4194304 ts 1/18446744073709551615 mtime 2017-06-23 09:30:56.632914) v10
2017-06-23 09:30:56.645524 7efe7bbef700 7 mds.1.server reply_client_request 0 ((0) Success) client_request(client.224716:12337764 create #100000003f1/laura 2017-06-23 09:30:56.632914 caller_uid=0, caller_gid=0{}) v2
  • create file fast ***********************************
  • create file under [/data/NFS1/test_agaim/vdb.1_1.dir] slowly **********************
    2017-06-23 09:32:48.575874 7efe81bfb700 4 mds.1.server handle_client_request client_request(client.224716:12338918 create #100000003f2/laura 2017-06-23 09:32:48.572909 caller_uid=0, caller_gid=0{}) v2
    2017-06-23 09:32:48.575883 7efe81bfb700 20 mds.1.server get_session have 0x5581e4d37380 client.224716 192.168.87.81:0/1136778258 state open
    2017-06-23 09:32:48.575887 7efe81bfb700 15 mds.1.server oldest_client_tid=12338918
    2017-06-23 09:32:48.575897 7efe81bfb700 7 mds.1.cache request_start request(client.224716:12338918 cr=0x5581e625a000)
    2017-06-23 09:32:48.575902 7efe81bfb700 7 mds.1.server dispatch_client_request client_request(client.224716:12338918 create #100000003f2/laura 2017-06-23 09:32:48.572909 caller_uid=0, caller_gid=0{}) v2
    2017-06-23 09:32:48.575921 7efe81bfb700 7 mds.1.server open w/ O_CREAT on #100000003f2/laura
    2017-06-23 09:32:48.575923 7efe81bfb700 7 mds.1.cache traverse: opening base ino 100000003f2 snap head
    2017-06-23 09:32:48.575927 7efe81bfb700 12 mds.1.cache traverse: path seg depth 0 'laura' snapid head
    2017-06-23 09:32:48.575937 7efe81bfb700 20 mds.1.cache.dir(100000003f2.011010101011*) lookup (head, 'laura')

2017-06-23 09:32:51.227585 7efe81bfb700 10 mds.beacon.mds2 handle_mds_beacon up:active seq 380 rtt 0.000785
2017-06-23 09:32:51.227588 7efe81bfb700 10 -- 192.168.87.83:6832/2030350718 dispatch_throttle_release 129 to dispatch throttler 129/104857600
2017-06-23 09:32:51.227592 7efe81bfb700 20 -- 192.168.87.83:6832/2030350718 done calling dispatch on 0x558200ac83c0
2017-06-23 09:32:53.231641 7efe7f3f6700 10 mds.1.journaler.mdlog(rw) _do_flush flushing 38330406859~7788

2017-06-23 09:32:53.233969 7efe7bbef700 20 mds.1.bal hit_dir 1 pop is 1.25, frag 011010101011* size 1197
2017-06-23 09:32:53.233967 7efe83b7a700 20 -- 192.168.87.83:6832/2030350718 >> 192.168.87.81:0/1136778258 conn(0x5581e55dd000 :6832 s=STATE_OPEN pgs=676 cs=1 l=0).prepare_send_message m client_caps(grant ino 2000048a5df 18352457 seq 2 caps=pAsxLsXsxFsxcrwb dirty=- wanted=pAsxXsxFxwb follows 0 size 0/4194304 ts 1/18446744073709551615 mtime 2017-06-23 09:32:48.572909) v10
2017-06-23 09:32:53.233982 7efe7bbef700 7 mds.1.server reply_client_request 0 ((0) Success) client_request(client.224716:12338918 create #100000003f2/laura 2017-06-23 09:32:48.572909 caller_uid=0, caller_gid=0{}) v2
  • create file slowly ***********************************

#10 Updated by Zheng Yan over 4 years ago

please upload detailed log for the slow case.

#11 Updated by yanmei ding over 4 years ago

Zheng Yan wrote:

please upload detailed log for the slow case.

This is a detailed log.
Thank you!

#12 Updated by yanmei ding over 4 years ago

yanmei ding wrote:

Zheng Yan wrote:

please upload detailed log for the slow case.

This is a detailed log.
Thank you!

#13 Updated by yanmei ding over 4 years ago

yanmei ding wrote:

yanmei ding wrote:

Zheng Yan wrote:

please upload detailed log for the slow case.

This is a detailed log.
Thank you!

I can't see the log,maybe I can't upload file

#14 Updated by Zheng Yan over 4 years ago

try uploading it somewhere else or send it to my email

#15 Updated by Patrick Donnelly over 4 years ago

  • Status changed from New to Need More Info

#16 Updated by yanmei ding over 4 years ago

v12.1.0 has solved the problem,please help close this issue,thank you!

#17 Updated by Patrick Donnelly over 4 years ago

  • Status changed from Need More Info to Resolved

Thanks for letting us know!

Also available in: Atom PDF