Project

General

Profile

Actions
Copy link

Bug #49365

closed

octopus: qa: "Cannot write to 'pjd-fstest-20090130-RC-aclfixes.tgz' (Invalid argument)."

Added by Patrick Donnelly about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jeff Layton
Category:
-
Target version:
Ceph - v17.0.0
% Done:

0%

Source:
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

2021-02-18T19:29:06.553 INFO:tasks.workunit:Running workunit suites/pjd.sh...
2021-02-18T19:29:06.554 DEBUG:teuthology.orchestra.run.smithi149:workunit test suites/pjd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ef0cbaddda96a295b3751035095dce0a63604552 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh
2021-02-18T19:29:06.638 INFO:tasks.workunit.client.0.smithi149.stderr:--2021-02-18 19:29:06--  http://download.ceph.com/qa/pjd-fstest-20090130-RC-aclfixes.tgz
2021-02-18T19:29:11.074 INFO:tasks.workunit.client.0.smithi149.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1
2021-02-18T19:29:11.075 INFO:tasks.workunit.client.0.smithi149.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected.
2021-02-18T19:29:11.075 INFO:tasks.workunit.client.0.smithi149.stderr:HTTP request sent, awaiting response... 200 OK
2021-02-18T19:29:11.075 INFO:tasks.workunit.client.0.smithi149.stderr:Length: 45586 (45K) [application/octet-stream]
2021-02-18T19:29:11.075 INFO:tasks.workunit.client.0.smithi149.stderr:Saving to: 'pjd-fstest-20090130-RC-aclfixes.tgz'
2021-02-18T19:29:11.076 INFO:tasks.workunit.client.0.smithi149.stderr:
2021-02-18T19:29:11.076 INFO:tasks.workunit.client.0.smithi149.stderr:     0K                                                         0% 0.00 =0s
2021-02-18T19:29:11.076 INFO:tasks.workunit.client.0.smithi149.stderr:
2021-02-18T19:29:11.076 INFO:tasks.workunit.client.0.smithi149.stderr:
2021-02-18T19:29:11.076 INFO:tasks.workunit.client.0.smithi149.stderr:Cannot write to 'pjd-fstest-20090130-RC-aclfixes.tgz' (Invalid argument).
2021-02-18T19:29:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 3
2021-02-18T19:29:11.077 INFO:tasks.workunit:Stopping ['suites/pjd.sh'] on client.0...

From: /ceph/teuthology-archive/yuriw-2021-02-18_17:59:30-kcephfs-octopus-testing-basic-smithi/5892872/teuthology.log

Repeated run of: /ceph/teuthology-archive/yuriw-2021-02-16_18:11:32-kcephfs-octopus-testing-basic-smithi/5888305/teuthology.log

This is with k-testing branch. May be some obscure bug?

Actions
Copy link
 #1

Updated by Patrick Donnelly about 3 years ago

I think this might be because of inline feature getting turned on. Same failure/error for dbench:

/ceph/teuthology-archive/yuriw-2021-02-16_18:11:32-kcephfs-octopus-testing-basic-smithi/5888328/teuthology.log

Actions
Copy link
 #2

Updated by Patrick Donnelly about 3 years ago

  • Status changed from New to In Progress
2021-02-18T02:24:58.661159+00:00 smithi163 kernel: [  558.670041] ------------[ cut here ]------------
2021-02-18T02:24:58.661205+00:00 smithi163 kernel: [  558.670069] ceph: write_begin called on still-inlined inode!
2021-02-18T02:24:58.661208+00:00 smithi163 kernel: [  558.670099] WARNING: CPU: 2 PID: 11937 at fs/ceph/addr.c:1236 ceph_write_begin+0x237/0x290 [ceph]
2021-02-18T02:24:58.661209+00:00 smithi163 kernel: [  558.670129] Modules linked in: ceph libceph fscache netfs xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc xfs overlay intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal ipmi_ssif intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper joydev lpc_ich mei_me mei wmi acpi_ipmi ipmi_si kvm_intel ipmi_devintf ipmi_msghandler acpi_power_meter ioatdma acpi_pad kvm irqbypass sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi lp parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c hid_generic usbhid hid igb i2c_algo_bit ixgbe dca nvme ahci ptp libahci nvme_core pps_core mdio
2021-02-18T02:24:58.661228+00:00 smithi163 kernel: [  558.670348] CPU: 2 PID: 11937 Comm: wget Not tainted 5.11.0-ceph-ge25079bca433 #1
2021-02-18T02:24:58.661230+00:00 smithi163 kernel: [  558.670353] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016
2021-02-18T02:24:58.661232+00:00 smithi163 kernel: [  558.670356] RIP: 0010:ceph_write_begin+0x237/0x290 [ceph]
2021-02-18T02:24:58.661233+00:00 smithi163 kernel: [  558.670375] Code: ff ff ff 41 bc ea ff ff ff e9 5c ff ff ff 48 c7 c7 10 82 14 c1 44 89 44 24 0c 48 89 34 24 c6 05 2f cb 05 00 01 e8 0a a3 97 e4 <0f> 0b 48 8b 34 24 44 8b 44 24 0c 48 85 f6 0f 85 10 fe ff ff 48 8b
2021-02-18T02:24:58.661235+00:00 smithi163 kernel: [  558.670379] RSP: 0018:ffffa973c1277c30 EFLAGS: 00010286
2021-02-18T02:24:58.661247+00:00 smithi163 kernel: [  558.670385] RAX: 0000000000000000 RBX: ffffa973c1277ca0 RCX: 0000000000000027
2021-02-18T02:24:58.661249+00:00 smithi163 kernel: [  558.670388] RDX: 0000000000000027 RSI: 0000000000000000 RDI: ffff8df11fc98ff8
2021-02-18T02:24:58.661251+00:00 smithi163 kernel: [  558.670391] RBP: ffff8dea80eb9038 R08: ffff8df11fc98ff0 R09: 0000000000000001
2021-02-18T02:24:58.661252+00:00 smithi163 kernel: [  558.670394] R10: 0000000000000001 R11: 0000000000000001 R12: ffffa973c1277e60
2021-02-18T02:24:58.661254+00:00 smithi163 kernel: [  558.670398] R13: ffff8dea80eb9038 R14: ffff8de9c312ce00 R15: 0000000000001000
2021-02-18T02:24:58.661255+00:00 smithi163 kernel: [  558.670401] FS:  00007f58ce5492c0(0000) GS:ffff8df11fc80000(0000) knlGS:0000000000000000
2021-02-18T02:24:58.661267+00:00 smithi163 kernel: [  558.670405] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2021-02-18T02:24:58.661269+00:00 smithi163 kernel: [  558.670408] CR2: 0000559f1ed18f98 CR3: 000000010852c005 CR4: 00000000003706e0
2021-02-18T02:24:58.661271+00:00 smithi163 kernel: [  558.670412] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2021-02-18T02:24:58.661272+00:00 smithi163 kernel: [  558.670415] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
2021-02-18T02:24:58.661273+00:00 smithi163 kernel: [  558.670418] Call Trace:
2021-02-18T02:24:58.661275+00:00 smithi163 kernel: [  558.670427]  generic_perform_write+0xc3/0x1e0
2021-02-18T02:24:58.661277+00:00 smithi163 kernel: [  558.670446]  ceph_write_iter+0xb2f/0xd50 [ceph]
2021-02-18T02:24:58.661289+00:00 smithi163 kernel: [  558.670492]  ? new_sync_write+0x11c/0x1b0
2021-02-18T02:24:58.661290+00:00 smithi163 kernel: [  558.670498]  new_sync_write+0x11c/0x1b0
2021-02-18T02:24:58.661292+00:00 smithi163 kernel: [  558.670513]  vfs_write+0x243/0x390
2021-02-18T02:24:58.661293+00:00 smithi163 kernel: [  558.670523]  ksys_write+0x68/0xe0
2021-02-18T02:24:58.661295+00:00 smithi163 kernel: [  558.670531]  do_syscall_64+0x2d/0x70
2021-02-18T02:24:58.661296+00:00 smithi163 kernel: [  558.670537]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
2021-02-18T02:24:58.661308+00:00 smithi163 kernel: [  558.670544] RIP: 0033:0x7f58cd058224
2021-02-18T02:24:58.661310+00:00 smithi163 kernel: [  558.670548] Code: 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 8d 05 c1 07 2e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5
2021-02-18T02:24:58.661312+00:00 smithi163 kernel: [  558.670552] RSP: 002b:00007ffd48b0c738 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
2021-02-18T02:24:58.661313+00:00 smithi163 kernel: [  558.670558] RAX: ffffffffffffffda RBX: 0000000000002000 RCX: 00007f58cd058224
2021-02-18T02:24:58.661314+00:00 smithi163 kernel: [  558.670561] RDX: 0000000000002000 RSI: 0000559f1ed14f10 RDI: 0000000000000004
2021-02-18T02:24:58.661316+00:00 smithi163 kernel: [  558.670563] RBP: 0000559f1ed14f10 R08: 0000000000002000 R09: 0000000000000000
2021-02-18T02:24:58.661317+00:00 smithi163 kernel: [  558.670566] R10: 0000559f1ed0f010 R11: 0000000000000246 R12: 0000559f1ed104a0
2021-02-18T02:24:58.661329+00:00 smithi163 kernel: [  558.670569] R13: 0000000000002000 R14: 00007f58cd32f760 R15: 0000000000002000
2021-02-18T02:24:58.661330+00:00 smithi163 kernel: [  558.670587] irq event stamp: 14161
2021-02-18T02:24:58.661332+00:00 smithi163 kernel: [  558.670590] hardirqs last  enabled at (14167): [<ffffffffa5115cef>] console_unlock+0x4bf/0x5a0
2021-02-18T02:24:58.661333+00:00 smithi163 kernel: [  558.670599] hardirqs last disabled at (14172): [<ffffffffa5115c62>] console_unlock+0x432/0x5a0
2021-02-18T02:24:58.661334+00:00 smithi163 kernel: [  558.670604] softirqs last  enabled at (13322): [<ffffffffa5e00306>] __do_softirq+0x306/0x42c
2021-02-18T02:24:58.661335+00:00 smithi163 kernel: [  558.670610] softirqs last disabled at (13317): [<ffffffffa5c010af>] asm_call_irq_on_stack+0xf/0x20
2021-02-18T02:24:58.661348+00:00 smithi163 kernel: [  558.670616] ---[ end trace 9112c35e13586d71 ]---

From: /ceph/teuthology-archive/yuriw-2021-02-16_18:11:32-kcephfs-octopus-testing-basic-smithi/5888305/remote/smithi163/syslog/kern.log.gz

Caused by new code: https://git.kernel.org/pub/scm/linux/kernel/git/jlayton/linux.git/tree/fs/ceph/addr.c?h=ceph-netfs-lib#n1237

Jeff is looking into it.

Actions
Copy link
 #3

Updated by Jeff Layton about 3 years ago

  • Project changed from CephFS to Linux kernel client
  • Status changed from In Progress to Resolved

The problem was the write_begin patch in the netfs lib patchset in testing branch. I resolved it in my own testing and pushed an updated branch to ceph-client/testing.

The main issue was that I needed to handle the case of writing to a new inode with no data.

Actions
Copy link

Also available in: Atom PDF