Project

General

Profile

Bug #50021

qa: snaptest-git-ceph failure during mon thrashing

Added by Patrick Donnelly 26 days ago. Updated 14 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure, snapshots
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-03-25T09:56:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:Checking out files:   2% (6/227)   ^MChecking out files:   3% (7/227)   ^MChecking out files:   4% (10/227)   ^MChecking out files:   5% (12/227)   ^MChecking out files:   6% (14/227)   ^MChecking out files:   7% (16/227)   ^MChecking out files:   7% (18/227)   ^MChecking out files:   8% (19/227)   ^MChecking out files:   8% (20/227)   ^MChecking out files:   9% (21/227)   ^MChecking out files:  10% (23/227)   ^MChecking out files:  11% (25/227)   ^MChecking out files:  12% (28/227)   ^MChecking out files:  13% (30/227)   ^MChecking out files:  14% (32/227)   ^MChecking out files:  15% (35/227)   ^MChecking out files:  15% (36/227)   ^MChecking out files:  16% (37/227)   ^MChecking out files:  16% (38/227)   ^MChecking out files:  17% (39/227)   ^MChecking out files:  18% (41/227)   ^MChecking out files:  19% (44/227)   ^MChecking out files:  19% (45/227)   ^MChecking out files:  20% (46/227)   ^MChecking out files:  21% (48/227)   ^MChecking out files:  22% (50/227)   ^MChecking out files:  23% (53/227)   ^MChecking out files:  24% (55/227)   ^MChecking out files:  25% (57/227)   ^MChecking out files:  26% (60/227)   ^MChecking out files:  27% (62/227)   ^MChecking out files:  28% (64/227)   ^MChecking out files:  29% (66/227)   ^MChecking out files:  30% (69/227)   ^MChecking out files:  31% (71/227)   ^MChecking out files:  32% (73/227)   ^MChecking out files:  33% (75/227)   ^MChecking out files:  34% (78/227)   ^MChecking out files:  35% (80/227)   ^MChecking out files:  36% (82/227)   ^MChecking out files:  37% (84/227)   ^MChecking out files:  37% (86/227)   ^MChecking out files:  38% (87/227)   ^MChecking out files:  39% (89/227)   ^MChecking out files:  39% (90/227)   ^MChecking out files:  40% (91/227)   ^MChecking out files:  40% (92/227)   ^MChecking out files:  40% (93/227)   ^MChecking out files:  41% (94/227)   ^MChecking out files:  41% (95/227)   ^MChecking out files:  42% (96/227)   ^MChecking out files:  42% (97/227)   ^MChecking out files:  43% (98/227)   ^MChecking out files:  43% (99/227)   ^MChecking out files:  44% (100/227)   ^MChecking out files:  44% (101/227)   ^MChecking out files:  44% (102/227)   ^MChecking out files:  45% (103/227)   ^MChecking out files:  45% (104/227)   ^MChecking out files:  46% (105/227)   ^MChecking out files:  46% (106/227)   ^MChecking out files:  47% (107/227)   ^MChecking out files:  47% (108/227)   ^MChecking out files:  48% (109/227)   ^MChecking out files:  48% (110/227)   ^MChecking out files:  48% (111/227)   ^MChecking out files:  49% (112/227)   ^MChecking out files:  49% (113/227)   ^MChecking out files:  50% (114/227)   ^MChecking out files:  50% (115/227)   ^MChecking out files:  51% (116/227)
2021-03-25T09:56:11.132 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...
2021-03-25T09:56:11.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2021-03-25T09:56:11.325 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 147, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 297, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 425, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi016 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0691d6bed3e3aaf89688b125297e25f6f6c3fae2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2021-03-25T09:56:11.783 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d943e434792545d5bcd96d9d39646764
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 147, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 297, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 425, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi016 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0691d6bed3e3aaf89688b125297e25f6f6c3fae2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'

From: /ceph/teuthology-archive/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/5995979/teuthology.log

It's not clear to me from the log what actually broke.

History

#1 Updated by Xiubo Li 23 days ago

Checked the client logs in `smithi016/log/ceph-client.0.25180.log.gz`, everything works well till now, I didn't see any exception.

2021-03-25T09:56:04.247 INFO:teuthology.orchestra.run.smithi016.stderr:2021-03-25T09:56:04.241+0000 7fbb9e669700  1 -- 172.21.15.16:0/3146238485 wait complete.
2021-03-25T09:56:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-03-25T09:56:07.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status

[Edit]
The 124 means the command timed out.

#2 Updated by Xiubo Li 23 days ago

Checked all the mds/osd/mon/client/kernel/misc related logs, didn't find any error during that exception around 2021-03-25T09:56:06.572.

#3 Updated by Xiubo Li 23 days ago

The exception occured just before the snap test at:

2021-03-25T09:56:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 124

And just after that remote process returned errno 124 and before the task ['fs/snaps'] was stopped by teuthology, the snap test was running well:

2021-03-25T09:56:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:Checking out files:   2% (6/227)   ^MChecking out files:   3% (7/227)   ^MChecking out files:   4% (10/227)   ^MChecking out files:   5% (12/227)   ^MChecking out files:   6% (14/227)   ^MChecking out files:   7% (16/227)   ^MChecking out files:   7% (18/227)   ^MChecking out files:   8% (19/227)   ^MChecking out files:   8% (20/227)   ^MChecking out files:   9% (21/227)   ^MChecking out files:  10% (23/227)   ^MChecking out files:  11% (25/227)   ^MChecking out files:  12% (28/227)   ^MChecking out files:  13% (30/227)   ^MChecking out files:  14% (32/227)   ^MChecking out files:  15% (35/227)   ^MChecking out files:  15% (36/227)   ^MChecking out files:  16% (37/227)   ^MChecking out files:  16% (38/227)   ^MChecking out files:  17% (39/227)   ^MChecking out files:  18% (41/227)   ^MChecking out files:  19% (44/227)   ^MChecking out files:  19% (45/227)   ^MChecking out files:  20% (46/227)   ^MChecking out files:  21% (48/227)   ^MChecking out files:  22% (50/227)   ^MChecking out files:  23% (53/227)   ^MChecking out files:  24% (55/227)   ^MChecking out files:  25% (57/227)   ^MChecking out files:  26% (60/227)   ^MChecking out files:  27% (62/227)   ^MChecking out files:  28% (64/227)   ^MChecking out files:  29% (66/227)   ^MChecking out files:  30% (69/227)   ^MChecking out files:  31% (71/227)   ^MChecking out files:  32% (73/227)   ^MChecking out files:  33% (75/227)   ^MChecking out files:  34% (78/227)   ^MChecking out files:  35% (80/227)   ^MChecking out files:  36% (82/227)   ^MChecking out files:  37% (84/227)   ^MChecking out files:  37% (86/227)   ^MChecking out files:  38% (87/227)   ^MChecking out files:  39% (89/227)   ^MChecking out files:  39% (90/227)   ^MChecking out files:  40% (91/227)   ^MChecking out files:  40% (92/227)   ^MChecking out files:  40% (93/227)   ^MChecking out files:  41% (94/227)   ^MChecking out files:  41% (95/227)   ^MChecking out files:  42% (96/227)   ^MChecking out files:  42% (97/227)   ^MChecking out files:  43% (98/227)   ^MChecking out files:  43% (99/227)   ^MChecking out files:  44% (100/227)   ^MChecking out files:  44% (101/227)   ^MChecking out files:  44% (102/227)   ^MChecking out files:  45% (103/227)   ^MChecking out files:  45% (104/227)   ^MChecking out files:  46% (105/227)   ^MChecking out files:  46% (106/227)   ^MChecking out files:  47% (107/227)   ^MChecking out files:  47% (108/227)   ^MChecking out files:  48% (109/227)   ^MChecking out files:  48% (110/227)   ^MChecking out files:  48% (111/227)   ^MChecking out files:  49% (112/227)   ^MChecking out files:  49% (113/227)   ^MChecking out files:  50% (114/227)   ^MChecking out files:  50% (115/227)   ^MChecking out files:  51% (116/227)
2021-03-25T09:56:11.132 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...

And no any warning/error found in the client/mds/osd logs.

I really doubt it was one bug in the remote process...

#4 Updated by Xiubo Li 23 days ago

  • Status changed from New to In Progress
  • Assignee set to Xiubo Li

#5 Updated by Xiubo Li 23 days ago

Okay, I was in wrong direction yesterday.

I think it was the `git clone ceph...` command's problem, it took too long:

The script is:


  1 #!/bin/sh -x
  2                                
  3 set -e
  4                                
  5 git clone git://git.ceph.com/ceph.git                                                                                                 
  6 cd ceph
  7 
  8 versions=`seq 1 21`
  ...

The logs are from 2021-03-25T06:56:06.474:

2021-03-25T06:56:06.474 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh...
2021-03-25T06:56:06.475 DEBUG:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0691d6bed3e3aaf89688b125297e25f6f6c3fae2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh
2021-03-25T06:56:06.574 INFO:tasks.workunit.client.0.smithi016.stderr:+ set -e
2021-03-25T06:56:06.574 INFO:tasks.workunit.client.0.smithi016.stderr:+ git clone git://git.ceph.com/ceph.git
2021-03-25T06:56:06.582 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into 'ceph'...
...

And the `cd ceph` at 2021-03-25T09:33:06.323:

2021-03-25T09:33:06.323 INFO:tasks.workunit.client.0.smithi016.stderr:+ cd ceph
2021-03-25T09:33:06.323 INFO:tasks.workunit.client.0.smithi016.stderr:+ seq 1 21
2021-03-25T09:33:06.340 INFO:tasks.workunit.client.0.smithi016.stderr:+ versions=1
2021-03-25T09:33:06.340 INFO:tasks.workunit.client.0.smithi016.stderr:2
2021-03-25T09:33:06.340 INFO:tasks.workunit.client.0.smithi016.stderr:3
2021-03-25T09:33:06.340 INFO:tasks.workunit.client.0.smithi016.stderr:4
2021-03-25T09:33:06.340 INFO:tasks.workunit.client.0.smithi016.stderr:5
2021-03-25T09:33:06.340 INFO:tasks.workunit.client.0.smithi016.stderr:6
2021-03-25T09:33:06.340 INFO:tasks.workunit.client.0.smithi016.stderr:7
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:8
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:9
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:10
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:11
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:12
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:13
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:14
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:15
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:16
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:17
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:18
2021-03-25T09:33:06.341 INFO:tasks.workunit.client.0.smithi016.stderr:19
2021-03-25T09:33:06.342 INFO:tasks.workunit.client.0.smithi016.stderr:20
2021-03-25T09:33:06.342 INFO:tasks.workunit.client.0.smithi016.stderr:21
2021-03-25T09:33:06.342 INFO:tasks.workunit.client.0.smithi016.stderr:+ ver=v0.1
2021-03-25T09:33:06.342 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo v0.1
2021-03-25T09:33:06.342 INFO:tasks.workunit.client.0.smithi016.stdout:v0.1
2021-03-25T09:33:06.342 INFO:tasks.workunit.client.0.smithi016.stderr:+ git reset --hard v0.1

It took 2h and 37m already.

Locally recently when I am cloning the ceph repo from the github, it sometimes will stuck and then may timeout with connection failure to the github, I need to try many times. Not sure is it the similar problem causing the git clone taking to long here ?

Tried without the VPN, for me mostly the git clone or git pull can be better and very fast.

@Patrick,

Maybe we could save a ceph repo somewhere locally in the test lab and clone it from there, will it make sense ?

#6 Updated by Patrick Donnelly 23 days ago

Xiubo Li wrote:

Okay, I was in wrong direction yesterday.

I think it was the `git clone ceph...` command's problem, it took too long:

The script is:
[...]

The logs are from 2021-03-25T06:56:06.474:

[...]

And the `cd ceph` at 2021-03-25T09:33:06.323:

[...]

It took 2h and 37m already.

Locally recently when I am cloning the ceph repo from the github, it sometimes will stuck and then may timeout with connection failure to the github, I need to try many times. Not sure is it the similar problem causing the git clone taking to long here ?

Tried without the VPN, for me mostly the git clone or git pull can be better and very fast.

@Patrick,

Maybe we could save a ceph repo somewhere locally in the test lab and clone it from there, will it make sense ?

!! Yes. We should definitely do that. We've done it for other QA downloads/git clones. Please talk with David Galloway via email about a mirror. I think we may already have one for ceph.git.

#7 Updated by Xiubo Li 23 days ago

Patrick Donnelly wrote:

Xiubo Li wrote:

[...]

@Patrick,

Maybe we could save a ceph repo somewhere locally in the test lab and clone it from there, will it make sense ?

!! Yes. We should definitely do that. We've done it for other QA downloads/git clones. Please talk with David Galloway via email about a mirror. I think we may already have one for ceph.git.

Sure.

#8 Updated by David Galloway 20 days ago

Xiubo Li wrote:

Patrick Donnelly wrote:

Xiubo Li wrote:

[...]

@Patrick,

Maybe we could save a ceph repo somewhere locally in the test lab and clone it from there, will it make sense ?

!! Yes. We should definitely do that. We've done it for other QA downloads/git clones. Please talk with David Galloway via email about a mirror. I think we may already have one for ceph.git.

Sure.

What repo do you want to mirror?

#9 Updated by Patrick Donnelly 16 days ago

David Galloway wrote:

Xiubo Li wrote:

Patrick Donnelly wrote:

Xiubo Li wrote:

[...]

@Patrick,

Maybe we could save a ceph repo somewhere locally in the test lab and clone it from there, will it make sense ?

!! Yes. We should definitely do that. We've done it for other QA downloads/git clones. Please talk with David Galloway via email about a mirror. I think we may already have one for ceph.git.

Sure.

What repo do you want to mirror?

ceph.git. Sorry for pulling you into this David; it's actually already using the ceph.com mirror.

@xiubo, I think we can chalk this up to random noise and NAB.

#10 Updated by Xiubo Li 16 days ago

Patrick Donnelly wrote:

David Galloway wrote:

Xiubo Li wrote:

Patrick Donnelly wrote:

Xiubo Li wrote:

[...]

@Patrick,

Maybe we could save a ceph repo somewhere locally in the test lab and clone it from there, will it make sense ?

!! Yes. We should definitely do that. We've done it for other QA downloads/git clones. Please talk with David Galloway via email about a mirror. I think we may already have one for ceph.git.

Sure.

What repo do you want to mirror?

ceph.git. Sorry for pulling you into this David; it's actually already using the ceph.com mirror.

@xiubo, I think we can chalk this up to random noise and NAB.

Yeah. Agree.

While not sure letting the clone command try under a `timeout 1200 git clone git://git.ceph.com:/ceph.git` and try 3 times will work here ? From my experience sometimes the stuck or slow socket connecting issue could be fixed up by retrying.

Make sense ?

#11 Updated by Patrick Donnelly 16 days ago

Xiubo Li wrote:

Patrick Donnelly wrote:

David Galloway wrote:

Xiubo Li wrote:

Patrick Donnelly wrote:

Xiubo Li wrote:

[...]

@Patrick,

Maybe we could save a ceph repo somewhere locally in the test lab and clone it from there, will it make sense ?

!! Yes. We should definitely do that. We've done it for other QA downloads/git clones. Please talk with David Galloway via email about a mirror. I think we may already have one for ceph.git.

Sure.

What repo do you want to mirror?

ceph.git. Sorry for pulling you into this David; it's actually already using the ceph.com mirror.

@xiubo, I think we can chalk this up to random noise and NAB.

Yeah. Agree.

While not sure letting the clone command try under a `timeout 1200 git clone git://git.ceph.com:/ceph.git` and try 3 times will work here ? From my experience sometimes the stuck or slow socket connecting issue could be fixed up by retrying.

Make sense ?

Makes sense.

#12 Updated by Xiubo Li 15 days ago

Patrick Donnelly wrote:

Xiubo Li wrote:

[...]

What repo do you want to mirror?

ceph.git. Sorry for pulling you into this David; it's actually already using the ceph.com mirror.

@xiubo, I think we can chalk this up to random noise and NAB.

Yeah. Agree.

While not sure letting the clone command try under a `timeout 1200 git clone git://git.ceph.com:/ceph.git` and try 3 times will work here ? From my experience sometimes the stuck or slow socket connecting issue could be fixed up by retrying.

Make sense ?

Makes sense.

Okay, will fix it.

#13 Updated by Xiubo Li 15 days ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 40611

#14 Updated by Patrick Donnelly 14 days ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF