Project

General

Profile

Bug #62667

Error running "ceph-container-build-ceph-base-push-imgs" jenkins job

Added by Yuri Weinstein 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Dan Mick
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Description of problem

Running https://2.jenkins.ceph.com/job/ceph-container-build-ceph-base-push-imgs/ ends up with errors

How to reproduce

Steps: Run Build Now (for pacific 16.2.14) via https://2.jenkins.ceph.com/job/ceph-container-build-ceph-base-push-imgs/

Time noticed / Event Window (in UTC)

Started seeing 8/30/23

Actual results

Console Output
Started by user Yuri Weinstein
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on 172.21.15.56+smithi056 (small x86_64 huge trusty) in workspace /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs
The recommended git tool is: NONE
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/ceph/ceph-container.git
 > git init /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container # timeout=10
Fetching upstream changes from https://github.com/ceph/ceph-container.git
 > git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > git fetch --tags --progress -- https://github.com/ceph/ceph-container.git +refs/heads/*:refs/remotes/origin/* # timeout=20
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/ceph/ceph-container.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: detected dubious ownership in repository at '/home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container'
To add an exception for this directory, call:

    git config --global --add safe.directory /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:87)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:623)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:852)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 8.43.84.3/8.43.84.3:54302
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1000)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143)
        at jdk.internal.reflect.GeneratedMethodAccessor622.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129)
        at com.sun.proxy.$Proxy224.execute(Unknown Source)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1225)_Please add logs, screenshots, and/or links to examples_

        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1308)
        at hudson.scm.SCM.checkout(SCM.java:540)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1240)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
        at hudson.model.Run.execute(Run.java:1900)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:101)
        at hudson.model.Executor.run(Executor.java:442)
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
The recommended git tool is: NONE
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/ceph/ceph-container.git
 > git init /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container # timeout=10
Fetching upstream changes from https://github.com/ceph/ceph-container.git
 > git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > git fetch --tags --progress -- https://github.com/ceph/ceph-container.git +refs/heads/*:refs/remotes/origin/* # timeout=20
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/ceph/ceph-container.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: detected dubious ownership in repository at '/home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container'
To add an exception for this directory, call:

    git config --global --add safe.directory /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:87)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:623)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:852)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 8.43.84.3/8.43.84.3:54302
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1000)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143)
        at jdk.internal.reflect.GeneratedMethodAccessor622.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129)
        at com.sun.proxy.$Proxy224.execute(Unknown Source)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1225)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1308)
        at hudson.scm.SCM.checkout(SCM.java:540)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1240)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
        at hudson.model.Run.execute(Run.java:1900)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:101)
        at hudson.model.Executor.run(Executor.java:442)
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
The recommended git tool is: NONE
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/ceph/ceph-container.git
 > git init /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container # timeout=10
Fetching upstream changes from https://github.com/ceph/ceph-container.git
 > git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > git fetch --tags --progress -- https://github.com/ceph/ceph-container.git +refs/heads/*:refs/remotes/origin/* # timeout=20
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/ceph/ceph-container.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: detected dubious ownership in repository at '/home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container'
To add an exception for this directory, call:

    git config --global --add safe.directory /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:87)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:623)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:852)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 8.43.84.3/8.43.84.3:54302
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1000)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143)
        at jdk.internal.reflect.GeneratedMethodAccessor622.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129)
        at com.sun.proxy.$Proxy224.execute(Unknown Source)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1225)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1308)
        at hudson.scm.SCM.checkout(SCM.java:540)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1240)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
        at hudson.model.Run.execute(Run.java:1900)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:101)
        at hudson.model.Executor.run(Executor.java:442)
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
The recommended git tool is: NONE
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/ceph/ceph-container.git
 > git init /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container # timeout=10
Fetching upstream changes from https://github.com/ceph/ceph-container.git
 > git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > git fetch --tags --progress -- https://github.com/ceph/ceph-container.git +refs/heads/*:refs/remotes/origin/* # timeout=20
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/ceph/ceph-container.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: detected dubious ownership in repository at '/home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container'
To add an exception for this directory, call:

    git config --global --add safe.directory /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:87)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:623)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:852)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 8.43.84.3/8.43.84.3:54302
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1000)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143)
        at jdk.internal.reflect.GeneratedMethodAccessor622.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129)
        at com.sun.proxy.$Proxy224.execute(Unknown Source)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1225)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1308)
        at hudson.scm.SCM.checkout(SCM.java:540)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1240)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
        at hudson.model.Run.execute(Run.java:1900)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:101)
        at hudson.model.Executor.run(Executor.java:442)
ERROR: Error cloning remote repo 'origin'
Finished: FAILURE

Expected results

Successful run and container pushed to https://quay.io/repository/ceph/ceph?tab=tags

Additional analysis

History

#1 Updated by Yuri Weinstein 3 months ago

  • Description updated (diff)
  • Crash signature (v1) updated (diff)

#2 Updated by Yuri Weinstein 3 months ago

  • Description updated (diff)

#3 Updated by Yuri Weinstein 3 months ago

  • Crash signature (v1) updated (diff)

#4 Updated by Yuri Weinstein 3 months ago

  • Description updated (diff)

#5 Updated by Laura Flores 3 months ago

I saw this yesterday, and I ran the suggested `git config --global --add safe.directory /home/jenkins-build/build/workspace/ceph-container-build-ceph-base-push-imgs/ceph-container` on smithi056. This solved the problem in the moment, but my guess is that if the workspace gets wiped out, so does that setting.

#6 Updated by Dan Mick 3 months ago

There were some build directories on several of the 2.jenkins.ceph.com builders that still had root permissions on git checkouts; that's the ultimate source of the error (the job runs as jenkins-build and git doesn't want to check out into a directory owned by root). I've run the ansible builder.yml role as follows:

ansible-playbook --vault-password-file=<vault-pass-file> -e api_uri="https://2.jenkins.ceph.com" --limit $(JENKINS_HOST=2.jenkins.ceph.com jenkins.tags -l -d '.*,') --tags chown builder.yml

and now nothing seems to be owned by root

#7 Updated by Dan Mick 3 months ago

  • Status changed from New to Resolved
  • Assignee set to Dan Mick

Also available in: Atom PDF