Infrastructure

Description of problem¶

on the teuthology machine as the teuthology user we use `/bin/bash /home/teuthology/bin/update-crontab.sh` to update the schedule for the suites we run. This is broken now

How to reproduce¶

Steps:

ssh teuthology
 sudo su - teuthology
 run /home/teuthology/bin/update-crontab.sh

Time noticed / Event Window (in UTC)¶

unknown

Actual results¶

teuthology@teuthology:~$ /home/teuthology/bin/update-crontab.sh
........
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f036b5d6740 (most recent call first):
/home/teuthology/bin/update-crontab.sh: line 38: 341631 Aborted                 chkcrontab $TEMP_DIR/new
Checking crontab in /tmp/FgPZWQ7s/new failed

Also see with -x option:

teuthology@teuthology:~$ /bin/bash -x /home/teuthology/bin/update-crontab.sh
+ REMOTE_CRONTAB_URL=https://raw.githubusercontent.com/ceph/ceph/main/qa/crontab/teuthology-cronjobs
++ mktemp -d /tmp/XXXXXXXX
+ TEMP_DIR=/tmp/YGWf1yWZ
+ CHKCRONTAB_PATH=/home/teuthology/bin/chkcrontab-venv
+ curl -s -o /tmp/YGWf1yWZ/new https://raw.githubusercontent.com/ceph/ceph/main/qa/crontab/teuthology-cronjobs
+ crontab -l
+ diff /tmp/YGWf1yWZ/old /tmp/YGWf1yWZ/new
128c128
< 20 01 * * 4 CEPH_BRANCH=quincy; MACHINE_NAME=smithi; /home/teuthology/bin/cron_wrapper teuthology-suite -v -c $CEPH_BRANCH -n 100 -m $MACHINE_NAME -s upgrade-clients/client-upgrade-octopus-quincy -k distro -e $CEPH_QA_EMAIL --suite-branch octopus --suite-repo https://github.com/ceph/ceph.git --ceph-repo https://github.com/ceph/ceph.git -p 50
---
......
+ '[' 1 -eq 0 ']'
+ '[' -x /home/teuthology/bin/chkcrontab-venv/bin/chkcrontab ']'
+ source /home/teuthology/bin/chkcrontab-venv/bin/activate
++ deactivate nondestructive
++ unset -f pydoc
++ '[' -z '' ']'
++ '[' -z '' ']'
++ '[' -n /bin/bash ']'
++ hash -r
++ '[' -z '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/home/teuthology/bin/chkcrontab-venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/home/teuthology/bin:/home/teuthology/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/teuthology/src/git.ceph.com_git_teuthology_main/virtualenv/bin:/home/teuthology/src/git.ceph.com_git_teuthology_main/virtualenv/bin
++ PATH=/home/teuthology/bin/chkcrontab-venv/bin:/home/teuthology/bin:/home/teuthology/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/teuthology/src/git.ceph.com_git_teuthology_main/virtualenv/bin:/home/teuthology/src/git.ceph.com_git_teuthology_main/virtualenv/bin
++ export PATH
++ '[' -z '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ '[' x '!=' x ']'
+++ basename /home/teuthology/bin/chkcrontab-venv
++ PS1='(chkcrontab-venv) '
++ export PS1
++ alias pydoc
++ '[' -n /bin/bash ']'
++ hash -r
+ chkcrontab /tmp/YGWf1yWZ/new
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fa525e21740 (most recent call first):
/home/teuthology/bin/update-crontab.sh: line 38: 350877 Aborted                 chkcrontab $TEMP_DIR/new
+ '[' 134 -eq 0 ']'
+ echo 'Checking crontab in /tmp/YGWf1yWZ/new failed'
Checking crontab in /tmp/YGWf1yWZ/new failed
+ exit 1

Expected results¶

script updates crontab with latest changes from github https://github.com/ceph/ceph/blob/main/qa/crontab/teuthology-cronjobs

Additional analysis¶