Project

General

Profile

Actions

Bug #62700

closed

postgres workunit failed with "PQputline failed"

Added by Xiubo Li 8 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Q/A
Tags:
backport_processed
Backport:
reef,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

https://pulpito.ceph.com/rishabh-2023-08-10_20:16:46-fs-wip-rishabh-2023Aug1-b4-testing-default-smithi/7365718/teuthology.log:

2023-08-11T12:27:13.213 INFO:teuthology.orchestra.run.smithi008.stderr:35500000 of 50000000 tuples (71%) done (elapsed 137.90 s, remaining 56.33 s)
2023-08-11T12:27:13.460 INFO:teuthology.orchestra.run.smithi008.stderr:35600000 of 50000000 tuples (71%) done (elapsed 138.15 s, remaining 55.88 s)
2023-08-11T12:27:13.643 INFO:teuthology.orchestra.run.smithi008.stderr:35700000 of 50000000 tuples (71%) done (elapsed 138.33 s, remaining 55.41 s)
2023-08-11T12:27:13.812 INFO:teuthology.orchestra.run.smithi008.stderr:35800000 of 50000000 tuples (71%) done (elapsed 138.50 s, remaining 54.94 s)
2023-08-11T12:27:13.884 INFO:teuthology.orchestra.run.smithi008.stderr:PQputline failed
2023-08-11T12:27:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-08-11T12:27:13.891 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/task/exec.py", line 66, in task
    remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" 
2023-08-11T12:27:14.014 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=131e6996218e42dbadd9f9c0b2bcd491
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/task/exec.py", line 66, in task
    remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" 
2023-08-11T12:27:14.017 DEBUG:teuthology.run_tasks:Unwinding manager background_exec
2023-08-11T12:27:14.027 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi008.front.sepia.ceph.com
2023-08-11T12:27:14.041 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally
2023-08-11T12:27:14.051 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec...
2023-08-11T12:27:14.052 INFO:teuthology.task.exec:Executing custom commands...
2023-08-11T12:27:14.052 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi008.front.sepia.ceph.com


Related issues 2 (0 open2 closed)

Copied to CephFS - Backport #62946: quincy: postgres workunit failed with "PQputline failed"RejectedXiubo LiActions
Copied to CephFS - Backport #62947: reef: postgres workunit failed with "PQputline failed"ResolvedXiubo LiActions
Actions #1

Updated by Xiubo Li 8 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 53279

The scale factor will depend on the node's performance and disk sizes being used to run the test, and 500 seems too large here as a common test case without knowing the nodes' configuration.

We should just follow the white-book:
https://github.com/ApsaraDB/PolarDB-for-PostgreSQL/blob/distributed/doc/polardb/benchmark.md

Actions #3

Updated by Patrick Donnelly 8 months ago

  • Category set to Testing
  • Target version set to v19.0.0
  • Source set to Q/A
  • Backport set to reef,quincy
Actions #4

Updated by Venky Shankar 8 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions #5

Updated by Backport Bot 8 months ago

  • Copied to Backport #62946: quincy: postgres workunit failed with "PQputline failed" added
Actions #6

Updated by Backport Bot 8 months ago

  • Copied to Backport #62947: reef: postgres workunit failed with "PQputline failed" added
Actions #7

Updated by Backport Bot 8 months ago

  • Tags set to backport_processed
Actions #8

Updated by Xiubo Li 7 months ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF