Project

General

Profile

Actions

Feature #2226

closed

osd: better filestore idempotency test

Added by Sage Weil about 12 years ago. Updated about 12 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Joao Eduardo Luis
Category:
OSD
Target version:
% Done:

100%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

- generate a pseudorandom sequence of operations that are non-idempotent
  - collection_move
  - long sequences of transactions that create object, modify it, and then move into place
  - clone
  etc.
- instrument a bunch of failure injection points in filestore.  these should include
  - before and after any random op
  - in the midst of the "interesting" ops that set a guard (e.g., collection_move)
- tester
  - run a sequence
  - inject a failure
  - restart and replay journal
  - compare state to a parallel instance (another FileStore instance in another directory) that runs sequence to same position but without a failure
  - either continue to the next failure, or start over and repeat

Subtasks 4 (0 open4 closed)

Subtask #2235: generate deterministic sequence of transactions (5)Resolved04/04/2012

Actions
Subtask #2236: filestore failure injection (3)Resolved04/04/2012

Actions
Subtask #2237: failure+replay tester (8)Resolved04/04/2012

Actions
Subtask #2249: teuthology task (3)Resolved04/04/2012

Actions
Actions #1

Updated by Sage Weil about 12 years ago

  • Target version set to v0.46
  • Translation missing: en.field_position set to 9
Actions #2

Updated by Sage Weil about 12 years ago

  • Description updated (diff)
  • Status changed from New to 12
  • Assignee set to Joao Eduardo Luis
Actions #3

Updated by Sage Weil about 12 years ago

  • Translation missing: en.field_story_points set to 19
  • Translation missing: en.field_position deleted (20)
  • Translation missing: en.field_position set to 20
Actions #4

Updated by Sage Weil about 12 years ago

  • Description updated (diff)
Actions #5

Updated by Sage Weil about 12 years ago

Thought about the a bit more. The filestore failure injection is easiest to implement with an _exit(1) or something, which makes me think we want to be able to do something like

test_idempotent_sequence mkfs /path/to/filestore
test_idempotent_sequence run 1 10000 --seed 123 --filestore-inject-failure-at 12345 /path/to/filestore --status-file /tmp/foo
<crash/exit>
test_idempotent_sequence replay /path/to/filestore
test_idempotent_sequence mkfs /path/to/clean
test_idempotent_sequence run 1 `cat /tmp/foo` /path/to/clean
test_idempotent_sequence diff /path/to/clean /path/to/filestore

...

Actions #6

Updated by Sage Weil about 12 years ago

  • Status changed from 12 to Resolved
Actions

Also available in: Atom PDF