Project

General

Profile

Bug #21388 » osd-78_dump.log

Yoann Moulin, 03/01/2018 09:24 AM

 
root@iccluster020:/var/lib/ceph/osd/cephprod-78/current/11.5f_head# ceph-objectstore-tool --cluster cephprod --data-path /var/lib/ceph/osd/cephprod-78 --pgid 11.5f --journal /var/lib/ceph/osd/cephprod-78/journal --debug .dir.c9724aff-5fa0-4dd9-b494-57bdb48fab4e.314528.19__head_087F8EDF__b dump
2018-03-01 10:21:46.739736 7ff5a9e57400 0 filestore(/var/lib/ceph/osd/cephprod-78) backend xfs (magic 0x58465342)
2018-03-01 10:21:46.740213 7ff5a9e57400 0 genericfilestorebackend(/var/lib/ceph/osd/cephprod-78) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2018-03-01 10:21:46.740229 7ff5a9e57400 0 genericfilestorebackend(/var/lib/ceph/osd/cephprod-78) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2018-03-01 10:21:46.740231 7ff5a9e57400 0 genericfilestorebackend(/var/lib/ceph/osd/cephprod-78) detect_features: splice() is disabled via 'filestore splice' config option
2018-03-01 10:21:46.751452 7ff5a9e57400 0 genericfilestorebackend(/var/lib/ceph/osd/cephprod-78) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2018-03-01 10:21:46.751533 7ff5a9e57400 0 xfsfilestorebackend(/var/lib/ceph/osd/cephprod-78) detect_feature: extsize is disabled by conf
2018-03-01 10:21:46.752154 7ff5a9e57400 0 filestore(/var/lib/ceph/osd/cephprod-78) start omap initiation
2018-03-01 10:21:46.752227 7ff5a9e57400 0 set rocksdb option compaction_readahead_size = 2097152
2018-03-01 10:21:46.752241 7ff5a9e57400 0 set rocksdb option compression = kNoCompression
2018-03-01 10:21:46.752248 7ff5a9e57400 0 set rocksdb option max_background_compactions = 8
2018-03-01 10:21:46.752292 7ff5a9e57400 0 set rocksdb option compaction_readahead_size = 2097152
2018-03-01 10:21:46.752301 7ff5a9e57400 0 set rocksdb option compression = kNoCompression
2018-03-01 10:21:46.752308 7ff5a9e57400 0 set rocksdb option max_background_compactions = 8
2018-03-01 10:21:46.752963 7ff5a9e57400 4 rocksdb: RocksDB version: 5.4.0

2018-03-01 10:21:46.752978 7ff5a9e57400 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2018-03-01 10:21:46.752980 7ff5a9e57400 4 rocksdb: Compile date Nov 30 2017
2018-03-01 10:21:46.752981 7ff5a9e57400 4 rocksdb: DB SUMMARY

2018-03-01 10:21:46.753052 7ff5a9e57400 4 rocksdb: CURRENT file: CURRENT

2018-03-01 10:21:46.753055 7ff5a9e57400 4 rocksdb: IDENTITY file: IDENTITY

2018-03-01 10:21:46.753062 7ff5a9e57400 4 rocksdb: MANIFEST file: MANIFEST-000391 size: 1268 Bytes

2018-03-01 10:21:46.753066 7ff5a9e57400 4 rocksdb: SST files in /var/lib/ceph/osd/cephprod-78/current/omap dir, Total Num: 7, files: 000158.sst 000316.sst 000317.sst 000383.sst 000384.sst 000385.sst 000387.sst

2018-03-01 10:21:46.753070 7ff5a9e57400 4 rocksdb: Write Ahead Log file in /var/lib/ceph/osd/cephprod-78/current/omap: 000392.log size: 0 ;

2018-03-01 10:21:46.753072 7ff5a9e57400 4 rocksdb: Options.error_if_exists: 0
2018-03-01 10:21:46.753073 7ff5a9e57400 4 rocksdb: Options.create_if_missing: 1
2018-03-01 10:21:46.753074 7ff5a9e57400 4 rocksdb: Options.paranoid_checks: 1
2018-03-01 10:21:46.753075 7ff5a9e57400 4 rocksdb: Options.env: 0x56467d2a7c80
2018-03-01 10:21:46.753077 7ff5a9e57400 4 rocksdb: Options.info_log: 0x56467e696500
2018-03-01 10:21:46.753078 7ff5a9e57400 4 rocksdb: Options.max_open_files: -1
2018-03-01 10:21:46.753079 7ff5a9e57400 4 rocksdb: Options.max_file_opening_threads: 16
2018-03-01 10:21:46.753080 7ff5a9e57400 4 rocksdb: Options.use_fsync: 0
2018-03-01 10:21:46.753082 7ff5a9e57400 4 rocksdb: Options.max_log_file_size: 0
2018-03-01 10:21:46.753083 7ff5a9e57400 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2018-03-01 10:21:46.753085 7ff5a9e57400 4 rocksdb: Options.log_file_time_to_roll: 0
2018-03-01 10:21:46.753086 7ff5a9e57400 4 rocksdb: Options.keep_log_file_num: 1000
2018-03-01 10:21:46.753087 7ff5a9e57400 4 rocksdb: Options.recycle_log_file_num: 0
2018-03-01 10:21:46.753088 7ff5a9e57400 4 rocksdb: Options.allow_fallocate: 1
2018-03-01 10:21:46.753088 7ff5a9e57400 4 rocksdb: Options.allow_mmap_reads: 0
2018-03-01 10:21:46.753089 7ff5a9e57400 4 rocksdb: Options.allow_mmap_writes: 0
2018-03-01 10:21:46.753090 7ff5a9e57400 4 rocksdb: Options.use_direct_reads: 0
2018-03-01 10:21:46.753101 7ff5a9e57400 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2018-03-01 10:21:46.753103 7ff5a9e57400 4 rocksdb: Options.create_missing_column_families: 0
2018-03-01 10:21:46.753104 7ff5a9e57400 4 rocksdb: Options.db_log_dir:
2018-03-01 10:21:46.753105 7ff5a9e57400 4 rocksdb: Options.wal_dir: /var/lib/ceph/osd/cephprod-78/current/omap
2018-03-01 10:21:46.753106 7ff5a9e57400 4 rocksdb: Options.table_cache_numshardbits: 6
2018-03-01 10:21:46.753107 7ff5a9e57400 4 rocksdb: Options.max_subcompactions: 1
2018-03-01 10:21:46.753108 7ff5a9e57400 4 rocksdb: Options.max_background_flushes: 1
2018-03-01 10:21:46.753109 7ff5a9e57400 4 rocksdb: Options.WAL_ttl_seconds: 0
2018-03-01 10:21:46.753110 7ff5a9e57400 4 rocksdb: Options.WAL_size_limit_MB: 0
2018-03-01 10:21:46.753111 7ff5a9e57400 4 rocksdb: Options.manifest_preallocation_size: 4194304
2018-03-01 10:21:46.753113 7ff5a9e57400 4 rocksdb: Options.is_fd_close_on_exec: 1
2018-03-01 10:21:46.753114 7ff5a9e57400 4 rocksdb: Options.advise_random_on_open: 1
2018-03-01 10:21:46.753115 7ff5a9e57400 4 rocksdb: Options.db_write_buffer_size: 0
2018-03-01 10:21:46.753116 7ff5a9e57400 4 rocksdb: Options.access_hint_on_compaction_start: 1
2018-03-01 10:21:46.753117 7ff5a9e57400 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1
2018-03-01 10:21:46.753118 7ff5a9e57400 4 rocksdb: Options.compaction_readahead_size: 2097152
2018-03-01 10:21:46.753119 7ff5a9e57400 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2018-03-01 10:21:46.753121 7ff5a9e57400 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2018-03-01 10:21:46.753122 7ff5a9e57400 4 rocksdb: Options.use_adaptive_mutex: 0
2018-03-01 10:21:46.753123 7ff5a9e57400 4 rocksdb: Options.rate_limiter: (nil)
2018-03-01 10:21:46.753124 7ff5a9e57400 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2018-03-01 10:21:46.753125 7ff5a9e57400 4 rocksdb: Options.bytes_per_sync: 0
2018-03-01 10:21:46.753126 7ff5a9e57400 4 rocksdb: Options.wal_bytes_per_sync: 0
2018-03-01 10:21:46.753127 7ff5a9e57400 4 rocksdb: Options.wal_recovery_mode: 2
2018-03-01 10:21:46.753128 7ff5a9e57400 4 rocksdb: Options.enable_thread_tracking: 0
2018-03-01 10:21:46.753128 7ff5a9e57400 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2018-03-01 10:21:46.753129 7ff5a9e57400 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2018-03-01 10:21:46.753130 7ff5a9e57400 4 rocksdb: Options.write_thread_max_yield_usec: 100
2018-03-01 10:21:46.753131 7ff5a9e57400 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2018-03-01 10:21:46.753132 7ff5a9e57400 4 rocksdb: Options.row_cache: None
2018-03-01 10:21:46.753135 7ff5a9e57400 4 rocksdb: Options.wal_filter: None
2018-03-01 10:21:46.753136 7ff5a9e57400 4 rocksdb: Options.avoid_flush_during_recovery: 0
2018-03-01 10:21:46.753137 7ff5a9e57400 4 rocksdb: Options.base_background_compactions: 1
2018-03-01 10:21:46.753138 7ff5a9e57400 4 rocksdb: Options.max_background_compactions: 8
2018-03-01 10:21:46.753139 7ff5a9e57400 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2018-03-01 10:21:46.753140 7ff5a9e57400 4 rocksdb: Options.delayed_write_rate : 16777216
2018-03-01 10:21:46.753141 7ff5a9e57400 4 rocksdb: Options.max_total_wal_size: 0
2018-03-01 10:21:46.753142 7ff5a9e57400 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2018-03-01 10:21:46.753143 7ff5a9e57400 4 rocksdb: Options.stats_dump_period_sec: 600
2018-03-01 10:21:46.753144 7ff5a9e57400 4 rocksdb: Compression algorithms supported:
2018-03-01 10:21:46.753146 7ff5a9e57400 4 rocksdb: Snappy supported: 0
2018-03-01 10:21:46.753147 7ff5a9e57400 4 rocksdb: Zlib supported: 0
2018-03-01 10:21:46.753148 7ff5a9e57400 4 rocksdb: Bzip supported: 0
2018-03-01 10:21:46.753149 7ff5a9e57400 4 rocksdb: LZ4 supported: 0
2018-03-01 10:21:46.753149 7ff5a9e57400 4 rocksdb: ZSTD supported: 0
2018-03-01 10:21:46.753151 7ff5a9e57400 4 rocksdb: Fast CRC32 supported: 0
2018-03-01 10:21:46.753583 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000391

2018-03-01 10:21:46.753794 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:

2018-03-01 10:21:46.753808 7ff5a9e57400 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2018-03-01 10:21:46.753811 7ff5a9e57400 4 rocksdb: Options.merge_operator:
2018-03-01 10:21:46.753813 7ff5a9e57400 4 rocksdb: Options.compaction_filter: None
2018-03-01 10:21:46.753814 7ff5a9e57400 4 rocksdb: Options.compaction_filter_factory: None
2018-03-01 10:21:46.753816 7ff5a9e57400 4 rocksdb: Options.memtable_factory: SkipListFactory
2018-03-01 10:21:46.753817 7ff5a9e57400 4 rocksdb: Options.table_factory: BlockBasedTable
2018-03-01 10:21:46.753859 7ff5a9e57400 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56467e3ec0f8)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x56467e687500
block_cache_name: LRUCache
block_cache_options:
capacity : 134217728
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2

2018-03-01 10:21:46.753867 7ff5a9e57400 4 rocksdb: Options.write_buffer_size: 67108864
2018-03-01 10:21:46.753868 7ff5a9e57400 4 rocksdb: Options.max_write_buffer_number: 2
2018-03-01 10:21:46.753871 7ff5a9e57400 4 rocksdb: Options.compression: NoCompression
2018-03-01 10:21:46.753872 7ff5a9e57400 4 rocksdb: Options.bottommost_compression: Disabled
2018-03-01 10:21:46.753874 7ff5a9e57400 4 rocksdb: Options.prefix_extractor: nullptr
2018-03-01 10:21:46.753875 7ff5a9e57400 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2018-03-01 10:21:46.753876 7ff5a9e57400 4 rocksdb: Options.num_levels: 7
2018-03-01 10:21:46.753877 7ff5a9e57400 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2018-03-01 10:21:46.753878 7ff5a9e57400 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2018-03-01 10:21:46.753879 7ff5a9e57400 4 rocksdb: Options.compression_opts.window_bits: -14
2018-03-01 10:21:46.753896 7ff5a9e57400 4 rocksdb: Options.compression_opts.level: -1
2018-03-01 10:21:46.753897 7ff5a9e57400 4 rocksdb: Options.compression_opts.strategy: 0
2018-03-01 10:21:46.753898 7ff5a9e57400 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2018-03-01 10:21:46.753899 7ff5a9e57400 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2018-03-01 10:21:46.753902 7ff5a9e57400 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2018-03-01 10:21:46.753904 7ff5a9e57400 4 rocksdb: Options.level0_stop_writes_trigger: 36
2018-03-01 10:21:46.753905 7ff5a9e57400 4 rocksdb: Options.target_file_size_base: 67108864
2018-03-01 10:21:46.753907 7ff5a9e57400 4 rocksdb: Options.target_file_size_multiplier: 1
2018-03-01 10:21:46.753908 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2018-03-01 10:21:46.753909 7ff5a9e57400 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2018-03-01 10:21:46.753910 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2018-03-01 10:21:46.753914 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2018-03-01 10:21:46.753916 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2018-03-01 10:21:46.753917 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2018-03-01 10:21:46.753918 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2018-03-01 10:21:46.753920 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2018-03-01 10:21:46.753921 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2018-03-01 10:21:46.753922 7ff5a9e57400 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2018-03-01 10:21:46.753937 7ff5a9e57400 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2018-03-01 10:21:46.753939 7ff5a9e57400 4 rocksdb: Options.max_compaction_bytes: 1677721600
2018-03-01 10:21:46.753940 7ff5a9e57400 4 rocksdb: Options.arena_block_size: 8388608
2018-03-01 10:21:46.753941 7ff5a9e57400 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2018-03-01 10:21:46.753942 7ff5a9e57400 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2018-03-01 10:21:46.753943 7ff5a9e57400 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2018-03-01 10:21:46.753945 7ff5a9e57400 4 rocksdb: Options.disable_auto_compactions: 0
2018-03-01 10:21:46.753947 7ff5a9e57400 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2018-03-01 10:21:46.753948 7ff5a9e57400 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2018-03-01 10:21:46.753950 7ff5a9e57400 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2018-03-01 10:21:46.753951 7ff5a9e57400 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2018-03-01 10:21:46.753952 7ff5a9e57400 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2018-03-01 10:21:46.753953 7ff5a9e57400 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2018-03-01 10:21:46.753955 7ff5a9e57400 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2018-03-01 10:21:46.753956 7ff5a9e57400 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2018-03-01 10:21:46.753957 7ff5a9e57400 4 rocksdb: Options.table_properties_collectors:
2018-03-01 10:21:46.753972 7ff5a9e57400 4 rocksdb: Options.inplace_update_support: 0
2018-03-01 10:21:46.753974 7ff5a9e57400 4 rocksdb: Options.inplace_update_num_locks: 10000
2018-03-01 10:21:46.753975 7ff5a9e57400 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2018-03-01 10:21:46.753978 7ff5a9e57400 4 rocksdb: Options.memtable_huge_page_size: 0
2018-03-01 10:21:46.753979 7ff5a9e57400 4 rocksdb: Options.bloom_locality: 0
2018-03-01 10:21:46.753981 7ff5a9e57400 4 rocksdb: Options.max_successive_merges: 0
2018-03-01 10:21:46.753982 7ff5a9e57400 4 rocksdb: Options.optimize_filters_for_hits: 0
2018-03-01 10:21:46.753983 7ff5a9e57400 4 rocksdb: Options.paranoid_file_checks: 0
2018-03-01 10:21:46.753984 7ff5a9e57400 4 rocksdb: Options.force_consistency_checks: 0
2018-03-01 10:21:46.753985 7ff5a9e57400 4 rocksdb: Options.report_bg_io_stats: 0
2018-03-01 10:21:46.755930 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/osd/cephprod-78/current/omap/MANIFEST-000391 succeeded,manifest_file_number is 391, next_file_number is 393, last_sequence is 19302434, log_number is 0,prev_log_number is 0,max_column_family is 0

2018-03-01 10:21:46.755957 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 390

2018-03-01 10:21:46.756157 7ff5a9e57400 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1519896106756145, "job": 1, "event": "recovery_started", "log_files": [392]}
2018-03-01 10:21:46.756168 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #392 mode 2
2018-03-01 10:21:46.756336 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 394

2018-03-01 10:21:46.782166 7ff5a9e57400 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1519896106782159, "job": 1, "event": "recovery_finished"}
2018-03-01 10:21:46.807505 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x56467e708000
2018-03-01 10:21:47.908247 7ff5a9e57400 0 filestore(/var/lib/ceph/osd/cephprod-78) mount(1757): enabling WRITEAHEAD journal mode: checkpoint is not enabled
2018-03-01 10:21:47.911122 7ff5a9e57400 1 journal _open /var/lib/ceph/osd/cephprod-78/journal fd 18: 10737418240 bytes, block size 4096 bytes, directio = 1, aio = 1
2018-03-01 10:21:47.911684 7ff5a9e57400 -1 journal do_read_entry(739758080): bad header magic
2018-03-01 10:21:47.911705 7ff5a9e57400 -1 journal do_read_entry(739758080): bad header magic
2018-03-01 10:21:47.911935 7ff5a9e57400 1 journal _open /var/lib/ceph/osd/cephprod-78/journal fd 18: 10737418240 bytes, block size 4096 bytes, directio = 1, aio = 1
2018-03-01 10:21:47.913099 7ff5a9e57400 1 filestore(/var/lib/ceph/osd/cephprod-78) upgrade(1364)
Cluster fsid=f9dfd27f-c704-4d53-9aa0-4a23d655c7c4
Supported features: compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo object,3=object locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded objects,12=transaction hints,13=pg meta object,14=explicit missing set,15=fastinfo pg attr,16=deletes in missing set}
On-disk features: compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo object,3=object locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded objects,12=transaction hints,13=pg meta object,14=explicit missing set,15=fastinfo pg attr,16=deletes in missing set}
191 pgs to scan
Scanning 20.651_head, 0/191 completed
Error getting attr on : 20.651_head,#-22:8a600000:::scrub_20.651:head#, (61) No data available
Scanning 11.5f_head, 1/191 completed
Error getting attr on : 11.5f_head,#-13:fa000000:::scrub_11.5f:head#, (61) No data available
Scanning 0.74_head, 2/191 completed
Error getting attr on : 0.74_head,#-2:2e000000:::scrub_0.74:head#, (61) No data available
Scanning 0.ebe_head, 3/191 completed
Error getting attr on : 0.ebe_head,#-2:7d700000:::scrub_0.ebe:head#, (61) No data available
Scanning 0.74b_head, 4/191 completed
Error getting attr on : 0.74b_head,#-2:d2e00000:::scrub_0.74b:head#, (61) No data available
Scanning 20.501_head, 5/191 completed
Error getting attr on : 20.501_head,#-22:80a00000:::scrub_20.501:head#, (61) No data available
Scanning 0.ab2_head, 6/191 completed
Error getting attr on : 0.ab2_head,#-2:4d500000:::scrub_0.ab2:head#, (61) No data available
Scanning 0.177_head, 7/191 completed
Error getting attr on : 0.177_head,#-2:ee800000:::scrub_0.177:head#, (61) No data available
Scanning 20.be_head, 8/191 completed
Error getting attr on : 20.be_head,#-22:7d000000:::scrub_20.be:head#, (61) No data available
Scanning 40.es0_head, 9/191 completed
Error getting attr on : 40.es0_head,0#-42:70000000:::scrub_40.es0:head#, (61) No data available
Scanning 20.110_head, 10/191 completed
Error getting attr on : 20.110_head,#-22:08800000:::scrub_20.110:head#, (61) No data available
Scanning 20.369_head, 11/191 completed
Error getting attr on : 20.369_head,#-22:96c00000:::scrub_20.369:head#, (61) No data available
Scanning 0.9c1_head, 12/191 completed
Error getting attr on : 0.9c1_head,#-2:83900000:::scrub_0.9c1:head#, (61) No data available
Scanning 0.e7_head, 13/191 completed
Error getting attr on : 0.e7_head,#-2:e7000000:::scrub_0.e7:head#, (61) No data available
Scanning 20.876_head, 14/191 completed
Error getting attr on : 20.876_head,#-22:6e100000:::scrub_20.876:head#, (61) No data available
Scanning 0.64f_head, 15/191 completed
Error getting attr on : 0.64f_head,#-2:f2600000:::scrub_0.64f:head#, (61) No data available
Scanning 0.d1e_head, 16/191 completed
Error getting attr on : 0.d1e_head,#-2:78b00000:::scrub_0.d1e:head#, (61) No data available
Scanning 0.3c3_head, 17/191 completed
Error getting attr on : 0.3c3_head,#-2:c3c00000:::scrub_0.3c3:head#, (61) No data available
Scanning 20.5a5_head, 18/191 completed
Error getting attr on : 20.5a5_head,#-22:a5a00000:::scrub_20.5a5:head#, (61) No data available
Scanning 0.191_head, 19/191 completed
Error getting attr on : 0.191_head,#-2:89800000:::scrub_0.191:head#, (61) No data available
Scanning 12.47_head, 20/191 completed
Error getting attr on : 12.47_head,#-14:e2000000:::scrub_12.47:head#, (61) No data available
Scanning 0.da4_head, 21/191 completed
Error getting attr on : 0.da4_head,#-2:25b00000:::scrub_0.da4:head#, (61) No data available
Scanning 0.e22_head, 22/191 completed
Error getting attr on : 0.e22_head,#-2:44700000:::scrub_0.e22:head#, (61) No data available
Scanning 20.6c3_head, 23/191 completed
Error getting attr on : 20.6c3_head,#-22:c3600000:::scrub_20.6c3:head#, (61) No data available
Scanning 0.c2d_head, 24/191 completed
Error getting attr on : 0.c2d_head,#-2:b4300000:::scrub_0.c2d:head#, (61) No data available
Scanning 20.681_head, 25/191 completed
Error getting attr on : 20.681_head,#-22:81600000:::scrub_20.681:head#, (61) No data available
Scanning 0.c30_head, 26/191 completed
Error getting attr on : 0.c30_head,#-2:0c300000:::scrub_0.c30:head#, (61) No data available
Scanning 0.c01_head, 27/191 completed
Error getting attr on : 0.c01_head,#-2:80300000:::scrub_0.c01:head#, (61) No data available
Scanning 20.825_head, 28/191 completed
Error getting attr on : 20.825_head,#-22:a4100000:::scrub_20.825:head#, (61) No data available
Scanning 0.132_head, 29/191 completed
Error getting attr on : 0.132_head,#-2:4c800000:::scrub_0.132:head#, (61) No data available
Scanning 0.66d_head, 30/191 completed
Error getting attr on : 0.66d_head,#-2:b6600000:::scrub_0.66d:head#, (61) No data available
Scanning 20.e43_head, 31/191 completed
Error getting attr on : 20.e43_head,#-22:c2700000:::scrub_20.e43:head#, (61) No data available
Scanning 20.b4c_head, 32/191 completed
Error getting attr on : 20.b4c_head,#-22:32d00000:::scrub_20.b4c:head#, (61) No data available
Scanning 0.be9_head, 33/191 completed
Error getting attr on : 0.be9_head,#-2:97d00000:::scrub_0.be9:head#, (61) No data available
Scanning 20.856_head, 34/191 completed
Error getting attr on : 20.856_head,#-22:6a100000:::scrub_20.856:head#, (61) No data available
Scanning 20.d2b_head, 35/191 completed
Error getting attr on : 20.d2b_head,#-22:d4b00000:::scrub_20.d2b:head#, (61) No data available
Scanning 0.f2d_head, 36/191 completed
Error getting attr on : 0.f2d_head,#-2:b4f00000:::scrub_0.f2d:head#, (61) No data available
Scanning 20.5c2_head, 37/191 completed
Error getting attr on : 20.5c2_head,#-22:43a00000:::scrub_20.5c2:head#, (61) No data available
Scanning 0.e33_head, 38/191 completed
Error getting attr on : 0.e33_head,#-2:cc700000:::scrub_0.e33:head#, (61) No data available
Scanning 0.838_head, 39/191 completed
Error getting attr on : 0.838_head,#-2:1c100000:::scrub_0.838:head#, (61) No data available
Scanning 20.5b5_head, 40/191 completed
Error getting attr on : 20.5b5_head,#-22:ada00000:::scrub_20.5b5:head#, (61) No data available
Scanning 0.23f_head, 41/191 completed
Error getting attr on : 0.23f_head,#-2:fc400000:::scrub_0.23f:head#, (61) No data available
Scanning 0.a07_head, 42/191 completed
Error getting attr on : 0.a07_head,#-2:e0500000:::scrub_0.a07:head#, (61) No data available
Scanning 0.eda_head, 43/191 completed
Error getting attr on : 0.eda_head,#-2:5b700000:::scrub_0.eda:head#, (61) No data available
Scanning 20.93_head, 44/191 completed
Error getting attr on : 20.93_head,#-22:c9000000:::scrub_20.93:head#, (61) No data available
Scanning 20.e0_head, 45/191 completed
Error getting attr on : 20.e0_head,#-22:07000000:::scrub_20.e0:head#, (61) No data available
Scanning 20.8ff_head, 46/191 completed
Error getting attr on : 20.8ff_head,#-22:ff100000:::scrub_20.8ff:head#, (61) No data available
Scanning 41.11f_head, 47/191 completed
Error getting attr on : 41.11f_head,#-43:f8800000:::scrub_41.11f:head#, (61) No data available
Scanning 41.263_head, 48/191 completed
Error getting attr on : 41.263_head,#-43:c6400000:::scrub_41.263:head#, (61) No data available
Scanning 20.23d_head, 49/191 completed
Error getting attr on : 20.23d_head,#-22:bc400000:::scrub_20.23d:head#, (61) No data available
Scanning 20.165_head, 50/191 completed
Error getting attr on : 20.165_head,#-22:a6800000:::scrub_20.165:head#, (61) No data available
Scanning 0.f55_head, 51/191 completed
Scanning 0.118_head, 52/191 completed
Scanning 20.1ee_head, 53/191 completed
Error getting attr on : 20.1ee_head,#-22:77800000:::scrub_20.1ee:head#, (61) No data available
Scanning 20.287_head, 54/191 completed
Scanning 40.79s2_head, 55/191 completed
Scanning 20.232_head, 56/191 completed
Scanning 0.c_head, 57/191 completed
Scanning 0.776_head, 58/191 completed
Scanning 20.7c0_head, 59/191 completed
Scanning 20.53b_head, 60/191 completed
Scanning 20.a92_head, 61/191 completed
Scanning 0.626_head, 62/191 completed
Scanning 15.5_head, 63/191 completed
Scanning 35.b_head, 64/191 completed
Scanning 20.266_head, 65/191 completed
Scanning 20.b5_head, 66/191 completed
Scanning 20.3c5_head, 67/191 completed
Scanning 20.83d_head, 68/191 completed
Scanning 41.2a_head, 69/191 completed
Scanning 0.6f_head, 70/191 completed
Scanning 20.bc4_head, 71/191 completed
Scanning 40.1s3_head, 72/191 completed
Scanning 20.d0a_head, 73/191 completed
Scanning 0.472_head, 74/191 completed
Scanning 12.51_head, 75/191 completed
Scanning 20.f10_head, 76/191 completed
Scanning 20.437_head, 77/191 completed
Scanning 20.9c9_head, 78/191 completed
Scanning 34.46_head, 79/191 completed
Scanning 20.5d2_head, 80/191 completed
Scanning 20.4f9_head, 81/191 completed
Scanning 35.14_head, 82/191 completed
Error getting attr on : 35.14_head,#-37:28000000:::scrub_35.14:head#, (61) No data available
Scanning 20.ff4_head, 83/191 completed
Scanning 20.4a9_head, 84/191 completed
Scanning 40.4fs4_head, 85/191 completed
Scanning 40.5s5_head, 86/191 completed
Scanning 41.25d_head, 87/191 completed
Error getting attr on : 41.25d_head,#-43:ba400000:::scrub_41.25d:head#, (61) No data available
Scanning 20.fbb_head, 88/191 completed
Scanning 0.57_head, 89/191 completed
Scanning 0.ec_head, 90/191 completed
Scanning 0.924_head, 91/191 completed
Scanning 41.24_head, 92/191 completed
Scanning 20.1b1_head, 93/191 completed
Scanning 0.614_head, 94/191 completed
Scanning 0.5a7_head, 95/191 completed
Scanning 6.7_head, 96/191 completed
Scanning 20.985_head, 97/191 completed
Scanning 20.9ae_head, 98/191 completed
Scanning 0.555_head, 99/191 completed
Scanning 20.d7e_head, 100/191 completed
Scanning 41.1bb_head, 101/191 completed
Scanning 0.d72_head, 102/191 completed
Scanning 0.84f_head, 103/191 completed
Scanning 20.455_head, 104/191 completed
Scanning 20.3ab_head, 105/191 completed
Scanning 20.bcd_head, 106/191 completed
Scanning 40.62s1_head, 107/191 completed
Scanning 0.323_head, 108/191 completed
Scanning 0.b0c_head, 109/191 completed
Scanning 20.bc6_head, 110/191 completed
Scanning 40.7cs6_head, 111/191 completed
Scanning 41.2f_head, 112/191 completed
Scanning 35.7c_head, 113/191 completed
Scanning 0.b9e_head, 114/191 completed
Scanning 20.c66_head, 115/191 completed
Scanning 0.851_head, 116/191 completed
Error getting attr on : 0.851_head,#-2:8a100000:::scrub_0.851:head#, (61) No data available
Scanning 0.fb3_head, 117/191 completed
Scanning 40.66s0_head, 118/191 completed
Error getting attr on : 40.66s0_head,0#-42:66000000:::scrub_40.66s0:head#, (61) No data available
Scanning 0.a48_head, 119/191 completed
Scanning 41.38_head, 120/191 completed
Error getting attr on : 41.38_head,#-43:1c000000:::scrub_41.38:head#, (61) No data available
Scanning 0.a87_head, 121/191 completed
Scanning 41.3ab_head, 122/191 completed
Scanning 0.dcf_head, 123/191 completed
Scanning 35.26_head, 124/191 completed
Error getting attr on : 35.26_head,#-37:64000000:::scrub_35.26:head#, (61) No data available
Scanning 20.1f9_head, 125/191 completed
Scanning 20.508_head, 126/191 completed
Scanning 41.3d_head, 127/191 completed
Scanning 0.719_head, 128/191 completed
Scanning 20.544_head, 129/191 completed
Scanning 7.14_head, 130/191 completed
Error getting attr on : 7.14_head,#-9:28000000:::scrub_7.14:head#, (61) No data available
Scanning 0.204_head, 131/191 completed
Scanning 20.11e_head, 132/191 completed
Scanning 0.542_head, 133/191 completed
Scanning 20.e61_head, 134/191 completed
Scanning 0.721_head, 135/191 completed
Scanning 20.96f_head, 136/191 completed
Scanning 0.b75_head, 137/191 completed
Scanning 20.883_head, 138/191 completed
Scanning 20.c5_head, 139/191 completed
Scanning 20.ea4_head, 140/191 completed
Scanning 41.83_head, 141/191 completed
Scanning 11.3d_head, 142/191 completed
Scanning 40.12s8_head, 143/191 completed
Scanning 0.a71_head, 144/191 completed
Scanning 20.ad3_head, 145/191 completed
Scanning 20.aaf_head, 146/191 completed
Scanning 41.1e4_head, 147/191 completed
Error getting attr on : 41.1e4_head,#-43:27800000:::scrub_41.1e4:head#, (61) No data available
Scanning 0.c26_head, 148/191 completed
Scanning 0.c80_head, 149/191 completed
Scanning 0.c61_head, 150/191 completed
Scanning 0.22f_head, 151/191 completed
Scanning 0.31b_head, 152/191 completed
Scanning 20.b55_head, 153/191 completed
Scanning 0.605_head, 154/191 completed
Scanning 0.857_head, 155/191 completed
Scanning 0.fd5_head, 156/191 completed
Scanning 0.a98_head, 157/191 completed
Scanning 20.1bd_head, 158/191 completed
Scanning 14.5_head, 159/191 completed
Scanning 20.944_head, 160/191 completed
Scanning 20.478_head, 161/191 completed
Scanning 20.46_head, 162/191 completed
Scanning 0.c28_head, 163/191 completed
Scanning 0.a64_head, 164/191 completed
Scanning 0.450_head, 165/191 completed
Scanning 20.3f1_head, 166/191 completed
Scanning 20.468_head, 167/191 completed
Scanning 41.24c_head, 168/191 completed
Scanning 0.18f_head, 169/191 completed
Scanning 40.47s8_head, 170/191 completed
Scanning 0.ec2_head, 171/191 completed
Scanning 0.901_head, 172/191 completed
Scanning 0.3ee_head, 173/191 completed
Scanning 20.e69_head, 174/191 completed
Scanning 40.31s5_head, 175/191 completed
Scanning 0.506_head, 176/191 completed
Scanning 40.7es7_head, 177/191 completed
Scanning 20.fc5_head, 178/191 completed
Scanning 0.168_head, 179/191 completed
Scanning 34.7b_head, 180/191 completed
Scanning 0.cff_head, 181/191 completed
Scanning 41.e_head, 182/191 completed
Scanning 0.521_head, 183/191 completed
Scanning 41.36a_head, 184/191 completed
Scanning 20.4ca_head, 185/191 completed
Scanning 20.5a0_head, 186/191 completed
Error getting attr on : 20.5a0_head,#-22:05a00000:::scrub_20.5a0:head#, (61) No data available
Scanning 0.48f_head, 187/191 completed
Scanning 20.1b_head, 188/191 completed
Scanning 34.64_head, 189/191 completed
Scanning 0.317_head, 190/191 completed
No object id 'cephprod' found or invalid JSON specified
2018-03-01 10:22:53.873600 7ff5a9e57400 1 journal close /var/lib/ceph/osd/cephprod-78/journal
2018-03-01 10:22:53.876150 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2018-03-01 10:22:53.878227 7ff5a9e57400 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl.cc:343] Shutdown complete
(5-5/6)