{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":6934395,"defaultBranch":"main","name":"rocksdb","ownerLogin":"facebook","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2012-11-30T06:16:18.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/69631?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1720825646.0","currentOid":""},"activityList":{"items":[{"before":"3db030d7ee1b887ce818ec6f6a8d10949f9e9a22","after":"72438a678872544809393b831c7273794c074215","ref":"refs/heads/main","pushedAt":"2024-07-12T23:06:09.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Support read & write with unsynced data in FaultInjectionTestFS (#12852)\n\nSummary:\nFollow-up to https://github.com/facebook/rocksdb/issues/12729 and others to fix FaultInjectionTestFS handling the case where a live WAL is being appended to and synced while also being copied for checkpoint or backup, up to a known flushed (but not necessarily synced) prefix of the file. It was tricky to structure the code in a way that could handle a tricky race with Sync in another thread (see code comments, thanks Changyu) while maintaining good performance and test-ability.\n\nFor more context, see the call to FlushWAL() in DBImpl::GetLiveFilesStorageInfo().\n\nAlso, the unit test for https://github.com/facebook/rocksdb/issues/12729 was neutered by https://github.com/facebook/rocksdb/issues/12797, and this re-enables the functionality it is testing.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12852\n\nTest Plan:\nunit test expanded/updated. Local runs of blackbox_crash_test.\n\nThe implementation is structured so that a multi-threaded unit test is not needed to cover at least the code lines, as the race handling is folded into \"catch up after returning unsynced and then a sync.\"\n\nReviewed By: cbi42\n\nDifferential Revision: D59594045\n\nPulled By: pdillinger\n\nfbshipit-source-id: 94667bb72255e2952586c53bae2c2dd384e85a50","shortMessageHtmlLink":"Support read & write with unsynced data in FaultInjectionTestFS (#12852)"}},{"before":"cec28aa90f0e38666c0b3485d197ecbe0c2a025f","after":"3db030d7ee1b887ce818ec6f6a8d10949f9e9a22","ref":"refs/heads/main","pushedAt":"2024-07-11T23:31:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix bug for recovering a prepared but not committed txn (#12856)\n\nSummary:\nThis PR fix a bug for recovering a prepared Transaction that can contain user-defined timestamps.\n\nThe `Transaction::Put` type of APIs expect the key provided to be user key without timestamps. When the original transaction added a key for a column family that enables user-defined timestamps, say of size 8. Internally `WriteBatch::Put` will leave a placeholder 8 bytes for the final commit timestamp. For example:\nhttps://github.com/facebook/rocksdb/blob/cec28aa90f0e38666c0b3485d197ecbe0c2a025f/db/write_batch.cc#L937\n\nWhen rebuilding this transaction from a `WriteBatch` from WAL log, we should consider this and remove the tailing 8 bytes of a key before adding it via the public Transaction write APIs.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12856\n\nTest Plan: Added unit test that would fail without this fix\n\nReviewed By: cbi42\n\nDifferential Revision: D59656399\n\nPulled By: jowlyzhang\n\nfbshipit-source-id: c716aefa4d548770b691efe96ac8e6d7dab458b9","shortMessageHtmlLink":"Fix bug for recovering a prepared but not committed txn (#12856)"}},{"before":"37b81bd28f03e7ef3744fd9ecc28cb4c4e2ae03a","after":"cec28aa90f0e38666c0b3485d197ecbe0c2a025f","ref":"refs/heads/main","pushedAt":"2024-07-11T04:39:40.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix SetOptions() failure in stress test (#12854)\n\nSummary:\nfix SetOptions() so that max_read_amp is at least level0_file_num_compaction_trigger.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12854\n\nTest Plan: monitor stress test new failure\n\nReviewed By: hx235\n\nDifferential Revision: D59618547\n\nPulled By: cbi42\n\nfbshipit-source-id: b83371f293b87097ee9cdd32d662e9965cde57e6","shortMessageHtmlLink":"Fix SetOptions() failure in stress test (#12854)"}},{"before":"6997dd909c96e55b8587d07ca07683541843f838","after":"37b81bd28f03e7ef3744fd9ecc28cb4c4e2ae03a","ref":"refs/heads/main","pushedAt":"2024-07-10T23:02:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Avoid SyncWAL if flushing during shutdown (#12853)\n\nSummary:\nhttps://github.com/facebook/rocksdb/issues/12746 added calls to FlushWAL/SyncWAL in db_stress during reopen, in order to ensure persistence of unpersisted data and avoid false alarms due to lack of prefix recovery support in db_stress reopen. However, there's no need to flush/sync the WAL if avoid_flush_during_shutdown is false, as the WAL will not be needed during recovery. This allows file systems that don't support SyncWAL (not thread safe) to avoid the need by requesting flush during shutdown.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12853\n\nReviewed By: hx235\n\nDifferential Revision: D59604138\n\nPulled By: anand1976\n\nfbshipit-source-id: 4c4470b3c956d6bf64f5b8a1a5727a8b888f1a5f","shortMessageHtmlLink":"Avoid SyncWAL if flushing during shutdown (#12853)"}},{"before":"d6f265f9d6bd4ee3c5356b6e7b7f7e2f5e2ea716","after":"6997dd909c96e55b8587d07ca07683541843f838","ref":"refs/heads/main","pushedAt":"2024-07-10T17:57:36.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Disable attribute group txn tests (#12851)\n\nSummary:\nTransactions are not yet supported in AttributeGroup APIs. Disabling `use_attribute_group` for txn tests\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12851\n\nTest Plan:\nVerified output that `--use_attribute_group=0`\n\n```\npython3 tools/db_crashtest.py whitebox --txn\n```\n```\npython3 tools/db_crashtest.py whitebox --optimistic_txn\n```\n\nReviewed By: hx235\n\nDifferential Revision: D59565635\n\nPulled By: jaykorean\n\nfbshipit-source-id: 7d618f475b6d2e5a53c3c59cdf1e694f3893ae58","shortMessageHtmlLink":"Disable attribute group txn tests (#12851)"}},{"before":"ebe2116240a4efac4226975b975917f889010aa2","after":"d6f265f9d6bd4ee3c5356b6e7b7f7e2f5e2ea716","ref":"refs/heads/main","pushedAt":"2024-07-09T23:56:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix race in multiops txn stress test (#12847)\n\nSummary:\n`MultiOpsTxnsStressListener::OnCompactionCompleted()` access `db_` and can be called while db_ is being destroyed in ~StressTest(). This causes TSAN to complain about data race. This PR fixes this issue by calling db_->Close() first to stop all background work. Also moved the cleanup out of StressTest destructor to avoid race between the listener and ~StressTest().\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12847\n\nTest Plan: monitor crash test failure.\n\nReviewed By: hx235\n\nDifferential Revision: D59492691\n\nPulled By: cbi42\n\nfbshipit-source-id: afcbab084cc9ac0904d6b04809b0888498ca8e66","shortMessageHtmlLink":"Fix race in multiops txn stress test (#12847)"}},{"before":"5ecb92760a3ea856c8f9440c822e459df895db62","after":"ebe2116240a4efac4226975b975917f889010aa2","ref":"refs/heads/main","pushedAt":"2024-07-09T22:39:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Remove false-postive assertion in `FaultInjectionTestFS::RenameFile` (#12828)\n\nSummary:\n**Context/Summary:**\nThe assertion `tlist.find(tdn.second) == tlist.end()` https://github.com/facebook/rocksdb/blame/9eebaf11cbd875435b572f05f0378ecdb761cc74/utilities/fault_injection_fs.cc#L1003 can catch us false positive.\n\nSome context\n(1) When fault injection is enabled and db open fails because of that, crash test will retry open without injected error in order to proceed with a clean open:\nhttps://github.com/facebook/rocksdb/blob/9eebaf11cbd875435b572f05f0378ecdb761cc74/db_stress_tool/db_stress_test_base.cc#L3559\nhttps://github.com/facebook/rocksdb/blob/9eebaf11cbd875435b572f05f0378ecdb761cc74/db_stress_tool/db_stress_test_base.cc#L3586-L3639\n(2)\na. `FaultInjectionTestFS::dir_to_new_files_since_last_sync` records files that are created but not yet synced.\nb. When we create CURRENT, we will first create a temp file and rename it as \"CURRENT\". As part of the renaming, we will [assert](https://github.com/facebook/rocksdb/blame/9eebaf11cbd875435b572f05f0378ecdb761cc74/utilities/fault_injection_fs.cc#L1003) `FaultInjectionTestFS::dir_to_new_files_since_last_sync ` doesn't already have a file named `CURRENT`.\n\nSuppose the following sequence of events happened:\n\n(1) 1st open, with metadata write error\n1. As part of creating CURRENT file, added \"CURRENT\" to `FaultInjectionTestFS::dir_to_new_files_since_last_sync_`\nhttps://github.com/facebook/rocksdb/blob/9eebaf11cbd875435b572f05f0378ecdb761cc74/utilities/fault_injection_fs.cc#L735\n2. `SyncDir()` here https://github.com/facebook/rocksdb/blob/9eebaf11cbd875435b572f05f0378ecdb761cc74/file/filename.cc#L412 failed with injected metadata write error. Therefore, \"CURRENT\" file didn't get removed from `FaultInjectionTestFS::dir_to_new_files_since_last_sync_` as it would if `SyncDir()` succeeded https://github.com/facebook/rocksdb/blob/9eebaf11cbd875435b572f05f0378ecdb761cc74/utilities/fault_injection_fs.h#L344\n\n(2) 2st open\n1. Attempted to create a CURRENT file and failed during renaming since `FaultInjectionTestFS::dir_to_new_files_since_last_sync_` already had a file called CURRENT. So will fail\n```\nassertion failed - tlist.find(tdn.second) == tlist.end()\n```\n\nThis PR fixed this by removing the assertion. It used to catch us some missing sync of some directory (e.,g https://github.com/facebook/rocksdb/pull/10573) so we will keep thinking about a better way to catch that.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12828\n\nTest Plan:\nCommand constantly failed before the fix but passed after the PR running for 10 minutes\n```\npython3 tools/db_crashtest.py --simple blackbox --interval=10 --WAL_size_limit_MB=1 --WAL_ttl_seconds=60 --acquire_snapshot_one_in=100 --adaptive_readahead=1 --adm_policy=2 --advise_random_on_open=1 --allow_concurrent_memtable_write=1 --allow_data_in_errors=True --allow_fallocate=1 --async_io=0 --auto_readahead_size=1 --avoid_flush_during_recovery=0 --avoid_flush_during_shutdown=0 --avoid_unnecessary_blocking_io=0 --backup_max_size=104857600 --backup_one_in=100000 --batch_protection_bytes_per_key=0 --bgerror_resume_retry_interval=100 --block_align=0 --block_protection_bytes_per_key=8 --block_size=16384 --bloom_before_level=1 --bloom_bits=10 --bottommost_compression_type=lz4hc --bottommost_file_compaction_delay=86400 --bytes_per_sync=0 --cache_index_and_filter_blocks=1 --cache_index_and_filter_blocks_with_high_priority=0 --cache_size=8388608 --cache_type=tiered_auto_hyper_clock_cache --charge_compression_dictionary_building_buffer=1 --charge_file_metadata=0 --charge_filter_construction=0 --charge_table_reader=0 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=10000 --checksum_type=kCRC32c --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000 --compact_range_one_in=1000000 --compaction_pri=3 --compaction_readahead_size=0 --compaction_ttl=1 --compress_format_version=1 --compressed_secondary_cache_ratio=0.5 --compressed_secondary_cache_size=0 --compression_checksum=0 --compression_max_dict_buffer_bytes=15 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=zstd --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=65536 --continuous_verification_interval=0 --daily_offpeak_time_utc= --data_block_index_type=1 --db_write_buffer_size=0 --default_temperature=kHot --default_write_temperature=kUnknown --delete_obsolete_files_period_micros=30000000 --delpercent=4 --delrangepercent=1 --destroy_db_initially=0 --detect_filter_construct_corruption=1 --disable_file_deletions_one_in=10000 --disable_manual_compaction_one_in=10000 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_custom_split_merge=0 --enable_do_not_compress_roles=0 --enable_index_compression=1 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=0 --enable_sst_partitioner_factory=1 --enable_thread_tracking=1 --enable_write_thread_adaptive_yield=0 --error_recovery_with_no_fault_injection=1 --exclude_wal_from_write_fault_injection=1 --fail_if_options_file_error=1 --fifo_allow_compaction=0 --file_checksum_impl=crc32c --fill_cache=1 --flush_one_in=1000000 --format_version=3 --get_all_column_family_metadata_one_in=1000000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=1000000 --get_properties_of_all_tables_one_in=100000 --get_property_one_in=100000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=2097152 --high_pri_pool_ratio=0 --index_block_restart_interval=2 --index_shortening=0 --index_type=2 --ingest_external_file_one_in=0 --initial_auto_readahead_size=16384 --inplace_update_support=0 --iterpercent=10 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100000 --last_level_temperature=kWarm --level_compaction_dynamic_level_bytes=0 --lock_wal_one_in=10000 --log_file_time_to_roll=60 --log_readahead_size=16777216 --long_running_snapshots=1 --low_pri_pool_ratio=0.5 --lowest_used_cache_tier=1 --manifest_preallocation_size=0 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=10 --max_auto_readahead_size=16384 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=1000000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=1073741824 --max_sequential_skip_in_iterations=1 --max_total_wal_size=0 --max_write_batch_group_size_bytes=16 --max_write_buffer_number=3 --max_write_buffer_size_to_maintain=2097152 --memtable_insert_hint_per_batch=0 --memtable_max_range_deletions=0 --memtable_prefix_bloom_size_ratio=0.1 --memtable_protection_bytes_per_key=8 --memtable_whole_key_filtering=0 --memtablerep=skip_list --metadata_charge_policy=1 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=0 --min_write_buffer_number_to_merge=2 --mmap_read=1 --mock_direct_io=False --nooverwritepercent=1 --num_file_reads_for_auto_readahead=1 --open_files=-1 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=8 --open_read_fault_one_in=0 --open_write_fault_one_in=0 --ops_per_thread=100000000 --optimize_filters_for_hits=0 --optimize_filters_for_memory=0 --optimize_multiget_for_io=1 --paranoid_file_checks=1 --partition_filters=1 --partition_pinning=3 --pause_background_one_in=1000000 --periodic_compaction_seconds=1000 --prefix_size=5 --prefixpercent=5 --prepopulate_block_cache=1 --preserve_internal_time_seconds=0 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=32 --read_fault_one_in=0 --readahead_size=524288 --readpercent=45 --recycle_log_file_num=0 --reopen=0 --report_bg_io_stats=0 --reset_stats_one_in=1000000 --sample_for_compression=0 --secondary_cache_fault_one_in=32 --secondary_cache_uri= --set_options_one_in=0 --skip_stats_update_on_db_open=0 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=68719476736 --sqfc_name=foo --sqfc_version=1 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=10 --stats_history_buffer_size=1048576 --strict_bytes_per_sync=1 --subcompactions=2 --sync=0 --sync_fault_injection=1 --table_cache_numshardbits=6 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=2 --uncache_aggressiveness=1582 --universal_max_read_amp=4 --unpartitioned_pinning=0 --use_adaptive_mutex=0 --use_adaptive_mutex_lru=1 --use_attribute_group=1 --use_delta_encoding=0 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=1 --use_multi_get_entity=1 --use_multiget=0 --use_put_entity_one_in=1 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=0 --use_write_buffer_manager=0 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000 --verify_compression=1 --verify_db_one_in=10000 --verify_file_checksums_one_in=1000 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=none --write_buffer_size=33554432 --write_dbid_to_manifest=1 --write_fault_one_in=8 --writepercent=35\n```\n\nReviewed By: cbi42\n\nDifferential Revision: D59241548\n\nPulled By: hx235\n\nfbshipit-source-id: 5bb49e6a94943273f47578a2caf3d08ca5b67e5f","shortMessageHtmlLink":"Remove false-postive assertion in FaultInjectionTestFS::RenameFile (#…"}},{"before":"b837d41ab1f09a7f0e2ebdb543d915eafc1ac342","after":"5ecb92760a3ea856c8f9440c822e459df895db62","ref":"refs/heads/main","pushedAt":"2024-07-09T19:09:24.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Create C API function to iterate over WriteBatch for custom Column Families (#12718)\n\nSummary:\nCreate C API function for iterating over WriteBatch for custom Column Families\nAdding function to C API that exposes column family specific methods to iterate over WriteBatch: put_cf, delete_cf and merge_cf. This is required when the one needs to read changes for any non-default column family. Without that functionality it is impossible to iterate over changes in WAL that are relevant to custom column families.\n\nFixes https://github.com/facebook/rocksdb/issues/12790\n\nTesting:\nAdded WriteBatch iteration test to \"columnfamilies\" section of C API unit tests\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12718\n\nReviewed By: cbi42\n\nDifferential Revision: D59483601\n\nPulled By: ajkr\n\nfbshipit-source-id: b68b900636304528a38620a8c3ad82fdce4b60cb","shortMessageHtmlLink":"Create C API function to iterate over WriteBatch for custom Column Fa…"}},{"before":"2e1b3f921f5205c3bfebdc24dff37c404c49c2de","after":"b837d41ab1f09a7f0e2ebdb543d915eafc1ac342","ref":"refs/heads/main","pushedAt":"2024-07-09T19:04:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Expose SizeApproximationFlags to C API (#12836)\n\nSummary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/12836\n\nReviewed By: cbi42\n\nDifferential Revision: D59502673\n\nPulled By: ajkr\n\nfbshipit-source-id: fc9f77d6740d8efa45d9357662f0f827dbd0511f","shortMessageHtmlLink":"Expose SizeApproximationFlags to C API (#12836)"}},{"before":"62b62cf135dc4e6b693aa9a07dc94112781fd11b","after":"2e1b3f921f5205c3bfebdc24dff37c404c49c2de","ref":"refs/heads/main","pushedAt":"2024-07-09T16:32:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Remove unreachable code (#12846)\n\nSummary:\nRemoving some unreachable code.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12846\n\nReviewed By: cbi42\n\nDifferential Revision: D59498423\n\nPulled By: jowlyzhang\n\nfbshipit-source-id: 6b2c51732d94b1f69a8ba7474b16a171d4e6d640","shortMessageHtmlLink":"Remove unreachable code (#12846)"}},{"before":"a97a1f3247b6d36f8256158ce45a497ea4daa416","after":"62b62cf135dc4e6b693aa9a07dc94112781fd11b","ref":"refs/heads/main","pushedAt":"2024-07-09T04:42:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix CondVar::TimedWait for Windows (#12815)\n\nSummary:\nBased on https://github.com/microsoft/STL/issues/369\nThey fixed the issue in `std::condition_variable_any` but not in `std::condition_variable`, which is currently used in rocksdb repo. So we need to implement the work around regardless of `_MSVC_STL_UPDATE`.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12815\n\nReviewed By: cbi42\n\nDifferential Revision: D59493690\n\nPulled By: ajkr\n\nfbshipit-source-id: ad0fc9ef9f2357347d21e271c2f1d0a3a97d89be","shortMessageHtmlLink":"Fix CondVar::TimedWait for Windows (#12815)"}},{"before":"f471e561905148d8001ed08e7ac0e80685e572f6","after":"a97a1f3247b6d36f8256158ce45a497ea4daa416","ref":"refs/heads/main","pushedAt":"2024-07-09T01:11:26.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix incorrect refillPeriodMicros unit in the document (#12832)\n\nSummary:\nThe default value for `refillPeriodMicros` is `100 * 1000`, which means 100ms (or 100,000us).\n\nThe document comments say 100,000ms (equivalent to 100 seconds), which is incorrect and misleading. This PR fixes this typo.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12832\n\nReviewed By: cbi42\n\nDifferential Revision: D59492336\n\nPulled By: ajkr\n\nfbshipit-source-id: c2f55a8b996fe078a1510fcbebaea92ec0075929","shortMessageHtmlLink":"Fix incorrect refillPeriodMicros unit in the document (#12832)"}},{"before":"110ce5f4a392d02167cee3439160f83d2929a2c8","after":"f471e561905148d8001ed08e7ac0e80685e572f6","ref":"refs/heads/main","pushedAt":"2024-07-08T23:01:57.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"fix the non initialized bug in StderrLogger. (#12839)\n\nSummary:\nThis PR is intended to fix a potential uninitialized variable bug.\n\nFixes https://github.com/facebook/rocksdb/issues/12837\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12839\n\nReviewed By: ajkr\n\nDifferential Revision: D59398888\n\nPulled By: cbi42\n\nfbshipit-source-id: 337391d7c1e73c0ff61797f88fbb4a8379500211","shortMessageHtmlLink":"fix the non initialized bug in StderrLogger. (#12839)"}},{"before":"b6c3495a7183f01901d3be01dc68f7e40a1a2e9b","after":"110ce5f4a392d02167cee3439160f83d2929a2c8","ref":"refs/heads/main","pushedAt":"2024-07-08T19:27:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"fix: Round-Robin pri under leveled compaction allows subcompactions b… (#12843)\n\nSummary:\n### Summary: Round-Robin pri under leveled compaction allows subcompactions by default is not compatible with PlainTable\n\n```c++\nbool Compaction::ShouldFormSubcompactions() const {\n if (cfd_ == nullptr) {\n return false;\n }\n\n // Round-Robin pri under leveled compaction allows subcompactions by default\n // and the number of subcompactions can be larger than max_subcompactions_\n if (cfd_->ioptions()->compaction_pri == kRoundRobin &&\n cfd_->ioptions()->compaction_style == kCompactionStyleLevel) {\n return output_level_ > 0;\n }\n\n if (max_subcompactions_ <= 1) {\n return false;\n }\n```\n\nPlainTable does not support Subcompaction, including when AdaptiveTable is applied to PlainTable. subcompaction by default will result in the following error in some scenarios.\n\n```c++\nvoid PlainTableIterator::Seek(const Slice& target) {\n if (use_prefix_seek_ != !table_->IsTotalOrderMode()) {\n // This check is done here instead of NewIterator() to permit creating an\n // iterator with total_order_seek = true even if we won't be able to Seek()\n // it. This is needed for compaction: it creates iterator with\n // total_order_seek = true but usually never does Seek() on it,\n // only SeekToFirst().\n status_ = Status::InvalidArgument(\n \"total_order_seek not implemented for PlainTable.\");\n offset_ = next_offset_ = table_->file_info_.data_end_offset;\n return;\n }\n```\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12843\n\nReviewed By: ajkr\n\nDifferential Revision: D59433477\n\nPulled By: cbi42\n\nfbshipit-source-id: fb780ba7f7e8efdfedb7480abf14dd38e0b63677","shortMessageHtmlLink":"fix: Round-Robin pri under leveled compaction allows subcompactions b… ("}},{"before":"1f589a3f730c3013b6b80373471b0cd2ae8fde1a","after":"b6c3495a7183f01901d3be01dc68f7e40a1a2e9b","ref":"refs/heads/main","pushedAt":"2024-07-05T16:34:24.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Update snappy dependency for Java releases. (#12207)\n\nSummary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/12207\n\nReviewed By: hx235\n\nDifferential Revision: D59299915\n\nPulled By: cbi42\n\nfbshipit-source-id: 3f5fa88b0c5e8366a08734f99db1d3de942cd60b","shortMessageHtmlLink":"Update snappy dependency for Java releases. (#12207)"}},{"before":"de16611a50508b6f43a14f788b044f23121363e2","after":"1f589a3f730c3013b6b80373471b0cd2ae8fde1a","ref":"refs/heads/main","pushedAt":"2024-07-02T20:18:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Clarify GetProperty API doc (#12829)\n\nSummary:\n**Context/Summary:** as titled since https://github.com/facebook/rocksdb/blob/9eebaf11cbd875435b572f05f0378ecdb761cc74/db/internal_stats.cc#L1162.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12829\n\nTest Plan: no code change\n\nReviewed By: pdillinger\n\nDifferential Revision: D59243565\n\nPulled By: hx235\n\nfbshipit-source-id: 074137b29bb12d9d965d154626a3289f85a39c52","shortMessageHtmlLink":"Clarify GetProperty API doc (#12829)"}},{"before":"0bb939611dd80a2a4cb07d4fb526cc39304d9a1d","after":"de16611a50508b6f43a14f788b044f23121363e2","ref":"refs/heads/main","pushedAt":"2024-07-02T20:05:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix WAL corruption in stress test (#12834)\n\nSummary:\nWe are seeing WAL corruption in crash tests where wal_compression and recycled_wal are enabled. With wal_compression, we write a SetCompression record when creating a WAL, which can happen during DB open time. Our current stress test set up may write directly to the underlying WAL file during DB open, while writing to a buffer under TestFSWritableFile later for sync fault injection. This causes the last synced position to be inaccurately tracked in TestFSWritableFile and causes reads to return incorrect data.\n\nThis PR removes the line that causes this mixture of WAL writes. Also updated TestFSWritableFile to avoid such a mixture of buffered and direct writes.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12834\n\nTest Plan:\nthe following command repros WAL corruption before this PR\n```\n ./db_stress --WAL_size_limit_MB=0 --WAL_ttl_seconds=60 --acquire_snapshot_one_in=10000 --adaptive_readahead=0 --adm_policy=1 --advise_random_on_open=1 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --allow_fallocate=0 --async_io=1 --auto_readahead_size=0 --avoid_flush_during_recovery=0 --avoid_flush_during_shutdown=0 --avoid_unnecessary_blocking_io=1 --backup_max_size=104857600 --backup_one_in=1000 --batch_protection_bytes_per_key=8 --bgerror_resume_retry_interval=1000000 --block_align=0 --block_protection_bytes_per_key=1 --block_size=16384 --bloom_before_level=0 --bloom_bits=8 --bottommost_compression_type=snappy --bottommost_file_compaction_delay=600 --bytes_per_sync=0 --cache_index_and_filter_blocks=0 --cache_index_and_filter_blocks_with_high_priority=1 --cache_size=33554432 --cache_type=auto_hyper_clock_cache --charge_compression_dictionary_building_buffer=1 --charge_file_metadata=0 --charge_filter_construction=1 --charge_table_reader=1 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=10000 --checksum_type=kxxHash64 --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000 --compact_range_one_in=1000000 --compaction_pri=4 --compaction_readahead_size=1048576 --compaction_ttl=0 --compress_format_version=1 --compressed_secondary_cache_size=16777216 --compression_checksum=1 --compression_max_dict_buffer_bytes=1099511627775 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=zstd --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --daily_offpeak_time_utc= --data_block_index_type=1 --db_write_buffer_size=134217728 --default_temperature=kHot --default_write_temperature=kCold --delete_obsolete_files_period_micros=21600000000 --delpercent=4 --delrangepercent=1 --destroy_db_initially=0 --detect_filter_construct_corruption=1 --disable_file_deletions_one_in=1000000 --disable_manual_compaction_one_in=10000 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_custom_split_merge=0 --enable_do_not_compress_roles=1 --enable_index_compression=1 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=1 --enable_sst_partitioner_factory=1 --enable_thread_tracking=1 --enable_write_thread_adaptive_yield=1 --error_recovery_with_no_fault_injection=0 --exclude_wal_from_write_fault_injection=1 --fail_if_options_file_error=0 --fifo_allow_compaction=1 --file_checksum_impl=crc32c --fill_cache=0 --flush_one_in=1000000 --format_version=2 --get_all_column_family_metadata_one_in=1000000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=1000000 --get_properties_of_all_tables_one_in=1000000 --get_property_one_in=100000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0.5 --index_block_restart_interval=13 --index_shortening=0 --index_type=0 --ingest_external_file_one_in=0 --initial_auto_readahead_size=0 --inplace_update_support=0 --iterpercent=10 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100 --last_level_temperature=kUnknown --level_compaction_dynamic_level_bytes=0 --lock_wal_one_in=1000000 --log_file_time_to_roll=0 --log_readahead_size=0 --long_running_snapshots=1 --low_pri_pool_ratio=0 --lowest_used_cache_tier=0 --manifest_preallocation_size=5120 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=0 --max_auto_readahead_size=524288 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=100000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=1073741824 --max_sequential_skip_in_iterations=8 --max_total_wal_size=0 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=10 --max_write_buffer_size_to_maintain=4194304 --memtable_insert_hint_per_batch=1 --memtable_max_range_deletions=0 --memtable_prefix_bloom_size_ratio=0.5 --memtable_protection_bytes_per_key=0 --memtable_whole_key_filtering=0 --memtablerep=skip_list --metadata_charge_policy=1 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=0 --min_write_buffer_number_to_merge=1 --mmap_read=0 --mock_direct_io=True --nooverwritepercent=1 --num_file_reads_for_auto_readahead=1 --open_files=-1 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=8 --open_read_fault_one_in=0 --open_write_fault_one_in=16 --ops_per_thread=100000000 --optimize_filters_for_hits=1 --optimize_filters_for_memory=1 --optimize_multiget_for_io=1 --paranoid_file_checks=0 --partition_filters=0 --partition_pinning=3 --pause_background_one_in=10000 --periodic_compaction_seconds=0 --prefix_size=8 --prefixpercent=5 --prepopulate_block_cache=1 --preserve_internal_time_seconds=0 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=0 --read_fault_one_in=1000 --readahead_size=16384 --readpercent=45 --recycle_log_file_num=1 --reopen=0 --report_bg_io_stats=1 --reset_stats_one_in=10000 --sample_for_compression=0 --secondary_cache_fault_one_in=0 --secondary_cache_uri= --set_options_one_in=0 --skip_stats_update_on_db_open=0 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=1048576 --sqfc_name=bar --sqfc_version=2 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=10 --stats_history_buffer_size=1048576 --strict_bytes_per_sync=1 --subcompactions=4 --sync=0 --sync_fault_injection=1 --table_cache_numshardbits=-1 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=3 --uncache_aggressiveness=113 --universal_max_read_amp=4 --unpartitioned_pinning=3 --use_adaptive_mutex=1 --use_adaptive_mutex_lru=1 --use_attribute_group=0 --use_delta_encoding=1 --use_direct_io_for_flush_and_compaction=1 --use_direct_reads=0 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=0 --use_multi_get_entity=1 --use_multiget=0 --use_put_entity_one_in=0 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=5 --use_write_buffer_manager=1 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_compression=0 --verify_db_one_in=10000 --verify_file_checksums_one_in=1000 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=zstd --write_buffer_size=33554432 --write_dbid_to_manifest=0 --write_fault_one_in=128 --writepercent=35 --preserve_unverified_changes=1 --db=/dev/shm/rocksdb_test/blackbox --expected_values_dir=/dev/shm/rocksdb_test/expected\nChoosing random keys with no overwrite\n...\n(Re-)verified 0 unique IDs\n2024/07/01-16:42:46 Initializing worker threads\nCrash-recovery verification passed :)\n2024/07/01-16:42:46 Starting database operations\n^C\n\n./db_stress --WAL_size_limit_MB=0 --WAL_ttl_seconds=60 --acquire_snapshot_one_in=10000 --adaptive_readahead=0 --adm_policy=1 --advise_random_on_open=1 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --allow_fallocate=0 --async_io=1 --auto_readahead_size=0 --avoid_flush_during_recovery=0 --avoid_flush_during_shutdown=0 --avoid_unnecessary_blocking_io=1 --backup_max_size=104857600 --backup_one_in=1000 --batch_protection_bytes_per_key=8 --bgerror_resume_retry_interval=1000000 --block_align=0 --block_protection_bytes_per_key=1 --block_size=16384 --bloom_before_level=0 --bloom_bits=8 --bottommost_compression_type=snappy --bottommost_file_compaction_delay=600 --bytes_per_sync=0 --cache_index_and_filter_blocks=0 --cache_index_and_filter_blocks_with_high_priority=1 --cache_size=33554432 --cache_type=auto_hyper_clock_cache --charge_compression_dictionary_building_buffer=1 --charge_file_metadata=0 --charge_filter_construction=1 --charge_table_reader=1 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=10000 --checksum_type=kxxHash64 --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000 --compact_range_one_in=1000000 --compaction_pri=4 --compaction_readahead_size=1048576 --compaction_ttl=0 --compress_format_version=1 --compressed_secondary_cache_size=16777216 --compression_checksum=1 --compression_max_dict_buffer_bytes=1099511627775 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=zstd --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --daily_offpeak_time_utc= --data_block_index_type=1 --db_write_buffer_size=134217728 --default_temperature=kHot --default_write_temperature=kCold --delete_obsolete_files_period_micros=21600000000 --delpercent=4 --delrangepercent=1 --destroy_db_initially=0 --detect_filter_construct_corruption=1 --disable_file_deletions_one_in=1000000 --disable_manual_compaction_one_in=10000 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_custom_split_merge=0 --enable_do_not_compress_roles=1 --enable_index_compression=1 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=1 --enable_sst_partitioner_factory=1 --enable_thread_tracking=1 --enable_write_thread_adaptive_yield=1 --error_recovery_with_no_fault_injection=0 --exclude_wal_from_write_fault_injection=1 --fail_if_options_file_error=0 --fifo_allow_compaction=1 --file_checksum_impl=crc32c --fill_cache=0 --flush_one_in=1000000 --format_version=2 --get_all_column_family_metadata_one_in=1000000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=1000000 --get_properties_of_all_tables_one_in=1000000 --get_property_one_in=100000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0.5 --index_block_restart_interval=13 --index_shortening=0 --index_type=0 --ingest_external_file_one_in=0 --initial_auto_readahead_size=0 --inplace_update_support=0 --iterpercent=10 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100 --last_level_temperature=kUnknown --level_compaction_dynamic_level_bytes=0 --lock_wal_one_in=1000000 --log_file_time_to_roll=0 --log_readahead_size=0 --long_running_snapshots=1 --low_pri_pool_ratio=0 --lowest_used_cache_tier=0 --manifest_preallocation_size=5120 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=0 --max_auto_readahead_size=524288 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=100000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=1073741824 --max_sequential_skip_in_iterations=8 --max_total_wal_size=0 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=10 --max_write_buffer_size_to_maintain=4194304 --memtable_insert_hint_per_batch=1 --memtable_max_range_deletions=0 --memtable_prefix_bloom_size_ratio=0.5 --memtable_protection_bytes_per_key=0 --memtable_whole_key_filtering=0 --memtablerep=skip_list --metadata_charge_policy=1 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=0 --min_write_buffer_number_to_merge=1 --mmap_read=0 --mock_direct_io=True --nooverwritepercent=1 --num_file_reads_for_auto_readahead=1 --open_files=-1 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=8 --open_read_fault_one_in=0 --open_write_fault_one_in=16 --ops_per_thread=100000000 --optimize_filters_for_hits=1 --optimize_filters_for_memory=1 --optimize_multiget_for_io=1 --paranoid_file_checks=0 --partition_filters=0 --partition_pinning=3 --pause_background_one_in=10000 --periodic_compaction_seconds=0 --prefix_size=8 --prefixpercent=5 --prepopulate_block_cache=1 --preserve_internal_time_seconds=0 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=0 --read_fault_one_in=1000 --readahead_size=16384 --readpercent=45 --recycle_log_file_num=1 --reopen=0 --report_bg_io_stats=1 --reset_stats_one_in=10000 --sample_for_compression=0 --secondary_cache_fault_one_in=0 --secondary_cache_uri= --set_options_one_in=0 --skip_stats_update_on_db_open=0 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=1048576 --sqfc_name=bar --sqfc_version=2 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=10 --stats_history_buffer_size=1048576 --strict_bytes_per_sync=1 --subcompactions=4 --sync=0 --sync_fault_injection=1 --table_cache_numshardbits=-1 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=3 --uncache_aggressiveness=113 --universal_max_read_amp=4 --unpartitioned_pinning=3 --use_adaptive_mutex=1 --use_adaptive_mutex_lru=1 --use_attribute_group=0 --use_delta_encoding=1 --use_direct_io_for_flush_and_compaction=1 --use_direct_reads=0 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=0 --use_multi_get_entity=1 --use_multiget=0 --use_put_entity_one_in=0 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=5 --use_write_buffer_manager=1 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_compression=0 --verify_db_one_in=10000 --verify_file_checksums_one_in=1000 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=zstd --write_buffer_size=33554432 --write_dbid_to_manifest=0 --write_fault_one_in=128 --writepercent=35 --preserve_unverified_changes=1 --db=/dev/shm/rocksdb_test/blackbox --expected_values_dir=/dev/shm/rocksdb_test/expected\nChoosing random keys with no overwrite\n...\nCrash-recovery verification passed :)\n2024/07/01-16:43:02 Starting database operations\nFailure in BackupEngine::CreateNewBackup with: Corruption: bad record length under specified BackupEngineOptions: share_table_files: 1, share_files_with_checksum: 1, share_files_with_checksum_naming: 2147483650, schema_version: 1, max_background_operations: 1, backup_rate_limiter: 0x7f2373676280, restore_rate_limiter: 0, current_temperatures_override_manifest: 1, CreateBackupOptions: flush_before_backup: 0, decrease_background_thread_cpu_priority: 0, background_thread_cpu_priority: 2, RestoreOptions: keep_log_files: 1 (Empty string or missing field indicates default option or value is used)\nVerification failed: Backup/restore failed: Corruption: bad record length\ndb_stress: db_stress_tool/db_stress_test_base.cc:528: void rocksdb::StressTest::ProcessStatus(rocksdb::SharedState*, std::string, const rocksdb::Status&, bool) const: Assertion `false' failed.\nReceived signal 6 (Aborted)\nInvoking GDB for stack trace...\n^CCouldn't get CS register: No such process.\nCouldn't get registers: No such process.\n[Inferior 1 (process 2097222) detached]\n```\n\nReviewed By: pdillinger\n\nDifferential Revision: D59260401\n\nPulled By: cbi42\n\nfbshipit-source-id: fdcdaaab2e14b527b26fbdfa819b4fe3f745a4de","shortMessageHtmlLink":"Fix WAL corruption in stress test (#12834)"}},{"before":"84296bc2489ab24e023fe3d9bc93160f6b91fe29","after":"0bb939611dd80a2a4cb07d4fb526cc39304d9a1d","ref":"refs/heads/main","pushedAt":"2024-07-02T06:32:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831)\n\nSummary:\nWe are seeing a number of crash test failures coming from checkpoint and backup code, likely from WalManager::GetSortedWalFiles -> ... -> WalManager::ReadFirstLine and this code path is not needed, because we don't need to know the sequence numbers of WAL files going into a checkpoint or backup. We can minimize the impact of whatever inconsistency is causing that problem by not relying on it where it's not needed.\n\nSimilarly, when we only need a roughly accurate set of current WAL files, we don't need to query all the archived WAL files (and redundantly the live ones again).\n\nSo this reduces filesystem queries and DB mutex acquires in creating backups and checkpoints.\n\nNeeded follow-up:\nFigure out what is causing various failures with an apparent inconsistency where GetSortedWalFiles fails on reading a WAL file. If it's an injected failure, perhaps it's not propagating that injected failure appropriately. It might also be an inconsistency between what the DB knows is flushed and what WalManager reads from the filesystem (which we know is dubious and should be phased out, which this is arguably another step toward). Or completing that phase-out might solve the problem without a full diagnosis.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12831\n\nTest Plan:\nexisting tests (easily caught when I went too far in initally developing this change)\n\nUpdate to BackupUsingDirectIO test so that there's a WAL file in what is backed up. (Was relying on some oddity.)\n\nReviewed By: cbi42\n\nDifferential Revision: D59252649\n\nPulled By: pdillinger\n\nfbshipit-source-id: 7ad4187a1c70caa59a6d6c1c643ef95232b929f5","shortMessageHtmlLink":"Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831)"}},{"before":"093f4ef82cdd599896517f5414ee9ae7f6af3b35","after":"84296bc2489ab24e023fe3d9bc93160f6b91fe29","ref":"refs/heads/main","pushedAt":"2024-07-01T23:43:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Reset seen_injected_error_ with seen_error_ (#12830)\n\nSummary:\n**Context/Summary** : as titled as seen_injected_error_ is a subcategory of seen_error_\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12830\n\nTest Plan: existing CI as it only affects crash test code\n\nReviewed By: jaykorean\n\nDifferential Revision: D59249018\n\nPulled By: hx235\n\nfbshipit-source-id: 20e4c22cade57e12a104a03999e4c841a3648b11","shortMessageHtmlLink":"Reset seen_injected_error_ with seen_error_ (#12830)"}},{"before":"9eebaf11cbd875435b572f05f0378ecdb761cc74","after":"093f4ef82cdd599896517f5414ee9ae7f6af3b35","ref":"refs/heads/main","pushedAt":"2024-07-01T23:17:18.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix db_rate_limiter_test for win (#12816)\n\nSummary:\nWe didn't implement file system prefetch for OS Win. During table open, it uses `FilePrefetchBuffer` instead and only do 1 read instead of 4 in BufferedIO.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12816\n\nReviewed By: jaykorean\n\nDifferential Revision: D59181835\n\nPulled By: ajkr\n\nfbshipit-source-id: 18b8f0247408cd1a80f289357ede5232ae5a3c66","shortMessageHtmlLink":"Fix db_rate_limiter_test for win (#12816)"}},{"before":"69ad597b4600d17085672d93993ff9e538ed4663","after":"9eebaf11cbd875435b572f05f0378ecdb761cc74","ref":"refs/heads/main","pushedAt":"2024-07-01T19:24:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix stress test `SetOptions()` setting incompatible options (#12827)\n\nSummary:\nTo fix errors like \"Verification failed: SetOptions failed: Invalid argument: max_successive_merges > 0 is incompatible with unordered_write\".\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12827\n\nTest Plan: no new crash test failure due to this option combination.\n\nReviewed By: hx235\n\nDifferential Revision: D59233002\n\nPulled By: cbi42\n\nfbshipit-source-id: 2a3e4d57a56f07bdda49ea36f0f9f6a30f17bbc3","shortMessageHtmlLink":"Fix stress test SetOptions() setting incompatible options (#12827)"}},{"before":"c7e94bc878013b80a4288c5ddf5344a8e21d668e","after":"69ad597b4600d17085672d93993ff9e538ed4663","ref":"refs/heads/main","pushedAt":"2024-07-01T17:59:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Disable fault injection for TestGetProperty (#12825)\n\nSummary:\n**Context/Summary:**\nSee titled; along with one more minor fix to other disabling\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12825\n\nTest Plan: CI won't show `Failed to get DB property: rocksdb.aggregated-table-properties`\n\nReviewed By: jaykorean\n\nDifferential Revision: D59231819\n\nPulled By: hx235\n\nfbshipit-source-id: a8e73c9e06eeceb4c6025a4885823a3eba25c359","shortMessageHtmlLink":"Disable fault injection for TestGetProperty (#12825)"}},{"before":"22fe23edc89e9842ed72b613de172cd80d3b00da","after":"c7e94bc878013b80a4288c5ddf5344a8e21d668e","ref":"refs/heads/main","pushedAt":"2024-07-01T16:45:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Disable WAL write error injection when reopen with WAL (#12820)\n\nSummary:\n**Context/Summary:**\n\nRight now we need to persist WAL data before closing for reopen when `reopen > 0` and `disable_wal = false` https://github.com/facebook/rocksdb/blame/71f9e6b5b36e3223e8dba29df75e4e5008818d16/db_stress_tool/db_stress_test_base.cc#L3479. Previous injected WAL write errors may not be cleared by the time of closing and ready or persisting WAL. To simplify, we disable any WAL write error injection when `reopen > 0` and `disable_wal = false`\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12820\n\nTest Plan:\nBelow command failed `Error persisting WAL data which is needed before reopening the DB: IO error: Writer has previous injected error.` with but passes after we disable WAL write error injection (exclude_wal_from_write_fault_injection=1, metadata_write_fault_one_in=0)\n```\n./db_stress --WAL_size_limit_MB=0 --WAL_ttl_seconds=0 --acquire_snapshot_one_in=0 --adaptive_readahead=0 --adm_policy=1 --advise_random_on_open=0 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --allow_fallocate=1 --async_io=0 --auto_readahead_size=0 --avoid_flush_during_recovery=1 --avoid_flush_during_shutdown=1 --avoid_unnecessary_blocking_io=0 --backup_max_size=104857600 --backup_one_in=1000 --batch_protection_bytes_per_key=0 --bgerror_resume_retry_interval=100 --block_align=0 --block_protection_bytes_per_key=8 --block_size=16384 --bloom_before_level=2147483646 --bloom_bits=8.890585558621982 --bottommost_compression_type=zstd --bottommost_file_compaction_delay=0 --bytes_per_sync=0 --cache_index_and_filter_blocks=1 --cache_index_and_filter_blocks_with_high_priority=0 --cache_size=8388608 --cache_type=auto_hyper_clock_cache --charge_compression_dictionary_building_buffer=1 --charge_file_metadata=0 --charge_filter_construction=1 --charge_table_reader=1 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=10000 --checksum_type=kxxHash --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000 --compact_range_one_in=0 --compaction_pri=0 --compaction_readahead_size=0 --compaction_ttl=0 --compress_format_version=2 --compressed_secondary_cache_size=16777216 --compression_checksum=1 --compression_max_dict_buffer_bytes=1 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=snappy --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --daily_offpeak_time_utc= --data_block_index_type=1 --db_write_buffer_size=1048576 --default_temperature=kCold --default_write_temperature=kCold --delete_obsolete_files_period_micros=21600000000 --delpercent=0 --delrangepercent=0 --destroy_db_initially=0 --detect_filter_construct_corruption=0 --disable_file_deletions_one_in=1000000 --disable_manual_compaction_one_in=1000000 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_custom_split_merge=1 --enable_do_not_compress_roles=1 --enable_index_compression=1 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=0 --enable_sst_partitioner_factory=0 --enable_thread_tracking=0 --enable_write_thread_adaptive_yield=1 --error_recovery_with_no_fault_injection=0 --exclude_wal_from_write_fault_injection=0 --fail_if_options_file_error=1 --fifo_allow_compaction=1 --file_checksum_impl=xxh64 --fill_cache=0 --flush_one_in=1000000 --format_version=5 --get_all_column_family_metadata_one_in=10000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=1000000 --get_properties_of_all_tables_one_in=1000000 --get_property_one_in=1000000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0 --index_block_restart_interval=1 --index_shortening=0 --index_type=0 --ingest_external_file_one_in=1000 --initial_auto_readahead_size=524288 --inplace_update_support=1 --iterpercent=0 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100 --last_level_temperature=kUnknown --level_compaction_dynamic_level_bytes=1 --lock_wal_one_in=10000 --log2_keys_per_lock=10 --log_file_time_to_roll=60 --log_readahead_size=0 --long_running_snapshots=0 --low_pri_pool_ratio=0 --lowest_used_cache_tier=1 --manifest_preallocation_size=5120 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=10 --max_auto_readahead_size=524288 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=100000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=1073741824 --max_sequential_skip_in_iterations=1 --max_total_wal_size=0 --max_write_batch_group_size_bytes=64 --max_write_buffer_number=10 --max_write_buffer_size_to_maintain=8388608 --memtable_insert_hint_per_batch=0 --memtable_max_range_deletions=0 --memtable_prefix_bloom_size_ratio=0.01 --memtable_protection_bytes_per_key=2 --memtable_whole_key_filtering=1 --memtablerep=skip_list --metadata_charge_policy=0 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=0 --min_write_buffer_number_to_merge=2 --mmap_read=0 --mock_direct_io=False --nooverwritepercent=1 --num_file_reads_for_auto_readahead=0 --open_files=500000 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=0 --open_read_fault_one_in=0 --open_write_fault_one_in=0 --ops_per_thread=2000 --optimize_filters_for_hits=1 --optimize_filters_for_memory=0 --optimize_multiget_for_io=1 --paranoid_file_checks=1 --partition_filters=0 --partition_pinning=3 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefix_size=5 --prefixpercent=0 --prepopulate_block_cache=0 --preserve_internal_time_seconds=3600 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=0 --read_fault_one_in=1000 --readahead_size=0 --readpercent=0 --recycle_log_file_num=1 --reopen=20 --report_bg_io_stats=0 --reset_stats_one_in=10000 --sample_for_compression=0 --secondary_cache_fault_one_in=0 --secondary_cache_uri= --skip_stats_update_on_db_open=1 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=68719476736 --sqfc_name=bar --sqfc_version=1 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=10 --stats_history_buffer_size=1048576 --strict_bytes_per_sync=1 --subcompactions=1 --sync=0 --sync_fault_injection=0 --table_cache_numshardbits=-1 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=1 --uncache_aggressiveness=3 --universal_max_read_amp=10 --unpartitioned_pinning=1 --use_adaptive_mutex=0 --use_adaptive_mutex_lru=1 --use_attribute_group=1 --use_delta_encoding=1 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=1 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=1 --use_multi_get_entity=0 --use_multiget=1 --use_put_entity_one_in=0 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=0 --use_write_buffer_manager=1 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000 --verify_compression=1 --verify_db_one_in=100000 --verify_file_checksums_one_in=1000000 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=none --write_buffer_size=33554432 --write_dbid_to_manifest=1 --write_fault_one_in=10 --writepercent=100\n```\n\nReviewed By: jaykorean\n\nDifferential Revision: D59119811\n\nPulled By: hx235\n\nfbshipit-source-id: bcc3839567b38f939a66aa55d539f2e6a6e94cba","shortMessageHtmlLink":"Disable WAL write error injection when reopen with WAL (#12820)"}},{"before":"8c1558a3e0ed7178a57fcf639e1443dc3f1d1f44","after":"22fe23edc89e9842ed72b613de172cd80d3b00da","ref":"refs/heads/main","pushedAt":"2024-06-29T02:53:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix unknown flag \"manual_wal_flush\" (#12823)\n\nSummary:\n- Fix `manual_wal_flush` -> `manual_wal_flush_one_in`\n- auto-formatter fixed format in other settings\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12823\n\nTest Plan: CI\n\nReviewed By: hx235\n\nDifferential Revision: D59177107\n\nPulled By: jaykorean\n\nfbshipit-source-id: 2400b2822f42299d03e150e3a098c62e7fdaf1f8","shortMessageHtmlLink":"Fix unknown flag \"manual_wal_flush\" (#12823)"}},{"before":"0b10e7dbbae4773e56ae6de5cb3d699faaa7ac1f","after":"8c1558a3e0ed7178a57fcf639e1443dc3f1d1f44","ref":"refs/heads/main","pushedAt":"2024-06-29T01:59:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add some documentation for Env related interfaces (#12813)\n\nSummary:\nAs titled. Added some documentations for some `Env` interfaces and removed some obsolete doc for `Options.env`.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12813\n\nReviewed By: ajkr\n\nDifferential Revision: D59119632\n\nPulled By: jowlyzhang\n\nfbshipit-source-id: 114b13f0f843cde6ebc0746156b80c94ea2ce143","shortMessageHtmlLink":"Add some documentation for Env related interfaces (#12813)"}},{"before":"5bb7f95ed6688b914841749e0d192986ce4c252f","after":"0b10e7dbbae4773e56ae6de5cb3d699faaa7ac1f","ref":"refs/heads/main","pushedAt":"2024-06-28T19:06:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Ignore more non-critical IO error in BlockCacheLookupForReadAheadSize() in crash test (#12822)\n\nSummary:\n**Context/Summary:**\nSimilar to https://github.com/facebook/rocksdb/pull/12814#issue-2376803461\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12822\n\nTest Plan: CI\n\nReviewed By: jaykorean\n\nDifferential Revision: D59166188\n\nPulled By: hx235\n\nfbshipit-source-id: 68c2eb7b319103ede0ba34944a0737440aecb17f","shortMessageHtmlLink":"Ignore more non-critical IO error in BlockCacheLookupForReadAheadSize…"}},{"before":"aec15eebec08429142fde04a4006303412def90c","after":"5bb7f95ed6688b914841749e0d192986ce4c252f","ref":"refs/heads/main","pushedAt":"2024-06-28T19:00:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Don't take archived log size into account when calculating log size for flush (#12680)\n\nSummary:\n**Context/Summary:**\nIt seems unreasonable to take the archived log size into account when calculating log size **for flush** in method CreateCheckpoint. If the user sets WAL_ttl_seconds or WAL_size_limit_MB, the argument _log_size_for_flush_ can easily be reached due to the size of the archived dir. As a result, the flush may always be triggered.\n**Test**\ncorverd by ./checkpoint_test\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12680\n\nReviewed By: jaykorean\n\nDifferential Revision: D59097904\n\nPulled By: ajkr\n\nfbshipit-source-id: 0ed29c1b078d8f40b85288541b008e00dbc517d3","shortMessageHtmlLink":"Don't take archived log size into account when calculating log size f…"}},{"before":"a31fe521732c6150003ea43f1e30f27f13be597c","after":"aec15eebec08429142fde04a4006303412def90c","ref":"refs/heads/main","pushedAt":"2024-06-27T06:05:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Ignore non-critical IO error in `BlockCacheLookupForReadAheadSize()` in crash test (#12814)\n\nSummary:\n**Context/Summary:**\n\nError in `BlockCacheLookupForReadAheadSize()` is not critical enough to return such error in read path. That's because the worst case is to not have any read ahead. See below comment. https://github.com/facebook/rocksdb/blob/a31fe521732c6150003ea43f1e30f27f13be597c/table/block_based/block_based_table_iterator.cc#L867-L871\n\nTherefore we should allow the read to return ok() even when we inject read error there.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12814\n\nTest Plan:\nBelow command failed with ` Didn't get expected error from PrefixScan` before the fix but passes after\n\n```\n./db_stress --WAL_size_limit_MB=0 --WAL_ttl_seconds=60 --acquire_snapshot_one_in=100 --adaptive_readahead=0 --adm_policy=3 --advise_random_on_open=0 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --allow_fallocate=1 --async_io=0 --auto_readahead_size=1 --avoid_flush_during_recovery=0 --avoid_flush_during_shutdown=1 --avoid_unnecessary_blocking_io=1 --backup_max_size=104857600 --backup_one_in=1000 --batch_protection_bytes_per_key=8 --bgerror_resume_retry_interval=1000000 --block_align=0 --block_protection_bytes_per_key=1 --block_size=16384 --bloom_before_level=5 --bloom_bits=29.31310447925055 --bottommost_compression_type=lz4hc --bottommost_file_compaction_delay=0 --bytes_per_sync=262144 --cache_index_and_filter_blocks=1 --cache_index_and_filter_blocks_with_high_priority=0 --cache_size=8388608 --cache_type=tiered_auto_hyper_clock_cache --charge_compression_dictionary_building_buffer=0 --charge_file_metadata=1 --charge_filter_construction=0 --charge_table_reader=1 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=1000000 --checksum_type=kxxHash64 --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_pri=0 --compaction_readahead_size=0 --compaction_ttl=0 --compress_format_version=2 --compressed_secondary_cache_ratio=0.6666666666666666 --compressed_secondary_cache_size=0 --compression_checksum=1 --compression_max_dict_buffer_bytes=0 --compression_max_dict_bytes=0 --compression_parallel_threads=1 --compression_type=lz4 --compression_use_zstd_dict_trainer=0 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --daily_offpeak_time_utc= --data_block_index_type=0 --db=/dev/shm/rocksdb_test/rocksdb_crashtest_whitebox --db_write_buffer_size=8388608 --default_temperature=kHot --default_write_temperature=kHot --delete_obsolete_files_period_micros=30000000 --delpercent=4 --delrangepercent=1 --destroy_db_initially=0 --detect_filter_construct_corruption=1 --disable_file_deletions_one_in=10000 --disable_manual_compaction_one_in=10000 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_custom_split_merge=0 --enable_do_not_compress_roles=1 --enable_index_compression=0 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=1 --enable_sst_partitioner_factory=0 --enable_thread_tracking=1 --enable_write_thread_adaptive_yield=1 --error_recovery_with_no_fault_injection=0 --exclude_wal_from_write_fault_injection=0 --expected_values_dir=/dev/shm/rocksdb_test/rocksdb_crashtest_expected --fail_if_options_file_error=0 --fifo_allow_compaction=0 --file_checksum_impl=big --fill_cache=1 --flush_one_in=1000 --format_version=3 --get_all_column_family_metadata_one_in=1000000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=10000 --get_properties_of_all_tables_one_in=100000 --get_property_one_in=1000000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0 --index_block_restart_interval=2 --index_shortening=0 --index_type=2 --ingest_external_file_one_in=1000 --initial_auto_readahead_size=0 --inplace_update_support=0 --iterpercent=10 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100 --last_level_temperature=kHot --level_compaction_dynamic_level_bytes=0 --lock_wal_one_in=1000000 --log2_keys_per_lock=10 --log_file_time_to_roll=60 --log_readahead_size=0 --long_running_snapshots=0 --low_pri_pool_ratio=0 --lowest_used_cache_tier=2 --manifest_preallocation_size=0 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=0 --max_auto_readahead_size=16384 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=100000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=1073741824 --max_sequential_skip_in_iterations=1 --max_total_wal_size=0 --max_write_batch_group_size_bytes=16 --max_write_buffer_number=10 --max_write_buffer_size_to_maintain=2097152 --memtable_insert_hint_per_batch=0 --memtable_max_range_deletions=0 --memtable_prefix_bloom_size_ratio=0.1 --memtable_protection_bytes_per_key=8 --memtable_whole_key_filtering=1 --memtablerep=skip_list --metadata_charge_policy=0 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=128 --min_write_buffer_number_to_merge=2 --mmap_read=0 --mock_direct_io=True --nooverwritepercent=1 --num_file_reads_for_auto_readahead=1 --open_files=100 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=8 --open_read_fault_one_in=0 --open_write_fault_one_in=0 --ops_per_thread=20000000 --optimize_filters_for_hits=0 --optimize_filters_for_memory=1 --optimize_multiget_for_io=0 --paranoid_file_checks=1 --partition_filters=1 --partition_pinning=2 --pause_background_one_in=10000 --periodic_compaction_seconds=0 --prefix_size=5 --prefixpercent=5 --prepopulate_block_cache=1 --preserve_internal_time_seconds=0 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=32 --read_fault_one_in=1000 --readahead_size=524288 --readpercent=45 --recycle_log_file_num=0 --reopen=20 --report_bg_io_stats=1 --reset_stats_one_in=1000000 --sample_for_compression=5 --secondary_cache_fault_one_in=32 --secondary_cache_uri= --skip_stats_update_on_db_open=1 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=1048576 --sqfc_name=foo --sqfc_version=1 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=600 --stats_history_buffer_size=0 --strict_bytes_per_sync=1 --subcompactions=2 --sync=0 --sync_fault_injection=0 --table_cache_numshardbits=6 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=1 --uncache_aggressiveness=203 --universal_max_read_amp=10 --unpartitioned_pinning=0 --use_adaptive_mutex=1 --use_adaptive_mutex_lru=1 --use_attribute_group=0 --use_delta_encoding=0 --use_direct_io_for_flush_and_compaction=1 --use_direct_reads=0 --use_full_merge_v1=1 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=0 --use_multi_get_entity=0 --use_multiget=1 --use_put_entity_one_in=0 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=0 --use_write_buffer_manager=0 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_compression=1 --verify_db_one_in=10000 --verify_file_checksums_one_in=1000 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=none --write_buffer_size=33554432 --write_dbid_to_manifest=0 --write_fault_one_in=1000 --writepercent=35\n```\n\nReviewed By: jaykorean\n\nDifferential Revision: D59092430\n\nPulled By: hx235\n\nfbshipit-source-id: 39558c34461ce92275cae706c33dfd00e6f0ecce","shortMessageHtmlLink":"Ignore non-critical IO error in BlockCacheLookupForReadAheadSize() …"}},{"before":"0d93c8a6cae309b5708641a8694ee0963014d633","after":"a31fe521732c6150003ea43f1e30f27f13be597c","ref":"refs/heads/main","pushedAt":"2024-06-27T01:20:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Remove the return value of `SetBGError()` (#12792)\n\nSummary:\nthe return value for `ErrorHandler::SetBGError(error)` seems to be not well-defined, it can be `bg_error_` (no matter if the `bg_error_` is set to the input error), ok status or [`recovery_error_`](https://github.com/facebook/rocksdb/blob/3ee4d5a11a882056b341a9a1694a71371a39f664/db/error_handler.cc#L669) from `StartRecoverFromRetryableBGIOError()`. The `recovery_error_` returned may be an OK status.\n\nWe have only a few places that use the return value of `SetBGError()` and they don't need to do so. Using the return value may even be wrong for example in https://github.com/facebook/rocksdb/blob/3ee4d5a11a882056b341a9a1694a71371a39f664/db/db_impl/db_impl_write.cc#L2365 where a non-ok `s` could be overwritten to OK. This PR changes SetBGError() to return void and clean up relevant code.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12792\n\nTest Plan: existing unit tests and go over all places where return value of `SetBGError()` is used.\n\nReviewed By: hx235\n\nDifferential Revision: D58904898\n\nPulled By: cbi42\n\nfbshipit-source-id: d58a20ba5a40e3f35367c6034a32c755088c3653","shortMessageHtmlLink":"Remove the return value of SetBGError() (#12792)"}},{"before":"41c6b4549ab0dccfb15b7af038f7803cbe8c5f23","after":"0d93c8a6cae309b5708641a8694ee0963014d633","ref":"refs/heads/main","pushedAt":"2024-06-26T21:59:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Decouple sync fault and write injection in FaultInjectionTestFS & fix tracing issue under WAL write error injection (#12797)\n\nSummary:\n**Context/Summary:**\n\nAfter injecting write error to WAL, we started to see crash recovery verification failure in prefix recovery. That's because the current tracing implementation traces every write before it writes to WAL even when the WAL write can fail with write error injection. One consequence of that is the traced writes in trace files does not corresponding to write sequence sequence anymore e.g, it has more traced writes that the actual assigned sequence number to successful writes. Therefore https://github.com/facebook/rocksdb/blob/b4a84efb4e842b782e976de5b22a4554c2f76edd/db_stress_tool/expected_state.cc#L674 won't restore the ExpectedState to the correct sequence number we want.\n\nIdeally, we should have a prepare-commit mechanism for tracing just like our ExpectedState so we can ignore the traced write if the write fails later. But for now, to simplify, we simply don't inject WAL error (and metadata write error cuz it could fail write when sync WAL dir fails)\n\nTo do so, we need to be able to exclude WAL from write injection but still allow sync fault injection in it to maintain its original sync fault testing coverage. This prompts us to decouple sync fault and write injection in FaultInjectionTestFS. And this is what this PR mainly about.\n\nSo now `FaultInjectionTestFS` works as the following:\n- If direct_writable is true, then `FaultInjectionTestFS` is bypassed for writable file\n- Otherwise, FaultInjectionTestFS` can buffer data for sync fault injection (if inject_unsynced_data_loss_ == true, global settings) and/or inject write error (if MaybeInjectThreadLocalError(), thread-local settings). WAL file can be optionally excluded from write injection\n\nBonus: better naming of relevant variables\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12797\n\nTest Plan:\n- The follow commands failed before this fix but passes after\n```\npython3 tools/db_crashtest.py --simple blackbox \\\n --interval=5 \\\n --preserve_unverified_changes=1 \\\n --threads=32 \\\n --disable_auto_compactions=1 \\\n --WAL_size_limit_MB=0 --WAL_ttl_seconds=0 --acquire_snapshot_one_in=0 --adaptive_readahead=0 --adm_policy=0 --advise_random_on_open=1 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --allow_fallocate=1 --async_io=0 --auto_readahead_size=0 --avoid_flush_during_recovery=1 --avoid_flush_during_shutdown=0 --avoid_unnecessary_blocking_io=0 --backup_max_size=104857600 --backup_one_in=0 --batch_protection_bytes_per_key=0 --bgerror_resume_retry_interval=1000000 --block_align=0 --block_protection_bytes_per_key=4 --block_size=16384 --bloom_before_level=2147483646 --bloom_bits=3.2003682301518492 --bottommost_compression_type=zlib --bottommost_file_compaction_delay=600 --bytes_per_sync=0 --cache_index_and_filter_blocks=1 --cache_index_and_filter_blocks_with_high_priority=1 --cache_size=33554432 --cache_type=fixed_hyper_clock_cache --charge_compression_dictionary_building_buffer=0 --charge_file_metadata=0 --charge_filter_construction=0 --charge_table_reader=1 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=0 --checksum_type=kxxHash64 --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=0 --compact_range_one_in=0 --compaction_pri=2 --compaction_readahead_size=0 --compaction_ttl=0 --compress_format_version=1 --compressed_secondary_cache_size=16777216 --compression_checksum=1 --compression_max_dict_buffer_bytes=549755813887 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=none --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --daily_offpeak_time_utc=00:00-23:59 --data_block_index_type=0 \\\n --db_write_buffer_size=0 --delete_obsolete_files_period_micros=0 --delpercent=0 --delrangepercent=0 --destroy_db_initially=0 --detect_filter_construct_corruption=0 --disable_file_deletions_one_in=0 --disable_manual_compaction_one_in=0 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=0 --enable_compaction_filter=0 --enable_custom_split_merge=0 --enable_do_not_compress_roles=1 --enable_index_compression=0 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=0 --enable_sst_partitioner_factory=0 --enable_thread_tracking=0 --enable_write_thread_adaptive_yield=0 --error_recovery_with_no_fault_injection=0 --fail_if_options_file_error=0 --fifo_allow_compaction=1 --file_checksum_impl=xxh64 --fill_cache=0 --flush_one_in=100 --format_version=4 --get_all_column_family_metadata_one_in=0 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=0 --get_properties_of_all_tables_one_in=0 --get_property_one_in=0 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0.5 --index_block_restart_interval=9 --index_shortening=1 --index_type=0 --ingest_external_file_one_in=0 --initial_auto_readahead_size=0 --inplace_update_support=0 --iterpercent=0 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=0 --last_level_temperature=kUnknown --level_compaction_dynamic_level_bytes=1 --lock_wal_one_in=0 --log2_keys_per_lock=10 --log_file_time_to_roll=0 --log_readahead_size=16777216 --long_running_snapshots=0 --low_pri_pool_ratio=0 --lowest_used_cache_tier=2 --manifest_preallocation_size=0 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=0 --max_auto_readahead_size=524288 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=1000 --max_key_len=3 --memtable_insert_hint_per_batch=0 --memtable_max_range_deletions=0 --memtable_prefix_bloom_size_ratio=0.5 --memtable_protection_bytes_per_key=8 --memtable_whole_key_filtering=0 --memtablerep=skip_list --metadata_charge_policy=0 --metadata_read_fault_one_in=0 --metadata_write_fault_one_in=0 --min_write_buffer_number_to_merge=1 --mmap_read=0 --mock_direct_io=False --nooverwritepercent=1 --num_file_reads_for_auto_readahead=0 --open_files=-1 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=0 --open_read_fault_one_in=0 --open_write_fault_one_in=0 --ops_per_thread=20000000 \\\n --optimize_filters_for_hits=1 --optimize_filters_for_memory=1 --optimize_multiget_for_io=0 --paranoid_file_checks=1 --partition_filters=0 --partition_pinning=3 --pause_background_one_in=0 --periodic_compaction_seconds=0 --prefix_size=1 --prefixpercent=0 --prepopulate_block_cache=0 --preserve_internal_time_seconds=0 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=0 --read_fault_one_in=0 --readahead_size=0 --readpercent=0 --recycle_log_file_num=0 --reopen=0 --report_bg_io_stats=0 --reset_stats_one_in=1000000 --sample_for_compression=5 --secondary_cache_fault_one_in=0 --secondary_cache_uri= --skip_stats_update_on_db_open=0 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=68719476736 --sqfc_name=bar --sqfc_version=1 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=10 --stats_history_buffer_size=0 --strict_bytes_per_sync=0 --subcompactions=1 --sync=0 --sync_fault_injection=1 --table_cache_numshardbits=0 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=3 --uncache_aggressiveness=9890 --universal_max_read_amp=-1 --unpartitioned_pinning=3 --use_adaptive_mutex=0 --use_adaptive_mutex_lru=1 --use_attribute_group=0 --use_delta_encoding=0 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=0 --use_multi_get_entity=0 --use_multiget=0 --use_put_entity_one_in=0 --use_sqfc_for_range_queries=0 --use_timed_put_one_in=0 --use_write_buffer_manager=0 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=0 --verify_checksum_one_in=0 --verify_compression=1 --verify_db_one_in=0 --verify_file_checksums_one_in=0 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=zstd --write_buffer_size=335544320 --write_dbid_to_manifest=1 --write_fault_one_in=100 --writepercent=100\n\n```\n- CI\n\nReviewed By: cbi42\n\nDifferential Revision: D58917145\n\nPulled By: hx235\n\nfbshipit-source-id: b6397036bea035a92341c2b05fb01872db2153d7","shortMessageHtmlLink":"Decouple sync fault and write injection in FaultInjectionTestFS & fix…"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEfkjJyAA","startCursor":null,"endCursor":null}},"title":"Activity · facebook/rocksdb"}