为什么缓冲区容量越大，“File：：read_to_end”越慢？

问：

通知：截至 2023 年 4 月 23 日，此问题的修复程序落在 rust-lang/rust：master 上。您很快就可以使用File：：read_to_end而不必担心这些。

我正在研究一个非常具体的问题，需要我读取数十万个文件，从几个字节到几百兆字节不等。由于大部分操作包括枚举文件和从磁盘移动数据，因此我求助于重用缓冲区进行文件读取，以期避免一些内存管理。Vec

就在这时，我遇到了意想不到的问题：缓冲区的容量越大，速度就越慢。首先读取 300MB 文件，然后读取一千个 1KB 文件比反之慢得多（只要我们不截断缓冲区）。file.read_to_end(&mut buffer)?

令人困惑的是，如果我将文件包装在 or use 中，则不会发生减速。Takeread_exact()

有谁知道这是怎么回事？它是否有可能在每次调用时（重新）初始化整个缓冲区？这是特定于 Windows 的怪癖吗？在处理此类问题时，您会推荐哪些（基于 Windows）的分析工具？

下面是一个简单的复制品，它演示了这些方法之间的巨大（在这台机器上是 50x+）性能差异，忽略磁盘速度：

use std::io::Read;
use std::fs::File;

// with a smaller buffer, there's basically no difference between the methods...
// const BUFFER_SIZE: usize = 2 * 1024;

// ...but the larger the Vec, the bigger the discrepancy.
// for simplicity's sake, let's assume this is a hard upper limit.
const BUFFER_SIZE: usize = 300 * 1024 * 1024;


fn naive() {
    let mut buffer = Vec::with_capacity(BUFFER_SIZE);

    for _ in 0..100 {
        let mut file = File::open("some_1kb_file.txt").expect("opening file");

        let metadata = file.metadata().expect("reading metadata");
        let len = metadata.len();
        assert!(len <= BUFFER_SIZE as u64);

        buffer.clear();
        file.read_to_end(&mut buffer).expect("reading file");

        // do "stuff" with buffer
        let check = buffer.iter().fold(0usize, |acc, x| acc.wrapping_add(*x as usize));

        println!("length: {len}, check: {check}");
    }
}

fn take() {
    let mut buffer = Vec::with_capacity(BUFFER_SIZE);

    for _ in 0..100 {
        let file = File::open("some_1kb_file.txt").expect("opening file");

        let metadata = file.metadata().expect("reading metadata");
        let len = metadata.len();
        assert!(len <= BUFFER_SIZE as u64);

        buffer.clear();
        file.take(len).read_to_end(&mut buffer).expect("reading file");

        // this also behaves like the straight `read_to_end` with a significant slowdown:
        // file.take(BUFFER_SIZE as u64).read_to_end(&mut buffer).expect("reading file");

        // do "stuff" with buffer
        let check = buffer.iter().fold(0usize, |acc, x| acc.wrapping_add(*x as usize));

        println!("length: {len}, check: {check}");
    }
}

fn exact() {
    let mut buffer = vec![0u8; BUFFER_SIZE];

    for _ in 0..100 {
        let mut file = File::open("some_1kb_file.txt").expect("opening file");

        let metadata = file.metadata().expect("reading metadata");
        let len = metadata.len() as usize;
        assert!(len <= BUFFER_SIZE);

        // SAFETY: initialized by `vec!` and within capacity by `assert!`
        unsafe { buffer.set_len(len); }
        file.read_exact(&mut buffer[0..len]).expect("reading file");

        // do "stuff" with buffer
        let check = buffer.iter().fold(0usize, |acc, x| acc.wrapping_add(*x as usize));

        println!("length: {len}, check: {check}");
    }
}

fn main() {
    let args: Vec<String> = std::env::args().collect();

    if args.len() < 2 {
        println!("usage: {} <method>", args[0]);
        return;
    }

    match args[1].as_str() {
        "naive" => naive(),
        "take" => take(),
        "exact" => exact(),
        _ => println!("Unknown method: {}", args[1]),
    }
}

尝试了几种模式组合，甚至没有显着差异。--releaseLTO+crt-static