如何使用 zlib 压缩缓冲区?

How to compress a buffer with zlib?

提问人:Richard Knop 提问时间:12/27/2010 最后编辑:Anurag SinghRichard Knop 更新时间:9/16/2020 访问量:65656

问:

zlib 网站上有一个使用示例:http://www.zlib.net/zlib_how.html

但是,在示例中,它们正在压缩文件。我想压缩存储在内存缓冲区中的二进制数据。我也不想将压缩缓冲区保存到磁盘。

基本上这是我的缓冲区:

fIplImageHeader->imageData = (char*)imageIn->getFrame();

如何用 zlib 压缩它?

我将不胜感激一些关于如何做到这一点的代码示例。

C++ 压缩 zlib

评论

0赞 Sergei Tachenov 12/27/2010
你提到的例子有什么问题?它确实压缩了内存中的缓冲区,只是它首先从文件中读取数据。你从其他地方获取数据,但其余的都是一样的——为什么不呢?

答:

3赞 Yippie-Ki-Yay 12/27/2010 #1

这不是您关于 zlib API 问题的直接答案,但您可能对与 zlib 配对的 boost::iostreams 库感兴趣。

这允许使用基本的“流”操作表示法使用驱动的打包算法,然后可以通过打开一些内存流并对其执行操作来轻松压缩数据。zlib<< data

在这种情况下,将自动为通过流的每个数据调用相应的打包过滤器。boost::iostreams

评论

0赞 Ben 8/25/2012
只是一个注释,但这意味着您必须包含 boost iostreams 库、boost zlib 库、libz 和 libzbz2,而仅包含 libz。因此,如果捆绑这些库是一个大小问题,那么在这种情况下最好避免提升。~本
14赞 kichik 12/27/2010 #2

您可以通过将 和 调用替换为指向数据的直接指针来轻松调整示例。对于 zlib 压缩(称为 deflate,因为您“取出所有数据”),您分配结构,调用,然后:fread()fwrite()z_streamdeflateInit()

  1. 填充要压缩的下一块数据next_in
  2. 设置为avail_innext_in
  3. 设置为应写入压缩数据的位置,该位置通常应是缓冲区内的指针,该指针会随着您的前进而前进next_out
  4. 设置为avail_outnext_out
  5. deflate
  6. 重复步骤 3-5,直到不为零(即输出缓冲区中的空间比 zlib 需要的要多 - 没有更多数据要写入)avail_out
  7. 在有要压缩的数据时重复步骤 1-6

最终,你打电话,你就完成了。deflateEnd()

你基本上是在给它输入和输出块,直到你用完了输入,它就没有输出了。

38赞 Jonas Engström 12/27/2010 #3

这是一个使用 zlib 打包缓冲区并将压缩内容保存在向量中的示例。

void compress_memory(void *in_data, size_t in_data_size, std::vector<uint8_t> &out_data)
{
 std::vector<uint8_t> buffer;

 const size_t BUFSIZE = 128 * 1024;
 uint8_t temp_buffer[BUFSIZE];

 z_stream strm;
 strm.zalloc = 0;
 strm.zfree = 0;
 strm.next_in = reinterpret_cast<uint8_t *>(in_data);
 strm.avail_in = in_data_size;
 strm.next_out = temp_buffer;
 strm.avail_out = BUFSIZE;

 deflateInit(&strm, Z_BEST_COMPRESSION);

 while (strm.avail_in != 0)
 {
  int res = deflate(&strm, Z_NO_FLUSH);
  assert(res == Z_OK);
  if (strm.avail_out == 0)
  {
   buffer.insert(buffer.end(), temp_buffer, temp_buffer + BUFSIZE);
   strm.next_out = temp_buffer;
   strm.avail_out = BUFSIZE;
  }
 }

 int deflate_res = Z_OK;
 while (deflate_res == Z_OK)
 {
  if (strm.avail_out == 0)
  {
   buffer.insert(buffer.end(), temp_buffer, temp_buffer + BUFSIZE);
   strm.next_out = temp_buffer;
   strm.avail_out = BUFSIZE;
  }
  deflate_res = deflate(&strm, Z_FINISH);
 }

 assert(deflate_res == Z_STREAM_END);
 buffer.insert(buffer.end(), temp_buffer, temp_buffer + BUFSIZE - strm.avail_out);
 deflateEnd(&strm);

 out_data.swap(buffer);
}

评论

0赞 Robert 2/1/2013
我刚刚读到一个有趣的方法,它使用一个标志来控制刷新状态,并在一个循环中完成。此外,值得注意的是,这同样适用于 std::string 代替向量,这非常适合通过网络或其他函数发送。
2赞 user673679 1/25/2015
这似乎很复杂。为什么要做这一切,而不仅仅是使用函数?compress
3赞 Josh Atkins 6/17/2015
压缩要求您知道输出大小并分配足够大的缓冲区;此方法允许您 realloc() 并具有动态扩展的缓冲区。
0赞 BullyWiiPlaza 8/4/2018
不错的代码,但会产生 3 次警告。您可能需要解决此问题。temp_bufferescapes the local scope
51赞 Huiwei 4/8/2012 #4

zlib.h具有您需要的所有功能:(或 ) 和 。有关答案,请参阅 zlib 的源代码。compresscompress2uncompress

ZEXTERN int ZEXPORT compress OF((Bytef *dest,   uLongf *destLen, const Bytef *source, uLong sourceLen));
/*
         Compresses the source buffer into the destination buffer.  sourceLen is
     the byte length of the source buffer.  Upon entry, destLen is the total size
     of the destination buffer, which must be at least the value returned by
     compressBound(sourceLen).  Upon exit, destLen is the actual size of the
     compressed buffer.

         compress returns Z_OK if success, Z_MEM_ERROR if there was not
     enough memory, Z_BUF_ERROR if there was not enough room in the output
     buffer.
*/

ZEXTERN int ZEXPORT uncompress OF((Bytef *dest,   uLongf *destLen, const Bytef *source, uLong sourceLen));
/*
         Decompresses the source buffer into the destination buffer.  sourceLen is
     the byte length of the source buffer.  Upon entry, destLen is the total size
     of the destination buffer, which must be large enough to hold the entire
     uncompressed data.  (The size of the uncompressed data must have been saved
     previously by the compressor and transmitted to the decompressor by some
     mechanism outside the scope of this compression library.) Upon exit, destLen
     is the actual size of the uncompressed buffer.

         uncompress returns Z_OK if success, Z_MEM_ERROR if there was not
     enough memory, Z_BUF_ERROR if there was not enough room in the output
     buffer, or Z_DATA_ERROR if the input data was corrupted or incomplete.  In
     the case where there is not enough room, uncompress() will fill the output
     buffer with the uncompressed data up to that point.
*/

评论

3赞 Hossein 5/6/2012
+1.如果您想要所有默认设置,这是该死的简单解决方案。即使您不需要默认设置,也可以修改和使用这些函数的源代码。
3赞 PlinyTheElder 1/17/2020
不幸的是,在使用 uncompress() 之前,我们必须知道未压缩数据的大小才能分配缓冲区。如果您之前在压缩数据时没有保存大小,那么您就不走运了。
5赞 BullyWiiPlaza 8/6/2018 #5

经典方式更方便的C++功能

下面是一个完整的示例,它演示了使用对象进行压缩和解压缩:C++std::vector

#include <cstdio>
#include <iosfwd>
#include <iostream>
#include <vector>
#include <zconf.h>
#include <zlib.h>
#include <iomanip>
#include <cassert>

void add_buffer_to_vector(std::vector<char> &vector, const char *buffer, uLongf length) {
    for (int character_index = 0; character_index < length; character_index++) {
        char current_character = buffer[character_index];
        vector.push_back(current_character);
    }
}

int compress_vector(std::vector<char> source, std::vector<char> &destination) {
    unsigned long source_length = source.size();
    uLongf destination_length = compressBound(source_length);

    char *destination_data = (char *) malloc(destination_length);
    if (destination_data == nullptr) {
        return Z_MEM_ERROR;
    }

    Bytef *source_data = (Bytef *) source.data();
    int return_value = compress2((Bytef *) destination_data, &destination_length, source_data, source_length,
                                 Z_BEST_COMPRESSION);
    add_buffer_to_vector(destination, destination_data, destination_length);
    free(destination_data);
    return return_value;
}

int decompress_vector(std::vector<char> source, std::vector<char> &destination) {
    unsigned long source_length = source.size();
    uLongf destination_length = compressBound(source_length);

    char *destination_data = (char *) malloc(destination_length);
    if (destination_data == nullptr) {
        return Z_MEM_ERROR;
    }

    Bytef *source_data = (Bytef *) source.data();
    int return_value = uncompress((Bytef *) destination_data, &destination_length, source_data, source.size());
    add_buffer_to_vector(destination, destination_data, destination_length);
    free(destination_data);
    return return_value;
}

void add_string_to_vector(std::vector<char> &uncompressed_data,
                          const char *my_string) {
    int character_index = 0;
    while (true) {
        char current_character = my_string[character_index];
        uncompressed_data.push_back(current_character);

        if (current_character == '\00') {
            break;
        }

        character_index++;
    }
}

// https://stackoverflow.com/a/27173017/3764804
void print_bytes(std::ostream &stream, const unsigned char *data, size_t data_length, bool format = true) {
    stream << std::setfill('0');
    for (size_t data_index = 0; data_index < data_length; ++data_index) {
        stream << std::hex << std::setw(2) << (int) data[data_index];
        if (format) {
            stream << (((data_index + 1) % 16 == 0) ? "\n" : " ");
        }
    }
    stream << std::endl;
}

void test_compression() {
    std::vector<char> uncompressed(0);
    auto *my_string = (char *) "Hello, world!";
    add_string_to_vector(uncompressed, my_string);

    std::vector<char> compressed(0);
    int compression_result = compress_vector(uncompressed, compressed);
    assert(compression_result == F_OK);

    std::vector<char> decompressed(0);
    int decompression_result = decompress_vector(compressed, decompressed);
    assert(decompression_result == F_OK);

    printf("Uncompressed: %s\n", uncompressed.data());
    printf("Compressed: ");
    std::ostream &standard_output = std::cout;
    print_bytes(standard_output, (const unsigned char *) compressed.data(), compressed.size(), false);
    printf("Decompressed: %s\n", decompressed.data());
}

在您的简单通话中:main.cpp

int main(int argc, char *argv[]) {
    test_compression();
    return EXIT_SUCCESS;
}

产生的输出:

Uncompressed: Hello, world!
Compressed: 78daf348cdc9c9d75128cf2fca495164000024e8048a
Decompressed: Hello, world!

Boost 方式

#include <iostream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/zlib.hpp>

std::string compress(const std::string &data) {
    boost::iostreams::filtering_streambuf<boost::iostreams::output> output_stream;
    output_stream.push(boost::iostreams::zlib_compressor());
    std::stringstream string_stream;
    output_stream.push(string_stream);
    boost::iostreams::copy(boost::iostreams::basic_array_source<char>(data.c_str(),
                                                                      data.size()), output_stream);
    return string_stream.str();
}

std::string decompress(const std::string &cipher_text) {
    std::stringstream string_stream;
    string_stream << cipher_text;
    boost::iostreams::filtering_streambuf<boost::iostreams::input> input_stream;
    input_stream.push(boost::iostreams::zlib_decompressor());

    input_stream.push(string_stream);
    std::stringstream unpacked_text;
    boost::iostreams::copy(input_stream, unpacked_text);
    return unpacked_text.str();
}

TEST_CASE("zlib") {
    std::string plain_text = "Hello, world!";
    const auto cipher_text = compress(plain_text);
    const auto decompressed_plain_text = decompress(cipher_text);
    REQUIRE(plain_text == decompressed_plain_text);
}

评论

1赞 HolyBlackCat 8/6/2018
请注意,这是无效的 C++。它没有 VLA。uLongf destination_length = compressBound(source_length); char destination_data[destination_length];
0赞 BullyWiiPlaza 8/6/2018
@HolyBlackCat:但是代码在没有警告的情况下编译。C++17
0赞 BullyWiiPlaza 8/6/2018
@HolyBlackCat:好吧,那我就用吧。我修复了代码。malloc()
1赞 HolyBlackCat 8/6/2018
甚至更好,或者.std::vector<uint8_t>std::unique_ptr<uint8_t[]>
5赞 angelvet 1/19/2019
在解压缩时,你不能简单地用来计算目标长度,正如 zlib 文档所说:“未压缩数据的大小必须事先由压缩器保存,并通过此压缩库范围之外的某种机制传输到解压缩器。compressBound