提问人:Richard Knop 提问时间:12/27/2010 最后编辑:Anurag SinghRichard Knop 更新时间:9/16/2020 访问量:65656
如何使用 zlib 压缩缓冲区?
How to compress a buffer with zlib?
问:
zlib 网站上有一个使用示例:http://www.zlib.net/zlib_how.html
但是,在示例中,它们正在压缩文件。我想压缩存储在内存缓冲区中的二进制数据。我也不想将压缩缓冲区保存到磁盘。
基本上这是我的缓冲区:
fIplImageHeader->imageData = (char*)imageIn->getFrame();
如何用 zlib 压缩它?
我将不胜感激一些关于如何做到这一点的代码示例。
答:
3赞
Yippie-Ki-Yay
12/27/2010
#1
这不是您关于 zlib API 问题的直接答案,但您可能对与 zlib
配对的 boost::iostreams
库感兴趣。
这允许使用基本的“流”操作表示法使用驱动的打包算法,然后可以通过打开一些内存流并对其执行操作来轻松压缩数据。zlib
<< data
在这种情况下,将自动为通过流的每个数据调用相应的打包过滤器。boost::iostreams
评论
0赞
Ben
8/25/2012
只是一个注释,但这意味着您必须包含 boost iostreams 库、boost zlib 库、libz 和 libzbz2,而仅包含 libz。因此,如果捆绑这些库是一个大小问题,那么在这种情况下最好避免提升。~本
14赞
kichik
12/27/2010
#2
您可以通过将 和 调用替换为指向数据的直接指针来轻松调整示例。对于 zlib 压缩(称为 deflate,因为您“取出所有数据”),您分配结构,调用,然后:fread()
fwrite()
z_stream
deflateInit()
- 填充要压缩的下一块数据
next_in
- 设置为
avail_in
next_in
- 设置为应写入压缩数据的位置,该位置通常应是缓冲区内的指针,该指针会随着您的前进而前进
next_out
- 设置为
avail_out
next_out
- 叫
deflate
- 重复步骤 3-5,直到不为零(即输出缓冲区中的空间比 zlib 需要的要多 - 没有更多数据要写入)
avail_out
- 在有要压缩的数据时重复步骤 1-6
最终,你打电话,你就完成了。deflateEnd()
你基本上是在给它输入和输出块,直到你用完了输入,它就没有输出了。
38赞
Jonas Engström
12/27/2010
#3
这是一个使用 zlib 打包缓冲区并将压缩内容保存在向量中的示例。
void compress_memory(void *in_data, size_t in_data_size, std::vector<uint8_t> &out_data)
{
std::vector<uint8_t> buffer;
const size_t BUFSIZE = 128 * 1024;
uint8_t temp_buffer[BUFSIZE];
z_stream strm;
strm.zalloc = 0;
strm.zfree = 0;
strm.next_in = reinterpret_cast<uint8_t *>(in_data);
strm.avail_in = in_data_size;
strm.next_out = temp_buffer;
strm.avail_out = BUFSIZE;
deflateInit(&strm, Z_BEST_COMPRESSION);
while (strm.avail_in != 0)
{
int res = deflate(&strm, Z_NO_FLUSH);
assert(res == Z_OK);
if (strm.avail_out == 0)
{
buffer.insert(buffer.end(), temp_buffer, temp_buffer + BUFSIZE);
strm.next_out = temp_buffer;
strm.avail_out = BUFSIZE;
}
}
int deflate_res = Z_OK;
while (deflate_res == Z_OK)
{
if (strm.avail_out == 0)
{
buffer.insert(buffer.end(), temp_buffer, temp_buffer + BUFSIZE);
strm.next_out = temp_buffer;
strm.avail_out = BUFSIZE;
}
deflate_res = deflate(&strm, Z_FINISH);
}
assert(deflate_res == Z_STREAM_END);
buffer.insert(buffer.end(), temp_buffer, temp_buffer + BUFSIZE - strm.avail_out);
deflateEnd(&strm);
out_data.swap(buffer);
}
评论
0赞
Robert
2/1/2013
我刚刚读到一个有趣的方法,它使用一个标志来控制刷新状态,并在一个循环中完成。此外,值得注意的是,这同样适用于 std::string 代替向量,这非常适合通过网络或其他函数发送。
2赞
user673679
1/25/2015
这似乎很复杂。为什么要做这一切,而不仅仅是使用函数?compress
3赞
Josh Atkins
6/17/2015
压缩要求您知道输出大小并分配足够大的缓冲区;此方法允许您 realloc() 并具有动态扩展的缓冲区。
0赞
BullyWiiPlaza
8/4/2018
不错的代码,但会产生 3 次警告。您可能需要解决此问题。temp_buffer
escapes the local scope
51赞
Huiwei
4/8/2012
#4
zlib.h
具有您需要的所有功能:(或 ) 和 。有关答案,请参阅 zlib 的源代码。compress
compress2
uncompress
ZEXTERN int ZEXPORT compress OF((Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen));
/*
Compresses the source buffer into the destination buffer. sourceLen is
the byte length of the source buffer. Upon entry, destLen is the total size
of the destination buffer, which must be at least the value returned by
compressBound(sourceLen). Upon exit, destLen is the actual size of the
compressed buffer.
compress returns Z_OK if success, Z_MEM_ERROR if there was not
enough memory, Z_BUF_ERROR if there was not enough room in the output
buffer.
*/
ZEXTERN int ZEXPORT uncompress OF((Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen));
/*
Decompresses the source buffer into the destination buffer. sourceLen is
the byte length of the source buffer. Upon entry, destLen is the total size
of the destination buffer, which must be large enough to hold the entire
uncompressed data. (The size of the uncompressed data must have been saved
previously by the compressor and transmitted to the decompressor by some
mechanism outside the scope of this compression library.) Upon exit, destLen
is the actual size of the uncompressed buffer.
uncompress returns Z_OK if success, Z_MEM_ERROR if there was not
enough memory, Z_BUF_ERROR if there was not enough room in the output
buffer, or Z_DATA_ERROR if the input data was corrupted or incomplete. In
the case where there is not enough room, uncompress() will fill the output
buffer with the uncompressed data up to that point.
*/
评论
3赞
Hossein
5/6/2012
+1.如果您想要所有默认设置,这是该死的简单解决方案。即使您不需要默认设置,也可以修改和使用这些函数的源代码。
3赞
PlinyTheElder
1/17/2020
不幸的是,在使用 uncompress() 之前,我们必须知道未压缩数据的大小才能分配缓冲区。如果您之前在压缩数据时没有保存大小,那么您就不走运了。
5赞
BullyWiiPlaza
8/6/2018
#5
经典方式更方便的C++功能
下面是一个完整的示例,它演示了使用对象进行压缩和解压缩:C++
std::vector
#include <cstdio>
#include <iosfwd>
#include <iostream>
#include <vector>
#include <zconf.h>
#include <zlib.h>
#include <iomanip>
#include <cassert>
void add_buffer_to_vector(std::vector<char> &vector, const char *buffer, uLongf length) {
for (int character_index = 0; character_index < length; character_index++) {
char current_character = buffer[character_index];
vector.push_back(current_character);
}
}
int compress_vector(std::vector<char> source, std::vector<char> &destination) {
unsigned long source_length = source.size();
uLongf destination_length = compressBound(source_length);
char *destination_data = (char *) malloc(destination_length);
if (destination_data == nullptr) {
return Z_MEM_ERROR;
}
Bytef *source_data = (Bytef *) source.data();
int return_value = compress2((Bytef *) destination_data, &destination_length, source_data, source_length,
Z_BEST_COMPRESSION);
add_buffer_to_vector(destination, destination_data, destination_length);
free(destination_data);
return return_value;
}
int decompress_vector(std::vector<char> source, std::vector<char> &destination) {
unsigned long source_length = source.size();
uLongf destination_length = compressBound(source_length);
char *destination_data = (char *) malloc(destination_length);
if (destination_data == nullptr) {
return Z_MEM_ERROR;
}
Bytef *source_data = (Bytef *) source.data();
int return_value = uncompress((Bytef *) destination_data, &destination_length, source_data, source.size());
add_buffer_to_vector(destination, destination_data, destination_length);
free(destination_data);
return return_value;
}
void add_string_to_vector(std::vector<char> &uncompressed_data,
const char *my_string) {
int character_index = 0;
while (true) {
char current_character = my_string[character_index];
uncompressed_data.push_back(current_character);
if (current_character == '\00') {
break;
}
character_index++;
}
}
// https://stackoverflow.com/a/27173017/3764804
void print_bytes(std::ostream &stream, const unsigned char *data, size_t data_length, bool format = true) {
stream << std::setfill('0');
for (size_t data_index = 0; data_index < data_length; ++data_index) {
stream << std::hex << std::setw(2) << (int) data[data_index];
if (format) {
stream << (((data_index + 1) % 16 == 0) ? "\n" : " ");
}
}
stream << std::endl;
}
void test_compression() {
std::vector<char> uncompressed(0);
auto *my_string = (char *) "Hello, world!";
add_string_to_vector(uncompressed, my_string);
std::vector<char> compressed(0);
int compression_result = compress_vector(uncompressed, compressed);
assert(compression_result == F_OK);
std::vector<char> decompressed(0);
int decompression_result = decompress_vector(compressed, decompressed);
assert(decompression_result == F_OK);
printf("Uncompressed: %s\n", uncompressed.data());
printf("Compressed: ");
std::ostream &standard_output = std::cout;
print_bytes(standard_output, (const unsigned char *) compressed.data(), compressed.size(), false);
printf("Decompressed: %s\n", decompressed.data());
}
在您的简单通话中:main.cpp
int main(int argc, char *argv[]) {
test_compression();
return EXIT_SUCCESS;
}
产生的输出:
Uncompressed: Hello, world!
Compressed: 78daf348cdc9c9d75128cf2fca495164000024e8048a
Decompressed: Hello, world!
Boost 方式
#include <iostream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/zlib.hpp>
std::string compress(const std::string &data) {
boost::iostreams::filtering_streambuf<boost::iostreams::output> output_stream;
output_stream.push(boost::iostreams::zlib_compressor());
std::stringstream string_stream;
output_stream.push(string_stream);
boost::iostreams::copy(boost::iostreams::basic_array_source<char>(data.c_str(),
data.size()), output_stream);
return string_stream.str();
}
std::string decompress(const std::string &cipher_text) {
std::stringstream string_stream;
string_stream << cipher_text;
boost::iostreams::filtering_streambuf<boost::iostreams::input> input_stream;
input_stream.push(boost::iostreams::zlib_decompressor());
input_stream.push(string_stream);
std::stringstream unpacked_text;
boost::iostreams::copy(input_stream, unpacked_text);
return unpacked_text.str();
}
TEST_CASE("zlib") {
std::string plain_text = "Hello, world!";
const auto cipher_text = compress(plain_text);
const auto decompressed_plain_text = decompress(cipher_text);
REQUIRE(plain_text == decompressed_plain_text);
}
评论
1赞
HolyBlackCat
8/6/2018
请注意,这是无效的 C++。它没有 VLA。uLongf destination_length = compressBound(source_length); char destination_data[destination_length];
0赞
BullyWiiPlaza
8/6/2018
@HolyBlackCat:但是代码在没有警告的情况下编译。C++17
0赞
BullyWiiPlaza
8/6/2018
@HolyBlackCat:好吧,那我就用吧。我修复了代码。malloc()
1赞
HolyBlackCat
8/6/2018
甚至更好,或者.std::vector<uint8_t>
std::unique_ptr<uint8_t[]>
5赞
angelvet
1/19/2019
在解压缩时,你不能简单地用来计算目标长度,正如 zlib 文档所说:“未压缩数据的大小必须事先由压缩器保存,并通过此压缩库范围之外的某种机制传输到解压缩器。compressBound
评论