How to change the buffer limit in Google's protobuf?
我在尝试解析大消息时收到此警告和错误。我知道默认限制超过 64MB。我现在正在使用 message.ParseFromIstream。有谁知道可以访问 CodedInputStream 对象来调用 SetTotalBytesLimit 函数？或任何其他方式来解决这个问题？
Reading dangerously large protocol message. If the message turns out
to be larger than 67108864 bytes, parsing will be halted for security
reasons. To increase the limit (or to disable these warnings), see
CodedInputStream::SetTotalBytesLimit() in
google/protobuf/io/coded_stream.h.
正确的解决方法：您应该尝试限制 protobuf 消息的大小。请参见：
https://developers.google.com/protocol-buffers/docs/techniques#streaming
快速而肮脏(不推荐阅读)的方法：
在protobuf库源文件coded_stream.h中，修改
只需阅读错误已经告诉您的函数的文档，就可以回答这个问题：
Hint: If you are reading this because your program is printing a
warning about dangerously large protocol messages, you may be confused
about what to do next. The best option is to change your design such
that excessively large messages are not necessary. For example, try to
design file formats to consist of many small messages rather than a
single large one. If this is infeasible, you will need to increase the
limit. Chances are, though, that your code never constructs a
CodedInputStream on which the limit can be set. You probably parse
messages by calling things like Message::ParseFromString(). In this
case, you will need to change your code to instead construct some sort
of ZeroCopyInputStream (e.g. an ArrayInputStream), construct a
CodedInputStream around that, then call
Message::ParseFromCodedStream() instead. Then you can adjust the
limit. Yes, it's more work, but you're doing something unusual.
来源
另外，遵循建议的第一部分并重新设计应用程序可能是一个非常好的主意。
这是来自代码 (
这个帖子很老了，但是最近深度学习受到了关注，并且库 Caffe 使用了 Protobuf，所以也许更多的人会偶然发现这个。我必须用 Caffe 做神经网络的东西，即使是最小的批量，整个网络也占用了很多内存。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | // Total Bytes Limit ----------------------------------------------- // To prevent malicious users from sending excessively large messages // and causing integer overflows or memory exhaustion, CodedInputStream // imposes a hard limit on the total number of bytes it will read. // Sets the maximum number of bytes that this CodedInputStream will read // before refusing to continue. To prevent integer overflows in the // protocol buffers implementation, as well as to prevent servers from // allocating enormous amounts of memory to hold parsed messages, the // maximum message length should be limited to the shortest length that // will not harm usability. The theoretical shortest message that could // cause integer overflows is 512MB. The default limit is 64MB. Apps // should set shorter limits if possible. If warning_threshold is not -1, // a warning will be printed to stderr after warning_threshold bytes are // read. For backwards compatibility all negative values get squashed to -1, // as other negative values might have special internal meanings. // An error will always be printed to stderr if the limit is reached. // // This is unrelated to PushLimit()/PopLimit(). // // Hint: If you are reading this because your program is printing a // warning about dangerously large protocol messages, you may be // confused about what to do next. The best option is to change your // design such that excessively large messages are not necessary. // For example, try to design file formats to consist of many small // messages rather than a single large one. If this is infeasible, // you will need to increase the limit. Chances are, though, that // your code never constructs a CodedInputStream on which the limit // can be set. You probably parse messages by calling things like // Message::ParseFromString(). In this case, you will need to change // your code to instead construct some sort of ZeroCopyInputStream // (e.g. an ArrayInputStream), construct a CodedInputStream around // that, then call Message::ParseFromCodedStream() instead. Then // you can adjust the limit. Yes, it's more work, but you're doing // something unusual. |