tree 37b433276f7c5e6a3345669573a671f44e2e42b4
parent 39a5f22c36bbcce3918f3eab6948bd583209d79b
author Willy Tarreau <w@1wt.eu> 1470667261 +0200
committer Willy Tarreau <w@1wt.eu> 1470668268 +0200
encoding latin1

BUG/MAJOR: compression: initialize avail_in/next_in even during flush

For quite some time, a few users have been experiencing random crashes
when compressing with zlib, from versions 1.2.3 to 1.2.8 included.

Upon thourough investigation in zlib's deflate.c, it appeared obvious
that avail_in and next_in are used during the flush operation and need
to be initialized, while admittedly it's not obvious in the documentation.

By simply forcing both values to -1 it's possible to immediately reproduce
the exact crash that these users have been experiencing :

  (gdb) bt
  #0  0x00007fa73ce10c00 in __memcpy_sse2 () from /lib64/libc.so.6
  #1  0x00007fa73e0c5d49 in ?? () from /lib64/libz.so.1
  #2  0x00007fa73e0c68e0 in ?? () from /lib64/libz.so.1
  #3  0x00007fa73e0c73c7 in deflate () from /lib64/libz.so.1
  #4  0x00000000004dca1c in deflate_flush_or_finish (comp_ctx=0x7b6580, out=0x7fa73e5bd010, flag=2) at src/compression.c:808
  #5  0x00000000004dcb60 in deflate_flush (comp_ctx=0x7b6580, out=0x7fa73e5bd010) at src/compression.c:835
  #6  0x00000000004dbc50 in http_compression_buffer_end (s=0x7c0050, in=0x7c00a8, out=0x78adf0 <tmpbuf.24662>, end=0) at src/compression.c:249
  #7  0x000000000048bb5f in http_response_forward_body (s=0x7c0050, res=0x7c00a0, an_bit=1048576) at src/proto_http.c:7173
  #8  0x00000000004cce54 in process_stream (t=0x7bffd8) at src/stream.c:1939
  #9  0x0000000000427ddf in process_runnable_tasks () at src/task.c:238
  #10 0x0000000000419892 in run_poll_loop () at src/haproxy.c:1573
  #11 0x000000000041a4a5 in main (argc=4, argv=0x7fffcda38348) at src/haproxy.c:1933

Note that for all reports deflate_flush_or_finish() was always involved.

The crash is very hard to reproduce when using regular traffic because it
requires that the combination of avail_in and next_in are inadequate so
that the memcpy() call reads out of bounds. But this can very likely
happen when the input buffer points to an area reused by another stream
when the flush has been interrupted due to a full output buffer. This
also explains why this report is recent, as dynamic buffer allocation
was introduced in 1.6.

Anyway it's not acceptable to call a function with a randomly set input
buffer. The deflate() function explicitly checks for the case where both
avail_in and next_in are null and doesn't use it in this case during a
flush, so this is the best solution.

Special thanks to Sasha Litvak, James Hartshorn and Paul Bauer for
reporting very useful stack traces which were critical to finding the
root cause of this bug.

This fix must be backported into 1.6 and 1.5, though 1.5 is less likely to
trigger this case given that it keeps its own buffers allocated all along
the session's life.