[MINOR] add options dontlog-normal and log-separate-errors
Some big traffic sites have trouble dealing with logs and tend to
disable them. Here are two new options to help cope with massive
logs.
- dontlog-normal only disables logging for 100% successful
connections, other ones will still be logged
- log-separate-errors will cause non-100% successful connections
to be logged at level "err" instead of level "info" so that a
properly configured syslog daemon can send them to a different
file for longer conservation.
diff --git a/doc/configuration.txt b/doc/configuration.txt
index 03a58e4..762343f 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -600,6 +600,7 @@
[no] option checkcache X - X X
[no] option clitcpka X X X -
[no] option contstats X X X -
+[no] option dontlog-normal X X X -
[no] option dontlognull X X X -
[no] option forceclose X - X X
option forwardfor X X X X
@@ -608,6 +609,8 @@
option httpchk X - X X
[no] option httpclose X X X X
option httplog X X X X
+[no] option log-separate-
+ errors X X X -
[no] option logasap X X X -
[no] option nolinger X X X X
[no] option persist X - X X
@@ -1999,6 +2002,29 @@
it is not enabled by default, as it has small performance impact (~0.5%).
+option dontlog-normal
+no option dontlog-normal
+ Enable or disable logging of normal, successful connections
+ May be used in sections : defaults | frontend | listen | backend
+ yes | yes | yes | no
+ Arguments : none
+
+ There are large sites dealing with several thousand connections per second
+ and for which logging is a major pain. Some of them are even forced to turn
+ logs off and cannot debug production issues. Setting this option ensures that
+ normal connections, those which experience no error, no timeout, no retry nor
+ redispatch, will not be logged. This leaves disk space for anomalies. In HTTP
+ mode, the response status code is checked and return codes 5xx will still be
+ logged.
+
+ It is strongly discouraged to use this option as most of the time, the key to
+ complex issues is in the normal logs which will not be logged here. If you
+ need to separate logs, see the "log-separate-errors" option instead.
+
+ See also : "log", "dontlognull", "log-separate-errors" and section 2.6 about
+ logging.
+
+
option dontlognull
no option dontlognull
Enable or disable logging of null connections
@@ -2299,6 +2325,29 @@
See also : section 2.6 about logging.
+
+option log-separate-errors
+no option log-separate-errors
+ Change log level for non-completely successful connections
+ May be used in sections : defaults | frontend | listen | backend
+ yes | yes | yes | no
+ Arguments : none
+
+ Sometimes looking for errors in logs is not easy. This option makes haproxy
+ raise the level of logs containing potentially interesting information such
+ as errors, timeouts, retries, redispatches, or HTTP status codes 5xx. The
+ level changes from "info" to "err". This makes it possible to log them
+ separately to a different file with most syslog daemons. Be careful not to
+ remove them from the original file, otherwise you would lose ordering which
+ provides very important information.
+
+ Using this option, large sites dealing with several thousand connections per
+ second may log normal traffic to a rotating buffer and only archive smaller
+ error logs.
+
+ See also : "log", "dontlognull", "dontlog-normal" and section 2.6 about
+ logging.
+
option logasap
no option logasap
@@ -4979,9 +5028,10 @@
environments with long sessions such as remote terminals. Sessions which match
the "monitor" rules are never logged. It is also possible not to emit logs for
sessions for which no data were exchanged between the client and the server, by
-specifying "option dontlognull" in the frontend. A few fields may slightly vary
-depending on some configuration options, those are marked with a star ('*')
-after the field name below.
+specifying "option dontlognull" in the frontend. Successful connections will
+not be logged if "option dontlog-normal" is specified in the frontend. A few
+fields may slightly vary depending on some configuration options, those are
+marked with a star ('*') after the field name below.
Example :
frontend fnt
@@ -5141,7 +5191,8 @@
generally only makes sense for download sites. A session which matches the
"monitor" rules will never logged. It is also possible not to log sessions for
which no data were sent by the client by specifying "option dontlognull" in the
-frontend.
+frontend. Successful connections will not be logged if "option dontlog-normal"
+is specified in the frontend.
Most fields are shared with the TCP log, some being different. A few fields may
slightly vary depending on some configuration options. Those ones are marked
@@ -5419,6 +5470,35 @@
with a '+' sign which means that real numbers are certainly larger.
+2.6.3.3) Raising log level upon errors
+--------------------------------------
+
+Sometimes it is more convenient to separate normal traffic from errors logs,
+for instance in order to ease error monitoring from log files. When the option
+"log-separate-errors" is used, connections which experience errors, timeouts,
+retries, redispatches or HTTP status codes 5xx will see their syslog level
+raised from "info" to "err". This will help a syslog daemon store the log in
+a separate file. It is very important to keep the errors in the normal traffic
+file too, so that log ordering is not altered. You should also be careful if
+you already have configured your syslog daemon to store all logs higher than
+"notice" in an "admin" file, because the "err" level is higher than "notice".
+
+
+2.6.3.4) Disabling logging of successful connections
+----------------------------------------------------
+
+Although this may sound strange at first, some large sites have to deal with
+multiple thousands of logs per second and are experiencing difficulties keeping
+them intact for a long time or detecting errors within them. If the option
+"dontlog-normal" is set on the frontend, all normal connections will not be
+logged. In this regard, a normal connection is defined as one without any
+error, timeout, retry nor redispatch. In HTTP, the status code is checked too,
+and a response with a status 5xx is not considered normal and will be logged
+too. Of course, doing is is really discouraged as it will remove most of the
+useful information from the logs. Do this only if you have no other
+alternative.
+
+
2.6.4) Timing events
--------------------
diff --git a/include/types/proxy.h b/include/types/proxy.h
index 924019c..48d0a82 100644
--- a/include/types/proxy.h
+++ b/include/types/proxy.h
@@ -115,6 +115,8 @@
#define PR_O2_SPLIC_ANY (PR_O2_SPLIC_REQ|PR_O2_SPLIC_RTR|PR_O2_SPLIC_AUT)
#define PR_O2_REQBUG_OK 0x00000008 /* let buggy requests pass through */
#define PR_O2_RSPBUG_OK 0x00000010 /* let buggy responses pass through */
+#define PR_O2_NOLOGNORM 0x00000020 /* don't log normal traffic, only errors and retries */
+#define PR_O2_LOGERRORS 0x00000040 /* log errors and retries at level LOG_ERR */
/* This structure is used to apply fast weighted round robin on a server group */
struct fwrr_group {
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 58e8407..6a4744e 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -125,6 +125,8 @@
#endif
{ "accept-invalid-http-request", PR_O2_REQBUG_OK, PR_CAP_FE, 0 },
{ "accept-invalid-http-response", PR_O2_RSPBUG_OK, PR_CAP_BE, 0 },
+ { "dontlog-normal", PR_O2_NOLOGNORM, PR_CAP_FE, 0 },
+ { "log-separate-errors", PR_O2_LOGERRORS, PR_CAP_FE, 0 },
{ NULL, 0, 0, 0 }
};
diff --git a/src/log.c b/src/log.c
index 976bc63..bc9fd0a 100644
--- a/src/log.c
+++ b/src/log.c
@@ -321,10 +321,15 @@
struct proxy *fe = s->fe;
struct proxy *be = s->be;
struct proxy *prx_log;
- int tolog;
+ int tolog, level, err;
char *svid;
struct tm tm;
+ /* if we don't want to log normal traffic, return now */
+ err = (s->flags & (SN_ERR_MASK | SN_REDISP)) || (s->conn_retries != be->conn_retries);
+ if (!err && (fe->options2 & PR_O2_NOLOGNORM))
+ return;
+
if (s->cli_addr.ss_family == AF_INET)
inet_ntop(AF_INET,
(const void *)&((struct sockaddr_in *)&s->cli_addr)->sin_addr,
@@ -353,7 +358,11 @@
tolog = fe->to_log;
svid = (tolog & LW_SVID) ? (s->srv != NULL) ? s->srv->id : "<NOSRV>" : "-";
- send_log(prx_log, LOG_INFO, "%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
+ level = LOG_INFO;
+ if (err && (fe->options2 & PR_O2_LOGERRORS))
+ level = LOG_ERR;
+
+ send_log(prx_log, level, "%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
" %s %s/%s %ld/%ld/%s%ld %s%lld"
" %c%c %d/%d/%d/%d/%s%u %ld/%ld\n",
pn,
diff --git a/src/proto_http.c b/src/proto_http.c
index 6ffc14e..a8612a4 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -755,7 +755,7 @@
struct proxy *be = s->be;
struct proxy *prx_log;
struct http_txn *txn = &s->txn;
- int tolog;
+ int tolog, level, err;
char *uri, *h;
char *svid;
struct tm tm;
@@ -763,6 +763,13 @@
int t_request;
int hdr;
+ /* if we don't want to log normal traffic, return now */
+ err = (s->flags & (SN_ERR_MASK | SN_REDISP)) ||
+ (s->conn_retries != be->conn_retries) ||
+ txn->status >= 500;
+ if (!err && (fe->options2 & PR_O2_NOLOGNORM))
+ return;
+
if (fe->logfac1 < 0 && fe->logfac2 < 0)
return;
prx_log = fe;
@@ -830,7 +837,11 @@
if (tv_isge(&s->logs.tv_request, &s->logs.tv_accept))
t_request = tv_ms_elapsed(&s->logs.tv_accept, &s->logs.tv_request);
- send_log(prx_log, LOG_INFO,
+ level = LOG_INFO;
+ if (err && (fe->options2 & PR_O2_LOGERRORS))
+ level = LOG_ERR;
+
+ send_log(prx_log, level,
"%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
" %s %s/%s %d/%ld/%ld/%ld/%s%ld %d %s%lld"
" %s %s %c%c%c%c %d/%d/%d/%d/%s%u %ld/%ld%s\n",