MAJOR: filters: Add filters support
This patch adds the support of filters in HAProxy. The main idea is to have a
way to "easely" extend HAProxy by adding some "modules", called filters, that
will be able to change HAProxy behavior in a programmatic way.
To do so, many entry points has been added in code to let filters to hook up to
different steps of the processing. A filter must define a flt_ops sutrctures
(see include/types/filters.h for details). This structure contains all available
callbacks that a filter can define:
struct flt_ops {
/*
* Callbacks to manage the filter lifecycle
*/
int (*init) (struct proxy *p);
void (*deinit)(struct proxy *p);
int (*check) (struct proxy *p);
/*
* Stream callbacks
*/
void (*stream_start) (struct stream *s);
void (*stream_accept) (struct stream *s);
void (*session_establish)(struct stream *s);
void (*stream_stop) (struct stream *s);
/*
* HTTP callbacks
*/
int (*http_start) (struct stream *s, struct http_msg *msg);
int (*http_start_body) (struct stream *s, struct http_msg *msg);
int (*http_start_chunk) (struct stream *s, struct http_msg *msg);
int (*http_data) (struct stream *s, struct http_msg *msg);
int (*http_last_chunk) (struct stream *s, struct http_msg *msg);
int (*http_end_chunk) (struct stream *s, struct http_msg *msg);
int (*http_chunk_trailers)(struct stream *s, struct http_msg *msg);
int (*http_end_body) (struct stream *s, struct http_msg *msg);
void (*http_end) (struct stream *s, struct http_msg *msg);
void (*http_reset) (struct stream *s, struct http_msg *msg);
int (*http_pre_process) (struct stream *s, struct http_msg *msg);
int (*http_post_process) (struct stream *s, struct http_msg *msg);
void (*http_reply) (struct stream *s, short status,
const struct chunk *msg);
};
To declare and use a filter, in the configuration, the "filter" keyword must be
used in a listener/frontend section:
frontend test
...
filter <FILTER-NAME> [OPTIONS...]
The filter referenced by the <FILTER-NAME> must declare a configuration parser
on its own name to fill flt_ops and filter_conf field in the proxy's
structure. An exemple will be provided later to make it perfectly clear.
For now, filters cannot be used in backend section. But this is only a matter of
time. Documentation will also be added later. This is the first commit of a long
list about filters.
It is possible to have several filters on the same listener/frontend. These
filters are stored in an array of at most MAX_FILTERS elements (define in
include/types/filters.h). Again, this will be replaced later by a list of
filters.
The filter API has been highly refactored. Main changes are:
* Now, HA supports an infinite number of filters per proxy. To do so, filters
are stored in list.
* Because filters are stored in list, filters state has been moved from the
channel structure to the filter structure. This is cleaner because there is no
more info about filters in channel structure.
* It is possible to defined filters on backends only. For such filters,
stream_start/stream_stop callbacks are not called. Of course, it is possible
to mix frontend and backend filters.
* Now, TCP streams are also filtered. All callbacks without the 'http_' prefix
are called for all kind of streams. In addition, 2 new callbacks were added to
filter data exchanged through a TCP stream:
- tcp_data: it is called when new data are available or when old unprocessed
data are still waiting.
- tcp_forward_data: it is called when some data can be consumed.
* New callbacks attached to channel were added:
- channel_start_analyze: it is called when a filter is ready to process data
exchanged through a channel. 2 new analyzers (a frontend and a backend)
are attached to channels to call this callback. For a frontend filter, it
is called before any other analyzer. For a backend filter, it is called
when a backend is attached to a stream. So some processing cannot be
filtered in that case.
- channel_analyze: it is called before each analyzer attached to a channel,
expects analyzers responsible for data sending.
- channel_end_analyze: it is called when all other analyzers have finished
their processing. A new analyzers is attached to channels to call this
callback. For a TCP stream, this is always the last one called. For a HTTP
one, the callback is called when a request/response ends, so it is called
one time for each request/response.
* 'session_established' callback has been removed. Everything that is done in
this callback can be handled by 'channel_start_analyze' on the response
channel.
* 'http_pre_process' and 'http_post_process' callbacks have been replaced by
'channel_analyze'.
* 'http_start' callback has been replaced by 'http_headers'. This new one is
called just before headers sending and parsing of the body.
* 'http_end' callback has been replaced by 'channel_end_analyze'.
* It is possible to set a forwarder for TCP channels. It was already possible to
do it for HTTP ones.
* Forwarders can partially consumed forwardable data. For this reason a new
HTTP message state was added before HTTP_MSG_DONE : HTTP_MSG_ENDING.
Now all filters can define corresponding callbacks (http_forward_data
and tcp_forward_data). Each filter owns 2 offsets relative to buf->p, next and
forward, to track, respectively, input data already parsed but not forwarded yet
by the filter and parsed data considered as forwarded by the filter. A any time,
we have the warranty that a filter cannot parse or forward more input than
previous ones. And, of course, it cannot forward more input than it has
parsed. 2 macros has been added to retrieve these offets: FLT_NXT and FLT_FWD.
In addition, 2 functions has been added to change the 'next size' and the
'forward size' of a filter. When a filter parses input data, it can alter these
data, so the size of these data can vary. This action has an effet on all
previous filters that must be handled. To do so, the function
'filter_change_next_size' must be called, passing the size variation. In the
same spirit, if a filter alter forwarded data, it must call the function
'filter_change_forward_size'. 'filter_change_next_size' can be called in
'http_data' and 'tcp_data' callbacks and only these ones. And
'filter_change_forward_size' can be called in 'http_forward_data' and
'tcp_forward_data' callbacks and only these ones. The data changes are the
filter responsability, but with some limitation. It must not change already
parsed/forwarded data or data that previous filters have not parsed/forwarded
yet.
Because filters can be used on backends, when we the backend is set for a
stream, we add filters defined for this backend in the filter list of the
stream. But we must only do that when the backend and the frontend of the stream
are not the same. Else same filters are added a second time leading to undefined
behavior.
The HTTP compression code had to be moved.
So it simplifies http_response_forward_body function. To do so, the way the data
are forwarded has changed. Now, a filter (and only one) can forward data. In a
commit to come, this limitation will be removed to let all filters take part to
data forwarding. There are 2 new functions that filters should use to deal with
this feature:
* flt_set_http_data_forwarder: This function sets the filter (using its id)
that will forward data for the specified HTTP message. It is possible if it
was not already set by another filter _AND_ if no data was yet forwarded
(msg->msg_state <= HTTP_MSG_BODY). It returns -1 if an error occurs.
* flt_http_data_forwarder: This function returns the filter id that will
forward data for the specified HTTP message. If there is no forwarder set, it
returns -1.
When an HTTP data forwarder is set for the response, the HTTP compression is
disabled. Of course, this is not definitive.
diff --git a/include/proto/filters.h b/include/proto/filters.h
new file mode 100644
index 0000000..d860424
--- /dev/null
+++ b/include/proto/filters.h
@@ -0,0 +1,138 @@
+/*
+ * include/proto/filters.h
+ * This file defines function prototypes for stream filters management.
+ *
+ * Copyright (C) 2015 Qualys Inc., Christopher Faulet <cfaulet@qualys.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation, version 2.1
+ * exclusively.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+#ifndef _PROTO_FILTERS_H
+#define _PROTO_FILTERS_H
+
+#include <types/channel.h>
+#include <types/filters.h>
+#include <types/proto_http.h>
+#include <types/proxy.h>
+#include <types/stream.h>
+
+#include <proto/channel.h>
+
+/* Useful macros to access per-channel values. It can be safely used inside
+ * filters. */
+#define CHN_IDX(chn) (((chn)->flags & CF_ISRESP) == CF_ISRESP)
+#define FLT_NXT(flt, chn) ((flt)->next[CHN_IDX(chn)])
+#define FLT_FWD(flt, chn) ((flt)->fwd[CHN_IDX(chn)])
+
+extern struct pool_head *pool2_filter;
+
+int flt_init(struct proxy *p);
+void flt_deinit(struct proxy *p);
+int flt_check(struct proxy *p);
+
+int flt_stream_start(struct stream *s);
+void flt_stream_stop(struct stream *s);
+
+int flt_http_headers(struct stream *s, struct http_msg *msg);
+int flt_http_start_chunk(struct stream *s, struct http_msg *msg);
+int flt_http_data(struct stream *s, struct http_msg *msg);
+int flt_http_last_chunk(struct stream *s, struct http_msg *msg);
+int flt_http_end_chunk(struct stream *s, struct http_msg *msg);
+int flt_http_chunk_trailers(struct stream *s, struct http_msg *msg);
+int flt_http_end(struct stream *s, struct http_msg *msg);
+void flt_http_reset(struct stream *s, struct http_msg *msg);
+
+void flt_http_reply(struct stream *s, short status, const struct chunk *msg);
+int flt_http_forward_data(struct stream *s, struct http_msg *msg, unsigned int len);
+
+int flt_start_analyze(struct stream *s, struct channel *chn, unsigned int an_bit);
+int flt_analyze(struct stream *s, struct channel *chn, unsigned int an_bit);
+int flt_end_analyze(struct stream *s, struct channel *chn, unsigned int an_bit);
+
+int flt_xfer_data(struct stream *s, struct channel *chn, unsigned int an_bit);
+
+void flt_register_keywords(struct flt_kw_list *kwl);
+struct flt_kw *flt_find_kw(const char *kw);
+void flt_dump_kws(char **out);
+
+static inline void
+flt_set_forward_data(struct filter *filter, struct channel *chn)
+{
+ filter->flags[CHN_IDX(chn)] |= FILTER_FL_FORWARD_DATA;
+}
+
+static inline void
+flt_reset_forward_data(struct filter *filter, struct channel *chn)
+{
+ filter->flags[CHN_IDX(chn)] &= ~FILTER_FL_FORWARD_DATA;
+}
+
+static inline int
+flt_want_forward_data(struct filter *filter, const struct channel *chn)
+{
+ return filter->flags[CHN_IDX(chn)] & FILTER_FL_FORWARD_DATA;
+}
+
+
+/* This function must be called when a filter alter incoming data. It updates
+ * next offset value of all filter's predecessors. Do not call this function
+ * when a filter change the size of incomding data leads to an undefined
+ * behavior.
+ *
+ * This is the filter's responsiblitiy to update data itself. For now, it is
+ * unclear to know how to handle data updates, so we do the minimum here. For
+ * example, if you filter an HTTP message, we must update msg->next and
+ * msg->chunk_len values.
+ */
+static inline void
+flt_change_next_size(struct filter *filter, struct channel *chn, int len)
+{
+ struct stream *s = chn_strm(chn);
+ struct filter *f;
+
+ list_for_each_entry(f, &s->strm_flt.filters, list) {
+ if (f == filter)
+ break;
+ FLT_NXT(f, chn) += len;
+ }
+}
+
+/* This function must be called when a filter alter forwarded data. It updates
+ * offset values (next and forward) of all filters. Do not call this function
+ * when a filter change the size of forwarded data leads to an undefined
+ * behavior.
+ *
+ * This is the filter's responsiblitiy to update data itself. For now, it is
+ * unclear to know how to handle data updates, so we do the minimum here. For
+ * example, if you filter an HTTP message, we must update msg->next and
+ * msg->chunk_len values.
+ */
+static inline void
+flt_change_forward_size(struct filter *filter, struct channel *chn, int len)
+{
+ struct stream *s = chn_strm(chn);
+ struct filter *f;
+ int before = 1;
+
+ list_for_each_entry(f, &s->strm_flt.filters, list) {
+ if (f == filter)
+ before = 0;
+ if (before)
+ FLT_FWD(f, chn) += len;
+ FLT_NXT(f, chn) += len;
+ }
+}
+
+
+#endif /* _PROTO_FILTERS_H */
diff --git a/include/proto/proto_http.h b/include/proto/proto_http.h
index ef564dc..9317a55 100644
--- a/include/proto/proto_http.h
+++ b/include/proto/proto_http.h
@@ -126,6 +126,7 @@
void http_init_txn(struct stream *s);
void http_end_txn(struct stream *s);
void http_reset_txn(struct stream *s);
+void http_end_txn_clean_session(struct stream *s);
void http_adjust_conn_mode(struct stream *s, struct http_txn *txn, struct http_msg *msg);
struct act_rule *parse_http_req_cond(const char **args, const char *file, int linenum, struct proxy *proxy);
@@ -284,6 +285,7 @@
case HTTP_MSG_DATA: return "MSG_DATA";
case HTTP_MSG_CHUNK_CRLF: return "MSG_CHUNK_CRLF";
case HTTP_MSG_TRAILERS: return "MSG_TRAILERS";
+ case HTTP_MSG_ENDING: return "MSG_ENDING";
case HTTP_MSG_DONE: return "MSG_DONE";
case HTTP_MSG_CLOSING: return "MSG_CLOSING";
case HTTP_MSG_CLOSED: return "MSG_CLOSED";