Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 1 | Multi script langyage Stream Processing Offload Agent |
| 2 | ----------------------------------------------------- |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 3 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 4 | This agent receive SPOP message and process it with script languages. The |
| 5 | language register callback with a message. Each callback receive the list |
| 6 | of arguments with types according with the language capabilities. The |
| 7 | callback write variables which are sent as response when the processing |
| 8 | is done. |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 9 | |
| 10 | |
Aleksandar Lazic | a714475 | 2019-07-03 08:16:17 +0000 | [diff] [blame] | 11 | Prerequirement |
| 12 | ---------------- |
| 13 | |
| 14 | You have to install the development packages, either from the |
| 15 | distribution repositories or from the source. |
| 16 | |
| 17 | CentOS/RHEL: yum install python-devel |
| 18 | |
| 19 | The current python version in use is 2.7. |
| 20 | |
| 21 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 22 | Compilation |
| 23 | --------------- |
| 24 | |
| 25 | Actually, the server support Lua and Python. Type "make" with the options: |
| 26 | USE_LUA=1 and/or USE_PYTHON=1. |
| 27 | |
Aleksandar Lazic | a714475 | 2019-07-03 08:16:17 +0000 | [diff] [blame] | 28 | You can add LUA_INC=.. LUA_LIB=.. to the make command to set the paths to |
| 29 | the lua header files and lua libraries. |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 30 | |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 31 | Start the service |
| 32 | --------------------- |
| 33 | |
| 34 | After you have compiled it, to start the service, you just need to use "spoa" |
| 35 | binary: |
| 36 | |
| 37 | $> ./spoa -h |
| 38 | Usage: ./spoa [-h] [-d] [-p <port>] [-n <num-workers>] |
| 39 | -h Print this message |
| 40 | -d Enable the debug mode |
| 41 | -p <port> Specify the port to listen on (default: 12345) |
| 42 | -n <num-workers> Specify the number of workers (default: 5) |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 43 | -f <file> Load script according with the supported languages |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 44 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 45 | The file processor is recognized using the extension. .lua or .luac for lua and |
| 46 | .py for python. Start example: |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 47 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 48 | $> ./spoa -d -f ps_lua.lua |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 49 | |
Daniel Corbett | c802921 | 2019-06-11 10:08:53 -0400 | [diff] [blame] | 50 | $> ./spoa -d -f ps_python.py |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 51 | |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 52 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 53 | Configure |
| 54 | ------------- |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 55 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 56 | Sample configuration are join to this server: |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 57 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 58 | spoa-server.conf : The HAProxy configuration file using SPOE server |
| 59 | spoa-server.spoe.conf : The SPOP description file used by HAProxy |
| 60 | ps_lua.lua : Processing Lua example |
| 61 | ps_python.py : Processing Python example |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 62 | |
| 63 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 64 | Considerations |
| 65 | ------------------ |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 66 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 67 | This server is a beta version. It works fine, but some improvement will be |
| 68 | welcome: |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 69 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 70 | Main process: |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 71 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 72 | * Improve log management: Today the log are sent on stdout. |
| 73 | * Improve process management: The dead process are ignored. |
| 74 | * Implement systemd integration. |
| 75 | * Implement threads: It would be fine to implement thread working. Shared |
| 76 | memory is welcome for managing database connection pool and something like |
| 77 | that. |
| 78 | * Add PHP support and some other languages. |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 79 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 80 | Python: |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 81 | |
Ilya Shipitsin | 1fae8db | 2020-03-14 17:47:28 +0500 | [diff] [blame] | 82 | * Improve reporting: Catch python error message and report it in the right |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 83 | place. Today the error are dumped on stdout. How using syslog for logging |
| 84 | stack traces ? |
Thierry FOURNIER | d8b5c77 | 2018-02-23 11:40:03 +0100 | [diff] [blame] | 85 | |
Thierry FOURNIER | 4d7bfa1 | 2018-02-25 21:28:05 +0100 | [diff] [blame] | 86 | Maybe some other things... |