Supervisor (http://supervisord.org) is a process management tool written in Python, which can be used conveniently to start, restart, and stop processes (not limited to Python processes). In addition to controlling individual processes, it can also start or stop multiple processes simultaneously. For example, if an unfortunate server issue causes all applications to be killed, you can use Supervisor to start all applications at once rather than typing commands one by one.
Installation
Supervisor can run on Linux and Mac OS X. As mentioned earlier, since Supervisor is written in Python, installation is straightforward using pip:
sudo pip install supervisor
On Ubuntu systems, you can also install it using apt-get.
Supervisord Configuration
Supervisor is quite powerful and offers rich functionality, though we may only need a small portion of it. After installation, you can write configuration files to meet your needs. For convenience, we divide the configuration into two parts: supervisord (Supervisor follows a C/S model; this is the server side, corresponding to the client side: supervisorctl) and application programs (i.e., the programs we want to manage).
First, let’s look at the supervisord configuration file. After installing Supervisor, you can run the echo_supervisord_conf command to output the default configuration items, or redirect them to a configuration file:
echo_supervisord_conf > /etc/supervisord.conf
By removing most comments and “irrelevant” sections, we can see these configurations:
[unix_http_server]
file=/tmp/supervisor.sock ; UNIX socket file, used by supervisorctl
;chmod=0700 ; Mode of the socket file, default is 0700
;chown=nobody:nogroup ; Owner of the socket file, format: uid:gid
;[inet_http_server] ; HTTP server, provides web management interface
;port=127.0.0.1:9001 ; IP and port for the Web management backend; if exposed to the public internet, security must be considered
;username=user ; Username for logging into the management backend
;password=123 ; Password for logging into the management backend
[supervisord]
logfile=/tmp/supervisord.log ; Log file, default is $CWD/supervisord.log
logfile_maxbytes=50MB ; Log file size; rotation occurs when exceeded, default is 50MB
logfile_backups=10 ; Number of backup log files retained, default is 10
loglevel=info ; Log level, default info; others: debug, warn, trace
pidfile=/tmp/supervisord.pid ; PID file
nodaemon=false ; Whether to start in the foreground, default is false, i.e., start as a daemon
minfds=1024 ; Minimum number of open file descriptors, default is 1024
minprocs=200 ; Minimum number of open processes, default is 200
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; Connect to supervisord via UNIX socket, path matches the 'file' in unix_http_server
;serverurl=http://127.0.0.1:9001 ; Connect to supervisord via HTTP
; Include other configuration files
[include]
files = relative/directory/*.ini ; Can be *.conf or *.ini
Save the above configuration to /etc/supervisord.conf (or any other file with appropriate access permissions), then start supervisord (specify the configuration file path via the -c option; if not specified, it searches for the configuration file in this order: $CWD/supervisord.conf, $CWD/etc/supervisord.conf, /etc/supervisord.conf):
supervisord -c /etc/supervisord.conf
To check if supervisord is running:
ps aux | grep supervisord
Program Configuration
We have now started supervisord, so we can add configuration files for the processes we want to manage. Although you can write all configuration items in the supervisord.conf file, it is not recommended. Instead, use the include directive to write different programs (groups) into separate configuration files.
For example, let’s create a directory /etc/supervisor/ to store these configuration files. Accordingly, modify the include section in /etc/supervisord.conf:
[include]
files = /etc/supervisor/*.conf
Assume there is a user center system written in Python using the Flask framework, named usercenter, using gunicorn (http://gunicorn.org/) as the web server. The project code is located at /home/leon/projects/usercenter, the gunicorn configuration file is gunicorn.py, and the WSGI callable is the app attribute in wsgi.py. So, starting it directly from the command line might look like this:
cd /home/leon/projects/usercenter
gunicorn -c gunicorn.py wsgi:app
Now, write a configuration file to manage this process (Note: When managed by supervisord, the daemon option for gunicorn must be set to False):
[program:usercenter]
directory = /home/leon/projects/usercenter ; Startup directory for the program
command = gunicorn -c gunicorn.py wsgi:app ; Startup command, same as manually starting from the command line
autostart = true ; Automatically start when supervisord starts
startsecs = 5 ; If no abnormal exit occurs within 5 seconds after startup, it is considered normally started
autorestart = true ; Automatically restart if the program exits abnormally
startretries = 3 ; Number of automatic retries on startup failure, default is 3
user = leon ; User to start the program with
redirect_stderr = true ; Redirect stderr to stdout, default is false
stdout_logfile_maxbytes = 20MB ; Size of stdout log file, default is 50MB
stdout_logfile_backups = 20 ; Number of stdout log file backups
; stdout log file. Note that if the specified directory does not exist, the program cannot start normally, so the directory must be created manually (supervisord will automatically create log files)
stdout_logfile = /data/logs/usercenter_stdout.log
; Environment variables can be added via environment. A common usage is modifying PYTHONPATH
; environment=PYTHONPATH=$PYTHONPATH:/path/to/somewhere
A configuration file requires at least one [program:x] section to tell supervisord which process to manage. In the syntax [program:x], x represents the program name, which will be displayed on the client (supervisorctl or web interface). In supervisorctl, this value is used to perform operations such as start, restart, and stop on the program.
Using Supervisorctl
Supervisorctl is a command-line client tool for supervisord. It needs to specify the same configuration file as supervisord when starting; otherwise, it searches for the configuration file in the same order as supervisord.
supervisorctl -c /etc/supervisord.conf
The above command enters the supervisorctl shell interface, where you can execute different commands:
> status # View program status
> stop usercenter # Stop the usercenter program
> start usercenter # Start the usercenter program
> restart usercenter # Restart the usercenter program
> reread # Read updated (added) configuration files, does not start newly added programs
> update # Restart programs whose configuration files have been modified
The above commands have corresponding outputs. Besides entering the supervisorctl shell interface, you can also run them directly in the bash terminal:
$ supervisorctl status
$ supervisorctl stop usercenter
$ supervisorctl start usercenter
$ supervisorctl restart usercenter
$ supervisorctl reread
$ supervisorctl update
Other Information
Besides supervisorctl, you can also configure supervisord to start a web management interface. This web backend uses Basic Auth for authentication.
In addition to controlling individual processes, you can configure groups for grouped management.
Frequently check log files, including supervisord logs and log files for each program. Program crashes or exception information are usually output to stderr, so you can check the corresponding log files to troubleshoot issues.
Supervisor has rich functionality and many other configuration options. You can find more information in the official documentation: http://supervisord.org/index.html