Add web server capacity when WebTrisul slows down under many users

Field	Value
App Note ID	AN-013
Audience	Trisul admins, NOC and network engineers
Applies to	All product modes

Summary

WebTrisul runs as an nginx reverse proxy in front of a pool of backend application processes. A default install starts 5 backend processes. If the web interface becomes sluggish or many people use it at the same time, you can raise the number of backend processes. In the current WebTrisul stack this is one setting, workers, in /usr/local/share/webtrisul/build/puma-config.rb, followed by a restart of the web service.

The problem

Every WebTrisul request except static files is served by a backend process. With the default pool of 5 processes, requests queue when many users work at the same time or when several heavy pages load together. Users see this as a slow or unresponsive interface.

The source HowTo describes an older stack built on thin servers, where the server count was set in three files: nginx.conf, thin-nginxd.conf and thin-config.yml. Current WebTrisul releases use the Puma application server instead. The thin files no longer exist, and the server count is set in a single place. This note gives the procedure for the current stack and maps each step of the older HowTo to its current equivalent.

How it works

The bundled nginx web server accepts browser connections on the web port (3000 for HTTP, 443 for HTTPS). It forwards every request to one upstream named puma_cluster, which points to the Unix socket /tmp/puma.webtrisul.sock.

The Puma application server listens on that socket. Puma runs a master process and a number of worker processes. The workers setting in the Puma configuration file sets how many worker processes start. Each worker runs one thread (threads 1, 1), so the worker count is the number of requests WebTrisul can process at the same time.

Because nginx sends all requests to the single socket, you do not change nginx.conf when you add workers.

Browser ──► nginx (listen 3000 or 443)
              │  upstream puma_cluster
              ▼
       unix:/tmp/puma.webtrisul.sock
              │
              ▼
        Puma master process
     ┌────┬────┼────┬────┐
     W1   W2   W3   W4   W5 ... Wn      (workers N, threads 1,1)

The HTTP service (webtrisuld) reads puma-config.rb. The HTTPS service (webtrisulssld) reads puma-configssl.rb. Both files are in /usr/local/share/webtrisul/build and both default to workers 5.

Reference: Customizing the web server stack

Prerequisites

• Root or sudo access to the Trisul Hub server.
• Knowledge of which web service you run: webtrisuld (HTTP) or webtrisulssld (HTTPS). See Using HTTPS.
• The number of workers you want. The default is 5. The documentation gives no sizing rule.

Procedure

Run the shell commands in this procedure as root or with sudo.

Step 1: Locate the configuration files

The web server configuration files are stored in /usr/local/share/webtrisul/build.

1. List the Puma configuration files:

   ls /usr/local/share/webtrisul/build/puma-config*.rb

2. Choose the file for the service you run:

Service	File to edit
webtrisuld (HTTP)	puma-config.rb
webtrisulssld (HTTPS)	puma-configssl.rb

Step 2: Set the number of backend workers

1. Open the file from Step 1 in a text editor.

2. Find the concurrency setting. By default it reads:

   workers 5

3. Replace 5 with the number of workers you want, for example workers 8.

4. Leave threads 1, 1 and worker_timeout 900 unchanged.

5. Save the file.

Check: The file contains exactly one workers line with the new value.

Step 3: Leave nginx.conf and the old thin files as they are

The older HowTo also asked you to edit nginx.conf, thin-nginxd.conf and thin-config.yml. On the current stack, do the following instead:

Older HowTo instruction	Current WebTrisul
nginx.conf: add server names to the upstream thin_cluster section	Do not edit. nginx.conf has a single upstream puma_cluster entry that points to the Puma socket. It serves any number of workers.
thin-nginxd.conf: update NO_OF_SERVERS	Not present. No second file needs the server count.
thin-config.yml: update servers	Replaced by workers in puma-config.rb or puma-configssl.rb (Step 2).

Step 4: Restart WebTrisul

1. Restart the service you run. For HTTP:

   systemctl restart webtrisuld

   For HTTPS:

   systemctl restart webtrisulssld

The restart stops and starts the Puma application server and nginx. Puma reads the new workers value when it starts.

Verify the result

1. List the Puma processes:

   ps -ef | grep puma

   You see one Puma master process and one worker process for each worker you configured.

2. Check the Puma logs in /usr/local/share/webtrisul/log for errors: puma.log, puma.stdout.log and puma.stderr.log.

3. Open WebTrisul in a browser and log in. The login page loads and dashboards render.

Troubleshooting

The web interface does not load after the restart. Check puma.stderr.log and puma.log in /usr/local/share/webtrisul/log for errors. Correct the workers line if the error points to it, and restart the service. Also check the nginx error log at /usr/local/share/webtrisul/log/nginx.error.log.

nginx.conf changed after a restart, and a file named nginx.conf.bak_<date>_<time> appeared. You or an earlier procedure added an upstream thin_cluster section to nginx.conf. Each time the webtrisuld service runs, it checks nginx.conf and nginxssl.conf for that section. If it finds one, it saves a backup with the .bak_ suffix, removes the thin sections and points the proxy to puma_cluster. This is expected. Do not add thin servers back. Use the workers setting instead.

The worker count did not change. You edited the file for the other service. webtrisuld reads only puma-config.rb, and webtrisulssld reads only puma-configssl.rb. Edit the file for the service you run and restart it.

You need application-level logs. The WebTrisul application logs are in /usr/local/var/log/trisul-hub/webtrisul (production.log for HTTP, productionssl.log for HTTPS). You can also view them in Web Admin → Manage → Web Server Logs.

Limitations

• The documentation gives no sizing rule for the number of workers. Each worker is a separate process on the Hub server.
• This note changes only the worker count. It does not cover changes to threads, worker_timeout or other nginx settings.
• The documentation does not state whether a WebTrisul package upgrade keeps an edited puma-config.rb or puma-configssl.rb.
• The procedure is verified against the documentation and the product configuration. It has not been lab-tested.

Related

• Using HTTPS
• Change the Trisul webserver port
• Web Server Logs
• Start/Stop Tasks
• AN-011 – Enable HTTPS for the web UI
• AN-012 – Change the web server port