Skip to main content

Recover Trisul from a full disk and stop it filling again

FieldValue
App Note IDAN-010
AudienceTrisul admins, NOC and network engineers, ISP operators
Applies toAll product modes (PCAP steps apply only where packets are stored)

Summary​

When the Trisul disk reaches 100%, Trisul stops working and the web interface may not respond. You recover by mounting a new volume and moving one or more data directories to it, either with the relocate context command or by hand with a symbolic link left in the old location. You then check free space with df, restart the web server and the Trisul components, and tune the SlicePolicy on the Probe and the Hub so the disk does not fill again.

The problem​

Trisul writes packet captures on the Probe and metrics, flows and alerts on the Hub. Both keep data for a fixed window that you size when you install Trisul. If traffic grows, or the window was sized for a larger disk than the one you have, the disk fills up. At 100% Trisul stops working and you may not be able to open the web interface, so you cannot fix it from the UI.

You need to free space quickly without deleting the data you keep for investigations, and then change the retention settings so the problem does not return.

How it works​

By default Trisul stores all its data under /usr/local/var. Each context has its own data directory on each node. For the default context the directory name is context0; a named context uses context_<name>.

NodeDataDirectory for the default contextRetention is set by
ProbePacket captures (PCAP)/usr/local/var/lib/trisul-probe/domain0/probe0/context0/capsDisk space: FileSizeMB × SliceCount
HubMetrics, flows, alerts/usr/local/var/lib/trisul-hub/domain0/hub0/context0/metersDays: SliceCount with a DAILY slice window

Both directories hold three storage areas: oper, ref and archive. Trisul writes new slices in oper. As slices age they slide into ref, then into archive, and are then deleted as the SlicePolicy specifies. A storage area can live on another volume: Trisul moves slices into it in the same way.

To free space you move one or more of these areas to a new volume. There are two ways:

  • relocate context in trisulctl_probe or trisulctl_hub moves the whole data directory of a context (Probe: packet captures, configuration database and logs; Hub: the database slices, configuration database and logs) to the new volume and rewrites the paths in the configuration file.
  • Move by hand: you move one storage area, for example archive, to the new volume and leave a symbolic link with the same name in the old location. Trisul keeps using the old path.

With the default Probe SlicePolicy the archive area holds 0 slices (8 in oper, 8 in ref). On a default install most packet data is in oper and ref, so moving archive alone may free little space.

Reference: Configure Retention Policy, Relocate Trisul Database

Storage areas before and after a manual move (Part C) Figure: Storage areas before and after a manual move (Part C)

Slices slide from oper to ref to archive. After a manual move, each archive path is a symbolic link (dashed) to a directory on the new volume, and Trisul keeps writing through it.

Prerequisites​

  • Root shell access to the Probe node and the Hub node. On a single-node install they are the same server.
  • A new disk or partition with enough free space for the data you will move.
  • For relocate context: the target directory must already exist, and its file system must have at least 1.1 times the current size of the context data free. The move can take a long time on a large database, so run it inside a screen or tmux session.
  • For a manual move: stop the context first, as relocate context does before it moves data. Run trisulctl_hub and enter stop context default.

Procedure​

Part A is needed in every case. Then use Part B (one command per node) or Part C (move storage areas by hand). Finish with Part D so the disk does not fill again.

Part A – Mount a new volume​

  1. Create a new partition with enough space.
  2. Mount it, for example on /mnt/trisul_extra.

Part B – Move all data of a context with relocate context​

This moves the oper, ref and archive areas of a Probe context, with its configuration database and logs, in one step.

  1. On the Probe node, start the Probe control utility:
/usr/local/bin/trisulctl_probe
  1. Run relocate context for the context. For the default context:
relocate context domain0 probe0 default
  1. The command stops the context if it is running, then shows the current location and size and asks for the new location. Enter the mount point of the new volume:
Current DB Root is at              : /usr/local/var
Current size                       : 5.67 MB
-----------------------------------------------------------------------------
Enter new location (enter to quit) : /mnt/trisul_extra
  1. Review the summary and type YES at the Confirm (Enter YES/no) ? : prompt. Do not interrupt the move.
  2. On a single-node install, also relocate the Hub data. Start /usr/local/bin/trisulctl_hub and run:
relocate context domain0 hub0 default
  1. Restart the context. In trisulctl_hub, run:
restart context default
quit
  1. Restart the web server:
systemctl restart webtrisuld

Check: The command ends with ** Successfully relocated context [context0] to [/mnt/trisul_extra/]. Run df to confirm the old disk has space again.

Part C – Move storage areas by hand​

Move one storage area at a time and check the result after each move.

  1. Move the Probe archive area to the new volume and leave a symbolic link in its place:
cd /usr/local/var/lib/trisul-probe/domain0/probe0/context0/caps/
mv archive /mnt/trisul_extra/probe_archive
ln -sf /mnt/trisul_extra/probe_archive archive
  1. Run df and check how much disk is now free.
  2. If not enough disk is free, repeat step 1 for the ref area:
cd /usr/local/var/lib/trisul-probe/domain0/probe0/context0/caps/
mv ref /mnt/trisul_extra/probe_ref
ln -sf /mnt/trisul_extra/probe_ref ref
  1. Run df again. If more space is still needed, repeat step 1 for the oper area in the same way.

Check: If disk usage has dropped to about 70–80%, you can stop here and skip step 5.

  1. If you still need space, move the Hub archive area to the new volume and leave a symbolic link in its place:
cd /usr/local/var/lib/trisul-hub/domain0/hub0/context0/meters/
mv archive /mnt/trisul_extra/hub_archive
ln -sf /mnt/trisul_extra/hub_archive archive
  1. Restart the web server:
systemctl start webtrisuld
  1. Start the Trisul components. Run trisulctl_hub and enter start context default.

Part D – Tune the SlicePolicy so the disk does not fill again​

Set retention so that the data fits on the disk with room to spare.

  1. On the Probe, open /usr/local/etc/trisul-probe/domain0/probe0/context0/trisulProbeConfig.xml as root. Packet storage is FileSizeMB (the size of each PCAP file, maximum 8000) multiplied by the total SliceCount of Operational, Reference and Archive. The default is 1000 MB × (8 + 8 + 0) = 16 GB. Lower FileSizeMB or the slice counts to fit your disk.
  2. Restart the Probe from Context: default → Admin Tasks → Start/Stop Tasks.
  3. On the Hub, open /usr/local/etc/trisul-hub/domain0/hub0/context0/trisulHubConfig.xml as root. In SlicePolicy, each SliceCount is a number of days with the default DAILY window. The default is 32 + 32 + 32 = 96 days. Lower the counts to keep fewer days.
  4. Save the file. You do not need to restart the Hub: the trisul_archiver task runs every night at 12:00 AM and applies the new slice counts.
  5. Optional: set UsageRedMark to a percentage to get an alert when disk usage crosses it. In the Probe template it is 90 for each packet storage area; on the Hub it is blank (disabled) by default.

Reference: Trisul Probe Configuration: SlicePolicy, Trisul Hub Configuration: SlicePolicy

For large deployments, also follow these practices:

  • Run Trisul for 3–4 days to measure data growth per day, and size the disks from that. Context: default → Admin Tasks → Storage Status shows the disk space used per day.
  • Prefer the XFS file system over EXT4, because it supports a large number of inodes.
  • Use LVM, so you can expand a full volume by adding disk capacity.
  • Use RAID-0 for Probe PCAPs, for higher write performance, and RAID-5 for the Hub, for resilience.
  • In large enterprises, mount archive on a NAS if you have one.
  • Use PCAP pruning rules, for example to stop storing video streaming packets.

Verify the result​

  1. Run df on each node. Usage on the original disk should be well below 100%. At about 70–80% you can stop moving data.
  2. Open the web interface. Log in as admin and check on Context: default → Admin Tasks → Start/Stop Tasks that the Hub and the Probe are running.
  3. Open Context: default → Admin Tasks → Storage Status. In Storage Pools, check Disk Free for each pool. After the nightly archiver run, Configured Days shows the Hub retention you set in Part D.

Troubleshooting​

Symptom or messageCauseFix
The target directory does not exist, please create the directory and retry 'reloc context..'The location you entered for relocate context does not existCreate and mount the directory, then run the command again.
Not enough space on new directory. Need <size> available <size>The new file system has less than 1.1 times the current size of the context data freeUse a larger volume, or move single storage areas by hand (Part C).
Error: Invalid context <name> in :<domain>/<node>The context name is wrongUse default for the default context, or the name of the named context.
X ERROR! Unable to find mount point/root directory : <path>The current data root of the context is not reachableMount the directory and retry, or migrate the configuration files manually.
Moving the Probe archive area freed almost no spaceWith the default Probe SlicePolicy, archive holds 0 slicesMove ref, and then oper if needed (Part C, steps 3 and 4).
The Hub disk keeps filling although the slice counts fitDeleteMode Type in trisulHubConfig.xml is MANUAL, which disables automatic deletion of old slicesSet Type back to AUTO (the default). If you use TRASH, set TrashLocation and consider MaxTrashDiskPercentage.

Limitations​

  • Moving data frees space but does not change retention. Without Part D the disk fills again.
  • Lowering a SliceCount deletes the oldest slices for good. An Archive SliceCount of 0 means slices are deleted when they leave ref.
  • A lower Hub SliceCount takes effect at the next nightly archiver run (12:00 AM), not immediately.
  • relocate context moves the whole context. To move only one storage area, use Part C.
  • The paths in Part C are for the default context. A named context uses context_<name> in place of context0.
In this page: