FTS Monitor
BACKEND SCRIPT
FTS stands for Full Text Search documents. These are extracted by Trisul and fts_monitor scripts can look at them and perform custom processing.
Currently there are only two FTS doc types built in to Trisul
- HTTP Headers – same format as on the wire
- SSL Certificates fully decoded as text – same structure as OpenSSL text dump of certificates
Common FTS Groups GUIDs
For quick reference these are the common FTS GUIDs For a full list Login as Admin > profil0 > All FTS Groups
| Guid | Info |
|---|---|
| {9FEB8ADE-ADBB-49AD-BC68-C6A02F389C71} | SSL Certificate FTS |
| {28217924-E7A5-4523-993C-44B52758D5A8} | HTTP Header FTS |
| {09B305DF-078C-4B9E-8E2F-EA64B7326880} | Full text dump of DNS records |
Structure
Table fts_monitor
The Lua table fts_monitor = {..} can contain one or more of the following handler functions.
| field | type | when called |
|---|---|---|
| fts_guid | String | Type of fts. Example {5AEE3F0B-9304-44BE-BBD0-0467052CF468} for SSL Certs.See Well known guids |
| onnewfts | Function( engine, fts) | A new fts was seen. Sent within 1 sec of seeing the fts |
| onbeginflush | Function( engine) | Before starting to flush all metrics to db |
| flushfilter | Function( engine, fts) | Return true if you want to save in DB, false to skip this |
| onflush | Function( engine, fts) | Called for each fts as they are being flushed |
| onendflush | engine | After all fts have been flushed for this interval |
Objects Reference
FTS
The object has the following fields
| field | return type | description |
|---|---|---|
| timestamp | number,number | The time when the item was seen. Seconds in tv_sec format, and Microseconds tv_usec.LUACopy local secs=alert:timestamp() - if you only want seconds local secs,usecs=alert:timestamp() - if you want seconds, usecs local printable = os.date(‘%c’, secs) — if you want printable |
| key | string | The unique string identifying the document |
| flow | A flow object | the IP flow from which this document was extracted |
| text | string | The text of the document. |
Example use of object
- Logging HTTP header fields.
- OCSP verification of certificate chains
- Logging certificate chains
Functions Reference
Function onnewfts
Purpose
Trisul created a new FTS document.
When called
When a new FTS document is created by Trisul.
Parameters
| engine | An engine object | use this object to add metrics, FTSs, or alerts into the Trisul framework |
|---|---|---|
| FTS | A FTS object | the FTS |
Return value
Ignored
Example
Function onbeginflush
When a stream window closes and before all the FTS documents in this window are about to be flushed to the Hub node.
Purpose
Signal begin of data flushing window.
When called
Before FTS documents are flushed to the hub node.
Parameters
| engine | An engine object | use this object to add metrics, FTSs, or alerts into the Trisul framework |
|---|---|---|
| timestamp | Timestamp | Timestamps seconds tv_sec |
Return value
Ignored
Example
Function onflush
Purpose
FTS doc flush event.
When called
Before each FTS doc is flushed to the hub node.
Parameters
| engine | An engineobject | use this object to add metrics, FTSs, or alerts into the Trisul framework |
|---|---|---|
| FTS | An FTS object | the FTS document |
Return value
Ignored
Example
Function flushfilter
Purpose
To control whether you want to flush a particular FTS document or not.
When called
Before each FTS document is flushed.
If you return false from this method, the “onflush”#function_onflush will never be called.
Parameters
| engine | An engine object | use this object to add metrics, FTSs, or alerts into the Trisul framework |
|---|---|---|
| FTS | A FTS object | the FTS document |
Return value
true
flush this FTS to the backend database node
false
dont flush this FTS, drop it
If you return false in all cases, then no FTS documents will be saved to the backend.
Example
Function onendflush
Purpose
Handle cleanup and summarization after all documents have been flushed in the streaming analysis window (default 1 minute)
When called
After all FTS documents have been flushed. the scenario is
onbeginflush()
onflush( doc-1 )
onflush( doc-2 )
...
onflush( doc-n )
onendflush()
Parameters
| engine | An engine object | use this object to add metrics, FTSs, or alerts into the Trisul framework |
|---|---|---|
| timestamp | Timestamp | Timestamps seconds tv_sec |
Return value
Ignored