Flow Tracker
BACKEND SCRIPT
Flow tracking is a streaming algorithm that captures real time snapshots of interesting flows. You need to decide what flows are interesting – they could be elephant flows transferring large volume of data, or long lived flows, or video flows between 10AM and 5PM, etc.
You can create flow trackers using the Trisul Web Interface – see Flow Tracker When you create Flow Trackers from the UI you are however only allowed to use fixed criteria like IP Addresses, Ports, Volume, or Duration. The LUA flow tracker described here gives you the ultimate control over the process. For example you can create a tracker that says : “Track all flows from 10.240.0.0/16 subnet but exclude ports < 1024 and 12001,12002,10004 and IP 45.19.22.123”. This kind of complex rule building allows you to be endlessly creative and sharply focused. The LUA based flow tracker also allows you to be stateful – you can be guided by an Intel feed, or DNS.
Structure
Table flowtracker
The table consists the following
| field | type | description |
|---|---|---|
| control | Table | Details about the flow tracker. See below |
| onbeginflush | function engine | Before starting a snapshot interval |
| getmetric | function engine, flow | Compute and return a metric for this flow, return 0 or nil to discard this from the flow tracker. |
| onendflush | function engine | After all snapshots have been flushed |
| onmetronome | function(engine, timestamp, tick_count, tick_interval) | called every second if present |
Table control
The control table within the flowtracker table specifies details about the new flow tracker. This information will automatically be discovered and loaded into Trisul. Once inside Trisul the user can modify parameters like BucketSize and TopCount. Therefore these should be seen as default values.
| field | type | description |
|---|---|---|
| control | Table | Details about the flow tracker. See below |
| name | string | name of the flow tracker |
| description | string | optional description |
| bucketsize | number | snapshot interval in seconds. top-K flows with highest metrics in this time bucket are saved snapshots |
| count | number | number of flows to save in each tracker snapshot |
Functions Reference
Function getmetric
Return a metric for the supplied flow.
Purpose
Trisul is a streaming analytics platform – hence it continuously applies streaming algorithms on data streams. One of the algorithms is Flow Tracker , which is a Top-K analysis of selected flows. The getmetric function you supply in LUA serves the following function :
- it is called for every flow
- it is called after flow termination or periodically for long running flows
- return 0 : If you are not interested this flow – maybe because it is in a wrong subnet or using some other rule
- return x : If you are interested in this flow you need to supply a metric that will be used for the Top-K algorithm
When called
Called before each flow is flushed. The flow is completed or a snapshot of a long running flow.
Parameters
| engine | A Backend Engine object | use this object to add metrics, alerts, or alerts into the Trisul framework |
|---|---|---|
| flow | A flow | a flow object |
Return value
Ignored
Example
Function onbeginflush
Purpose
Called before a snapshot flush starts.
When called
Before a flush starts. Use ths if you need to perform some per-snapshot computation.
Parameters
| engine | An engine object | use this object to add metrics, alerts, or alerts into the Trisul framework |
|---|---|---|
| timestamp | Timestamp | Timestamps seconds tv_sec |
Return value
Ignored
Example
Function onendflush
Purpose
Wrap up a sequence of flush operations. You can do some cleanup operations here.
When called
When a snapshot flush is complete. use this to perform some per-snapshot computation.
Parameters
| engine | An engine object | use this object to add metrics, alerts, or alerts into the Trisul framework |
|---|---|---|
| timestamp | Timestamp | Timestamps seconds tv_sec |
Return value
Ignored
Example
Function onmetronome
Purpose
Plug into a metronome.
When called
If you define a onmetronome(..) function you will be plugged into the Trisul metronome heartbeat mechanism. This method will be called every metronome tick(roughly every second). The context in which this method is called is threadsafe and you can add metrics to the Engine from here.
Parameters
| engine | An Engine object | use this object to add metrics, resources, or alerts into the Trisul framework |
|---|---|---|
| timestamp | Number | Current timestamp (tv_sec epoch seconds) |
| tick_count | Number | An incremeting tick counter |
| tick_interval | Number | The tick interval, in seconds. |
Return value
Ignored