FAQ

What is a CorReactive?

CorReactive is a correlation engine which is based on Esper library to perform complex event processing. CorReactive is a correlation engine, it does not parse events, it expects events represented in JSON format.

How to install CorReactive?

It is required that JAVA 1.8 must be installed
Simply extract distributive to any directory and launch engine with bin/run.bat. Linux launcher is in progress (you can modify run.but to sh version it is not hard).
Currently only windows version was tested. If you have any problem running it in Linux please write me.

How to feed events to CorReactive?

CorReactive currently supports two inputs:

• Redis input which must be configured in conf/engine.json

    "inputs":[
        {
            "type": "redis",
            "config":{ 
                "host": "localhost",
                "port": 6379,
                "db": 0,
                "queue":"events",
                "batch_count":1000,
                "reconnect_timeout":60
            }
        }
    ]

• Http POST method. Simple post event JSON to http://engine_host:httpPort/api/events with tool like curl or RESTClient plugin for firefox. By default 2015 port is used but you can change it in conf/engine.json

The simpliest way to feed events to CorReactuve is to use Logstash with redis output.

How to describe event schema

Event fields types are desribed in conf/types/event.json. Currently following types of fileds are supported:

• String
• Integer
• Long
• Double
• Boolean
• Date
• Object
• Map
• Array

Simply add new fields to conf/types/event.json and reload engine.
You can define as much event types as you want, simply add new json file to conf/types/ directory and it will be converted to Esper type during loading.
When you feed any json to CorReactive it analyzes @cr_type field and tryes to convert an event to corresponding type which must be defined in conf/types/@cr_type.json file. If @cr_type is not present then engine will treat the event as type of "event".

How to write correlation rules?

CorReactive is based on Esper library. The best way to understand Esper is to read official documentation. Lots of ready to use examples are available here.
Correlation rules are placed in files located in conf/modules directory. Each file is treated like EPL module tha's why it is possible to place more than one EPL query or statement in one file, please read manual.
If you want to fire an alert, than special @Alert annotation must be placed before statement definition like this:

@Alert(name='NetworkScan',outID=1)
@Name('NetworkScan')
SELECT src_ip,window(dst_ip)
FROM event (type='firewall').win:time(1 min)
GROUP BY src_ip 
HAVING count(distinct dst_ip) > 50 
output first every 1 hour    

where

• name parameter is a name of alert, it will be placed in special "alert" field of alert.
• outID is id of outptut to which alert will be redirected

How to try Esper rules or debug them

Please use official Esper app from Esper developers.

What does @Persist annotation?

It is special annotation which was prepaired by author of CorReactive to automatically save data in Esper window every 5 minutes. Saved data is automatically restored during loading stage. You can use it like this:

@Persist
CREATE WINDOW 
LoginsIP.std:unique(ip) as (ip string, login string, last_seen string);

INSRT INTO LoginsIP
SELECT src_ip as ip, login.toLowerCase() as login, timestamp as last_seen 
FROM Event(
    type='windows' AND eventid='4624' AND src_ip IS NOT NULL 
    AND login IS NOT NULL AND login!='ANONYMOUS LOGON' 
    AND login NOT LIKE '%$' );

What does @Load annotation?

It is special annotation which was prepaired by author of CorReactive to automatically load data to Esper window every 5 minutes from csv file, which must be located in var/winload directory. You can use it like this:

@Load(file="data.csv",format="csv")
create window LoginsIP.win:keepall() as select login,src_ip from event;

The first line of csv file must contains the window columns names, next lined should contains data like this:

src_ip;login
192.168.0.1;john
192.168.0.2;ivanov

What does @Enrich annotation?

It is special annotations which was prepaired by author of CorReactive. It provides ability to add aditional data to generated alert. Forexample after detecting network scanning you can instruct engine to get process list from pc with source ip from alert and place the list to process_list field. Currently two enrichment sources are supported:

• Esper window data from which are obtained via ondemand query
  @Enrich(dst="enrichmentLogin",type="window", param="select src_ip from LoginsIP where login='%{login}'")
• Command output
  @Enrich(dst="nsLookupOut",type="cmd",param="nslookup %{src_ip}")

where

• dst is name of field where enrichment data will be placed
• type is type of enrichment (window or cmd)
• param is a parameter string which will be passed to enrichment plugin. To refference field from alert you can use %{field_name} syntax which will be converted to corresponding filed value.

How to get alerts from CorReactive?

All statements which prepended with @Alert annotation could genereate alerts. If an alert is genereated it is placed to output queue. Particular output is selected based on a outID parameter of @Alert annotation. Currently supported only redis outptut, which you can configure like this in conf/engine.json

"outputs":[
        {
            "type":"redis",
            "id":1,
            "config":{
                "host": "localhost",
                "queue":"alerts",
                "port": 6379,
                "db": 0,
                "batch_count":1,
                "reconnect_timeout":600
            }
        }
    ]

It is possible to do enrichment data wich was got by enrichment.

REST API

Send JSON event to engine
POST /api/events

View registered modules
GET /api/modules/registered

View all registered statements
GET api/modules/statements

Manual reload data in window (see @Load annotation)
POST /api/window/reload/{moduleName}/{windowName}

Perform on demand query

POST /api/query
{
    'query':'select * from LoginsIP'
}

Deploy ALL modyles.
Warning:all modules which will not be present in request will be deleted from disk. This method was created for web based GUI which is in plan, so please use it carefully.

POST api/modules/deploy
[
    {
        'uri':'module_name1',
        'text':'%Module contents%'
    },
    {
        ...
    }        
]

Validate module content

POST api/modules/validate
{
    'uri':'test module',
    'text':'text for validation'
}

How to integrate CorReactive with Logstash?

1) Configure Logstash output to feed events to CorReactive. Please read redis output manual. In high load environment do not forget to use batch_events parameter!

output {  
    redis {
        host => "127.0.0.1"
        db => 0
        data_type => "list"
        batch => true
        batch_events=>500
        key => "events"
        codec => json   
    }
}

2) Configure CorReactive to read events from Redis. You can read events from multiple Redis sources

"inputs":[
    {
        "type": "redis",
        "config":{ 
            "host": "localhost",
            "port": 6379,
            "db": 0,
            "queue":"events",
            "batch_count":500,
            "reconnect_timeout":60
        }
    }
]

3) Configure CorReactive to place alert in Redis. Next grab them with Logstash redis input.

"outputs":[
    {
        "type":"redis",
        "id":1,
        "config":{
            "host": "localhost",
            "queue":"alerts",
            "port": 6379,
            "db": 0,
            "reconnect_timeout":60,
            "batch_count":1
        }
    }
]

4) Configure esper event types in conf/types directory
5) Configutre esper modules in conf/modules