Overview
Junos Space Security Director, when used with Log Collector, provides the additional capabilities of analytics based on system log messages (syslogs). This document describes the REST end points exposed for querying information from logs as aggregated data and searches on logs.
All API access is over HTTPS and accessed from the [[:spaceserver]]/api/juniper/ecm/log-scoop domain. All data is sent and received in JSON form. All the APIs are HTTP POST, since the query size is big. They do not create or modify any data in system, but just fetch the queried result.
Authentication
The API needs to be authenticated using the Junos Space Platform user's credentials. The RBAC access is applied as per the user's permissions.
Parameters
Filter is an optional parameter. The request parameter needs to be attached with each POST request in JSON form. The header of the request should have the accept and content-type information.
Content-Type of ‘application/json’:
Accept 'appliction/json'
Time Handling
All timestamps are consumed and returned in ISO 8601 format:
YYYY-MM-DDTHH:MM:SSZ
Examples:
"2007-04-05T14:30Z" or "2007-04-05T12:30-02:00"
Time Interval
The following three formats are supported.
- <start>/<end>
- <start></duration>
- <duration></end>
Examples:
Start and end, such as "2007-03-01T13:00:00Z/2008-05-11T15:30:00Z"
Start and duration, such as "2007-03-01T13:00:00Z/P1Y2M10DT2H30M"
Duration and end, such as "P1Y2M10DT2H30M/2008-05-11T15:30:00Z"
Root Endpoints
The following lists the available end points, followed by an example request and response of each type.
- Aggregation Query: /api/juniper/ecm/log-scoop/aggregate
- Time Line Query: /api/juniper/ecm/log-scoop/time-aggregate
- Non Aggregation Query: /api/juniper/ecm/log-scoop/logs
Aggregated Query
The API is useful in fetching the information based on count or sum of a given attribute. An example of this is the top source address and their counts with a given filter criteria.
End Point: /api/juniper/ecm/log-scoop/aggregate
HTTP Method: POST
Request Structure:
{
"request":{
"aggregation":"COUNT",
"aggregation-attributes":"source-address",
"time-interval":"P1M/2015-07-02T00:30:15+05:00",
"size":"5",
"order":"ascending",
"filters": {
"and": [
{
"or": [
{
"filter": {
"key": "username",
"operator": "EQUALS",
"value": "root"
}
},
{
"filter": {
"key": "source-address",
"operator": "EQUALS",
"value": "1.1.1.1"
}
}
]
}
]
}
}
}
Response:
{
"response": {
"header": {
"total-count": 5,
"message": "Successfully data retrieved",
"response-code": 200
},
"result": [
{
"key": "12.12.12.12",
"value": 304134
},
{
"key": "15.15.15.15",
"value": 202532
},
{
"key": "11.11.11.11",
"value": 202009
},
{
"key": "18.18.19.19",
"value": 101008
},
{
"key": "21.21.21.21",
"value": 100955
}
]
}
}
Multiple Aggregation Query
This option is useful in querying multipoint aggregation. An example is - which attack happened, how many times, and from which place to which place.
End Point: /api/juniper/ecm/log-scoop/aggregate
HTTP Method: POST
Request Structure:
{
"request": {
"aggregation":"COUNT",
"aggregation-attributes": ["attack-name", "src-geo.country-code2","dst-geo.country-code2"],
"time-interval":"P1D/2015-07-22T20:30:15+05:30",
"size":"2",
"order":"ascending"
}
}
Response Structure:
{
"response": {
"header": {
"total-count": 3,
"message": "Result count mismatch.",
"response-code": 201,
"key": "aggregation-attribute",
"value": ["attack-name", "src-geo.country-code2", "dst-geo.country-code2" ]
},
"result": [ {
"keys": [ "TCP:C2S:AMBIG:C2S-SYN-DATA",
"US",
"US" ],
"value": 716
},
{
"keys": [ "TCP:C2S:AMBIG:C2S-SYN-DATA",
"US",
"SY" ],
"value": 11
},
{
"keys": [ "APP:CITRIX:PROVISIONINGSERV-UF",
"SY",
"US" ],
"value": 267
}
]
}
}
Time Line Query
This query is similar to the aggregate query, with the addition of time series information on the data.
End Point: /api/juniper/ecm/log-scoop/time-aggregate
HTTP Method: POST
Request Structure
{
"request":{
"aggregation":"COUNT",
"aggregation-attributes":"source-address",
"time-interval":"P1M/2015-07-01T00:30:15+05:00",
"size":"2",
"order":"ascending",
"slots":"10",
"filters": {
"and": [
{
"or": [
{
"filter": {
"key": "username",
"operator": "EQUALS",
"value": "root"
}
},
{
"filter": {
"key": "source-address",
"operator": "EQUALS",
"value": "1.1.1.1"
}
}
]
}
]
}
}
}
Response:
{
"response": {
"header": {
"total-count": 10,
"message": "Successfully data retrieved",
"response-code": 200,
"key": "time-series",
"value": [
"2015-06-01T01:00:15-2015-06-04T01:00:15",
"2015-06-04T01:00:15-2015-06-07T01:00:15",
"2015-06-07T01:00:15-2015-06-10T01:00:15",
"2015-06-10T01:00:15-2015-06-13T01:00:15",
"2015-06-13T01:00:15-2015-06-16T01:00:15",
"2015-06-16T01:00:15-2015-06-19T01:00:15",
"2015-06-19T01:00:15-2015-06-22T01:00:15",
"2015-06-22T01:00:15-2015-06-25T01:00:15",
"2015-06-25T01:00:15-2015-06-28T01:00:15",
"2015-06-28T01:00:15-2015-07-01T01:00:15"
]
},
"result": [
{
"time-value": [
{
"key": "12.12.12.12",
"value": 304134
},
{
"key": "15.15.15.15",
"value": 202532
}
]
},
""
]
}
}
Search on Event Query
This API can be used to fetch the logs in JSON form, with a given search criteria.
End Point: /api/juniper/ecm/log-scoop/logs
HTTP Method: POST
Request Structure:
{
"request": {
"time-interval": "P10D/2015-07-01T05:30:15+05:30",
"size": "2",
"order": "ascending",
"slots": "10",
"resolve-addresses": true,
"filters": {
"and": [
{
"or": [
{
"filter": {
"key": "username",
"operator": "EQUALS",
"value": "root"
}
},
{
"filter": {
"key": "source-address",
"operator": "EQUALS",
"value": "1.1.1.1"
}
}
]
}
]
}
}
}
Response:
{
"response": {
"header": {
"total-count": 2,
"message": "Successfully data retried",
"response-code": 200
},
"result": [
{
"index": "logstash-2015.06.26-13",
"host": "10.207.99.44",
"id": "AU4wDbd51mDx7k7neYEc",
"timestamp": "2015-06-26T13:31:42.111Z",
"priority": 0,
"severity": 0,
"facility": 0,
"facility-label": "kernel",
"severity-label": "Emergency",
"username": "root]",
"country-code": "US"
},
{
"index": "logstash-2015.06.26-13",
"host": "10.207.99.44",
"id": "AU4wDbjQ1mDx7k7neZFk",
"timestamp": "2015-06-26T13:31:42.110Z",
"priority": 0,
"severity": 0,
"facility": 0,
"facility-label": "kernel",
"severity-label": "Emergency",
"username": "root]",
"country-code": "US",
"nat-destination-address-host-name": "172.19.51.235",
"source-address-host-name": "1.1.1.1",
"destination-address-host-name": "172.19.51.235",
"nat-source-address-host-name": "172.19.51.235"
}
]
}
}
Finding Attributes to Query
For Data Coming in the Log
You can pick the attribute names from the structured logs themselves. For example, in the following log:
1 2013-03-05T09:28:12.945 junosdut RT_FLOW - RT_FLOW_SESSION_CREATE [junos@2636.1 .1.1.2.36 source-address="1.1.1.1" source-port="0" destination-address="2.1.1.1" destination-port="46613" service-name="icmp" nat-source-address="1.1.1.1" nat-sou rce-port="0" nat-destination-address="2.1.1.1" nat-destination-port="46613" src-n at-rule-name="None" dst-nat-rule-name="None" protocol-id="1" policy-name="3" source-zone-name="AS_AV_UF_Zone1.90" destination-zone-name="AS_AV_UF_Zone1.91" session-id-32=" 31117" username="N/A" roles="N/A" packet-incoming-interface="fe-0/0/2.0"
The key value pairs appear as:
destination-address="2.1.1.1" destination-port="46613"
These keys can form the query attributes.
For Data Augmented by Log Collector
There are a few attributes that are not part of logs directly, but are added by the log collection system. These are listed below:
- "host" -- IP address using which device is sending logs
- "src-geo.country-name" -- Country Name of origin of traffic
- "src-geo.country-code2" -- two letter contry code
- "src-geo.country-latitude"
- "src-geo.country-longitude"
- "dst-geo.country-name" -- Country Name of destination of the traffic
- "dst-geo.country-code2" -- two letter country code
- "dst-geo.country-latitude"
- "dst-geo.country-longitude"