Security Management

Expand all | Collapse all

Junos Space log collector could not display any log

  • 1.  Junos Space log collector could not display any log

    Posted 01-25-2017 00:33

    Hi,

    Could anyone help to figure out what's the reason why my security director could not display any logs? I have checked all of these:

    1. Log collector time is synced with Junos Space

    2. SRX devices are managed by Junos Space

    3. Here's the SRX log configuraiton:

    show system syslog

    host 172.31.1.6 {
    any any;
    structured-data;

    show security log
    mode stream;
    format sd-syslog;
    source-address 172.31.1.254;
    stream logcollector {
    severity info;
    format sd-syslog;
    category all;
    host {
    172.31.1.6;
    }

    4. I run tcpdump in my log-collector, it could receviced traffic log:

    show security log
     Msg: 1 2017-01-25T16:27:31.926+08:00 SG-vSRX RT_FLOW - APPTRACK_SESSION_CLOSE [junos@2636.1.1.1.2.96 reason="idle Timeout" source-address="172.31.1.3" source-port="47999" destination-address="172.16.2.6" destination-port="161" service-name="None" application="UNKNOWN" nested-application="UNKNOWN" nat-source-address="172.31.1.3" nat-source-port="47999" nat-destination-address="172.16.2.6" nat-destination-port="161" src-nat-rule-name="N/A" dst-nat-rule-name="N/A" protocol-id="17" policy-name="local-to-spokes" source-zone-name="trust" destination-zone-name="VPN" session-id-32="106716" packets-from-client="16" bytes-from-client="3470" packets-from-server="16" bytes-from-server="5814" elapsed-time="60" username="N/A" roles="N/A" encrypted="No"]

     

    In Security Director:

    Dashborad: No data available

    Administration-Logging Management-Logging devices: nothing

    Administration-Logging Management-Logging node: could display log collecort correcttly: 

    LOG-COLLECTOR Log Receiver and Indexer 172.31.1.6 NA UP UP 15.2.R2_12 Wed Jan 25 2017 14

    Monitor-events & Logs: display "loading data for Insight Bar"...

     

    Thanks.

     



  • 2.  RE: Junos Space log collector could not display any log

     
    Posted 01-25-2017 00:41

    Hi Michael,

     

    Please check below link for troubleshooing:

     

    http://forums.juniper.net/t5/Network-Management/FAQ-Log-Collector-Deployment/ta-p/292769

     

    Regards,

    PL



  • 3.  RE: Junos Space log collector could not display any log

    Posted 01-25-2017 00:51

    [root@LOG-COLLECTOR ~]# healthcheckOSLC
    --pre checks in progress--

    LS network check Done
    LS process is active
    ES health status - green
    ERROR: No traffic between LS to ES for the current hour
    ES server credential(root) - valid

    Failure in Pre-checks, Do you still want to continue?[y/n]y

    Performing OSLC health check

    System health check in progress...
    Tomcat health check in progress...
    Logstash health check in progress...
    Elasticsearch health check in progress...

    Generating report ....

    -----------
    Overall Summary
    -----------
    System Health - GREEN
    Logstash Health - WARNING
    Elasticsearch Health - WARNING
    Tomcat Health - WARNING

    Press enter to view detail report?

    System HEALTH
    -------------

    host ip --> 172.31.1.6
    Release version --> VMware(R) Tools version 8.6.0
    Total memory(GB) --> 16.0
    CPU configured --> 4
    Time Zone configured --> SGT
    Interface card installed are --> 2
    check firewall status - ipv4 --> active
    check firewall status - ipv6 --> active

    Logstash HEALTH
    -------------

    host ip --> 172.31.1.6
    process status --> running
    watchdog status --> active
    process id --> 1961
    process running time --> 01:55:52
    memory (MB) --> 3923
    memory allocated percen(%) --> 25.0
    logrotate config --> (rotate size)100M ,(rotate limit)5
    logrotate hourly status --> active
    syslogforward status --> not active
    ES ip configured in output.conf --> 127.0.0.1
    Live traffic to 514 port(logstash) --> active
    feed files check --> domain/device feed has no data!

    Elasticsearch HEALTH
    -------------

    host ip --> 172.31.1.6
    process status --> running
    watchdog status --> active
    process id --> 1495
    process running time --> 01:56:11
    memory (MB) --> 6278
    memory allocated percen(%) --> 40.0
    logrotate config --> (rotate size)100M ,(rotate limit)5
    logrotate hourly status --> active
    whitelisted IP's in elasticsearch --> "localhost", "127.0.0.1", "172.31.1.6"
    disk roll over limit(GB) --> 400.0
    number of data nodes --> 1
    cluster status --> green
    mapping status --> mapping not applied
    current index --> no index found
    indexer configured type(5k/10k/20K) --> 5K setup
    target node --> collector-indexer
    master node ip --> no master node ip found
    firewall status - ipv4 --> active
    firewall status - ipv6 --> active
    traffic from Logstash to Elasticsearch --> no traffic

    Tomcat HEALTH
    -------------

    host ip --> 172.31.1.6
    process status --> running
    watchdog status --> active
    process id --> 906
    process running time --> 01:56:54
    memory (MB) --> 256
    logrotate config --> (rotate size)100M ,(rotate limit)5
    logrotate hourly status --> not active
    8080 port accessability --> 200

    Capturing tcpdump inprogress...
    25-01-2017-08-45-56-syslog-capture.pcap Completed.


    Completed in 90.7134530544 seconds!

    [root@LOG-COLLECTOR ~]# route -n
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    172.31.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
    169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
    0.0.0.0 172.31.1.254 0.0.0.0 UG 0 0 0 eth0

     

    threadpool.search.type: cached


    ### Authentication ###
    http.basic.enabled: true
    http.basic.user: "admin"
    http.basic.password: "58dd311734e74638f99c93265713b03c391561c6ce626f8a745d1c7ece7675fa"
    http.basic.ipwhitelist: ["localhost", "127.0.0.1", "172.31.1.6"]
    [root@LOG-COLLECTOR ~]# cat /etc/hosts
    172.31.1.6 LOG-COLLECTOR localhost.localdom localhost
    127.0.0.1 localhost.localdom localhost
    [root@LOG-COLLECTOR ~]#

     

    Seems all of above are correct.



  • 4.  RE: Junos Space log collector could not display any log

    Posted 01-25-2017 00:55
    • Check Logging related statistics under
         - Administration->Logging Management->Logging Nodes
         - Administration->Logging Management->Statistics & Troubleshooting

    nothing here......

     



  • 5.  RE: Junos Space log collector could not display any log

     
    Posted 01-25-2017 00:59

    please check if 514 port is enabled:

     

    #netstat -antp | grep 514

     

    If not then try to reboot log collector.

    Also do you have hostname configured for SRX?



  • 6.  RE: Junos Space log collector could not display any log

    Posted 01-25-2017 01:03

    Here's the output:

    [root@LOG-COLLECTOR ~]# netstat -antp | grep 514
    tcp 0 0 :::514 :::* LISTEN 1961/java

     

    What do you mean hostname for SRX? Do you mean if I use "set system host-name xxx". If so, yes.

     

    Thanks.



  • 7.  RE: Junos Space log collector could not display any log

    Posted 01-25-2017 17:46

    Hi Expert,

     

    May I have your kindly suggestion to solve the issue?

     

    Thanks.



  • 8.  RE: Junos Space log collector could not display any log

     
    Posted 01-25-2017 18:34

    Need to look at the logs, so it is better to open JTAC case or attach /var/log/jboss/server/server1/SD.log and from log collector: /var/log/elasticsearch/log-collector.log

     

    Did you try to restart elasticsearch & logstash service? if not please try to restart once.

     

    service elasticsearch stop

    service logstash stop

    service elasticsearch start

    service logstash start



  • 9.  RE: Junos Space log collector could not display any log

    Posted 01-25-2017 18:42

    Thanks for the reply, I've tried to reboot the log collector server for many times. And seems I could not stop logstash service as below:

     

    [root@LOG-COLLECTOR ~]# service elasticsearch stop
    Stopping elasticsearch: [ OK ]
    [root@LOG-COLLECTOR ~]# service logstash stop
    Killing logstash (pid 16149) with SIGTERM
    Waiting logstash (pid 16149) to die...
    Waiting logstash (pid 16149) to die...
    Waiting logstash (pid 16149) to die...
    Waiting logstash (pid 16149) to die...
    Waiting logstash (pid 16149) to die...
    logstash stop failed; still running.
    [root@LOG-COLLECTOR ~]# service elasticsearch start
    service logstash startStarting elasticsearch: [ OK ]
    [root@LOG-COLLECTOR ~]# service logstash start
    logstash is already running

     

    Thanks.



  • 10.  RE: Junos Space log collector could not display any log

     
    Posted 01-25-2017 18:45

    kill -9 16149



  • 11.  RE: Junos Space log collector could not display any log

     
    Posted 01-25-2017 18:52

    Did you try to restart jboss service? if not please restart jboss service from Junos Space CLI as LC seems to be working fine.

     

    service jboss restart



  • 12.  RE: Junos Space log collector could not display any log

    Posted 01-25-2017 22:38

    At last, I reinstall Junos space and log collector. Now it works. Thanks.



  • 13.  RE: Junos Space log collector could not display any log

    Posted 10-11-2018 07:08

    Hi dude

    I've had the same behaviour: SD 18.1R2.44 with Integrated Log-Collector. The vm has 16GB memory and 1.5TB non-SSD disk. 

    The TAC suggested to update the ram size (because elasticsearch use it for 98%) I did that to 24GB but this not solved the issue (elasticsearch is continung to use all the memory). I had to re-install space+logcollector in order to restore the Monitor view.

     

    regards

    Entoni



  • 14.  RE: Junos Space log collector could not display any log

    Posted 01-25-2017 19:10

    attached is the log