Management

 View Only
last person joined: 2 days ago 

Ask questions and share experiences with Junos Space and network management.
Back to discussions
Expand all | Collapse all

The Junos Space web page does not load correctly. The server was turned off due to power supply.

  • 1.  The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 05-21-2024 13:24

    Good afternoon.

    The virtual server is quite old and was inherited (I had no previous experience with Junos Space).
    It is no longer possible to log into Junos Space correctly via the Web. The page loads empty - just white color of page. In this case, I can view the server certificate via browser (it is self-signed and has expired).
    What happened since the last successful login
    1. The web page certificate has expired.
    2. The power supply was turned off in the building and all servers went down by power.

    I can connect to the server itself via SSH. I restarted the services httpd, jboss, mysql - all restarted without the Failed status. Rebooted the server. I turned off NTP and rolled back the server time a month ago, but this did not help (for a while the certificate had not expired since I could not find how to install the certificate from Debug cli [all instructions only via the web]).

    Next, we tried to restore this server from a backup, but encountered another problem. The restored server on VMWare did not have an IP address, and when starting the vmware remote console, at the moment when the server should shown a login password  prompt, it gave a full black screen where you could write text, but the server did not give any feedback (there was no login password prompt ). It can be booted into another mode, I press the button at startup, but it was also not possible to set the address through any mode on the network interface, because of lack knowledge.

    Please help me understand
    1) How can I diagnose why the Junos Space web page is not loading. What diagnostic tool logs can be used via cli?
    2) Has anyone encountered that during normal server loading it is not possible to do anything via the VM-WARE remote console. How to overcome this?

    [root@space-0050569a3d7b ~]# service httpd restart

    Stopping webProxy:                                         [  OK  ]

    Starting webProxy: [Wed May 15 13:28:49 2024] [warn] module headers_module is already loaded, skipping

    httpd: Could not reliably determine the server's fully qualified domain name, using 10.41.3.181 for ServerName

                                                                             [  OK  ]

    [root@space-0050569a3d7b ~]# cat /var/log/httpd/error_log

    [Wed May 15 12:31:25 2024] [notice] Balancer mycluster changed

    [Wed May 15 12:31:30 2024] [notice] Balancer mycluster changed

    [Wed May 15 12:32:01 2024] [notice] Balancer mycluster changed

    [Wed May 15 12:32:27 2024] [notice] Balancer mycluster changed

    body.xml:1: parser error : Document is empty

    FD97A41459FD495A43E3BF922B40DB23<RECORD><NAME>SRS</NAME><OPERATION>4</OPERATION>

    ^

    [Wed May 15 12:33:54 2024] [notice] Balancer mycluster changed

    [Wed May 15 12:38:50 2024] [notice] caught SIGTERM, shutting down

    [Wed May 15 12:42:25 2024] [warn] RSA server certificate CommonName (CN) `10.41.3.182' does NOT match server name!?

    [Wed May 15 12:42:25 2024] [warn] httpd version 2.2.34 mismatch detected

    [Wed May 15 12:42:25 2024] [notice] ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/) configured.

    [Wed May 15 12:42:25 2024] [notice] ModSecurity: APR compiled version="1.3.9"; loaded version="1.3.9"

    [Wed May 15 12:42:25 2024] [notice] ModSecurity: PCRE compiled version="7.8 "; loaded version="7.8 2008-09-05"

    [Wed May 15 12:42:25 2024] [notice] ModSecurity: LUA compiled version="Lua 5.1"

    [Wed May 15 12:42:25 2024] [notice] ModSecurity: LIBXML compiled version="2.7.6"

    [Wed May 15 12:42:25 2024] [notice] Original server signature: Apache

    [Wed May 15 12:42:25 2024] [warn] module headers_module is already loaded, skipping

    [Wed May 15 12:42:26 2024] [warn] RSA server certificate CommonName (CN) `10.41.3.182' does NOT match server name!?

    [Wed May 15 12:42:26 2024] [warn] httpd version 2.2.34 mismatch detected

    [Wed May 15 12:42:26 2024] [notice] Advertise initialized for process 10482

    [Wed May 15 12:42:26 2024] [notice] Apache/2.2.34 (Unix) mod_ssl/2.2.34 OpenSSL/1.0.1e-fips mod_cluster/1.2.6.Final Server configured -- resuming normal operations



    ------------------------------
    Aleksandr Maksimov
    ------------------------------


  • 2.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 30 days ago

    Also, send the o/p for:

     openssl x509 -text -inform PEM -in /etc/ssl/certs/jmp_default.crt | grep After
     openssl x509 -text -inform PEM -in /etc/ssl/certs/jmp.crt | grep After
     openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
     openssl x509 -text -inform PEM -in /var/chroot/mysql/var/lib/mysql/ca.pem | grep After
     openssl x509 -text -inform PEM -in /etc/sysconfig/certs/*.pem | grep After
     openssl x509 -text -inform PEM -in /etc/pki/jmp-softwareManager/certs/JuniperRootCA.pem | grep After
     openssl x509 -text -inform PEM -in /etc/pki/jmp-softwareManager/certs/jmp_root_cert.crt | grep After
     openssl x509 -text -inform PEM -in /etc/pki/tls/certs/ca-bundle.crt | grep After



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 3.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 30 days ago

    [root@space-0050569a3d7b ~]#  openssl x509 -text -inform PEM -in /etc/ssl/certs/jmp_default.crt | grep After
                Not After : Apr 25 10:31:23 2024 GMT
    [root@space-0050569a3d7b ~]#  openssl x509 -text -inform PEM -in /etc/ssl/certs/jmp.crt | grep After
                Not After : Apr 25 10:31:23 2024 GMT
    [root@space-0050569a3d7b ~]#  openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
                Not After : Apr 25 10:33:06 2024 GMT
    [root@space-0050569a3d7b ~]#  openssl x509 -text -inform PEM -in /var/chroot/mysql/var/lib/mysql/ca.pem | grep After
    Error opening Certificate /var/chroot/mysql/var/lib/mysql/ca.pem
    139896370272072:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/var/chroot/mysql/var/lib/mysql/ca.pem','r')
    139896370272072:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:
    unable to load certificate
    [root@space-0050569a3d7b ~]#  openssl x509 -text -inform PEM -in /etc/sysconfig/certs/*.pem | grep After
                Not After : Apr 25 10:31:55 2024 GMT
    [root@space-0050569a3d7b ~]#  openssl x509 -text -inform PEM -in /etc/pki/jmp-softwareManager/certs/JuniperRootCA.pem | grep After
                Not After : Nov  1 01:59:25 2037 GMT
    [root@space-0050569a3d7b ~]# openssl x509 -text -inform PEM -in /etc/pki/jmp-softwareManager/certs/jmp_root_cert.crt | grep After
                Not After : Aug 29 07:14:59 2021 GMT
    [root@space-0050569a3d7b ~]#  openssl x509 -text -inform PEM -in /etc/pki/tls/certs/ca-bundle.crt | grep After
                Not After : Dec 31 09:37:37 2030 GMT



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 4.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 28 days ago

    Looking at the openssl output, most of the certs are expired.

    You can try below workaround:

    change date to (1 year ago)
    date --set='-1 year'

    Generate new certificate using (KB29266) and upload the new certificate to GUI.

    revert the date change:
    date --set='+1 year'



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 5.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 30 days ago

    I generated a self-signed certificate according to instruction ( https://supportportal.juniper.net/s/article/Junos-Space-Generating-a-replacement-web-Certificate-for-Junos-Space-CSR-or-Self-signed?language=en_US ) but I couldn't find how to install it correctly via CLI. 

    [root@space-0050569a3d7b ~]# ls |grep  spa
    space_self_cert.key
    space_self_cert.pem



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 6.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 30 days ago

    Hi,

    Restore it to the point where it was.

    Can you provide the output for:

    # df -hT
    # free -h
    # cat /etc/hosts
    # cat /etc/redhat-release
    # ls -al /etc | grep -i hotp

    Upload a log file: /var/log/jboss/servers/server1/server.log



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 7.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 28 days ago

    Hi,

    Thank you for your help.

    [root@space-0050569a3d7b ~]# df -hT
    Filesystem           Type   Size  Used Avail Use% Mounted on
    /dev/mapper/jmpvgnocf-lvroot
                         ext4   288G  3.7G  269G   2% /
    tmpfs                tmpfs   16G  4.0K   16G   1% /dev/shm
    /dev/sda1            ext4    93M   33M   55M  38% /boot
    /dev/mapper/jmpvgnocf-lvtmp
                         ext4    48G  311M   46G   1% /tmp
    /dev/mapper/jmpvgnocf-lvvar
                         ext4   111G  104G  887M 100% /var
    /dev/mapper/jmpvgnocf-lvlog
                         ext4    39G   18G   19G  48% /var/log
    [root@space-0050569a3d7b ~]# free -h
                 total       used       free     shared    buffers     cached
    Mem:           31G        17G        13G       143M        38M       1.1G
    -/+ buffers/cache:        16G        14G
    Swap:         7.8G         0B       7.8G
    [root@space-0050569a3d7b ~]# cat /etc/hosts
    127.0.0.1       localhost.localdomain   localhost
    10.41.3.181     space-0050569a3d7b      space-0050569a3d7b server1 server2
    10.41.3.182     jmp-CLUSTER     jmp-CLUSTER
    228.41.3.182    jmp-MCAST       jmp-MCAST
    [root@space-0050569a3d7b ~]# cat /etc/redhat-release
    Space release 18.4R1.5 (391560)
    [root@space-0050569a3d7b ~]# ls -al /etc | grep -i hotp
    [root@space-0050569a3d7b ~]#

    server.log was very big . I put it to archive. 

    In the end of file we have a lot of same lines:

    2024-05-13 11:38:17,416 WARN  [org.jboss.modcluster.load.impl.DynamicLoadBalanceFactorProvider] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Platform is Initializing will return -1 to httpd
    2024-05-13 11:38:27,417 WARN  [org.jboss.modcluster.load.impl.DynamicLoadBalanceFactorProvider] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Platform is Initializing will return -1 to httpd



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 8.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 29 days ago

    Sounds like you also encountered:

    https://supportportal.juniper.net/s/article/Junos-Space-gets-stuck-during-boot-process

    But have since gotten past this.



    ------------------------------
    Matthew Paskey
    ------------------------------

    Original Message


  • 9.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 28 days ago
      |   view attached

    Hello, 

    Thank you for advice. 

    1. I tried with date. 

    date --set='-1 year'
    service jboss restart
    service httpd restart
    service mysql restart

    [root@space-0050569a3d7b ~]# date
    Wed May 24 06:55:04 UTC 2023

    but page still doesn't correctly opening (tried from many browsers and different PC). White page without any information or errors. So, I suppose problem isn't in the certificate. 

    2.  https://supportportal.juniper.net/s/article/Junos-Space-gets-stuck-during-boot-process => Your account doesn't provide access to restricted content in the Juniper Support Portal. Unfortunately. 

    3.  I cleared /var/log/jboss/servers/server1/server.log file, reboot the server and download it. Now it smaller and I can attached it. 



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Attachment(s)

    txt
    Server.log.txt   78 KB 1 version
    Original Message


  • 10.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 28 days ago

    Hi,

    Error in server.log "

     No space left on device

    "

    I didnt catch that in earlier update of yours... /var/ partition is 100% full.

    You need to delete/remove some file... maybe some of the log file is consuming more memory.

    Please check and send the o/p for: 

    find /var -type f -exec du -Sh {} + | sort -rh | head -n 5



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 11.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 18 days ago

    Sorry for delay in reply! I was on work trip.

    [root@space-0050569a3d7b ~]# find /var -type f -exec du -Sh {} + | sort -rh | head -n 5
    5.3G    /var/chroot/mysql/var/lib/mysql/build_db/DeviceImageContentsEntity.ibd
    2.0G    /var/log/vpn-monitor/vpnlogparser.log
    1.9G    /var/chroot/mysql/var/lib/mysql/build_db/Job.ibd
    1.3G    /var/chroot/mysql/var/lib/mysql/appvisibility_db/ApplicationStatistics.ibd
    1.1G    /var/lib/pgsql/9.4/data/base/16385/16589

    [root@space-0050569a3d7b /]# du -h --max-depth=1 /var/
    4.0K    /var/games
    55G     /var/lib
    257M    /var/tmp
    4.0K    /var/preserve
    1.7G    /var/cache
    41G     /var/chroot
    4.0K    /var/crash
    8.0K    /var/specialNodeCerts
    248K    /var/run
    4.0K    /var/nis
    4.0K    /var/cvs
    4.0K    /var/yp
    96K     /var/spool
    4.0K    /var/opt
    17M     /var/www
    8.0K    /var/backups
    4.0K    /var/local
    1.7G    /var/opennms
    32K     /var/db
    16K     /var/lost+found
    28K     /var/lock
    8.0K    /var/empty
    4.8G    /var/log    = VAR/LOG/ was 19Gb - I deleted old logs file. - now  it 4.8G. Then I rebooted the server. But it didn't help
    5.7G    /var/jboss
    4.0K    /var/account
    109G    /var/


    [root@space-0050569a3d7b /]#  du -h --max-depth=1 /var/log/
    8.0K    /var/log/pluto
    174M    /var/log/nma
    48K     /var/log/redis
    20K     /var/log/libvirt
    15M     /var/log/sa
    23M     /var/log/slipstream
    4.0K    /var/log/glusterfs
    32K     /var/log/ConsoleKit
    2.9M    /var/log/packet-capture
    264M    /var/log/jboss
    4.0K    /var/log/ntpstats
    29M     /var/log/audit
    1.1M    /var/log/jmp-geo
    4.0K    /var/log/cassandra
    16K     /var/log/lost+found
    231M    /var/log/secmgt-skyatp-proxy
    245M    /var/log/httpd
    1.3G    /var/log/opennms
    4.0K    /var/log/prelink
    8.0K    /var/log/samba
    2.8M    /var/log/space-debug
    2.0G    /var/log/vpn-monitor
    131M    /var/log/elasticsearch
    4.0K    /var/log/aide
    4.8G    /var/log/

    [root@space-0050569a3d7b /]# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/jmpvgnocf-lvroot
                          288G  3.7G  269G   2% /
    tmpfs                  16G  4.0K   16G   1% /dev/shm
    /dev/sda1              93M   33M   55M  38% /boot
    /dev/mapper/jmpvgnocf-lvtmp
                           48G  309M   46G   1% /tmp
    /dev/mapper/jmpvgnocf-lvvar
                          111G  104G  889M 100% /var
    /dev/mapper/jmpvgnocf-lvlog
                           39G  9.6G   27G  27% /var/log

    [root@space-0050569a3d7b var]# ls -lh
    total 456K
    drwxr-xr-x.  2 root    root    4.0K Dec 10  2018 account
    drwxr-xr-x.  3 root    root    4.0K Dec 10  2018 backups
    drwxrwx---. 21 root    space   4.0K Apr 25  2019 cache
    drwxr-x---   3 root    root    4.0K Apr 25  2019 chroot
    drwxr-xr-x.  2 root    root    4.0K Apr 25  2019 crash
    drwxr-xr-x.  2 root    root    4.0K Nov 22  2013 cvs
    drwxr-xr-x.  4 root    root    4.0K Dec 10  2018 db
    drwxr-xr-x.  3 root    root    4.0K Dec 10  2018 empty
    drwxr-xr-x.  2 root    root    4.0K Sep 23  2011 games
    drwxr-x---. 22 jboss   jboss   4.0K Apr  4  2023 jboss
    drwxr-xr-x. 37 root    root    4.0K Jun  3 03:07 lib
    drwxr-xr-x.  2 root    root    4.0K Sep 23  2011 local
    drwxr-xr-x.  6 root    root    4.0K Jun  3 11:31 lock
    drwxrwx--x. 26 root    space    32K Jun  3 11:30 log
    drwx------.  2 root    root     16K Dec 10  2018 lost+found
    lrwxrwxrwx.  1 root    root      10 Dec 10  2018 mail -> spool/mail
    drwxr-xr-x.  2 root    root    4.0K Sep 23  2011 nis
    drwxr-x---.  7 opennms opennms 4.0K Dec 10  2018 opennms
    drwxr-xr-x.  2 root    root    4.0K Sep 23  2011 opt
    drwxr-xr-x.  2 root    root    4.0K Sep 23  2011 preserve
    drwxr-xr-x. 31 root    root    4.0K Jun  3 11:46 run
    drwxrwxr--   2 apache  space   4.0K Apr 29  2019 specialNodeCerts
    drwxr-xr-x.  7 root    root    4.0K Apr 25  2019 spool
    drwxrwxrwt. 13 root    root    316K Jun  3 11:47 tmp
    drwxr-xr-x. 12 root    root    4.0K Apr 25  2019 www
    drwxr-xr-x.  2 root    root    4.0K Sep 23  2011 yp

    So clearing /var/log - not change size of /var. 



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 12.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 18 days ago

    Please check in /var/lib/

    /var/log is different partition 



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 13.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 17 days ago

    Space in lib folder related to database of pgsql. But I don't know what to do with it. 

    How can I decrease/delete anything without causing problems with Junos Space from this DB ? 

    [root@space-0050569a3d7b lib]# du -h --max-depth=1 /var/lib/
    55G     /var/lib/pgsql
    4.0K    /var/lib/games
    12K     /var/lib/unbound
    5.6M    /var/lib/csync2
    28K     /var/lib/nfs
    8.0K    /var/lib/ntp
    4.0K    /var/lib/dav
    16K     /var/lib/samba
    16K     /var/lib/cassandra
    8.0K    /var/lib/secmgt-skyatp-proxy
    4.0K    /var/lib/dnsmasq
    4.0K    /var/lib/mysql-files
    12K     /var/lib/stateless
    4.0K    /var/lib/rsyslog
    44K     /var/lib/libvirt
    6.9M    /var/lib/yum
    184K    /var/lib/alternatives
    4.0K    /var/lib/readahead
    53M     /var/lib/rpm
    28K     /var/lib/polkit-1
    8.0K    /var/lib/authconfig
    4.0K    /var/lib/misc
    8.0K    /var/lib/dbus
    8.0K    /var/lib/plymouth
    4.0K    /var/lib/aide
    4.0K    /var/lib/mysql
    8.0K    /var/lib/elasticsearch
    4.0K    /var/lib/mod_security
    8.0K    /var/lib/udev
    28K     /var/lib/iscsi
    20K     /var/lib/net-snmp
    4.0K    /var/lib/postfix
    4.0K    /var/lib/prelink
    8.0K    /var/lib/redis
    28K     /var/lib/heartbeat
    55G     /var/lib/

    [root@space-0050569a3d7b lib]# du -h --max-depth=1 /var/lib/pgsql/9.4/data/base/
    6.6M    /var/lib/pgsql/9.4/data/base/13004
    6.8M    /var/lib/pgsql/9.4/data/base/1
    54G     /var/lib/pgsql/9.4/data/base/16385
    6.6M    /var/lib/pgsql/9.4/data/base/13009
    4.0K    /var/lib/pgsql/9.4/data/base/pgsql_tmp
    54G     /var/lib/pgsql/9.4/data/base/

    [root@space-0050569a3d7b 16385]# ls -lh | grep G
    total 54G
    -rw------- 1 postgres postgres 1.0G Jun  4 09:03 16570
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16570.1
    -rw------- 1 postgres postgres 1.0G Jun  1 05:33 16570.2
    -rw------- 1 postgres postgres 1.0G Jun  1 05:33 16570.3
    -rw------- 1 postgres postgres 1.0G Jun  1 05:33 16570.4
    -rw------- 1 postgres postgres 1.0G Jun  1 05:33 16570.5
    -rw------- 1 postgres postgres 1.0G Jun  1 05:33 16570.6
    -rw------- 1 postgres postgres 1.0G Jun  1 05:33 16570.7
    -rw------- 1 postgres postgres 1.0G Jun  1 05:33 16570.8
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16576
    -rw------- 1 postgres postgres 1.0G Jun  1 01:53 16576.1
    -rw------- 1 postgres postgres 1.0G Jun  1 01:53 16576.2
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16578
    -rw------- 1 postgres postgres 1.0G Jun  4 09:47 16578.1
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16579
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16580
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16581
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16581.1
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16582
    -rw------- 1 postgres postgres 1.0G Jun  1 02:33 16582.1
    -rw------- 1 postgres postgres 1.0G Jun  1 02:33 16582.2
    -rw------- 1 postgres postgres 1.0G Jun  1 02:33 16582.3
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16583
    -rw------- 1 postgres postgres 1.0G Jun  1 02:55 16583.1
    -rw------- 1 postgres postgres 1.0G Jun  1 02:55 16583.2
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16584
    -rw------- 1 postgres postgres 1.0G Jun  1 03:27 16584.1
    -rw------- 1 postgres postgres 1.0G Jun  1 03:27 16584.2
    -rw------- 1 postgres postgres 1.0G Jun  1 03:27 16584.3
    -rw------- 1 postgres postgres 1.0G Jun  1 03:28 16584.4
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16585
    -rw------- 1 postgres postgres 1.0G Jun  1 04:00 16585.1
    -rw------- 1 postgres postgres 1.0G Jun  1 04:00 16585.2
    -rw------- 1 postgres postgres 1.0G Jun  1 04:00 16585.3
    -rw------- 1 postgres postgres 1.0G Jun  1 04:00 16585.4
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16586
    -rw------- 1 postgres postgres 1.0G Jun  1 04:33 16586.1
    -rw------- 1 postgres postgres 1.0G Jun  1 04:33 16586.2
    -rw------- 1 postgres postgres 1.0G Jun  1 04:33 16586.3
    -rw------- 1 postgres postgres 1.0G Jun  1 04:33 16586.4
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16587
    -rw------- 1 postgres postgres 1.0G Jun  1 05:11 16587.1
    -rw------- 1 postgres postgres 1.0G Jun  1 05:10 16587.2
    -rw------- 1 postgres postgres 1.0G Jun  1 05:11 16587.3
    -rw------- 1 postgres postgres 1.0G Jun  1 05:11 16587.4
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16588
    -rw------- 1 postgres postgres 1.0G Jun  4 09:48 16589



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 14.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 17 days ago

    Pgsql is related to Network Monitoring (opennms) feature of Space…  works on SNMP.

    You can delete it if you do not use it…  you can reinitialize the DB later again.



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 15.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 17 days ago

    How I can delete DB without problem for Junos Space  ?

    I found some information in the internet - connect to server via CLI, then this algorithm.  I should do it by this instruction or it is not relevant to my case ? 

    Maybe there some other way how safety clear all date in this database ?

    sudo su - postgres

    psql -d database_name (???)

    Run the SQL query to identify which tables are occupying the most space.
    SELECT 2 relname as "Table", 3 pg_size_pretty(pg_total_relation_size(relid)) As "Size", 4 pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as "External Size" 5FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;

    TRUNCATE table_name;
    or
    DELETE FROM table_name WHERE condition;

    then
    VACUUM FULL VERBOSE; 
    REINDEX DATABASE your_database_name;

    \q


    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 16.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 17 days ago

    Hi ,

    for now, you can delete it directly to solve UI issue using rm -rf /var/lib/pgsql/9.4/data/base/16385

    You can re-create the complete data directory later using initdb



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 17.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 17 days ago
      |   view attached
    Step 1.
    rm -rf /var/lib/pgsql/9.4/data/base/16385
    Step 2.
    [root@space-0050569a3d7b base]#  du -h --max-depth=1 /var/
    4.0K    /var/games
    998M    /var/lib
    258M    /var/tmp
    4.0K    /var/preserve
    1.7G    /var/cache
    41G     /var/chroot
    4.0K    /var/crash
    8.0K    /var/specialNodeCerts
    248K    /var/run
    4.0K    /var/nis
    4.0K    /var/cvs
    4.0K    /var/yp
    96K     /var/spool
    4.0K    /var/opt
    17M     /var/www
    8.0K    /var/backups
    4.0K    /var/local
    1.7G    /var/opennms
    32K     /var/db
    16K     /var/lost+found
    28K     /var/lock
    8.0K    /var/empty
    4.8G    /var/log
    5.7G    /var/jboss
    4.0K    /var/account
    56G     /var/
    Step 3. 
    Reboot
    Step 4. 
    Check access to web = Same situation.  White page.
    Step 5.
    Collect Error logs from file Server.log (attached)
    Step 6. 
    After 10 mins 
    [root@space-0050569a3d7b /]# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/jmpvgnocf-lvroot
                          288G  3.7G  269G   2% /
    tmpfs                  16G  4.0K   16G   1% /dev/shm
    /dev/sda1              93M   33M   55M  38% /boot
    /dev/mapper/jmpvgnocf-lvtmp
                           48G  310M   46G   1% /tmp
    /dev/mapper/jmpvgnocf-lvvar
                          111G   96G  9.2G  92% /var
    /dev/mapper/jmpvgnocf-lvlog
                           39G  4.9G   32G  14% /var/log
       
    [root@space-0050569a3d7b /]#  du -h --max-depth=1 /var/lib/
    46G     /var/lib/pgsql
    4.0K    /var/lib/games
    12K     /var/lib/unbound
    5.6M    /var/lib/csync2
    28K     /var/lib/nfs
    8.0K    /var/lib/ntp
    4.0K    /var/lib/dav
    16K     /var/lib/samba
    16K     /var/lib/cassandra
    8.0K    /var/lib/secmgt-skyatp-proxy
    4.0K    /var/lib/dnsmasq
    4.0K    /var/lib/mysql-files
    12K     /var/lib/stateless
    4.0K    /var/lib/rsyslog
    44K     /var/lib/libvirt
    6.9M    /var/lib/yum
    184K    /var/lib/alternatives
    4.0K    /var/lib/readahead
    53M     /var/lib/rpm
    28K     /var/lib/polkit-1
    8.0K    /var/lib/authconfig
    4.0K    /var/lib/misc
    8.0K    /var/lib/dbus
    8.0K    /var/lib/plymouth
    4.0K    /var/lib/aide
    4.0K    /var/lib/mysql
    8.0K    /var/lib/elasticsearch
    4.0K    /var/lib/mod_security
    8.0K    /var/lib/udev
    28K     /var/lib/iscsi
    20K     /var/lib/net-snmp
    4.0K    /var/lib/postfix
    4.0K    /var/lib/prelink
    8.0K    /var/lib/redis
    28K     /var/lib/heartbeat
    46G     /var/lib/
    After 20 mins
    Now /var/ full again. 


    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Attachment(s)

    txt
    Server.log2.txt   37 KB 1 version
    Original Message


  • 18.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 16 days ago

    (Open a JTAC case for complete troubleshooting)

    You can try Workaround :

    1. Delete the directory:

    rm -rf rm -rf /var/lib/pgsql/9.4/data/base/

    2. Stop opennms service:

    service jmp-opennms stop

    But you are getting certificate issue so you can below for that: 

    change date to (1 year ago)
    date --set='-1 year'

    Generate new certificate using (KB29266) and upload the new certificate to GUI.

    revert the date change:
    date --set='+1 year'

    2024-06-04 13:30:41,820 ERROR [stderr] (pool-12-thread-1) javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed


    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 19.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 16 days ago
      |   view attached

    We have got a huge progress! Thank you a lot !

    What I did.

    1 service jmp-opennms stop

    2 rm -rf rm -rf /var/lib/pgsql/9.4/data/base/

    3 change date to (1 year ago)  => date --set='-1 year'

    4 service httpd restart

    5  service mysql restart

    6 service jboss restart

    After 5 min I checked web access - it start works. 

    Then I tried to install the certificate. To generate it I had to use addition command because Junos Space don't like key.

    openssl rsa -traditional -in space_self_cert.key -out space_self_cert_nopass-traditional.key

    Then system tell me that certificate is now valid because it created in 2024. I changed date forward date --set='+1 year' and try again.

    But now I got the error: "Validation error Not All nodes are up in the fabric"

    I went to Administration - Fabric and see status "Down".  

    And here I am stuck. Maybe some services are down ? How to understand what is a problem now ? 



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 20.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 15 days ago

    The problem why Node was down = Service opennms nor running. I revert to snapshot and repeat all action.  

    1. service jmp-opennms stop  OK
    2. service jmp-opennms start  OK
    3. service jmp-opennms stop OK
    4. rm -rf rm -rf /var/lib/pgsql/9.4/data/
    5. service postgresql-9.4 initdb  OK
    6. [root@space-0050569a3d7b /]# service jmp-opennms start
      opennms is stopped..
      Starting OpenNMS...
      Postgres already starting, restarting it...
      Stopping postgresql-9.4 service:                           [  OK  ]
      Starting postgresql-9.4 service:                           [  OK  ]
      Starting OpenNMS: OpenNMS was unable to connect to the 'opennms' database configured in opennms-datasources.xml.
                                                                 [FAILED]
    7. [root@space-0050569a3d7b /]# find ./ -name opennms-datasources.xml
      ./var/opennms/etc-pristine/opennms-datasources.xml
      ./opt/opennms/etc/opennms-datasources.xml
    8. cat  ./opt/opennms/etc/opennms-datasources.xml #(end of file)

        <jdbc-data-source name="opennms"
                          database-name="opennms"
                          class-name="org.postgresql.Driver"
                          url="jdbc:postgresql://localhost:5432/opennms"
                          user-name="opennms"
                          password="opennms" />

        <jdbc-data-source name="opennms-admin"
                          database-name="template1"
                          class-name="org.postgresql.Driver"
                          url="jdbc:postgresql://localhost:5432/template1"
                          user-name="postgres"
                          password="postgres" />

    === I stuck again====



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 21.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 15 days ago

    Hi,

    Status is showing down and unknown because server certificate is expired not because of opennms service.

    You should open JTAC case now to fix.



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 22.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 14 days ago

    Unfortunately in this company we already don't have active support contract. 

    But in any case - thank you a lot for your help. It helped me significantly to understand the reason of the problem.

    Just for your information - We have got a snapshot of the server from 2023 year.  Yesterday we revert to this version and we have the same problem. So problem for sure not related to the power outages. Just to certificate expiration.  And after 20-25 mins of work - /var/lib/pgsql/9.4/data/base/ ate all the space for /var. 

    I will continue to search solution how to update certificate in this case. If I found some way to do it - I will write instruction for this case. 



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 23.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

     
    Posted 14 days ago

    If you do not have many devices or configurations then it is better to deploy a new server on the latest release.

    -PL



    ------------------------------
    Pravin Lokhande
    ------------------------------

    Original Message


  • 24.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 14 days ago

    New step =) 

    I generated certificate on different server and before started - changed the this server time to 2020 year. (KB29266)   

    Using commands

    openssl req -x509 -days 5650 -newkey rsa:2048 -keyout space_self_cert.key -out space_self_cert.pem -config self-config.conf -sha256
    openssl rsa -traditional -in space_self_cert.key -out space_self_cert_nopass-traditional.key

    Then change time on Junos space 1 year befor and install new certificate.

    Then I rebooted server. Web page started to open, but my current password for web isn't suit now. 

    I decided to reset super for jboss password, but got error.

    [root@space-0050569a3d7b ~]#   mysql -ujboss -p$(grep mysql.jboss /etc/sysconfig/JunosSpace/pwd | awk -F= '{print $2}') build_db -e "UPDATE USER set password='ok89Nva6qHxytSHsP8AeLg==' where name='super'"
    Warning: Using a password on the command line interface can be insecure.
    ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1) 

    If check certificate now it doesn't tell that certificate expired but return code: 21 in the end of list.

    [root@space-0050569a3d7b ~]#   openssl s_client -connect 10.41.3.182:443 -state -debug

    ....

    Verify return code: 21 (unable to verify the first certificate)



    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message


  • 25.  RE: The Junos Space web page does not load correctly. The server was turned off due to power supply.

    Posted 14 days ago

    Loki - you share some addition instruction. One question about it. There IP = 10.219.50.230 (it is IP of WEB portal or it is IP of interface Junos Server (because I have 10.41.3.181 = Server and 10.41.3.182 = Junos Space WEB Portal). I used instruction with 10.41.3.181.

    Have this picture 

    [root@space-0050569a3d7b ~]# openssl x509 -text -inform PEM -in /etc/pki/tls/certs/localhost.crt | grep After
                Not After : Jun  8 13:58:45 2029 GMT
    [root@space-0050569a3d7b ~]# openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
                Not After : Jun  8 14:04:23 2029 GMT
    [root@space-0050569a3d7b ~]# openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After
                Not After : Jun  8 14:05:26 2029 GMT
    [root@space-0050569a3d7b ~]# openssl s_client -connect 10.41.3.182:443 -state -debug
    ...
    Verify return code: 21 (unable to verify the first certificate)
    and in logs have in the end error related to this IP 
    2024-06-07 14:29:18,672 ERROR [org.hibernate.tool.hbm2ddl.SchemaUpdate] (MSC service thread 1-16) HHH000388: Unsuccessful: create index type_vendor_title_index on SpaceIDPSigVulnerabilityObj (vendorKey)
    2024-06-07 14:29:18,673 ERROR [org.hibernate.tool.hbm2ddl.SchemaUpdate] (MSC service thread 1-16) Specified key was too long; max key length is 767 bytes
    2024-06-07 14:29:25,842 ERROR [net.juniper.jmp.apps.datastorage.attr.SchemaBuilder] (pool-12-thread-1) Cannot find schema: /usr/local/jboss/jmp-tmp/net/juniper/jmp/var/schemas/dmi-space/dmi/junos-qfx/releases/11.1/rpc-reply.xsd
    2024-06-07 14:29:30,776 ERROR [net.juniper.jmp.cmp.schema.schemaMgr.LGSchemaMgr] (pool-12-thread-1) Error while updating looking glass column details due to : null
    2024-06-07 14:29:42,499 ERROR [net.juniper.jmp.cmp.resourceDistributionManager.ejb.ResourceDistributionManagerBean] (pool-12-thread-1) Validate IPv4 address 10.41.3.181
    2024-06-07 14:29:43,304 ERROR [net.juniper.jmp.cmp.resourceDistributionManager.ejb.ResourceDistributionManagerBean] (pool-12-thread-1) After jboss starting, the new node ip is 10.41.3.181
    2024-06-07 14:29:43,304 ERROR [net.juniper.jmp.cmp.resourceDistributionManager.ejb.ResourceDistributionManagerBean] (pool-12-thread-1) After jboss starting, the new node v6 ip is null
    2024-06-07 14:29:43,304 ERROR [net.juniper.jmp.cmp.resourceDistributionManager.ejb.ResourceDistributionManagerBean] (pool-12-thread-1) After jboss starting, the new dev ip is null
    2024-06-07 14:29:43,304 ERROR [net.juniper.jmp.cmp.resourceDistributionManager.ejb.ResourceDistributionManagerBean] (pool-12-thread-1) After jboss starting, the new dev v6 ip is null
    2024-06-07 14:29:44,904 ERROR [net.juniper.jmp.cmp.resourceDistributionManager.ejb.FabricManagerNmaUtil] (pool-12-thread-1) getV6VIPNMA throws exception: java.lang.NullPointerException


    ------------------------------
    Aleksandr Maksimov
    ------------------------------

    Original Message