We're newcomers in the Pulse Secure/Juniper Community and technology as well, seeking assistance for a resolution of a problem summarized in the subject of this post.
Our desktops/laptops are running Win10 ver 1803 (OS Build 17134.590).
The gateways are:
Model: srx300Junos: 15.1X49-D70.3JUNOS Software Release [15.1X49-D70.3]
We own two SRX300 gateways setup to connect 2 business locations. PuTTy ssh can connect to both gateways w/o a problem even if the problem explained below is active.
The Pulse secure VPN client installed on the Win10 machines is v. 5.2.11 (1195). We didn't buy Pulse Secure VPN client. We downloaded a copy of it from Pulse Secure's site and we use it with a license that comes by default with the purchase of an SRX 300.
When we restart the SRX gateway error 1453 goes away but it comes back about 12-15 hrs later.
The body of network connection error 1453 reads as follows:
Network errors can be caused by temporary conditions such as an invalid URL , a server not available, and so on. Please try the operation again. Restart your system and try the operation again. If the problem persists, contact your network administrator.
We select Firewall(SRX) in the Type of Connection on Pulse and we're very sure we are using the correct Server URL.
To alleviate error 1453 we restart the gateway of the location the VPN client is connecting to, an hr or so before we open for business and that keeps our remote users' connectivity going for half a day, sometimes more than a day but on avg it lasts 12-15 hrs as mentioned above.
There's plenty of disk apce on the appliance. That's how, at least, we interpret the results showing below:
user@location1> show system storage detailFilesystem 1024-blocks Used Avail Capacity Mounted on/dev/da0s1a 2528708 251256 2075156 11% /devfs 1 1 0 100% /dev/dev/md0 20012 11820 6592 64% /junos/cf/packages 2528708 251256 2075156 11% /junos/cf/packagesdevfs 1 1 0 100% /junos/cf/dev/dev/md1 808018 808018 0 100% /junos/cf 20012 11820 6592 64% /junos/cfdevfs 1 1 0 100% /junos/dev//cf/packages 2528708 251256 2075156 11% /junos/cf/packages1procfs 4 4 0 100% /proc/dev/bo0s3e 189552 80 174308 0% /config/dev/bo0s3f 2218426 114380 1926572 6% /cf/var/dev/md2 687956 20384 612536 3% /mfs/cf/var/jail 2218426 114380 1926572 6% /jail/var/cf/var/jails/rest-api 2218426 114380 1926572 6% /web-api/var/cf/var/log 2218426 114380 1926572 6% /jail/var/logdevfs 1 1 0 100% /jail/dev/dev/md3 1884 292 1442 17% /jail/mfs
We tried troubleshooting the problem at hand with Pulse Secure official tech support but the moment we talk about VPN out to a Juniper hardware product they immediately point us to Juniper tech support, hence reaching out to Juniper Community, just in case someone else has experienced something similar and has an advise/recommendation how to resolve it.
Any help will be much appreciated
Can you try the same connection but form a PC that is running Windows 7? or Windows 10 but with a version prior 1803?
Some imcopatibility problems between Pulse software and Windows 10 were introduced after windows version 1803. This is one exmaple of it:
Also take a packet capture with Wireshark on your PC when the problem is happening, I would like to confirm if the PC is sending packets to the SRX when the problem happens and if the SRX is responding.
Thanks for replying.
I forgot to mention that one of our remote users is using Win7 SP1 and she experiences the same connection error when the other one who uses Win10 Build 1803 can not connect.
Per your advise when the error re-appears
1. I'll test a Win10 machine with Build < 1803
2. capture packets with Wireshark
When the connection error came back I initiated Pulse Secure and while kept clicking 'Retry' to create more VPN traffic I started collecting packets with Wireshark. After a few secs I stopped and filtered for the destimation IP. That's what shows in the jpg attached.
As far as I know black packets are problematic ones so I understand if the jpg is likely not enough to offer further advice what to troubleshoot next. If that's the case how can I share the pcapng file securely ? I'm concerned sharing it in the open b/c of the destination public IP address.
I tried capturing packets again this morning, only this time I filtered all traffic captured for packets that have the public IP as source
or destination. Example of the syntax w/ a non-routable IP follows below.
ip.src== 192.168.0.1 && ip.dst == 192.168.0.5
All VPN traffic in this collection and on my previous reply is going through my Wless interface.
The pattern of packets between this attempt and the one before seems the same. Anything I need to change so I can provide more info to troubleshoot the connection error ?
correction on filter on previous reply.
the filter I applied was
ip.src== Public_IP_Address || ip.dst == Public_IP_Address
where Public_IP_Address is the same on both sides of the OR
On both packet captures we can see that the PC is sending SYN messages in order to create a TCP connection with the SRX, however it is not receiving any replies and start retransmiting the SYN messages (black packets). We need to find out if the SRX is indeed not responding or if any device in between is dropping those packets at the time of the issue.
1. We can tell that the SRX's configuration is fine because the PC can connect from time to time.
2. Create a filter on the SRX with a counter that will increase if it receives the packets from the PC on its external interface. This way we can monitor the counter increases during the outages:
set firewall family inet filter FILTER term COUNTER from source-address [Public_IP_of_PC]
set firewall family inet filter FILTER term COUNTER from destination-port 443
set firewall family inet filter FILTER term COUNTER then count DynVPN
set firewall family inet filter FILTER term COUNTER then accept
set firewall family inet filter FILTER term ALLOW-ELSE then accept
set interfaces [SRX_External_Interface] unit 0 family inet filter input FILTER
Review the counter with the following operational command: > show firewall
3. You could also try taking a pcap on the external interface of the SRX during the time of the issue:
4. Confirm if during the time of the issue, hosts with different public IP addresses (connecting from different locations) experience the same issue. This will help us to isolate a problem with the SRX or a device sitting in front of the SRX.
Thanks for your recommendations @lpaniagua
In the end the problem I had was internal.
Juniper's JTAC team investigated the SRX300 Gateway, where Pulse Secure VPN client suppose to connect, while the VPN connectivity was failing and found out that it was caused by an over-utilization of its Routing Engine.
Next, we will show the Juniper commands the JTAC engineer ran on the SRX in config mode
user@SRXGateway# run show chassis routing-engineRouting Engine status:Temperature 45 degrees C / 113 degrees FCPU temperature 59 degrees C / 138 degrees FTotal memory 4096 MB Max 983 MB used ( 24 percent)Control plane memory 2624 MB Max 656 MB used ( 25 percent)Data plane memory 1472 MB Max 309 MB used ( 21 percent)5 sec CPU utilization:User 50 percentBackground 0 percentKernel 45 percentInterrupt 0 percentIdle 5 percent <---- idle was down to 0% when we initally executed the command at the time the VPN client was failing to connectLast reboot reason 0x200:normal shutdownLoad averages: 1 minute 5 minute 15 minute
user@SRXGateway# run show system processes extensive | except 0.00last pid: 49064; load averages: 2.33, 1.67, 1.38 up 3+07:22:08 10:40:48160 processes: 16 running, 130 sleeping, 2 zombie, 12 waiting
Mem: 304M Active, 144M Inact, 1574M Wired, 385M Cache, 112M Buf, 1572M FreeSwap:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND........................
.......................49023 nobody 3 84 0 11600K 5032K ucondt 0 0:01 1.46% httpd <---- was in the low 90% when the VPN client could not connect
Finally he executed
run show security flow session interface st0.21
and that returned a list of pair of communicating IPs, their Session IDs and the Policy Names that allow them.
The overwhelming majority of the records were indicating 3 specific IP addresses trying to aggressively connect to the SRX itself. All IPs were assigned on Hardware in the internal network and there was no config to get accessed externally, via DDNS for example.
Two of those IPs were given to IP power switches [to power up hardware when power from the grid restores after an outage] and the 3rd one on a server.
The IP power switches by design ping 5 different external preset targets every few seconds; if those become unreacheable then they would power-cycle one or both of their AC power outlets according to the user's configuration.
Unexpectedly, the SRX300 logs showed the traffic from the IP power switches wasn't icmp, as expected, but plain http, so the fact it was directed to the SRX raised suspicions of having been hacked, so I removed them from the network.
The server among other SW had 2 very resources intensive network mngt SW [SolarWinds and PRTG Network Monitor] that were hitting the Gateway to extract descriptive network info and stats for reporting purposes.
Both pieces of SW were completely removed [down to the Windows registry level] from the server and the machine was deeply scanned with a couple of AVs to assess/ensure its virus/spyware clean status.
The Juniper engineer also disabled http mngt access on the SRX
[edit system services web-management] <--- config setting he changed
and we agreed not to restart the SRX again but monitor Pulse connectivity daily. So far it has been 6 days and we haven't experienced any Pulse secure VPN outage hoping it'll stay that way.
However, Juniper highly recommends NOT to use Pulse Secure as a VPN client accessing their gateways, especially from Win10 machines [albeit, from personal experience Pulse Secure still works from Win7 and it's pretty stable and reliable].
Instead they propose to use NCP. Below is a KB article about how to set it up on the SRX and a PDF with NCP set up instructions for the Client:
I hope the above info is helpful providing enough insight to help others in their troubleshooting efforts.
For those users who will make the decision to purchase an NCP client here is the generic URL of their site: https://www.ncp-e.com/en/exclusive-remote-access-solution/vpn-client/#c12977
In our case we don't need a volume license so, in order to obtain a few of those one will need what it's called an "NCP Exclusive Entry Client for Windows" as it shows at
How-To-Buy site of NCP Exclusive Entry Client for Windows, MacOS and Android devices with lots of FAQs
With your permission we'll reply back after we run the installation of an NCP client so others can benefit from our findings. Please bear with us as it'll be our first time to set it up.
As you have noticed the pulse client is no longer supported and is having issues connecting on the most recent windows system patching.
So for official and tested support you would need to purchase the NCP licenses for the clients.
A free open source alternative you could test, but also would not be officially supported would be Shrew Soft.
Thanks for your input @spuluka. I wasn't aware of Shrew Soft.
With lots of help from Juniper's JTAC team we managed to configure NCP client and make it talk VPN to SRX300.
Here's an excerpt of the SRX config with generic values specifically for the NCP client-Gateway VPN communication
ALL THE CREDIT FOR WRITING THE EXCERPT BELOW GOES TO THE JTAC TEAM and not me.
set security ike proposal NCP-PROP authentication-method pre-shared-keysset security ike proposal NCP-PROP dh-group group5set security ike proposal NCP-PROP authentication-algorithm sha1set security ike proposal NCP-PROP encryption-algorithm aes-128-cbcset security ike proposal NCP-PROP lifetime-seconds 86400
set security ike policy NCP-POL mode aggressiveset security ike policy NCP-POL proposals NCP-PROPset security ike policy NCP-POL pre-shared-key ascii-text juniper123
set security ike gateway NCP-GW ike-policy NCP-POLset security ike gateway NCP-GW dynamic user-at-hostname "email@example.com"set security ike gateway NCP-GW dynamic connections-limit 2set security ike gateway NCP-GW dynamic ike-user-type shared-ike-idset security ike gateway NCP-GW external-interface ge-0/0/0.0set security ike gateway NCP-GW aaa access-profile ncp-vpn-profileset security ike gateway NCP-GW version v1-only
set security ipsec proposal NCP_IPSEC_PRO protocol espset security ipsec proposal NCP_IPSEC_PRO authentication-algorithm hmac-sha1-96set security ipsec proposal NCP_IPSEC_PRO encryption-algorithm aes-128-cbcset security ipsec proposal NCP_IPSEC_PRO lifetime-seconds 28800
set security ipsec policy NCP_IPSEC_POL proposals NCP_IPSEC_PROset security ipsec policy NCP_IPSEC_POL perfect-forward-secrecy keys group5
set security ipsec vpn NCP-IPSEC bind-interface st0.0set security ipsec vpn NCP-IPSEC ike gateway NCP-GWset security ipsec vpn NCP-IPSEC ike ipsec-policy NCP_IPSEC_POLset security ipsec vpn NCP-IPSEC traffic-selector TS1 local-ip 0.0.0.0/0set security ipsec vpn NCP-IPSEC traffic-selector TS1 remote-ip 0.0.0.0/0
set security zones security-zone trust interfaces st0.0 host-inbound-traffic system-services allset security zones security-zone trust interfaces st0.0 host-inbound-traffic protocols allset interfaces st0 unit 0 family inet
set access profile ncp-vpn-profile authentication-order passwordset access profile ncp-vpn-profile client test firewall-user password test123set access profile ncp-vpn-profile address-assignment pool NCP-pool
set access address-assignment pool NCP-pool family inet network 10.1.1.0/24set access address-assignment pool NCP-pool family inet xauth-attributes primary-dns 184.108.40.206/32set access firewall-authentication web-authentication default-profile ncp-vpn-profile
Also, do not forget to check the security policies, and confirm you may need a policy from Tunnel to Internal resources. Ensure that traffic started by checking the encrypted packets flow is increasing executing
# run show security ipsec sa
in order to get the TUNNEL_ID and then run
# run show security ipsec statistics index TUNNEL_ID
to confirm the encypted packets traffic is increasing
It is our experience that the NCP VPN client is VERY robust and connects really fast.
Lastly, if your Juniper Gateways are SRX300s please make sure you buy the Exclusive Entry client for Windows from https://www.ncp-e.com/en/exclusive-remote-access-solution/vpn-client/#c12977
I hope this helps someone else in a similar technical deadlock.
Please feel free to close this thread.