We have all suffered through the wait time of committing configurations. It doesn’t matter which OS or hardware platform, we all despise waiting for five minutes or MORE only to end up receiving some sort of error. I believe Einstein’s theory of relativity covers how commit times are perceived longer during those midnight maintenance windows in production… or I could be misinterpreting time dilation.
The good news on this topic is that Juniper has been continuously working on ways to improve the commit times of configurations. The catch is that you will need to upgrade to get these features as they have been trickling into the code for the previous five years.
IMAGE 1: Commit and Configuration Visual
The commit above may be done via CLI or NETCONF. The first step within the commit, the commit check process, invokes all underlaying infrastructure processes to validate the configuration. Each process has the power to reject the configuration. Once all processes pass the "check", the Junos OS continues to process the candidate configuration into the active configuration. The commit check is the most time consuming part of the commit and is the focus of this article. If one wants to bypass the commit check entirely, please learn more about JET and fast programmatic configuration.
COMMIT OPTIMIZATION 1: FAST SYNCHRONIZE
Command: set system commit fast-synchronize
Release: Junos OS 12.2
Summary: This command allows commits to run in parallel on the master and backup routing engine. Recommended use when the REs have almost identical configurations.
COMMIT OPTIMIZATION 2: GROUPS INHERITANCE
Command: set system commit persist-groups-inheritance
Release: Junos OS 13.2
Summary: Full inheritance paths of config groups will be built into the internal database so commit times are decreased when wildcards are used in config groups.
COMMIT OPTIMIZATION 3: DAEMON VALIDATION WITH DELTAS
Default behavior. No command.
Release: Junos OS 13.3
Summary: MGD (Management daemon) passes configurations to the various other daemons within the Junos OS and they individually perform validations on the delta configurations before reporting back to MGD.
COMMIT OPTIMIZATION 4: PRIVATE COMMIT
Default behavior. Command to turn off: set system commit private no-fast-diff
Summary: Private commit was optimized by including the use of binary databases. Now a patch is generated and applied to the shared database at commit.
COMMIT OPTIMIZATION 5: juniper.conf TEXT FILE CREATION
Command: set system commit delta-export
Release: Junos OS 14.2
Summary: Only the delta in the candidate configuration, rather than the entire candidate configuration, are exported.
COMMIT OPTIMIZATION 6: SYNC BOTH ROUTING ENGINES
Default behavior. Command to turn off: set system commit no-delta-synchronize
Summary: For both commit synchronize and commit fast-synchronize, the master routing engine generates only the delta to synchronize across both REs.
COMMIT OPTIMIZATION 7: CONSTRAINT VALIDATION
Default behavior. Command to turn off: set system commit no-delta-constraints
Summary: Related to commit optimization 2, groups inheritance. Once persist-groups is configured in 14.2 and forward, the check is performed on the delta which have constraints instead of the entire configuration.
COMMIT OPTIMIZATION 8: DELTA OF GROUP INHERITANCE
Default behavior. Command to turn off: set system commit no-delta-build-persist-groups
Summary: Also related to commit optimization 2, groups inheritance. At commit time, only the changed objects would need to be updated in the database.
COMMIT OPTIMIZATION 9: MULTI-PROCESSING DAEMON VALIDATION
Default behavior. Command to turn off: set system commit no-parallel-daemon-validation
Release: Junos OS 15.1
Summary: If one runs an image with FreeBSD 10.X, then SMP (symmetric multi-processing) allows the leveraging of multiple cores to validate sections of the configuration in parallel.
COMMIT OPTIMIZATION 10: FILE PROPAGATION
Release: Junos OS 16.1
Summary: Junos performs a FFP (foreign file propagation) on the configuration delta as opposed to the entire configuration.
The most troublesome configurations with respect to commit times are BNG configurations. The chart below takes a sample configuration and shows the difference our Juniper test engineers saw.
IMAGE 2: Time Improvements with a BNG Configuration
Beyond being annoyed by the wait, commit times are increasingly important as we move forward with automation and event-driven networks. We will continue to find ways to improve the Junos OS and commit times are just one example of what we are doing here at Juniper.
WARNING: Virtual devices lag behind our hardware devices in these "commit" improvements. However, we are working on that, as well.
how do we verify and make sure delta-export and persist-groups-inheritance are working correctly. Is there a way to make sure apart from log in "commit | dispay detail" stating "using delta export to export juniper.conf" ?
For us w the srx550M (4G model), we have the ability to pull a core from the vPFE and dedicate it to "control plane" ( really ppmd and/or bfdd - BGP runs in the other control plane core) and in the future maybe LACP and some other bits. While this is working for the control plane flapping, the price was losing 1 of the 5 cores bundled together for the vPFE so we've lost a fair bit of forwarding performance.
We need 4 bfd sessions (3x500ms) and i was really hoping for something like they made available for the smaller (less core SRX boxes), but on the 550, we only got the pull a core from the vPFE option. We're testing the new code now (550m got the improvement in d110) to verify the NAT/PAT tput at our pkt sizes (market data). We're hoping we can achieve the forwarding performance we need (wo the core) so we don't have to impliment rules about when to deploy the 2G and 4G boxes. Below are the two options available for the SRX branch line.
I'm still working to see if we can get both of the options above for the 550M boxes we're using, which hopefully will get us similar tput without any control plane instability.
Hey @Duaneo did you see any imporvements to your SRX after running D100 or later?
Thanks for sharing it !!
This is awesome, Jessica!
Some enhancements related to BFD are added in next SRX release (15.1X49-D100) and will fix this behavior. This release will be availble end of this month.
Fast/er commit times are nice, but we're seeing control plane flapping (BFD) during commits now on 15.1X code (4G SRX) that we don't see on our 12.1 or 12.3x (2G SRX) boxes.. ;-(
Hey @jgarrison this is a great one-stop shop for info around the commit times, and breaking down the various functions. You are not wrong, the time when performing a commit is at least doubled when working in a maintenance window, or when you have been called out 😉