Cisco NSO on a DevNet Expert Lab Exam #
Cisco NSO plays a key role on DevNet Expert Lab Exam. It’s listed in a DevNet Expert Lab Exam blueprint ⬇
2.10 Create a basic Cisco NSO service package to meet given business
and technical requirements. The service would generate a network
configuration on the target device platforms using the "cisco-ios-cli"
NED and be of type "python-and-template"
2.10.a Create a service template from a provided NSO device configuration
2.10.b Create a basic YANG module for the service containers
(including lists, leaf lists, data types, leaf references,
and single argument "when" and "must" conditions)
2.10.c Create basic actions to verify operational status of the service
2.10.d Monitor service status by reviewing the NCS Python VM log file
Devnet Expert candidates should be prepared to confront an NSO-related tasks. It’s good to have NSO installed on CWS (Candidate Workstation) to immerse into a system, that is almost the same as a real-life lab environment even more.
Cisco NSO in your lab #
Cisco NSO is available for non-production use as a “Free Trial”. You can download it here from Cisco Software Center. You can install it on CWS to lab multiple scenarios.
The problem with a free trial is that you are allowed to download only the latest version of NSO. At the time of writing this blog post, latest version of NSO Free Trial is 6.2, whereas on DevNet Expert Lab Exam, version 5.5 is used instead. This is explicitly pointed out in a
Software List
Cisco Network Services Orchestrator (NSO) local installation 5.5
Cisco NSO IOS CLI NED 6.69
Cisco NSO NX-OS CLI NED 5.21
So far I haven’t discovered major differences between 5.5 and 6.2 release in features used on a Lab Exam.
In this article, I want to show you my struggles with deploying NSO on CWS in a different version than it is supposed to be. It quickly turned out it’s not that easy to deploy it while being limited by CWS a bit, but we are experts, right? Things like this happen to us daily - I treat this as a nice case to troubleshoot.
In a rush? Jump straight to the “TLDR” section below to apply the fixes!"
Installing NSO #
On your CWS - in the /home/expert/nso directory you can find a file called “INSTALL_NSO_README” which contains URL to the “Getting and Installing NSO” documentation which guides you through the installation process.
I followed the steps described there and had my NSO installed.
(main) expert@expert-cws:~/Downloads/nso$ ./nso-6.2.linux.x86_64.installer.bin --local-install ~/nso-local
INFO Using temporary directory /tmp/ncs_installer.4726 to stage NCS installation bundle
INFO Unpacked ncs-6.2 in /home/expert/nso-local
INFO Found and unpacked corresponding DOCUMENTATION_PACKAGE
INFO Found and unpacked corresponding EXAMPLE_PACKAGE
INFO Found and unpacked corresponding JAVA_PACKAGE
INFO Generating default SSH hostkey (this may take some time)
INFO SSH hostkey generated
INFO Generating self-signed certificates for HTTPS
INFO Environment set-up generated in /home/expert/nso-local/ncsrc
INFO NSO installation script finished
INFO Found and unpacked corresponding NETSIM_PACKAGE
INFO NCS installation complete
In the installation directory, I’ve found several NEDs installed by default:
(main) expert@expert-cws:~/nso-local/packages/neds$ ls
a10-acos-cli-3.0 cisco-asa-cli-6.6 cisco-ios-cli-3.8 cisco-iosxr-cli-3.5 dell-ftos-cli-3.0
alu-sr-cli-3.4 cisco-ios-cli-3.0 cisco-iosxr-cli-3.0 cisco-nx-cli-3.0 juniper-junos-nc-3.0
Although I wanted to use different NED versions, so I downloaded them to my disk:
(main) expert@expert-cws:~/Downloads/nso-local$ ls
ncs-6.2-cisco-asa-6.18.7.tar.gz ncs-6.2-cisco-ios-6.103.tar.gz ncs-6.2-cisco-iosxr-7.53.tar.gz ncs-6.2-cisco-nx-5.25.6.tar.gz
then I extracted every single package using tar -xvf <filename>.tar.gz. Once finished - I moved them to my local-install NSO directory to packages/neds directory:
(main) expert@expert-cws:~/nso-local/packages/neds$ ls
a10-acos-cli-3.0 cisco-asa-cli-6.6 cisco-ios-cli-6.103 cisco-iosxr-cli-7.53 dell-ftos-cli-3.0
alu-sr-cli-3.4 cisco-ios-cli-3.0 cisco-iosxr-cli-3.0 cisco-nx-cli-3.0 juniper-junos-nc-3.0
cisco-asa-cli-6.18 cisco-ios-cli-3.8 cisco-iosxr-cli-3.5 cisco-nx-cli-5.25
Now I was ready to start my local instance of NSO, but first I had to source the ncsrc to have access to NSO utilities. The process might be familiar to you if you ever worked with Python venvs.
(main) expert@expert-cws:~/nso-local$ source ncsrc
Once done, I could execute the ncs-setup utility to set up my NSO instance. Along with this command, I provided a path to NEDs that I extracted to the packages/neds directory:
(main) expert@expert-cws:~/nso-local$ ncs-setup \
--package ~/nso-local/packages/neds/cisco-asa-cli-6.18 \
--package ~/nso-local/packages/neds/cisco-ios-cli-6.103 \
--package ~/nso-local/packages/neds/cisco-iosxr-cli-7.53 \
--package ~/nso-local/packages/neds/cisco-nx-cli-5.25 \
--dest ~/nso-instance
(main) expert@expert-cws:~/nso-local$
Starting NSO service and the first issues #
Now it was the time to start ncs service.
(main) expert@expert-cws:~/nso-instance$ ncs
NCS package upgrade failed with reason 'User java class "com.tailf.packages.ned.asa.UpgradeNedSettings" exited with status 1'
Daemon died status=13
Uh-oh! As you can see I received an error from NCS saying that NCS package upgrade has failed. When I had my first attempt with NSO, I checked the status of NCS service and it turned out it didn’t start after getting the error:
(main) expert@expert-cws:~/nso-instance$ ncs --status | grep status
connection refused (status)
I decided to execute ncs command again and this time it worked.
(main) expert@expert-cws:~/nso-instance$ ncs --status | grep status
status: started
Consequences of ignoring error code #
I thought that the error code I got before was not that important and I agreed that my ASA NED might behave not as expected so I executed ncs again and the service started. I was able to log in to the console with no issues:
(main) expert@expert-cws:~/nso-instance$ ncs_cli -C -u admin
admin connected from 127.0.0.1 using console on expert-cws
but… I couldn’t connect to the device being managed by NSO:
admin@ncs(config-device-core-rtr01)# connect
result false
info Device core-rtr01 does not have a Network Element Driver registered
Before I started investigating this I realized I had another issue - I couldn’t log in to GUI. I used default credentials but I was rejected by the system.
Troubleshooting #
First, I started with troubleshooting login issues. I thought that for some reason default credentials are different. A quick look at the docs to see how to change the default admin password - you can just modify the admin username in the aaa section. Sounds like a plan.
I jumped into CLI and then I wanted to see the structure of aaa section in an NSO configuration but… this section was missing.
admin@ncs# show running-config | include aaa
admin@ncs#
I decided to do a test and I created another NCS instance but this time - without passing ASA NED in the ncs-setup command, since NCS had some issues with it because of "com.tailf.packages.ned.asa.UpgradeNedSettings" error.
This time everything worked - I could log in to the devices, the aaa section was there and I could log in to the GUI with no issues.
Output from the instance without ASA NED:
admin@ncs# show running-config | include aaa
aaa authentication users user admin
aaa authentication users user oper
aaa authentication users user private
aaa authentication users user public
aaa admin-accounting enable false
aaa authentication users user admin
aaa authentication groups group aaa-r
aaa authentication groups group admin-r
aaa authentication groups group root-system
<output ommited>
Now I was sure there was something wrong with an ASA NED that caused NSO to be not fully initialized (missing config sections for example).
I went back to the CLI of NSO that didn’t work properly and I decided to take a look at the error I got when attempting to connect to the device. Connection attempt returned info Device core-rtr01 does not have a Network Element Driver registered so I checked the status of NEDs:
admin@ncs# show packages package oper-status
packages package cisco-asa-cli-6.18
oper-status java-uninitialized
packages package cisco-ios-cli-6.103
oper-status java-uninitialized
packages package cisco-iosxr-cli-7.53
oper-status java-uninitialized
packages package cisco-nx-cli-5.25
oper-status java-uninitialized
java-uninitialized - that was suspicious. I checked the status of Java VM:
admin@ncs# show java-vm status
java-vm status not-connected
admin@ncs#
I decided to pay more attention to Java VM. I was looking for an error log and thankfully NSO saved logs to the logs directory. A file called ncs-java-vm.log took my attention.
(main) expert@expert-cws:~/nso-instance/logs$ cat ncs-java-vm.log
Starting java-vm with options:
' -classpath ./state/packages-in-use/1/cisco-asa-cli-6.18/shared-jar/*:./state/packages-in-use/1/cisco-asa-cli-6.18/private-jar/*:/home/expert/nso-local/java/jar/* -Dport=4569 -Djvm.restart.enabled=false -Djvm.restart.errCount=3 -Djvm.restart.duration=60 -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF-8 -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=OFF'
Error: LinkageError occurred while loading main class com.tailf.packages.ned.asa.UpgradeNedSettings
java.lang.UnsupportedClassVersionError: com/tailf/packages/ned/asa/UpgradeNedSettings has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
I like moments like this when I find a clear, obvious reason for what is wrong. In this case, it turned out that CWS has outdated Java runtime and this specific version was not compatible with NSO 6.2 which requires Java Runtime in a version that supports class file in a version of 61.0.
Fixing issues with Java VM #
I started with verifying the current Java version:
(main) expert@expert-cws:~/nso-instance$ java --version
openjdk 11.0.13 2021-10-19
OpenJDK Runtime Environment (build 11.0.13+8-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.13+8-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)
In CWS, Java comes with an openjdk package. That was the package I had to upgrade. But what Java version should I have had to choose? 🤔
I went back to the error log and I saw that version 61.0 refers to class file, not to Java itself. I googled it a bit and I’ve found a really good website that maps Java version to class file versions.
This lines up with my error log:
and what I saw in java --version command output:
openjdk 11.0.13 2021-10-19
Java 11 supports Class file in version 55.0.
Back then I realized that I needed to upgrade to openjdk 17 or above to have support for class file in version 61.0.
I switched the user to root and then started the process of installing openjdk-17
root@expert-cws:/home/expert/nso-instance# apt install openjdk-17-jdk
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless
Suggested packages:
openjdk-17-demo openjdk-17-source visualvm fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei | fonts-wqy-zenhei
The following NEW packages will be installed:
openjdk-17-jdk openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless
0 upgraded, 4 newly installed, 0 to remove and 15 not upgraded.
Need to get 286 MB of archives.
After this operation, 445 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
0% [Connecting to 192.168.5.254 (192.168.5.254)]
Of course - the proxy for APT was set to the internal Cisco server. I had to get rid of that configuration so packages could be downloaded directly from repositories located on the Internet.
To disable the proxy for apt, I navigated to the /etc/apt/apt.conf.d/ directory to see the configuration files. One file took my attention - 90curtin-aptproxy
root@expert-cws:/home/expert/nso-instance# cd /etc/apt/apt.conf.d/
root@expert-cws:/etc/apt/apt.conf.d# ls
01autoremove 01-vendor-ubuntu 15update-stamp 20archive 20dbus 20snapd.conf 50command-not-found 60icons 70debconf 99update-notifier
01autoremove-kernels 10periodic 20apt-esm-hook.conf 20auto-upgrades 20packagekit 50appstream 50unattended-upgrades 60icons-hidpi 90curtin-aptproxy
Once previewed - I understood what was going on:
root@expert-cws:/etc/apt/apt.conf.d# cat 90curtin-aptproxy
Acquire::http::Proxy "http://192.168.5.254:3128";
Acquire::https::Proxy "http://192.168.5.254:3128";
I commented that lines out…
root@expert-cws:/etc/apt/apt.conf.d# cat 90curtin-aptproxy
#Acquire::http::Proxy "http://192.168.5.254:3128";
#Acquire::https::Proxy "http://192.168.5.254:3128";
but this was not enough. I still had to modify the sources.list in the /etc/apt/ directory. Fortunately, Cisco team that was preparing CWS saved the default sources.list file as sources.list.save, so I could just restore it.
I temporarily renamed sources.list to src.temp so I could revert it back one process is done, then in place of the modified sources.list file I placed the original one, which was saved as sources.list.save
root@expert-cws:/etc/apt# mv sources.list src.temp
root@expert-cws:/etc/apt# cp sources.list.save sources.list
After that, I executed apt update command to update a list of packages:
root@expert-cws:/etc/apt# apt update
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Hit:3 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal InRelease
Get:4 https://apt.releases.hashicorp.com focal InRelease [16.4 kB]
Get:5 https://dl.google.com/linux/chrome/deb stable InRelease [1,825 B]
Hit:6 https://download.docker.com/linux/ubuntu focal InRelease
Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Err:4 https://apt.releases.hashicorp.com focal InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY AA16FCBCA621E701
Hit:2 https://packages.microsoft.com/repos/code stable InRelease
Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Err:5 https://dl.google.com/linux/chrome/deb stable InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY E88979FB9B30ACF2
Get:9 http://archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [970 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/main Translation-en [506 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 DEP-11 Metadata [494 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/main DEP-11 48x48 Icons [98.4 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal/main DEP-11 64x64 Icons [163 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal/main DEP-11 64x64@2 Icons [15.8 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 c-n-f Metadata [29.5 kB]
<output ommited>
Success. I went back to openjdk installation process. I installed openjdk-17-jdk package.
root@expert-cws:/etc/apt# apt install openjdk-17-jdk
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless
Suggested packages:
openjdk-17-demo openjdk-17-source visualvm fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei | fonts-wqy-zenhei
The following NEW packages will be installed:
openjdk-17-jdk openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless
0 upgraded, 4 newly installed, 0 to remove and 561 not upgraded.
Need to get 118 MB of archives.
After this operation, 274 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Once the installation was completed, I checked Java version again to confirm the default Java version is now 17:
root@expert-cws:/etc/apt# java --version
openjdk 17.0.10 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-120.04.1)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-120.04.1, mixed mode, sharing)
Bringing Java VM back to life #
Java upgrade went off without a hitch ✅ I created a fresh NSO instance again using ncs-setup utility and then I started ncs service:
(main) expert@expert-cws:~$ ncs-setup --package ~/nso-local/packages/neds/cisco-asa-cli-6.18 \
--package ~/nso-local/packages/neds/cisco-ios-cli-6.103 \
--package ~/nso-local/packages/neds/cisco-iosxr-cli-7.53 \
--package ~/nso-local/packages/neds/cisco-nx-cli-5.25 \
--dest ~/nso-instance-java-17
(main) expert@expert-cws:~$ cd ~/nso-instance-java-17/
(main) expert@expert-cws:~/nso-instance-java-17$ ncs
(main) expert@expert-cws:~/nso-instance-java-17$ ncs --status | grep status
status: started
db=running id=33 priority=1 path=/ncs:devices/device/live-status-protocol/device-type
This time NCS service started on a first shot without any issues. I jumped into cli to see if “aaa” section is on the right place:
(main) expert@expert-cws:~/nso-instance-java-17$ ncs_cli -C -u admin
admin connected from 127.0.0.1 using console on expert-cws
admin@ncs# show running-config | include aaa
aaa authentication users user admin
aaa authentication users user oper
aaa authentication users user private
aaa authentication users user public
rule tailf-aaa-password-change
module-name tailf-aaa
path /aaa/authentication/users/user[name='$USER']/change-password
rule tailf-aaa-user
module-name tailf-aaa
admin@ncs#
Success! ✅ I also verified web GUI access:
What about our NEDs installed and Java VM? 🤔
admin@ncs# show packages package oper-status
PACKAGE
PROGRAM META FILE
CODE JAVA PYTHON BAD NCS PACKAGE PACKAGE CIRCULAR DATA LOAD ERROR
NAME UP ERROR UNINITIALIZED UNINITIALIZED VERSION NAME VERSION DEPENDENCY ERROR ERROR INFO WARNINGS
-----------------------------------------------------------------------------------------------------------------------------------------
cisco-asa-cli-6.18 X - - - - - - - - - - -
cisco-ios-cli-6.103 X - - - - - - - - - - -
cisco-iosxr-cli-7.53 X - - - - - - - - - - -
cisco-nx-cli-5.25 X - - - - - - - - - - -
admin@ncs# show java-vm status
java-vm status running
admin@ncs#
It works! ✅
TLDR #
- Disable apt proxy - comment out “Acquire” lines in
/etc/apt/apt.conf.d/90curtin-aptproxyfile:
root@expert-cws:/etc/apt/apt.conf.d# cat 90curtin-aptproxy
#Acquire::http::Proxy "http://192.168.5.254:3128";
#Acquire::https::Proxy "http://192.168.5.254:3128";
- Roll back the original
sources.list
root@expert-cws:/etc/apt# mv sources.list src.temp
root@expert-cws:/etc/apt# cp sources.list.save sources.list
- Execute
apt update - Install
openjdk-17-jdkpackage
root@expert-cws:/etc/apt# apt install openjdk-17-jdk
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless
Suggested packages:
openjdk-17-demo openjdk-17-source visualvm fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei | fonts-wqy-zenhei
The following NEW packages will be installed:
openjdk-17-jdk openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless
0 upgraded, 4 newly installed, 0 to remove and 561 not upgraded.
Need to get 118 MB of archives.
After this operation, 274 MB of additional disk space will be used.
Do you want to continue? [Y/n]
- Verify if your java is upgraded
root@expert-cws:/etc/apt# java --version
openjdk 17.0.10 2024-01-16
- Create a new instance of NSO using
ncs-setuputility. - Confirm that NEDs and Java VM are fully operational.
Summary #
I felt huge relief once I fixed that. After hours spent on troubleshooting, I finally was able to lab and learn NSO! 🤠 If NSO in version 5.5 would be available for download I assume this will not happen because Java 11.0 would be enough to start Java VM.
My approach is to show you what reality from Automation Engineer’s perspective looks like and what problems I usually deal with daily. This case requires some effort and I hope you enjoy it.
I hope this article will help you solve the issues with NSO deployed on CWS. Thank you for your time, I’m so happy having you here! 🫡
As always - feel free to contact me via email or just follow/invite me on LinkedIn!