router image

Netmiko and TextFSM

By Kirk Byers
2018-06-04

In Netmiko 2.0.0, I added support for direct integration to TextFSM particularly for ntc-templates.

I have talked about TextFSM previously here, but let's review a bit.


What does TextFSM and ntc-templates allow you to do?

Simply stated, it allows you to take unstructured data and convert it to structured data. Or worded differently, it takes a block of text and converts it to lists and dictionaries (or some combination thereof).

And even better, if someone has already written the TextFSM template for you, then you might be able to just get structured data directly. See the ntc-templates index file which maps platform and show command to TextFSM templates.

So to recap, TextFSM is a complex, regular expression state-machine that converts from blocks of string to structured data (lists and dictionaries) and in the context of ntc-templates various people have already done the parsing for you (for some set of platforms and show commands).

Now this is all fine, but how do we use TextFSM and ntc-templates with Netmiko. First we have to get our environment setup correctly. Let's look at this process.



Setting Up the Environment

Starting with a clean Python3 virtual environment:

$ virtualenv-3.6 -p /usr/bin/python36 netmiko_tfsm
Running virtualenv with interpreter /usr/bin/python36
Using base prefix '/usr'
New python executable in
/home/kbyers/VENV/netmiko_tfsm/bin/python36
Also creating executable in
/home/kbyers/VENV/netmiko_tfsm/bin/python
Installing setuptools, pip, wheel...done.

$ deactivate
$ source netmiko_tfsm/bin/activate
$ pip list
Package Version
---------- -------
pip 10.0.1
setuptools 39.2.0
wheel 0.31.1  

Then install the latest version of Netmiko.

$ pip install netmiko

At this point, you should be able to successfully import the Netmiko library.

$ python
Python 3.6.5 (default, Apr 26 2018, 00:14:31)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more
information.
>>> import netmiko
>>> netmiko.__version__
'2.1.1'  


Now you need to install ntc-templates using a 'git clone' action. The easiest way is to just install this into your home directory.

$ cd ~
$ git clone https://github.com/networktocode/ntc-templates?__s=XXXXXXXX
Cloning into 'ntc-templates'...
remote: Counting objects: 3345, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 3345 (delta 24), reused 28 (delta 12), pack-reused
3297
Receiving objects: 100% (3345/3345), 713.69 KiB | 9.91 MiB/s,
done.
Resolving deltas: 100% (1772/1772), done.  

At this point, you should have the ntc-templates index in the following directory:

$ cd ~
$ ls ~/ntc-templates/templates/index  

I mentioned this earlier, but the index file is just a mapping between platform, command, and the corresponding TextFSM template to use. This includes possible abbreviated versions of the command (for example, 'sh ip int br' and 'show ip interface brief').


Now, Netmiko has been configured to automatically look in ~/ntc-template/templates/index for the ntc-templates index file. Alternatively, you can explicitly tell Netmiko where to look for the TextFSM template directory by setting the following environment variable (note, there must be an index file in this directory):

export NET_TEXTFSM=/path/to/ntc-templates/templates/  


How to use Netmiko with TextFSM?

Now that both Netmiko and ntc-templates have been installed, how do we tell Netmiko to use the TextFSM parsing capability. First, look at the ntc-templates index file and make sure your show command has a corresponding TextFSM template for your platform.

After you have done that, then you add the 'use_textfsm=True' argument to either the send_command() method or to the send_command_timing() method. Here are a couple of examples:

$ ipython
Python 3.6.5 (default, Apr 26 2018, 00:14:31)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for
help.

In [1]: from netmiko import Netmiko

In [2]: from getpass import getpass

In [3]: password = getpass()
Password:

In [4]: net_connect = Netmiko(host='device.domain.com', username='admin', password=password, device_type='cisco_ios')

In [5]: net_connect.find_prompt()
Out[5]: 'pynet-rtr1#'

In [6]: net_connect.send_command("show ip int brief", use_textfsm=True)
Out[6]:
[{'intf': 'FastEthernet0',
  'ipaddr': 'unassigned',
  'status': 'down',
  'proto': 'down'},
 {'intf': 'FastEthernet1',
  'ipaddr': 'unassigned',
  'status': 'down',
  'proto': 'down'},
 {'intf': 'FastEthernet2',
  'ipaddr': 'unassigned',
  'status': 'down',
  'proto': 'down'},
 {'intf': 'FastEthernet3',
  'ipaddr': 'unassigned',
  'status': 'down',
  'proto': 'down'},
 {'intf': 'FastEthernet4',
  'ipaddr': '10.220.88.20',
  'status': 'up',
  'proto': 'up'},
 {'intf': 'Vlan1', 'ipaddr': 'unassigned', 'status': 'down', 'proto': 'down'}]   

Wow, that is structured data being returned!



Now what about the failure case? In other words, when the template lookup fails or when you do not have ntc-templates installed properly, in that situation, you will just get back unstructured data (i.e. a big string):

In [8]: output = net_connect.send_command("show arp", use_textfsm=True)

In [9]: print(output)
'Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.220.88.1            34   0062.ec29.70fe  ARPA   FastEthernet4
Internet  10.220.88.20            -   c89c.1dea.0eb6  ARPA   FastEthernet4
Internet  10.220.88.21          183   1c6a.7aaf.576c  ARPA   FastEthernet4
Internet  10.220.88.28          162   5254.aba8.9aea  ARPA   FastEthernet4
Internet  10.220.88.29          166   5254.abbe.5b7b  ARPA   FastEthernet4
Internet  10.220.88.30          163   5254.ab71.e119  ARPA   FastEthernet4
Internet  10.220.88.32          121   5254.abc7.26aa  ARPA   FastEthernet4
Internet  10.220.88.33           94   5254.ab3a.8d26  ARPA   FastEthernet4
Internet  10.220.88.35          130   5254.abfb.af12  ARPA   FastEthernet4
Internet  10.220.88.37           60   0001.00ff.0001  ARPA   FastEthernet4
Internet  10.220.88.38           46   0002.00ff.0001  ARPA   FastEthernet4
Internet  10.220.88.39           13   6464.9be8.08c8  ARPA   FastEthernet4'
   

Note, 'show ip arp' is supported in ntc-templates, but 'show arp' isn't (you can see this by looking at the ntc-templates index file). Also, note, I added string quotes to the output above to make the formatting in the browser cleaner (output is in fact a string, however).



How about one more working example:

In [14]: net_connect.send_command("show version", use_textfsm=True)

Out[14]:
[{'version': '15.4(2)T1',
  'rommon': 'System',
  'hostname': 'pynet-rtr1',
  'uptime': '6 days, 4 hours, 59 minutes',
  'running_image': 'c880data-universalk9-mz.154-2.T1.bin',
  'hardware': ['881'],
  'serial': ['FTX0000001X'],
  'config_register': '0x2102'}]   


If you want to learn more about network automation, Python, and Ansible—then join my email-list. I also periodically run a free Python for Network Engineers email course which you can sign-up for here.

router image

Kirk Byers

router image

Netmiko and TextFSM

By Kirk Byers
2018-06-04

In Netmiko 2.0.0, I added support for direct integration to TextFSM particularly for ntc-templates.

I have talked about TextFSM previously here, but let's review a bit.


What does TextFSM and ntc-templates allow you to do?

Simply stated, it allows you to take unstructured data and convert it to structured data. Or worded differently, it takes a block of text and converts it to lists and dictionaries (or some combination thereof).

And even better, if someone has already written the TextFSM template for you, then you might be able to just get structured data directly. See the ntc-templates index file which maps platform and show command to TextFSM templates.

So to recap, TextFSM is a complex, regular expression state-machine that converts from blocks of string to structured data (lists and dictionaries) and in the context of ntc-templates various people have already done the parsing for you (for some set of platforms and show commands).

Now this is all fine, but how do we use TextFSM and ntc-templates with Netmiko. First we have to get our environment setup correctly. Let's look at this process.



Setting Up the Environment

Starting with a clean Python3 virtual environment:

$ virtualenv-3.6 -p /usr/bin/python36 netmiko_tfsm
Running virtualenv with interpreter /usr/bin/python36
Using base prefix '/usr'
New python executable in
/home/kbyers/VENV/netmiko_tfsm/bin/python36
Also creating executable in
/home/kbyers/VENV/netmiko_tfsm/bin/python
Installing setuptools, pip, wheel...done.

$ deactivate
$ source netmiko_tfsm/bin/activate
$ pip list
Package Version
---------- -------
pip 10.0.1
setuptools 39.2.0
wheel 0.31.1  

Then install the latest version of Netmiko.

$ pip install netmiko

At this point, you should be able to successfully import the Netmiko library.

$ python
Python 3.6.5 (default, Apr 26 2018, 00:14:31)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more
information.
>>> import netmiko
>>> netmiko.__version__
'2.1.1'  


Now you need to install ntc-templates using a 'git clone' action. The easiest way is to just install this into your home directory.

$ cd ~
$ git clone https://github.com/networktocode/ntc-templates?__s=XXXXXXXX
Cloning into 'ntc-templates'...
remote: Counting objects: 3345, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 3345 (delta 24), reused 28 (delta 12), pack-reused
3297
Receiving objects: 100% (3345/3345), 713.69 KiB | 9.91 MiB/s,
done.
Resolving deltas: 100% (1772/1772), done.  

At this point, you should have the ntc-templates index in the following directory:

$ cd ~
$ ls ~/ntc-templates/templates/index  

I mentioned this earlier, but the index file is just a mapping between platform, command, and the corresponding TextFSM template to use. This includes possible abbreviated versions of the command (for example, 'sh ip int br' and 'show ip interface brief').


Now, Netmiko has been configured to automatically look in ~/ntc-template/templates/index for the ntc-templates index file. Alternatively, you can explicitly tell Netmiko where to look for the TextFSM template directory by setting the following environment variable (note, there must be an index file in this directory):

export NET_TEXTFSM=/path/to/ntc-templates/templates/  


How to use Netmiko with TextFSM?

Now that both Netmiko and ntc-templates have been installed, how do we tell Netmiko to use the TextFSM parsing capability. First, look at the ntc-templates index file and make sure your show command has a corresponding TextFSM template for your platform.

After you have done that, then you add the 'use_textfsm=True' argument to either the send_command() method or to the send_command_timing() method. Here are a couple of examples:

$ ipython
Python 3.6.5 (default, Apr 26 2018, 00:14:31)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for
help.

In [1]: from netmiko import Netmiko

In [2]: from getpass import getpass

In [3]: password = getpass()
Password:

In [4]: net_connect = Netmiko(host='device.domain.com', username='admin', password=password, device_type='cisco_ios')

In [5]: net_connect.find_prompt()
Out[5]: 'pynet-rtr1#'

In [6]: net_connect.send_command("show ip int brief", use_textfsm=True)
Out[6]:
[{'intf': 'FastEthernet0',
  'ipaddr': 'unassigned',
  'status': 'down',
  'proto': 'down'},
 {'intf': 'FastEthernet1',
  'ipaddr': 'unassigned',
  'status': 'down',
  'proto': 'down'},
 {'intf': 'FastEthernet2',
  'ipaddr': 'unassigned',
  'status': 'down',
  'proto': 'down'},
 {'intf': 'FastEthernet3',
  'ipaddr': 'unassigned',
  'status': 'down',
  'proto': 'down'},
 {'intf': 'FastEthernet4',
  'ipaddr': '10.220.88.20',
  'status': 'up',
  'proto': 'up'},
 {'intf': 'Vlan1', 'ipaddr': 'unassigned', 'status': 'down', 'proto': 'down'}]   

Wow, that is structured data being returned!



Now what about the failure case? In other words, when the template lookup fails or when you do not have ntc-templates installed properly, in that situation, you will just get back unstructured data (i.e. a big string):

In [8]: output = net_connect.send_command("show arp", use_textfsm=True)

In [9]: print(output)
'Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.220.88.1            34   0062.ec29.70fe  ARPA   FastEthernet4
Internet  10.220.88.20            -   c89c.1dea.0eb6  ARPA   FastEthernet4
Internet  10.220.88.21          183   1c6a.7aaf.576c  ARPA   FastEthernet4
Internet  10.220.88.28          162   5254.aba8.9aea  ARPA   FastEthernet4
Internet  10.220.88.29          166   5254.abbe.5b7b  ARPA   FastEthernet4
Internet  10.220.88.30          163   5254.ab71.e119  ARPA   FastEthernet4
Internet  10.220.88.32          121   5254.abc7.26aa  ARPA   FastEthernet4
Internet  10.220.88.33           94   5254.ab3a.8d26  ARPA   FastEthernet4
Internet  10.220.88.35          130   5254.abfb.af12  ARPA   FastEthernet4
Internet  10.220.88.37           60   0001.00ff.0001  ARPA   FastEthernet4
Internet  10.220.88.38           46   0002.00ff.0001  ARPA   FastEthernet4
Internet  10.220.88.39           13   6464.9be8.08c8  ARPA   FastEthernet4'
   

Note, 'show ip arp' is supported in ntc-templates, but 'show arp' isn't (you can see this by looking at the ntc-templates index file). Also, note, I added string quotes to the output above to make the formatting in the browser cleaner (output is in fact a string, however).



How about one more working example:

In [14]: net_connect.send_command("show version", use_textfsm=True)

Out[14]:
[{'version': '15.4(2)T1',
  'rommon': 'System',
  'hostname': 'pynet-rtr1',
  'uptime': '6 days, 4 hours, 59 minutes',
  'running_image': 'c880data-universalk9-mz.154-2.T1.bin',
  'hardware': ['881'],
  'serial': ['FTX0000001X'],
  'config_register': '0x2102'}]   


If you want to learn more about network automation, Python, and Ansible—then join my email-list. I also periodically run a free Python for Network Engineers email course which you can sign-up for here.

router image

Kirk Byers