TextFSM and Structured Data

Author: Kirk Byers

Date: 2015-10-22

Let's face it string processing is tedious.

Now in the legacy networking world where we don't have APIs, we are largely stuck screen scraping. We are stuck programmatically connecting to a human-interface and retrieving strings from it. And when we obtain these strings—we don't have dictionaries or lists. We don't have JSON or XML—we have big, ugly strings.

In order to extract the information that you want, you must process the strings: a bit of regular expressions here, a dash of string splitting there, and then cram it all into a data structure. Each conversion likely being custom to the specific 'show' command output.

What's worse is that each engineer will likely repeat this custom string parsing. Sharing of code is difficult here.

It would be nice if there was an easier way. It would be nice if we could obtain structured data from legacy devices. It would be nice if engineers could somehow share code that simplified this string parsing.

TextFSM can help.

Jason Edelman wrote about TextFSM in February of 2015.

So what is TextFSM?

TextFSM is a Python library created by Google. It requires that you define a template consisting of variables and rules. You then process strings against this template and from this you can obtain structured data (voila). Let me show you an example.

In the file 'show_ip_bgp.txt', I have approximately 100 lines of bgp-table output. From this output, I have manually stripped certain header information (note, this could also be done programmatically). Consequently, I have a file that starts as follows:

$ head -20 show_ip_bgp.txt 
     Network       Next Hop  Met LocPrf Wght Path
 *   1.0.0.0/24    12.0.1.63               0 7018 15169 i
 *   1.0.4.0/24    12.0.1.63               0 7018 3257 4826 38803 56203 i
 *   1.0.5.0/24    12.0.1.63               0 7018 3257 4826 38803 56203 i
 *   1.0.6.0/24    12.0.1.63               0 7018 3257 4826 38803 56203 56203 56203 i
 *   1.0.38.0/24   12.0.1.63               0 7018 3491 24155 i
 *   1.0.64.0/18   12.0.1.63               0 7018 701 2516 7670 18144 i
 *   1.0.128.0/24  12.0.1.63               0 7018 6453 4651 9737 23969 i
 *   1.0.128.0/19  12.0.1.63               0 7018 2914 38040 9737 i
 *   1.0.128.0/18  12.0.1.63               0 7018 2914 38040 9737 i
 *   1.0.128.0/17  12.0.1.63               0 7018 2914 38040 9737 i
 *   1.0.129.0/24  12.0.1.63               0 7018 3257 38040 9737 23969 i
 *   1.0.130.0/24  12.0.1.63               0 7018 6453 4651 9737 23969 i
 *   1.0.131.0/24  12.0.1.63               0 7018 6453 4651 9737 23969 i
 *   1.0.138.0/24  12.0.1.63               0 7018 6453 4651 9737 23969 i
 *   1.0.139.0/24  12.0.1.63               0 7018 6453 4651 9737 23969 i
 *   1.0.142.0/24  12.0.1.63               0 7018 6453 4651 9737 23969 i
 *   1.0.160.0/21  12.0.1.63               0 7018 6453 4651 9737 23969 i
 *   1.0.160.0/19  12.0.1.63               0 7018 2914 38040 9737 i
 *   1.0.192.0/21  12.0.1.63               0 7018 6453 4651 9737 23969 i

When I process this file using a TextFSM template, I am able to obtain the following:

FSM Table:
['valid_best', 'network', 'next_hop', 'as_path']
['*', '1.0.0.0/24', '12.0.1.63', '0 7018 15169 i']
['*', '1.0.4.0/24', '12.0.1.63', '0 7018 3257 4826 38803 56203 i']
['*', '1.0.5.0/24', '12.0.1.63', '0 7018 3257 4826 38803 56203 i']
['*', '1.0.6.0/24', '12.0.1.63', '0 7018 3257 4826 38803 56203 56203 56203 i']
['*', '1.0.38.0/24', '12.0.1.63', '0 7018 3491 24155 i']
['*', '1.0.64.0/18', '12.0.1.63', '0 7018 701 2516 7670 18144 i']
['*', '1.0.128.0/24', '12.0.1.63', '0 7018 6453 4651 9737 23969 i']
['*', '1.0.128.0/19', '12.0.1.63', '0 7018 2914 38040 9737 i']
['*', '1.0.128.0/18', '12.0.1.63', '0 7018 2914 38040 9737 i']
['*', '1.0.128.0/17', '12.0.1.63', '0 7018 2914 38040 9737 i']
['*', '1.0.129.0/24', '12.0.1.63', '0 7018 3257 38040 9737 23969 i']
['*', '1.0.130.0/24', '12.0.1.63', '0 7018 6453 4651 9737 23969 i']
['*', '1.0.131.0/24', '12.0.1.63', '0 7018 6453 4651 9737 23969 i']
['*', '1.0.138.0/24', '12.0.1.63', '0 7018 6453 4651 9737 23969 i']
['*', '1.0.139.0/24', '12.0.1.63', '0 7018 6453 4651 9737 23969 i']
['*', '1.0.142.0/24', '12.0.1.63', '0 7018 6453 4651 9737 23969 i']
['*', '1.0.160.0/21', '12.0.1.63', '0 7018 6453 4651 9737 23969 i']

Note, I omitted some information from the output that wasn't essential to the current discussion. Additionally, the as_path field needs further refinement, but this easily could be done in a post-processing step (or through improvements in the TextFSM template).

The key point being—I have converted a series of strings (lines of a file) to a set of list-like objects i.e. I have converted unstructured data to structured data.

So remember the TextFSM process is as follows:

network device output;
processed by a TextFSM template;
produces structured data.

Now this is fairly exciting...

Jason (and people working with him) have taken this output processing a few steps farther in the ntc-templates library.

First, they have taken advantage of a capability in TextFSM that allows you to dynamically select a template based on vendor, platform, and command information. Second, they have created a set of standardized templates.

Now what this enables is that you can execute a show command, pass the output through a template, and obtain structured data back and you don't have to do any string parsing. Here is an example.

Note in this example, I am relying on the templates from ntc-ansible library. Also the code below is a modified version of the code in the ntc-ansible, ntc_show_command module.

def main():

    # SSH connection information
    ip_addr = raw_input("IP address: ")
    port = 8022
    username = 'pyclass'
    password = getpass()
    device_type = 'cisco_ios'

    # Template file and TextFSM index
    index_file = 'index'
    template_dir = '/home/kbyers/NTC_ANS/templates'

    # Command to execute
    command = 'show ip int brief'

    # Establish SSH connection to device
    device = ConnectHandler(device_type=device_type,
                            ip=ip_addr, port=port, username=username,
                            password=password, verbose=False)

    # Retrieve output from remote device
    rawtxt = device.send_command(command)

    # Create CliTable object
    cli_table = clitable.CliTable(index_file, template_dir)
    attrs = {'Command': command, 'platform': device_type}

    # Dynamically parse the output from the router against the template
    cli_table.ParseCmd(rawtxt, attrs)
    
    # Convert from clitable format to list-dict format
    structured_data = clitable_to_dict(cli_table)

    print
    pprint(structured_data)
    print

There are a few minor things not shown—namely some libraries that are imported and a function named clitable_to_dict.

So what does this code do? Basically, it connects using SSH to a Cisco router. It then executes the 'show ip int brief' command. Finally, it uses TextFSM and a function to dynamically process this data and to return a Python data structure.

When I execute this code, here is what I obtain:

$ python test_textfsm.py
IP address: ******* (IP hidden)
Password:
 
[{'intf': 'FastEthernet0',
  'ipaddr': 'unassigned',
  'proto': 'down',
  'status': 'down'},
 {'intf': 'FastEthernet1',
  'ipaddr': 'unassigned',
  'proto': 'down',
  'status': 'down'},
 {'intf': 'FastEthernet2',
  'ipaddr': 'unassigned',
  'proto': 'down',
  'status': 'down'},
 {'intf': 'FastEthernet3',
  'ipaddr': 'unassigned',
  'proto': 'down',
  'status': 'down'},
 {'intf': 'FastEthernet4',
  'ipaddr': '10.220.88.21',
  'proto': 'up',
  'status': 'up'},
 {'intf': 'Vlan1', 'ipaddr': 'unassigned', 'proto': 'down', 'status': 'down'},
 {'intf': 'Vlan400',
  'ipaddr': 'unassigned',
  'proto': 'down',
  'status': 'down'}]

I now have a Python list where each list element is a dictionary of interface information.

Given all of the above, if network engineers could build a set of standard templates, then we could obtain structured output for various commands (for certain vendors-platforms). The ntc-ansible library already has some templates for cisco_nxos, cisco_ios, cisco_wlc, and hp_comware.

Kudos to Jason (and people helping him) for starting to bring this together.

Note, I intentionally omitted the specifics of TextFSM templates. These templates are complex and use regular expressions extensively.

Kirk Byers

@kirkbyers