router image

Expanding Netmiko's Secure Copy Support

By Kirk Byers
2018-03-07

I have recently been working on expanding the Secure Copy file transfer capabilities that are included in Netmiko. Ultimately this code should make OS upgrades and other file transfer operations easier to accomplish.

At this point, I have working and tested code on Cisco IOS, IOS-XE, NX-OS, IOS-XR, Juniper Junos, and Arista EOS. This is for both Secure Copy 'get' and 'put' operations and also includes associated methods that check whether the file exists, that verify disk space is available and that perform an MD5 comparison on the transferred file.

Note, the Netmiko driver generally requires an SCP server to be enabled on the remote network device.


In addition to adding Secure Copy support for additional platforms, I have also built a higher level 'file_transfer' function that performs a reliable file_transfer. This now should be the main interface to using Netmiko's Secure Copy.

Let's look at the this function's definition; here are the main arguments to the file_transfer function in Netmiko (as of today):

def file_transfer(ssh_conn, source_file, dest_file, 
                  file_system=None, direction='put',
                  disable_md5=False, overwrite_file=False):

Looking at the arguments, we can see that 'file_transfer' takes a Netmiko SSH connection (ssh_conn), a source file, and a destination file.


We can also see that have an optional argument for specifying the file_system. Additionally, there are default file systems included in the Netmiko code. The default file-systems are as follows:

Cisco IOS/IOS-XE/IOS-XR        Auto detects the file-system
Cisco NX-OS                    bootflash:
Arista EOS                     /mnt/flash
Juniper Junos                  /var/tmp

The default behavior of the file_transfer function is an idempotent file transfer.

In other words, Netmiko will first check if the file already exists, then calculate and compare the file's MD5. If the file already exists and has the correct MD5, then Netmiko will not transfer the file.

By default, 'file_transfer' has the overwrite_file argument set to False, so if the file already exists (and the MD5 doesn't match), then it won't overwrite the file. You can change this behavior by specifying overwrite_file=True.

You can also disable the MD5 calculation. You could potentially choose to do this because the MD5 calculation on large files is very slow on some network devices.


Let's look at an example:

#!/usr/bin/env python
from __future__ import print_function, unicode_literals

from getpass import getpass
from netmiko import ConnectHandler, file_transfer

password = getpass()

cisco = { 
    'device_type': 'cisco_ios',
    'host': 'device1.domain.com',
    'username': 'admin',
    'password': password,
    'file_system': 'flash:'
}
arista = { 
    'device_type': 'arista_eos',
    'host': 'device2.domain.com',
    'username': 'admin',
    'password': password,
    'file_system': '/mnt/flash',
}
junos = { 
    'device_type': 'juniper_junos',
    'host': 'device3.domain.com',
    'username': 'admin',
    'password': password,
    'file_system': '/var/tmp',
}
nxos = { 
    'device_type': 'cisco_nxos',
    'host': 'device4.domain.com',
    'username': 'admin',
    'password': password,
    'file_system': 'bootflash:'
}

source_file = 'test1.txt'
dest_file = 'testa.txt'
direction = 'put'

for net_device in (cisco, arista, junos, nxos):
    file_system = net_device.pop('file_system')

    # Create the Netmiko SSH connection
    ssh_conn = ConnectHandler(**net_device)
    transfer_dict = file_transfer(ssh_conn,
                                  source_file=source_file, 
                                  dest_file=dest_file,
                                  file_system=file_system, 
                                  direction=direction,
                                  overwrite_file=True)
    print(transfer_dict)
    pause = input("Hit enter to continue: ")

The 'file_transfer' function will return a dictionary with the following structure:

{
    'file_exists': True, 
    'file_transferred': True, 
    'file_verified': True
} 

The 'file_verified' key will indicate whether the MD5 check passed. In general, this should always be True or a Python exception should be raised (unless you explicitly disable the MD5 check using disable_md5=True).

The 'file_exists' key indicates the transferred file now exists. In the case of a 'put' operation, this will mean the file is on the remote system. In the case of a 'get' operation, this will mean the file is now on the local system (the system that is running the Python script).

The 'file_transferred' key indicates whether the file needed to be transferred or was already correct.


Here is the output of running this script. Note, I have added some comments and spacing for clarity:

$ python test_file_transfer.py 
Password: 
# Cisco IOS Device
{'file_exists': True, 'file_transferred': True, 'file_verified': True}
Hit enter to continue:

# Arista Device 
{'file_exists': True, 'file_transferred': True, 'file_verified': True}
Hit enter to continue: 

# Juniper Device
{'file_exists': True, 'file_transferred': True, 'file_verified': True}
Hit enter to continue:

# NX-OS Device 
{'file_exists': True, 'file_transferred': True, 'file_verified': True}
Hit enter to continue:  

Here is the before and after on each one of these devices:

# Cisco IOS Before 
pynet-rtr1#dir flash:/testa.txt
%Error opening flash:/testa.txt (File not found)

# Cisco IOS After 
pynet-rtr1#dir flash:/testa.txt
Directory of flash:/testa.txt

   67  -rw-          12   Mar 6 2018 13:45:10 -08:00  testa.txt 
# Arista Before 
pynet-sw1#dir flash:/testa.txt
Directory of flash:/testa.txt

% Error listing directory flash:/testa.txt (No such file or directory)

# Arista After 
pynet-sw1#dir flash:/testa.txt
Directory of flash:/testa.txt

       -rwx          12            Mar 6 13:47  testa.txt

4027953152 bytes total (3504869376 bytes free) 
# Juniper Before 
admin@pynet-jnpr-srx1> start shell sh 
$ cd /var/tmp
$ ls -al testa.txt
ls: testa.txt: No such file or directory

# Juniper After 
$ ls -al testa.txt
-rw-r--r--  1 pyclass  wheel  12 Mar  6 16:49 testa.txt 
# NX-OS Before 
nxos1# dir bootflash:/testa.txt
No such file or directory

# NX-OS After 
nxos1# dir bootflash:/testa.txt
         12    Mar 06 22:07:30 2018  testa.txt

Usage for bootflash://sup-local
  301174784 bytes used
 1293701120 bytes free
 1594875904 bytes total  

The file_transfer function is available in Netmiko 2.1.0 and later. I will also (probably) try to create a wrapper for this so it can be used directly from NAPALM and similarly I will try to make an Ansible module where it can be used.

As with all things open-source, make sure you test and validate this function properly in your environment before using it.


If you want to learn more about network automation, Python, and Ansible—then join my email-list. I also periodically run a free Python for Network Engineers email course which you can sign-up for here.

router image

Kirk Byers

router image

Expanding Netmiko's Secure Copy Support

By Kirk Byers
2018-03-07

I have recently been working on expanding the Secure Copy file transfer capabilities that are included in Netmiko. Ultimately this code should make OS upgrades and other file transfer operations easier to accomplish.

At this point, I have working and tested code on Cisco IOS, IOS-XE, NX-OS, IOS-XR, Juniper Junos, and Arista EOS. This is for both Secure Copy 'get' and 'put' operations and also includes associated methods that check whether the file exists, that verify disk space is available and that perform an MD5 comparison on the transferred file.

Note, the Netmiko driver generally requires an SCP server to be enabled on the remote network device.


In addition to adding Secure Copy support for additional platforms, I have also built a higher level 'file_transfer' function that performs a reliable file_transfer. This now should be the main interface to using Netmiko's Secure Copy.

Let's look at the this function's definition; here are the main arguments to the file_transfer function in Netmiko (as of today):

def file_transfer(ssh_conn, source_file, dest_file, 
                  file_system=None, direction='put',
                  disable_md5=False, overwrite_file=False):

Looking at the arguments, we can see that 'file_transfer' takes a Netmiko SSH connection (ssh_conn), a source file, and a destination file.


We can also see that have an optional argument for specifying the file_system. Additionally, there are default file systems included in the Netmiko code. The default file-systems are as follows:

Cisco IOS/IOS-XE/IOS-XR        Auto detects the file-system
Cisco NX-OS                    bootflash:
Arista EOS                     /mnt/flash
Juniper Junos                  /var/tmp

The default behavior of the file_transfer function is an idempotent file transfer.

In other words, Netmiko will first check if the file already exists, then calculate and compare the file's MD5. If the file already exists and has the correct MD5, then Netmiko will not transfer the file.

By default, 'file_transfer' has the overwrite_file argument set to False, so if the file already exists (and the MD5 doesn't match), then it won't overwrite the file. You can change this behavior by specifying overwrite_file=True.

You can also disable the MD5 calculation. You could potentially choose to do this because the MD5 calculation on large files is very slow on some network devices.


Let's look at an example:

#!/usr/bin/env python
from __future__ import print_function, unicode_literals

from getpass import getpass
from netmiko import ConnectHandler, file_transfer

password = getpass()

cisco = { 
    'device_type': 'cisco_ios',
    'host': 'device1.domain.com',
    'username': 'admin',
    'password': password,
    'file_system': 'flash:'
}
arista = { 
    'device_type': 'arista_eos',
    'host': 'device2.domain.com',
    'username': 'admin',
    'password': password,
    'file_system': '/mnt/flash',
}
junos = { 
    'device_type': 'juniper_junos',
    'host': 'device3.domain.com',
    'username': 'admin',
    'password': password,
    'file_system': '/var/tmp',
}
nxos = { 
    'device_type': 'cisco_nxos',
    'host': 'device4.domain.com',
    'username': 'admin',
    'password': password,
    'file_system': 'bootflash:'
}

source_file = 'test1.txt'
dest_file = 'testa.txt'
direction = 'put'

for net_device in (cisco, arista, junos, nxos):
    file_system = net_device.pop('file_system')

    # Create the Netmiko SSH connection
    ssh_conn = ConnectHandler(**net_device)
    transfer_dict = file_transfer(ssh_conn,
                                  source_file=source_file, 
                                  dest_file=dest_file,
                                  file_system=file_system, 
                                  direction=direction,
                                  overwrite_file=True)
    print(transfer_dict)
    pause = input("Hit enter to continue: ")

The 'file_transfer' function will return a dictionary with the following structure:

{
    'file_exists': True, 
    'file_transferred': True, 
    'file_verified': True
} 

The 'file_verified' key will indicate whether the MD5 check passed. In general, this should always be True or a Python exception should be raised (unless you explicitly disable the MD5 check using disable_md5=True).

The 'file_exists' key indicates the transferred file now exists. In the case of a 'put' operation, this will mean the file is on the remote system. In the case of a 'get' operation, this will mean the file is now on the local system (the system that is running the Python script).

The 'file_transferred' key indicates whether the file needed to be transferred or was already correct.


Here is the output of running this script. Note, I have added some comments and spacing for clarity:

$ python test_file_transfer.py 
Password: 
# Cisco IOS Device
{'file_exists': True, 'file_transferred': True, 'file_verified': True}
Hit enter to continue:

# Arista Device 
{'file_exists': True, 'file_transferred': True, 'file_verified': True}
Hit enter to continue: 

# Juniper Device
{'file_exists': True, 'file_transferred': True, 'file_verified': True}
Hit enter to continue:

# NX-OS Device 
{'file_exists': True, 'file_transferred': True, 'file_verified': True}
Hit enter to continue:  

Here is the before and after on each one of these devices:

# Cisco IOS Before 
pynet-rtr1#dir flash:/testa.txt
%Error opening flash:/testa.txt (File not found)

# Cisco IOS After 
pynet-rtr1#dir flash:/testa.txt
Directory of flash:/testa.txt

   67  -rw-          12   Mar 6 2018 13:45:10 -08:00  testa.txt 
# Arista Before 
pynet-sw1#dir flash:/testa.txt
Directory of flash:/testa.txt

% Error listing directory flash:/testa.txt (No such file or directory)

# Arista After 
pynet-sw1#dir flash:/testa.txt
Directory of flash:/testa.txt

       -rwx          12            Mar 6 13:47  testa.txt

4027953152 bytes total (3504869376 bytes free) 
# Juniper Before 
admin@pynet-jnpr-srx1> start shell sh 
$ cd /var/tmp
$ ls -al testa.txt
ls: testa.txt: No such file or directory

# Juniper After 
$ ls -al testa.txt
-rw-r--r--  1 pyclass  wheel  12 Mar  6 16:49 testa.txt 
# NX-OS Before 
nxos1# dir bootflash:/testa.txt
No such file or directory

# NX-OS After 
nxos1# dir bootflash:/testa.txt
         12    Mar 06 22:07:30 2018  testa.txt

Usage for bootflash://sup-local
  301174784 bytes used
 1293701120 bytes free
 1594875904 bytes total  

The file_transfer function is available in Netmiko 2.1.0 and later. I will also (probably) try to create a wrapper for this so it can be used directly from NAPALM and similarly I will try to make an Ansible module where it can be used.

As with all things open-source, make sure you test and validate this function properly in your environment before using it.


If you want to learn more about network automation, Python, and Ansible—then join my email-list. I also periodically run a free Python for Network Engineers email course which you can sign-up for here.

router image

Kirk Byers