Background

The concept of a continuous integration and validation for network changes is being recognized as a must-have for modern networks. Currently in the industry there are many companies and projects working on automation or some form of automation in combination with validation. At Tesuto, we have built a large-scale emulation platform that enables network operators to fully clone their network in the cloud and perform any number of tasks with it. The purpose of this blog post is to discuss how to use a cloned network to perform validation of network changes prior to production rollout. This change validation coverage goes beyond just configuration file validation. Test coverage extends to the deep library of network validators available through the use of NAPALM. In addition, Tesuto’s emulation platform validates the behavior of your control plane with the exact software version and topolog you have in production environment.

This document will cover a single example use case – there are many others – of creating a CI environment using a GitLab repository to store network configuration files. Whether you host a private GitLab server or use a free public account, this outline is equally valid. This example is written with Python code. The code is PEP-8 compliant but is written as example code and therefore not as Pythonic as it might be in a normal use case. Clarity has been prioritized over succinctness. It has also been designed to be as standalone as possible. Minimal imports have been done to keep a virtual-environment footprint as small as possible.

The code

The code is available publicly at https://gitlab.com/tesuto/public/gitlab-example.

Tesuto configuration files

In order to properly operate, several files are required for use. These constraints are mostly due to the static nature of this example and can be customized based on your particular needs. The layout of this repository is as follows, and the code in its current state mostly requires adherence to this structure to properly operate.
Network device configuration files are stored in the configs/ directory. In our example, we have two router configurations stored as core1.cfg and core2.cfg. In this example, we are using Arista devices and these are standard Arista configuration files.

Three custom files (devices.yaml, topologies.yaml, and validators.yaml) are required to allow the CI script to properly configure the emulation and network devices in the Tesuto cloud.

  • devices.yaml
    example:

    ---
     - name: core1
       config: configs/core1.cfg
       version: 302946505480339457
       interfaces: Ethernet[1-10]
     - name: core2
       config: configs/core2.cfg
       version: 302946505480339457
       interfaces: Ethernet[1-10]
    

    This YAML describes each device that is part of the validation process. Each configuration file in the configs/ directory needs an entry in this file to configure it in the Tesuto cloud.

    • The name: tag specifies the hostname for the device. This name is used to retrieve configuration files as well as link topologies.
    • The config: tag specifies the path to the configuration file for this device.
    • The version: tag specifies the Tesuto ID of the version of software to use for this device.
    • The interfaces: tag specifies the interfaces to configure for the device. In our example using Arista devices we specify Ethernet[1-10]. This is Tesuto shorthand to configure interfaces Ethernet1 – Ethernet10.
  • topologies.yaml
    ---
     - device: core1
       interface: Ethernet1
       neighbor: core2
       neighbor_interface: Ethernet1
    

    This YAML describes the physical links between the devices.

    • The device: tag specifies the hostname of a device.
    • The interface: tag specifies the interface to link on the specified device.
    • The neighbor: tag specifies the hostname of the device on the remote side of the connection.
    • The neighbor_interface: tag specifies the interface on the remote device.
  • validators.yaml
    ---
     - comment: Core1 Ethernet1 LLDP Neighbor Check
       script_name: LLDP Neighbors
       parse_params:
         run_on:
           - core1
       state_params:
         - get_lldp_neighbors:
             _mode: strict
             Ethernet1:
               - hostname: core2
                 port: Ethernet1
    

    This YAML file outlines the validators to run as part of this emulation. In our simple example we are only checking the LLDP neighbor for two interfaces on our core1 device. The full list of validations is extensive and beyond the scope of this document. For a full list of validators please see the Napalm validators documentation.

    • The comment: tag specifies a short description of the validator.
    • The script_name: tag specifies the name of the Tesuto module to use to perform the validation.
    • The parse_params: tag specifies a POSIX regular expression to determine what device in the emulation the validator should be run on.
    • The state_params: tag outlines the NAPALM validator configuration.

Configuring the repository

To work with the CI tools locally we can simply run python3 .tesuto.py to run the script.

AUTH_TOKEN Gitlab Secret Variable
In order to interact with the Tesuto API you will need to configure a GitLab Secret variable. The name of the variable is AUTH_TOKEN. The value for this variable is your authentication token specified as part of your Tesuto account. If you do not have a token and need one, please contact 5nines@tesuto.com.

IMPORTANT: This token has read/write access to your Tesuto account. It should be configured for use only on protected branches in the repository. For more information on using secret variables please see this GitLab documentation page.

.gitlab-ci.yml:

image: "python:3.6"
before_script:
 - python3 --version
 - pip3 install requests PyYAML

validate:
 stage: test
 script:
   - python3 .tesuto.py

In order to have GitLab automatically run the pipeline in the manner we wish we must configure the .gitlab-ci.yml file. The example above breaks down as follows:

  • The image: tag specifies the default OS image to use for the Pipeline. Being that this example is built using Python, we use the recent Python:3.6 image.
  • The before_script: tells the pipeline what to configure in the default image prior to executing our code. In this case we need the requests and PyYAML modules to be present for our script to operate properly.
  • The validate: tag is the name of our process.
  • The stage: tag controls when this pipeline will be executed. GitLab supports a number of stages. In this example we are setting the stage to test so the pipeline is triggered on any repository change.
  • The script: tag specifies the list of items to run for this pipeline. In this case, python3 .tesuto.py will run the validation script.

Network validation

For a GitLab pipeline in our use case the determination of whether a pipeline succeeded is binary. If the script exits cleanly upon completion, that is considered a successful validation for the pipeline. In order for the script to exit cleanly, the emulation and all associated configurations must be properly provisioned in the Tesuto cloud as well as all the validators as part of the emulation passing successfully. If any error is encountered while interacting with the Tesuto API or as a result of a validator failure, the script exits with an error code which causes a pipeline failure.

To examine what actions the .tesuto.py script is taking we can look at the __main__ function.

if __name__ == '__main__':
    """ Run Tesuto emulation validation. """
    print('Tesuto Network Validation CI')
    cfg = parse_config()
    cfg['emulation_id'] = create_emulation(cfg)
    create_devices(cfg)
    create_topologies(cfg)
    create_validators(cfg)
    cfg['job_id'] = create_job(cfg)
    watch_job(cfg)
    check_validations(cfg)
    do_cleanup(cfg)

The order of operation for the script is as follows:

  1. Create a new emulation.
    1. When using GitLab-CI each pipeline is given a unique ID exposed as an environment variable to the runner. The name of the variable is CI_JOB_ID and this is used as the emulation name. This allows for easy tracking in Tesuto when wanting to view a particular pipeline’s emulation.
  2. Create devices
    1. The devices specified in the devices.yaml file will be created in this emulation. The configuration files will also be uploaded to Tesuto for applying to the devices during the validator processing.
  3. Create topologies
    1. Create logical network links between the devices specified in topologies.yaml.
  4. Create validators
    1. Create NAPALM validators outline in validators.yaml. We will write about this more in the future but for information regarding validator support please see http://napalm.readthedocs.io/en/latest/validate/.
  5. Schedule a job
    1. At this point the emulation is created and fully configured in Tesuto and is ready for processing. A job in Tesuto is an action to perform. In this case it is to initialize the emulation and run the validators against it.
  6. Watch the job
    1. The job will take some time to complete. Must emulations without exotic validators will complete in 10-15 minutes. While the job is running in Tesuto we want the pipeline in GitLab to stay active. In this case we simply periodically check the Tesuto API until either the job is complete or the timeout is reached.
  7. Check the validation results
    1. Upon successful job completion the results of each validator are checked and if there are any failures then the pipeline is failed.
  8. Cleanup based on configuration
    1. It is possible to have the pipeline automatically delete the emulation from the Tesuto cloud once it completed successfully. To do this you can set a variable in the the GitLab repository configuration of DELETE_EMULATION = True.

Closing thoughts

It is important to note that is one basic method for performing network validation using the Tesuto API. The possibilities for sophisticated and rigorous testing of software version, configuration, and topology changes are many. If you have ideas on types of network change validation support you would like to see, please let us know at 5nines@tesuto.com.

If you are reading this and you want network emulation and change validation but don’t have the time or technical resources to build a solution for yourselves, please contact us. We can help from general advice all the way up to professional services that can deliver a turnkey solution for your organization.

This is the first in series of examples of how to eliminate change induced outages by utilizing an emulated network.

Pipeline environment variables

The .tesuto.py script supports a number of environment variables in order to customize it for your use case.

Variable

Description

Default

Required

AUTH_TOKEN

Required API authentication token. For use with Gitlab, this can be configured as a secret variable in the repository configuration.

Yes

API_URL

An optional url and should only be set for dev purposes.

https://api.tesuto.com/v1

No

CI_JOB_ID

Dynamically set by Gitlab-CI. It is the unique CI job id and is used as a unique emulation name. If not running in Gitlab and this value is not present, the current epoch timestamp will be used for the emulation name. Can be overridden by environment value.

No

TIMEOUT

The default timeout for a CI process time is 15 minutes. This can be overridden by setting the timeout value in seconds.

900

No

CHECK_INTERVAL

How often to poll the API for job status updates. Defaults to 30 seconds.

30

No

DELETE_EMULATION

Auto delete the emulation on success. Default is false.

False

No

Caveats

  • The Python code in this document is written for Python3. While it should be fully operational in Python >2.6 environments, it has not been fully tested. The use of Python3 is recommended.
  • At Tesuto, we love GitLab and use their software for all of our repositories. This example is built using the GitLab CI/CD tools. If you are a GitHub user simply replacing the .gitlab-ci.yml file with the appropriate GitHub CI settings will be all that is required. We also plan to add a GitHub example in the near future.
    • This document also assumes the reader has a basic of understanding of working with Git repositories in addition to knowledge of using the GitLab CI/CD tools. For further information on this please see: https://about.gitlab.com/features/gitlab-ci-cd.
  • To utilize the Tesuto API for validation an active account is required. For information about becoming a Tesuto customer or to request a free trial account, please contact 5nines@tesuto.com.
  • In the near future (read: ~30 days), Tesuto will be releasing a publicly available Python module importable via PIP. This module will simplify the process of interacting with the Tesuto API and shorten the code required to utilize CI. When this module is released we will add another example repository illustrating how to build CI using the public module. In the meantime, this module is self contained and should serve as an example for interacting with the API for any Python application.