Robot Framework Guide

This page contains a how-to guide for writing custom integration tests using the VPP Agent’s libraries for the Robot framework.


Robot Test setup

The following text describes how to write robot test suites (locally or in any other environment).

you need to create a test file in format:

../testname.robot

Example:

../tests/robot/suites/crud/acl_crud.robot

Resource setup in the next step is based on where you have created the file.

Setup test

When you have the file, you need to define (libraries, resources, tags, variables). - Base libraries used are “OperatingSystem” and “Collections”. OperatingSystem - is for use with the files- create, delete, copy, check if exist file, … . This is mostly used for configuration, make logs and other. Collections - Append To List, Combine Lists, Convert To Dictionary, Convert To List, Copy Dictionary, Copy List, Count Values In List, Dictionaries Should Be Equal, Should Contain Match, Should Not Contain Match, … . - Resources are defined in our libraries. - Tags are used to skip tests or mark them as non-critical. “Robot Tags” - Example:

Library      OperatingSystem
Library      Collections

Resource     ../../variables/${VARIABLES}_variables.robot
Resource     ../../libraries/all_libs.robot

Force Tags        crud     IPv4    #(ExpectedFailure, IPv6, traffic, misc, sfc)

Suite Setup       Testsuite Setup
Suite Teardown    Testsuite Teardown
Test Setup        TestSetup
Test Teardown     TestTeardown

*** Variables ***
...

If you need to add another library, then you add lib into -All libraries:

../vpp-agent/tests/robot/libraries/all_libs.robot

Configuring the test environment

  • Base configuration:
Configure Environment
    [Tags]    setup
    Configure Environment 1

Setup nodes(docker) for the test.

  • Configuration keyword is based on add VPP nodes and run them:
Configure Environment 1
    Add Agent VPP Node    agent_vpp_1
    Add Agent VPP Node    agent_vpp_2
    Execute In Container    agent_vpp_1    echo $MICROSERVICE_LABEL
    Execute In Container    agent_vpp_2    echo $MICROSERVICE_LABEL
    Execute In Container    agent_vpp_1    ls -al
    Execute On Machine    docker    ${DOCKER_COMMAND} images
    Execute On Machine    docker    ${DOCKER_COMMAND} ps -as
  • Mostly used configurations:
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/configurations.robot

Sometimes, you can use next keyword for other needed configuration. Its means interfaces, fill files, routes, … .

  • Specific configuration example:
Configure Environment
    [Tags]    setup
    ${DATA_FOLDER}=       Catenate     SEPARATOR=/       ${CURDIR}         ${TEST_DATA_FOLDER}
    Set Suite Variable          ${DATA_FOLDER}
    Configure Environment 2        acl_basic.conf

Test Setup, Teardown

  • Mostly used in the test is “make snapshots ETCD”. This part is needed for better test debugging. Add them to the end of the test, in part- Keywords.
*** Keywords ***
TestSetup
    Make Datastore Snapshots    ${TEST_NAME}_test_setup

TestTeardown
    Make Datastore Snapshots    ${TEST_NAME}_test_teardown

Test body

Test cases use keywords from our libraries. After configuration, setup and teardown you can start with testing. Before starting the test, it is a good practice to check whether there is no existing configuration from previous runs.

Show Something Before Setup
    ${interfaces}=    vpp_term: Show Interfaces    agent_vpp_1

This shows only the configuration. You can use keywords comparing output, or check if the output is empty(used in next step- Interface Not Exists). The next step is to configure something (in example interface):

Add Something
    vpp_term: Interface Not Exists  node=agent_vpp_1    mac=${MAC_TAP1}
    Put TAP Interface With IP    node=agent_vpp_1    name=${NAME_TAP1}    mac=${MAC_TAP1}    ip=${IP_TAP1}    prefix=${PREFIX}    host_if_name=linux_${NAME_TAP1}

Every configuration must be checked (sometimes a timeout is needed, in that case, use the Wait Until keyword):

Check Something Is Created
    ${interfaces}=       vat_term: Interfaces Dump    node=agent_vpp_1
    Wait Until Keyword Succeeds   ${WAIT_TIMEOUT}   ${SYNC_SLEEP}    vpp_term: Interface Is Created    node=agent_vpp_1    mac=${MAC_TAP1}
    ${actual_state}=    vpp_term: Check TAP interface State    agent_vpp_1    ${NAME_TAP1}    mac=${MAC_TAP1}    ipv4=${IP_TAP1}/${PREFIX}    state=${UP_STATE}

Always is good add other configuration from same thing and check if is created(delete “No Operation” and use same configuration from “Add Something” and “Check Something Is Created”. You need only change interface name, mac, ip):

Add Something_Other
    No Operation

Check Something_Other Is Created
    No Operation

After this, you need to check if the first thing (interface) is still configured and without failure:

Check Something Is Still Configured
    ${actual_state}=    vpp_term: Check TAP interface State    agent_vpp_1    ${NAME_TAP1}    mac=${MAC_TAP1}    ipv4=${IP_TAP1}/${PREFIX}    state=${UP_STATE}

In the CRUD, test should try to update configured data and check whether they are correct:

Update Something
    Put TAP Interface With IP    node=agent_vpp_1    name=${NAME_TAP1}    mac=${MAC_TAP1_2}    ip=${IP_TAP1_2}    prefix=${PREFIX}    host_if_name=linux_${NAME_TAP1}

Check Something Is Created
    Wait Until Keyword Succeeds   ${WAIT_TIMEOUT}   ${SYNC_SLEEP}    vpp_term: Interface Is Created    node=agent_vpp_1    mac=${MAC_TAP1_2}
    ${actual_state}=    vpp_term: Check TAP interface State    agent_vpp_1    ${NAME_TAP1}    mac=${MAC_TAP1_2}    ipv4=${IP_TAP1_2}/${PREFIX}    state=${UP_STATE}

Also need:

Check Something_Other Has Not Changed
    No Operation

Try delete the configuration and check it:

Delete Something
    Delete VPP Interface    agent_vpp_1    ${NAME_TAP1}

Check Something Has Been Deleted
    Wait Until Keyword Succeeds   ${WAIT_TIMEOUT}   ${SYNC_SLEEP}    vpp_term: Interface Not Exists  node=agent_vpp_1    mac=${MAC_TAP1_2}

After delete, check other configuration for changes (there should not be any):

Check Something_Other Is Still Configured
    No Operation

Last part can be in your own manage. This example could you help with debugging:

Show Interfaces And Other Objects After Setup
    vpp_term: Show Interfaces    agent_vpp_1
    Write To Machine    agent_vpp_1_term    show int addr
    Write To Machine    agent_vpp_1_term    show h
    Write To Machine    agent_vpp_1_term    show br
    Write To Machine    agent_vpp_1_term    show br 1 detail
    Write To Machine    agent_vpp_1_term    show vxlan tunnel
    Write To Machine    agent_vpp_1_term    show err
    vat_term: Interfaces Dump    agent_vpp_1
    Execute In Container    agent_vpp_1    ip a

Reference guide

In this part are examples of keywords, short descriptions and libraries witch you can use for tests.

Docker container keywords

  • Add Agent Node, Add Agent VPP Node, Execute In Container, Write Command to Container, …, are here:
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/docker.robot

Read/Write ETCD

  • Get, Put, keywords for read, write into ETCD:
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/etcdctl.robot

Linux interfaces and Linux commands

  • If you will use Linux interfaces and commands inside Linux for tests you need this library:
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/linux.robot

Inside are keywords for create, delete or read interfaces and interface status. Also, are there the keywords for ping in tests(traffic tests).

Create, Delete, Update

  • Most keywords for create, delete, update, show interfaces and routes in VPP-Agent are here:
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/pretty_keywords.robot

SSH Connection, Test Setup/Teardown

  • Keyword used for Open SSH Connection, Testsuite Setup, Test Setup, Testsuite Teardown, Test Teardown, Logs, Datastore, Dump, Snapshots:
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/setup-teardown.robot

SSH usage

  • SSH - Execute On Machine, Write To Machine:
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/ssh.robot

Read/Write VAT terminal

  • VAT terminal keywords are used for check status interfaces, bridge domains, … in docker. You read (“Write To Machine”) from docker.
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/vat_term.robot

Start/Stop VPP

  • Start, stop keywords for VPP(in docker) and write into VPP:
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/vpp.robot

Read/Write VPP terminal

  • Direct keywords to show (interfaces, acl, BD, route, DNAT,…), ping and other commands in VPP(not docker, inside VPP):
https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/vpp_term.robot

VXLAN keywords

https://github.com/ligato/vpp-agent/blob/master/tests/robot/libraries/vxlan.robot

Wait until Keyword

  • is used when some time is needed to provide to the test:
Wait Until Keyword Succeeds   ${WAIT_TIMEOUT}   ${SYNC_SLEEP}    ...

Test examples

Tests example directory- vpp-agent/tests/robot/examples/.

  • CRUD test example:
  • on github:
https://github.com/ligato/vpp-agent/blob/dev/tests/robot/examples/example_crud_test.robot
  • full example:
*** Settings ***
Library      OperatingSystem
Library      Collections

Resource     ../../variables/${VARIABLES}_variables.robot

Resource     ../../libraries/all_libs.robot

Force Tags        crud     IPv4
Suite Setup       Testsuite Setup
Suite Teardown    Testsuite Teardown
Test Setup        TestSetup
Test Teardown     TestTeardown

*** Variables ***
${VARIABLES}=        common
${ENV}=              common
${WAIT_TIMEOUT}=     20s
${SYNC_SLEEP}=       3s

${NAME_TAP1}=        vpp1_tap1
${NAME_TAP2}=        vpp1_tap2
${MAC_TAP1}=         12:21:21:11:11:11
${MAC_TAP1_2}=       22:21:21:11:11:11
${MAC_TAP2}=         22:21:21:22:22:22
${IP_TAP1}=          20.20.1.1
${IP_TAP1_2}=        21.20.1.2
${IP_TAP2}=          20.20.2.1
${PREFIX}=           24
${MTU}=              4800
${UP_STATE}=         up


*** Test Cases ***
Configure Environment
    [Tags]    setup
    Configure Environment 1

Show Something Before Setup
    ${interfaces}=    vpp_term: Show Interfaces    agent_vpp_1

Add Something
    vpp_term: Interface Not Exists  node=agent_vpp_1    mac=${MAC_TAP1}
    Put TAP Interface With IP    node=agent_vpp_1    name=${NAME_TAP1}    mac=${MAC_TAP1}    ip=${IP_TAP1}    prefix=${PREFIX}    host_if_name=linux_${NAME_TAP1}

Check Something Is Created
    ${interfaces}=       vat_term: Interfaces Dump    node=agent_vpp_1
    Wait Until Keyword Succeeds   ${WAIT_TIMEOUT}   ${SYNC_SLEEP}    vpp_term: Interface Is Created    node=agent_vpp_1    mac=${MAC_TAP1}
    ${actual_state}=    vpp_term: Check TAP interface State    agent_vpp_1    ${NAME_TAP1}    mac=${MAC_TAP1}    ipv4=${IP_TAP1}/${PREFIX}    state=${UP_STATE}

Add Something_Other
    No Operation

Check Something_Other Is Created
    No Operation

Check Something Is Still Configured
    ${actual_state}=    vpp_term: Check TAP interface State    agent_vpp_1    ${NAME_TAP1}    mac=${MAC_TAP1}    ipv4=${IP_TAP1}/${PREFIX}    state=${UP_STATE}

Update Something
    Put TAP Interface With IP    node=agent_vpp_1    name=${NAME_TAP1}    mac=${MAC_TAP1_2}    ip=${IP_TAP1_2}    prefix=${PREFIX}    host_if_name=linux_${NAME_TAP1}

Check Something Is Created
    Wait Until Keyword Succeeds   ${WAIT_TIMEOUT}   ${SYNC_SLEEP}    vpp_term: Interface Is Created    node=agent_vpp_1    mac=${MAC_TAP1_2}
    ${actual_state}=    vpp_term: Check TAP interface State    agent_vpp_1    ${NAME_TAP1}    mac=${MAC_TAP1_2}    ipv4=${IP_TAP1_2}/${PREFIX}    state=${UP_STATE}

Check Something_Other Has Not Changed
    No Operation

Delete Something
    Delete VPP Interface    agent_vpp_1    ${NAME_TAP1}

Check Something Has Been Deleted
    Wait Until Keyword Succeeds   ${WAIT_TIMEOUT}   ${SYNC_SLEEP}    vpp_term: Interface Not Exists  node=agent_vpp_1    mac=${MAC_TAP1_2}

Check Something_Other Is Still Configured
    No Operation

Show Interfaces And Other Objects After Setup
    vpp_term: Show Interfaces    agent_vpp_1
    Write To Machine    agent_vpp_1_term    show int addr
    Write To Machine    agent_vpp_1_term    show h
    Write To Machine    agent_vpp_1_term    show br
    Write To Machine    agent_vpp_1_term    show br 1 detail
    Write To Machine    agent_vpp_1_term    show vxlan tunnel
    Write To Machine    agent_vpp_1_term    show err
    vat_term: Interfaces Dump    agent_vpp_1
    Execute In Container    agent_vpp_1    ip a

*** Keywords ***

TestSetup
    Make Datastore Snapshots    ${TEST_NAME}_test_setup

TestTeardown
    Make Datastore Snapshots    ${TEST_NAME}_test_teardown