Control flows
This guide aims to explain the key-value scheduling algorithm using examples
with UML control-flow diagrams, each covering a specific scenario using real
configuration items from vpp and linux plugins of the agent. The control-flow
diagrams are structured to describe all the interactions between KVScheduler, NB
plane and KVDescriptor-s during transactions. To improve readability, the examples
use shortened keys (without prefixes) or even alternative and more descriptive
aliases as object identifiers. For example, my-route is used as a placeholder
for user-defined route, which otherwise would be identified by key composed
of destination network, outgoing interface and next hop address. Moreover, most
of the configuration items that are automatically pre-created in the SB plane
(i.e. OBTAINED), normally retrieved from VPP and Linux during the first resync
(e.g. default routes, physical interfaces, etc.), are omitted from the diagrams
since they do not play any role in the presented scenarios.
The UML diagrams are plotted as SVG images, also containing links to images presenting the state of the graph with values at the end of every transaction. But to be able to access these links, the UML diagrams have to be clicked at to open them as standalone inside another web browser tab. From within github web-UI the links are not directly accessible.
Example: AF-Packet interface
AF-Packet is VPP interface type attached to a host OS interface, capturing
all its incoming traffic and also allowing to inject Tx packets through a special
type of socket.
The requirement is that the host OS interface exists already before the AF-Packet
interface gets created. The challenge is that the host interface may not be
from the scope of items configured by the agent. Instead, it could be
a pre-existing physical device or interface created by an external process or
an administrator during the agent run-time. In such cases, however, there would
be no key-value pair to reference from within AF-Packet dependencies. Therefore,
KVScheduler allows to notify about external objects through
PushSBNotification(key, value, metadata) method. Values received through
notifications are denoted as OBTAINED and will not be removed by resync even
though they are not requested to be configured by NB. Obtained values are
allowed to have their own descriptors, but from the CRUD operations only
Retrieve() is ever called to refresh the graph. Create, Delete and Update
are never used, since obtained values are updated externally and the agent is
only notified about the changes after they have already happened.
Linux interface plugin ships with InterfaceWatcher descriptor, which retrieves
and notifies about Linux interfaces in the network namespace of the agent
(so-called default network namespace). Linux interfaces are assigned unique
keys using their host names, e.g.: linux/interface/host-name/eth1
The AF-Packet interface then defines dependency referencing the key with the
host name of the interface it is supposed to attach to (cannot attach
to interfaces from other namespaces).
In this example, the host interface gets created after the request to configure
AF-Packet is received. Therefore, the scheduler keeps the AF-Packet in the
PENDING state until the notification is received.
Example: Bridge Domain
Using bridge domain it is demonstrated how derived values can be used to “break” item into multiple parts with their own CRUD operations and dependencies.
Bridge domain groups multiple interfaces to share the same flooding or broadcast characteristics. Empty bridge domain has no dependencies and can be created independently from interfaces. But to put an interface into a bridge domain, both the interface and the domain must be created first. One solution for the KVScheduler framework would be to handle bridge domain as a single key-value pair depending on all the interfaces it is configured to contain. But this is a rather coarse-grained approach that would prevent the existence of the bridge domain even when only a single interface is missing. Moreover, with KVDB, request to remove interface could overtake update of the bridge domain configuration un-listing the interface, which would cause the bridge domain to be temporarily removed and shortly afterwards fully re-created.
The concept of derived values allowed to specify binding between bridge
domain and every bridged interface as a separate derived value, handled
by its own BDInterfaceDescriptor descriptor, where Create() operation puts
interface into the bridge domain, Delete() breaks the binding, etc.
The bridge domain itself has no dependencies and will be configured as long as
it is demanded by NB.
The bindings, however, will each have a dependency on its associated interface
(and implicitly on the bridge domain it is derived from).
Even if one or more interfaces are missing or are being deleted, the remaining
of the bridge domain will remain unaffected and continuously functional.
The control-flow diagram shows that bridge domain is created even if the
interface that it is supposed to contain gets configured later. The binding
remains in the PENDING state until the interface is configured.
Example: Interface Re-creation
The example uses VPP interface with attached route to outline the control-flow
of item re-creation. A specific configuration update may not be supported by SB
to perform incrementally - instead the given item may need to be deleted and
re-created with the new configuration. Using UpdateWithRecreate() method,
a descriptor is able to tell the KVScheduler if the given item requires
full re-creation for the configuration update to be applied.
This example demonstrates re-creation using a VPP TAP interface and a NB request
to change the RX ring size, which is not supported for an already created
interface. Furthermore, the interface has an L3 route attached to it. The route
cannot exists without the interface, therefore it must be deleted and moved into
the PENDING state before interface re-creation, and configured back again once
the re-creation procedure has finalized.
Example: Retry of failed operation
This example demonstrates that for best-effort transactions (e.g. every resync),
the KVScheduler allows to enable automatic and potentially repeated retry
of failed operations.
In this case, a TAP interface my-tap fails to get created. Before terminating
the transaction, the scheduler retrieves the current value of my-value, the
state of which cannot be assumed since the creation failed somewhere in-progress.
Then it schedules a so-called retry transaction, which will attempt to fix
the failure by re-applying the same configuration. Since the value retrieval
has not found my-tap to be configured, the retry transaction will repeat
the Create(my-tap) operation and succeed in our example.
Example: Transaction revert
An update transaction (i.e. not resync) can be configured to run in either best-effort mode, allowing partial completion, or to terminate upon first failure and revert already applied changes so that no visible effects are left in the system (i.e. the true transaction definition). This behaviour is only supported with GRPC or localclient as the agent NB interface. With KVDB, the agent will run in the best-effort mode to get as close to the desired configuration as it is possible.
In this example, a transaction is planned and executed to create a VPP interface
my-tap with an attached route my-route. While the interface is successfully
created, the route, on the other hand, fails to get configured. The scheduler
then triggers the revert procedure. First the current value of my-route is
retrieved, the state of which cannot be assumed since the creation failed
somewhere in-progress. The route is indeed not found to be configured, therefore
only the interface must be deleted to undo already executed changes. Once
the interface is removed, the state of the system is back to where it was before
the transaction started. Finally, the transaction error is returned back to the
northbound plane.
Example: Unnumbered interface
Turning interface into unnumbered allows to enable IP processing without assigning it an explicit IP address. An unnumbered interface can “borrow” an IP address of another interface, already configured on VPP, which conserves network and address space.
The requirement is that the interface from which the IP address is supposed
to be borrowed has to be already configured with at least one IP address assigned.
Normally, the agent represents a given VPP interface using a single key-value
pair. Depending on this key alone would only ensure that the target interface is
already configured when a dependent object is being created. In order to be able
to restrict an object existence based on the set of assigned IP addresses to
an interface, every VPP interface value must derive single (and unique)
key-value pair for each assigned IP address. This will enable to reference IP
address assignments and build dependencies around them.
For unnumbered interfaces alone it would be sufficient to derive single value
from every interface, with key that would allow to determine if the interface
has at least one IP address assigned,
something like: vpp/interface/<interface-name>/has-ip/<true/false>.
Dependency for an unnumbered interface could then reference key of this value:
vpp/interface/<interface-name-to-borrow-IP-from>/has-ip/true
For more complex cases, which are outside of the scope of this example, it may
be desirable to define dependency based not only on the presence but also on the
value of an assigned IP address. Therefore we derive (empty) value for each
assigned IP address with key template:
"vpp/interface/address/<interface-name>/<address>".
This complicates situation for unnumbered interfaces a bit, since they are not
able to reference key of a specific value. Instead, what they would need is to
match any address-representing key, so that the dependency gets satisfied when
at least one of them exists for a given interface. With wildcards this could
be expressed as: "vpp/interface/address/<interface-name>/*.
The KVScheduler offers even more generic (i.e. expressive) solution than
wildcards: dependency expressed using callback denoted as AnyOf. The callback
is a predicate, returning true or false for a given key. The semantics is
similar to that of the wildcards. The dependency is considered satisfied, when
for at least one of the existing (configured/derived) keys, the callback returns
true.
Lastly, to allow a (to-be-)unnumbered interface to exist even if the IP address(es)
to borrow are not available yet, the call to turn interface into unnumbered is
derived from the interface value and processed by a separate descriptor:
UnnumberedIfDescriptor. It is this derived value that uses AnyOf callback
to trigger IP address borrowing only once the IP addresses become available.
For the time being, the interface is available at least in the L2 mode.
The example also demonstrates that when the borrowed IP address is being removed, the unnumbered interface will not get un-configured, instead it will only return the address before it is unassigned and turn back into the L2 mode.
Scenario: Create VPP interface via KVDB
This is the most basic scenario covered in the guide. On the other hand,
the attached control-flow diagram is the most verbose - it includes all the
interacting components (listed from the top to the bottom layer):
* NB (KVDB): contains configuration of a single my-tap interface
* Orchestrator: listing KVDB and propagating the configuration to the KVScheduler
* KVScheduler: planning and executing the transaction operations
* Interface Descriptor: implementing CRUD operations for VPP interfaces
* Interface Model: builds and parses keys identifying VPP interfaces
Scenario: Create VPP interface via GRPC
Variant of this example, with GRPC used as the NB interface for the agent instead of KVDB. The transaction control-flow is collapsed, however, since there are actually no differences between the two cases. For KVScheduler it is completely irrelevant how the desired configuration gets conveyed into the agent. The advantage of GRPC over KVDB is that the transaction error value gets propagated back to the client, which is then able to react to it accordingly. On the other hand, with KVDB the client does not have to maintain a connection with the agent and the configuration can be submitted even when the agent is restarting or not running at all.
Example: Route waiting for the associated interface
This example shows how route, received from KVDB before the associated
interface, gets delayed and stays in the PENDING state until the interface
configuration is received and applied. This is achieved simply by listing the
key representing the interface among the dependencies for the route.
