Test scenario definition

In the current release, it is possible to supply a configuration file in JSON format for running your tests.

In the following sections, we will provide a detailed explanation of the structure and all parameters in the JSON configuration file.

Structure

A quick short reference for the structure:

{
    "server": "<bb_server_name_or_ip:str>",
    "meeting_point": "<bb_meeting_point_name_or_ip:str>",
    "source": {
        "name": "<source_name:str>",
        "interface": "<bb_interface_name:str>",
        "uuid":"<endpoint_uuid:str>",
        "ipv4": "<ipv4_address:str>|dhcp|true",
        "netmask": "<ipv4_address:str>",
        "gateway": "<ipv4_address:str>",
        "nat": "<nat_resolution:bool>",
        "ipv6": "dhcp|slaac|true"
    },
    "destination": {
        "name": "<destination_name:str>",
        "interface": "<bb_interface_name:str>",
        "uuid":"<endpoint_uuid:str>",
        "ipv4": "<ipv4_address:str>|dhcp|true",
        "netmask": "<ipv4_address:str>",
        "gateway": "<ipv4_address:str>",
        "nat": "<nat_resolution:bool>",
        "ipv6": "dhcp|slaac|true"
    },
    "duration": "<trial_duration:int>",
    "max_iterations": "<max_iterations:int>",
    "frame_configs": [
        {
            "size": "<frame_length:int>",
            "initial_bitrate": "<initial_frame_rate:int>",
            "tolerated_frame_loss": "<frame_loss:float>",
            "accuracy": "<accuracy:float>",
            "expected_bitrate": "<expected_frame_rate:int>"
        }
    ]
}

JSON schema

The complete structure and documentation of the file is available in Configuration file JSON schema, and documented below.

Server

ByteBlower server address
Host name or IP address of the ByteBlower Server.
type	string
oneOf	format	hostname
	format	ipv4
	format	ipv6

Meeting point

ByteBlower meeting point address
Host name or IP address of the ByteBlower meeting point.
Added in version 1.0.0: Support for ByteBlower Endpoint.
type	string
oneOf	format	hostname
	format	ipv4
	format	ipv6

Ports

The "source" and "destination" blocks define ByteBlower ports (simulated hosts) at a given ByteBlower interface. Where a ByteBlower interface is the physical connection on either the non-trunking interface (direct connection at a ByteBlower server) or a trunking interface (connection at a physical port on the ByteBlower switch).

Each port supports IPv4 address assignment using static IPv4 configuration or DHCP. IPv6 address configuration is possible through DHCPv6 or Stateless autoconfiguration (SLAAC).

When an (IPv4) port is located behind a NAT gateway, then the test framework will perform additional setup flow steps to resolve the NAT’s public IPv4 address and UDP port (when required).

Since version 1.2.0, the ByteBlower Test Framework also supports ByteBlower Endpoints. They can be configured by:

Providing the meeting point address and the UUID of the endpoint.
Setting the IPv4 or IPv6 parameter to true.

Configuration parameters for a ByteBlower Port or Endpoint
Configuration parameters for a ByteBlower Port or Endpoint
type	object
properties
name	Name of this ByteBlower traffic endpoint.
	(typically `Source` for the tester entity, and `Destination` for the NUT).
	type	string
interface	Name of the physical ByteBlower interface where this ByteBlower port is created.
	ByteBlower Port only. Mutually exclusive with `"uuid"`.
	type	string
	pattern	^(nontrunk[-][0-9]+\|trunk[-][0-9]+[-][0-9]+)$
uuid	Device identifier of the ByteBlower Endpoint.
	ByteBlower Endpoint only. Mutually exclusive with `"interface"`. Note Using an Endpoint also requires the `"meeting_point"` to be provided and `"ipv4"` or `"ipv6"` set to `true` for this port configuration. Added in version 1.0.0: Support for ByteBlower Endpoint.
	type	string
	anyOf	format	uuid
		pattern	^[0-9a-zA-Z]$
vlans	List of VLAN (stack) configurations. See details in VLAN (stack) configuration.
	Note Not supported by ByteBlower Endpoint.
	type	array
	items	VLAN configuration
ipv4	IPv4 addressing method
	Enable IPv4 on this traffic endpoint.
	Supports the following values: IPv4 address for static IPv4 IP address (ByteBlower Port only). `"dhcp"` for dynamic IPv4 address allocation (ByteBlower Port only). `true`: To use the IPv4 address(es) of the system (ByteBlower Endpoint only). Default: `"dhcp"` (ByteBlower Port) or undefined (ByteBlower Endpoint). Note For ByteBlower Endpoint, you must specify either `"ipv4": true` or `"ipv6": true`. Changed in version 1.0.0: Support for ByteBlower Endpoint.
	oneOf	type	string
		oneOf	enum	dhcp
			format	ipv4
		type	boolean
netmask	Netmask of the IPv4 network.
	Default: Assigned by DHCP or `255.255.255.0` for static IPv4 address configuration. Note Will be supported for IPv6 once manual address configuration is supported. Not supported by ByteBlower Endpoint.
	type	string
	format	ipv4
gateway	(Default) Gateway of the IPv4 network.
	Default: Assigned by DHCP or none for static IPv4 address configuration. Note Will be supported for IPv6 once manual address configuration is supported. Not supported by ByteBlower Endpoint.
	type	string
	format	ipv4
nat	Define whether this port is located behind a NAT gateway.
	Defaults to `false`. Note Enabled by default for ByteBlower Endpoint.
	type	boolean
ipv6	IPv6 addressing method
	Enable IPv6 on this traffic endpoint.
	Supports the following values: `"dhcp"`: to let the port perform DHCPv6 (ByteBlower Port only) `"slaac"`: to let the port perform stateless address auto-configuration (ByteBlower Port only). `true`: To use the IPv6 address(es) of the system (ByteBlower Endpoint only). Default: undefined. Note For ByteBlower Endpoint, you must specify either `"ipv4": true` or `"ipv6": true`. Changed in version 1.0.0: Support for ByteBlower Endpoint.
	oneOf	type	string
		oneOf	enum	dhcp, slaac
			format	ipv6
		type	boolean

VLAN (stack) configuration

type	object
properties
protocol_id	VLAN protocol ID (TPID).
	Default value is: `"0x8100"` for a single VLAN configuration `"0x88a8"` for the outer VLAN(s) in a VLAN stack configuration `"0x8100"` for the most inner VLAN in a VLAN stack configuration. Note Configuration of the VLAN protocol ID (TPID) requires ByteBlower Server version >= 2.20.0.
	type	integer
	maximum	65535
	minimum	0
id	VLAN ID (12-bit; 0-4095)
	type	integer
	maximum	4095
	minimum	0
priority	3 bits defining the VLAN PCP bits. Default: 0
	type	integer
	maximum	3
	minimum	0
drop_eligible	1-bit flag. Default: 0
	type	integer
	maximum	1
	minimum	0

Frame configurations

This defines a list of frame configurations for each frame size. These configurations are related to the RFC 2544 throughput test specifications.

As stated in the RFC 2544 basic requirements, at least five frame sizes SHOULD be tested for at least 60 seconds per test trial. Also, frame size with the largest MTU supported by the protocol under test SHOULD be used.

List of frame size configurations to test
type	array
items	config-schema.json#/$defs/frame_config

type	object
properties
size	Frame length in Bytes without the FCS (Frame Check Sequence, 4 Bytes)
	type	integer
	minimum	42
initial_bitrate	Initial bitrate to be used in the first trial of the RFC 2544 throughput test.
	type	number
	exclusiveMinimum	0
tolerated_frame_loss	The maximum acceptable frame loss during the test.
	type	number
	minimum	0
accuracy	Defines the test accuracy (precision) at which the test should be ended.
	type	number
	minimum	0
expected_bitrate	The expected maximum Ethernet layer 2 speed for frames without the 4 bytes of FCS. Usually, it is around the maximum theoretical speed of the tested network.
	type	number
	minimum	0

Default frames configuration

Frame configurations can be removed entirely from the configuration file. In case no frame configuration is provided, default frame configurations in the byteblower.test_cases.rfc_2544.definitions module will be used.

[
    {
        "size": 60,
        "initial_bitrate": 3e8,
        "tolerated_frame_loss": 1e-3,
        "expected_bitrate": 3.7e8,
        "accuracy": 1e5
    },
    {
        "size": 124,
        "initial_bitrate": 6e8,
        "tolerated_frame_loss": 1e-3,
        "expected_bitrate": 4.5e8,
        "accuracy": 1e5
    },
    {
        "size": 252,
        "initial_bitrate": 8e8,
        "tolerated_frame_loss": 1e-3,
        "expected_bitrate": 5.7e8,
        "accuracy": 1e5
    },
    {
        "size": 508,
        "initial_bitrate": 8e8,
        "tolerated_frame_loss": 1e-3,
        "expected_bitrate": 6.6e8,
        "accuracy": 1e5
    },
    {
        "size": 1020,
        "initial_bitrate": 1e9,
        "tolerated_frame_loss": 1e-3,
        "expected_bitrate": 7.15e8,
        "accuracy": 1e5
    },
    {
        "size": 1276,
        "initial_bitrate": 1e9,
        "tolerated_frame_loss": 1e-3,
        "expected_bitrate": 7.25e8,
        "accuracy": 1e5
    },
    {
        "size": 1514,
        "initial_bitrate": 1e9,
        "tolerated_frame_loss": 1e-3,
        "expected_bitrate": 7.35e8,
        "accuracy": 1e5
    }
]

These values could be used as is. However, you preferably should change these default values to comply to your network and test specifications.

Other parameters

maximum_run_time

Maximum run time of a single test trial in seconds
type	number
exclusiveMinimum	0

max_iterations

Maximum number of iteration to test per frame size
type	integer
minimum	1

Configuration file example

Using ByteBlower Ports

{
    "$schema": "https://api.byteblower.com/test-framework/json/test-cases/rfc-2544/config-schema.json",
    "server": "byteblower-server.example.com.",
    "source": {
        "name": "Source",
        "interface": "nontrunk-1",
        "ipv4": "192.168.5.2",
        "netmask": "255.255.255.0",
        "gateway": "192.168.5.254"
    },
    "destination": {
        "name": "Destination",
        "interface": "trunk-1",
        "ipv4": "dhcp",
        "nat": true
    },
    "duration": 60,
    "max_iterations": 25,
    "frame_configs": [
        {
            "size": 60,
            "initial_bitrate": 3e8,
            "tolerated_frame_loss": 1e-4,
            "expected_bitrate": 1.7e8,
            "accuracy": 1e3
        },
        {
            "size": 124,
            "initial_bitrate": 6e6,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 4.5e8,
            "accuracy": 1e4
        },
        {
            "size": 252,
            "initial_bitrate": 8e6,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 5.7e8,
            "accuracy": 1e3
        },
        {
            "size": 508,
            "initial_bitrate": 8e8,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 6.6e8,
            "accuracy": 1e3
        },
        {
            "size": 1020,
            "initial_bitrate": 1e9,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 7.15e8,
            "accuracy": 1e2
        },
        {
            "size": 1276,
            "initial_bitrate": 1e9,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 7.25e8,
            "accuracy": 1e2
        },
        {
            "size": 1514,
            "initial_bitrate": 1e9,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 7.35e8,
            "accuracy": 1e2
        }
    ]
}

Using ByteBlower Endpoint

{
    "$schema": "https://api.byteblower.com/test-framework/json/test-cases/rfc-2544/config-schema.json",
    "server": "byteblower-server.example.com.",
    "meeting_point": "byteblower-meeting-point.example.com.",
    "source": {
        "name": "Source",
        "interface": "nontrunk-1",
        "ipv4": "192.168.5.2",
        "netmask": "255.255.255.0",
        "gateway": "192.168.5.254"
    },
    "destination": {
        "name": "Destination",
        "uuid": "4d9a5fdb-32b4-4523-ace9-5972518de13b",
        "ipv4": true
    },
    "duration": 60,
    "max_iterations": 25,
    "frame_configs": [
        {
            "size": 60,
            "initial_bitrate": 3e6,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 1.7e8,
            "accuracy": 1e4
        },
        {
            "size": 124,
            "initial_bitrate": 6e6,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 4.5e8,
            "accuracy": 1e4
        },
        {
            "size": 252,
            "initial_bitrate": 8e6,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 5.7e8,
            "accuracy": 1e3
        },
        {
            "size": 508,
            "initial_bitrate": 8e8,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 6.6e8,
            "accuracy": 1e3
        },
        {
            "size": 1020,
            "initial_bitrate": 1e9,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 7.15e8,
            "accuracy": 1e2
        },
        {
            "size": 1276,
            "initial_bitrate": 1e9,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 7.25e8,
            "accuracy": 1e2
        },
        {
            "size": 1514,
            "initial_bitrate": 1e9,
            "tolerated_frame_loss": 1e-3,
            "expected_bitrate": 7.35e8,
            "accuracy": 1e2
        }
    ]
}