The paper proposes an advanced model of routing that allows balancing the load on the criteria of minimum/maximum network channel load and service quality for each flow in SDN. There is developed a technique for classifying IoT flows with the recommendation on the importance of certain parameters for particular classes of traffic taken into account. Tab. 2 show the requirements for QoS of existing services operating within the Internet of Things [48] (IoT automated emergency call—A, Real time Monitoring temperature—B, Real time Internet of Video Things (IoVT)—C, IoT-Alert (Photo/text/Email)—D, Un Real time IoVT (Video on Demand)—E, Un Real time IoVT (720p60)—F, Un Real time IoVT (1080p60)—G). The primary goal of QoS is to provide priority, including dedicated throughput—C, delay—T, jitter—J, packet loss—P (required by some real-time and non real-time traffic). We also introduce two additional criteria for the classification of IoT flows the first criterion is the sensitivity to the mixing of packet—P_m and the second criterion is the priority of the IoT subscribers–C_p.

To calculate the relative priority of a flow for a particular path, we defined the relative coefficients for each flow. The relative coefficients are the ratio of the minimum values of the service quality parameters to the current values obtained from the network monitoring.

The relative coefficient of packet loss—p, packet delay—t, packet jitter—j, throughput—c, sensitivity to the mixing of packets—pm and priority of the subscribers—cp are defined as shown in Eq. (1):

(1) p=PminP,t=TminT,j=JminJ,c=CCmax,pm=PmminPm,cp=CpCpmax.

The results of the calculations are shown in Tab. 3.

Tab. 4 is filled with numbers 1–3, which correspond to low, medium, and high significance of the QoS parameters, respectively [49].

These parameters can be specified by the operator. Moreover, each subscriber is assigned a priority for each type of traffic. If the priority is not explicitly specified in the service agreement, then the IoT subscribers are assigned the lowest priority of all possible by default.

Thus, the relative priority for each category of IoT applications (services) is determined by Eq. (2).

(2) PIoTk=∑m=14XkmYkm∑k=17∑m=14XkmYkm

where PIoTk—the relative priority of the i-th IoT service;

k—IoT service type number;

m—number service quality parameter;

X_km—relative priority of parameter m for IoT service k;

Y_km—the importance of parameter m for IoT service k.

The final result of this formula is a fraction within the range from zero to one. The greater value, the higher priority of an IoT flow. The formula can be applied to any number of flows and various quality requirements for the IoT service.

Considering above technique of prioritizing IoT flows, we divide them into three categories. The first type flows of should be delivered optimally with respect to the criterion of cost, which considers the following quality of service parameters: delay and jitter. Such flows are very sensitive to delay and jitter in the mission-critical IoT applications, so multi routing is forbidden for them, that is the whole flow can be transmitted one path only. The second type flows are less sensitive to the time parameters of the QoS, and therefore it is allowed to balance such flows. However, when balancing, there is a limit on the minimum sub-flow size. This constrain is essential because the second kind flows are usually TCP flows. They are sensitive to loss and mixing of packets, and as the number of sub-flows increases their size decreases, so the probability of loss and mixing due to multi routing increases. In this case, the delivery of the second types flows is guaranteed with QoS parameters within acceptable limits. The third type flows can be delivered by any routing with no guarantee of quality of service. Therefore, such flows are used to load low-load paths and to balance the load distribution between channels in a network.

When the network is operating in normal mode, without overload or bottlenecks, the monitoring system polls the network devices at a consistently long interval. In particular, the monitoring system polls network interface loads, switch flow table loads, and switch central input buffer loads that identify interface overloads. The monitoring system polls nodes with a certain interval of time and monitors the time parameters of the flows of the first category. In case of average growth in interface load, transmission delay and jitter, the system switches to the state of close monitoring of the specified interface and increases the intensity of delay measurement. In addition, when the interface load level reaches 0.9, the system starts polling the device buffer loads.

The developed monitoring system facilitates the implementation of this approach through adaptive monitoring of the used resources of channels and devices. The search for the optimum path is based on the mathematical theory used to solve the two well-known problems: the Multi-Commodity Flow Problem and the Constrained Shortest Path Problem. The main goal is to find the optimum set of paths in a network for all flows with a minimum total cost. The main constrain is that the total flow rate through a channel cannot exceed channel throughput.

Our model combines both of the abovementioned problems and allows finding the shortest path for each flow, subject to the given constraints. Suppose we have a network of nodes where each channel (link) is characterized by a delay, a loss and a channel throughput. In addition, each path is characterized by the cost calculated as a weighted sum of delays and packet losses. The model assumes that for each type of service/traffic there is a set of flows that can have completely different input and output nodes in the network. The goal is to find a set of optimum network paths for each flow with the minimum cost subjected to certain constraints, in particular such as the maximum delay, packet loss, and available link throughput.

Suppose a network is represented by the graph G = (V, E), where V is the set of nodes, and E is the set of arcs between each pair of nodes. The arcs, that is, the links, are characterized by available channel throughput, delays, packet loss, and the cost of transfer per unit flow. As a result, the cost can be calculated by following Eq. (3):

(3) Wij=θ⋅dij+ξ⋅pij,∀(i,j)∈E

where θ and ξ—the weighs for delay and loss, respectively.

This formula allows adjusting the cost of the path with respect to either delay or loss of packets for a particular flow. This allows adjusting these settings to meet the quality requirements for each flow. For example, multimedia traffic has restrictions on end-to-end delays, so one can set θ=1 and ξ=0 to take into account the delay only. Each individual flow k is identified by a relative priority PIoTk, whose value lies within the range from zero to one. The set of different IoT traffic flows to be transferred over the network is denoted by K. Each flow is characterized by five parameters:

Gxk—the node (host) where the k-th flow enters the network;

Gyk—the node (host) where the k-th flow leaves the network;

fk—the rate of the k-th flow that must be delivered from the input node to the output one;

Pmaxk≥0—the maximum allowed packet loss for the k-th flow;

Dmaxk≥0—the maximum allowed delay for the k-th flow.

The goal of optimization is to path all flows in the network through the paths with the minimum cost.

Sets:

nodes: n∈V;

arcs: (i,j)∈E;

channels (links): (i,j)∈E∪(j,i)∈E;

Variables:

0≤xijk≤fk—the data volume of the k-th flow passing through the link (i,j).

Parameters:

bij≥0—available link throughput(i,j);

0≤ρ≤1—the maximum channel load (link utilization) in the network;

0≤r≤1—relative cost priority over the maximum channel load ρ;

Cmax≥0—the maximum link throughput;

cij≥0—channel cost (i,j) calculated as θ⋅dij+ξ⋅pij;

θ—weight for delay;

ξ—weight for packet loss;

Gxk∈V—source of the k-th flow;

Gyk∈V—destination of the k-th flow;

fk—the rate of the k-th flow;

pij≥0—actual packet loss in the link (i,j);

dij≥0—actual link delay (i,j);

Bk≥0—the throughput required for the k-th flow.

The routing parameters are controlled based on the relative priority of a flow, which is calculated according to the technique presented above in this section. All values of the relative priority PIoTk of a flow are within the range from 0 to 1. Let us introduce the parameters TCHighPriority∈TC and TCBestEffort∈TC, with the condition TCHighPriority<TCBestEffort. The parameter TCHighPriority contains the relative priority which the relative priorities PIoTk of all other flows are compared with. If the relative priority of a certain flow is greater than TCHighPriority, then the flow belongs to the first category and has high priority. This means it cannot be split into sub-flows for load balancing. If the relative priority of the flow is less than TCHighPriority, however larger than TCBestEffort, then the flow belongs to the second category and has an average priority. This means the flow can be split into sub-flows and choose for them the paths with the quality of service no lower than the shortest available path. Such flows have a limitation that determines the minimum volume of a sub-flow. This allows controlling the levels of redistribution of sub-flows in the network to avoid packet mixing for the TCP flows. The quality of service is not guaranteed for all the flows with the priority lower than TCBestEffort. They can be split into sub-flows of arbitrary volume. For these sub-flows, the paths with minimum channel load are selected.

The objective function (4) minimizes cost with respect to the parameters of the quality of service in links, as well as with respect to the coefficient of the minimum/maximum load of links, which depends on the type of relative priority of a flow.

(4) F(x)=ρ+r∑(i,j)∈E∑k∈Kwijxijk→min,

Constrain in Eq. (5) corresponds to the conservation of the flow

(5) ∑(i,j)∈Exijk−∑(i,j)∈Exjik={fk,i=sk,−fk,i=tk, ∀i∈V, ∀k∈K0,i≠sk, tk,

Constrain in Eq. (6) takes into account the allowed level of flow distribution with respect to its relative priority:

(6) xijk≥{BH,PIoTk>TCHighPriority, ∀(i,j)∈E,BL,TCBestEffort<PIoTk<TCHighPriority, ∀(i,j)∈E.0,PIoTk<TCBestEffort, ∀(i,j)∈E,

where BL and BH are the minimum link throughput of sub-flows when balancing the main flow of the second and third category, respectively.

The following two constraints in Eqs. (7) and (8) consider the maximum allowed E2E packet loss and delay, which should not exceed critical values Dmaxk, Pmaxk, for a flow with relative priority k. In the case of routing of the third category, these parameters are given the maximum possible value:

(7) ∑(i,j)∈Epij≤{Pmaxk,xijk>0, ∀k∈K;0,xijk=0, ∀k∈K;

(8) ∑(i,j)∈Edij≤{Dmaxk,xijk>0, ∀k∈K;[8pt]0,xijk=0, ∀k∈K;

Inequality in Eq. (9) imposes a constrain on the available throughput of each link, taking into account all the flows k through these links. Moreover, this constrain is lower in case of routing flows of the second and the third categories, which causes their transmission in ways with low loading efficiency and with poor quality of service:

(9) ∑k∈Kxijk≤{ρ⋅Cmax,∀(i,j)∈E, k<TCHighPrioritybij,∀(i,j)∈E

The last constrain in Eq. (10) establishes the range of the variable and ensures that the variable acquires values within the flow rate:

(10) 0≤xijk≤fk∀(i,j)∈E,∀k∈K,

The proposed mathematical model allows setting the maximum allowed packet loss and delay for the first category flows. These values are essential for choosing the best path for the QoS criterion. By using weights, one can match the weight of each flow based on its requirements. On the one hand, one can set the maximum allowed loss and delay and, as a result of solving a linear programming problem, obtain a path with the minimum transfer cost that meets the requirements of the QoS parameters. On the other hand, one can change the maximum allowed packet loss and delay, then recalculate the optimum paths to find the path that ensures the required quality.

Let us analyze the structure of the communication network that consists of the N network devices. Numeric values of the metric W(i,j) can be presented as adjacency matrix W:

(11) W=(w1,1⋯w1,N⋮⋱⋮wN,1⋯wN,N),

It is worth mentioning that the matrix of parameters W is not symmetric (W(i,j)≠W(j,i)). Minimization of target function metric W(i,j) that can be under several constraints or limits is called multi criteria optimization. The main difficulty encountered in solving problems of this type is the ambiguity of the optimal solution at the point where one of the criteria reaches its maximum, while the other may be far from not only the maximum, but even from any arbitrarily admissible value. Finding a metric by the “ideal point” method requires that all values have the same dimensionality. To do this, let us model the channel characteristics as follows:

for the number of lost packets

(12) Pi,j=min,Pi,j=(1−Ni,jMi,j),

where Ni,j is a number of received packets, Mi,j is a number of sent packets and Mi,j>0

for the flow delay

(13) Di,j=min,Di,j=(1−dmindi,j),

Thus, the metric based on the two parameters can be calculated as:

(14) Wi,j=θ(Di,j)2+ξ(Pi,j)22.

where θ and ξare the weight coefficients that change their values in the range between 0 and 1, and their sum must be equal to 1:

Changing the values of weight coefficients in the metric W(i,j), we create an apparatus for controlling the value of a particular parameter when evaluating the channel characteristic from node i to node j.

In terms of mathematics, the “ideal point” is the one that has coordinates that correspond to Dmin,Pmin, under the fixed weight coefficients θ and ξ.

This part describes the implementation of our novel centralized routing model based on the multiple QoS requirements and IoT priority flow approach described in Section 3.1. The mathematical routing model is implemented as an external module on top of the Floodlight Open SDN Controller. This module is implemented in Java Programming Language. To solve the problem of finding the optimum path in the SDN, we also used AMPL (A Mathematical Programming Language) and the CPLEX optimizer [50].

For our experimental tests, we developed a network topology in the Mininet emulator. This topology composed of 7 Open vSwitches, 1 the Floodlight SDN controller and 6 IoT traffic generators (G1–G6). Link throughput between all nodes for all ports is set equal to 100 Mbps. The experimental SDN testbed is depicted in Fig. 2.

For the first time we tested the average packet delay vs. link utilization of the developed SDN switch ports (Fig. 3). For this we also used a traffic generator, iperf, to generate traffic and transfer from host to client. The load was generated from 10 to 100 Mbp in 10 Mbps steps.

During the experiment, IoT traffic was generated in the network using a multi-service traffic generation system proposed in paper [51]. The matrix of requirements C_ij for link throughput in Mbps between nodes is given below.

(15) Cij=1234561012,516,5597211012,52014133516,502112,6541014,51201311581868015612891550

Node No 7 is intermediate (it represents the aggregation level), so no IoT devices (servers) are connected to it. According to the matrix of requirements, the list of flows for all traffic categories was generated. A set of subscribers and the IoT services they use are generated for the channel (link) throughput of 100 Mbps. The list of flows for 9 subscribers is given in Tab. 5 (C is the link throughput, Mbps; Cp is the priority of IoT subscribers, PIoTk is the relative priority of the i-th IoT service).

Each subscriber uses a specific set of IoT services. In case an empty cell at the intersection of a column and a row, this service should be considered as the one with no guarantee of quality of service if a subscriber uses it. All other IoT services have throughput requirements and service parameters for a particular user secured by a service agreement between a subscriber and the network operator. As can be seen from Tab. 4, each subscriber is assigned a priority within the appropriate priority range for a particular type of service. This generation is carried out for each pair of servers used to generate subscribers’ load.

The next step was to calculate the relative priorities of IoT flows, taking into account the classification of service categories proposed in Section 3.1 of the paper.

For the first category of services (mission-critical IoT applications), which determines real-time flows that are extremely sensitive to fluctuations of time belongs the following ones: IoT automated emergency call (A), monitoring temperature (B), real time IoVT (C).

The results of the calculation of the relative priorities for all the subscribers according to the technique improved in the 3.1 section of this work, are given in Tab. 5.

We have compared the proposed routing model with the deterministic multiconstrained centralized QoS routing model (DMCQR) presented in work [45].

To verify the effectiveness of the proposed solutions for mission-critical IoT applications at high network loads was used real time IoVT (Internet of Video Things) service, broadcasting video from the server G6. The network is filled with IoT traffic in accordance with the requirements given in the above matrix generated by the IoT multiservice traffic generators G1 to G6 connected to 6 OvS switch.

As a result of the functioning of DMCQR, the network was filled with background multiservice traffic generated by IoT devices. All paths for all requirements are given in Tab. 6.

The next step was to generate load with only the real time IoVT service from the G6 IoT traffic generator. As a result of operation of the DMCQR, the paths for IoVT flows are selected as shown in Tab. 7.

The network awareness module can use OpenFlow to send HTTP requests in order to obtain real-time link throughput usage. The histogram of link utilization was obtained when using the DMCQR and is depicted in Fig. 4.

As a result of routing with the DMCQR model, unbalanced loading of links occurred in the network. Particular attention should be focused on link 4-6 which has significant delays and a high probability of packet loss due to insufficient link throughput. The links 5–6 and 6–7 are in a state near overload with a high probability of delays for the real time IoVT flow. In addition, there are no alternative transmission paths with sufficient throughput for a 9 Mbps flow from G6 to G4 hosts. Therefore, according to the DMCQR approach, the optimal path for an IoVT flows of 9.05 Mbps from G6 to G4 on the basis of QoS criterion is path 6–4.

We have a detailed analysis of the IoT flows leading to overload of link 4–6 according to the DMCQR model. The total load from the background multi-service IoT traffic according to Tabs. 5 and 6 is calculated as follows.

f(G2→G5)=F3+B2+F4+D2+D7=3.99+1.01+5.5+1.6+1.9=14 [Mbps]f(G4→G5)=F2+C5+D5+A2+D9+D2=5.57+2.15+1.68+0.19+1.81+1.6=13 [Mbps]f(G4→G6)=G7+A2+A3+C6+A6=8.3+0.19+0.16+2.17+0.18=11 [Mbps]f(G5→G2)=G8+C9+B8+B3+A5=12.98+2.42+1.09+0.16+1.17+0.18=18 [Mbps]f(G5→G4)=F7+A3+B8+B4=5.5+0.16+1.09+1.25=8 [Mbps]f(G6→G2)=G6+B3+A6+A3=10.49+1.17+0.18+0.16=12 [Mbps]f(G6→G4)=G4+F3+A2+A4=10.63+3.99+0.19+0.19=15 [Mbps]

The load in link 4-6 generated by background IoT traffic and the load in channel 4–6 generated by background IoT traffic and additional real-time IoVT traffic are depicted in the Figs. 5 and 6 respectively.

The total load from the real time IoVT service generated by IoT traffic generator G6 according to the Tabs. 4 and 6 is calculated as follows.

f ( G 6 → G 1 ) = C 1 + C 2 + C 3 + C 4 + C 5 + C 6 + C 7 + C 8 =1.91+2.09+2.13+2.21+2.15+2.17+1.95+2.33+2.42=19.36 [ Mbps ] f ( G 6 → G 2 ) = C 2 + C 3 + C 4 + C 5 + C 6 + C 7 + C 8 =2.09+2.13+2.21+2.15+2.17+1.95+2.33+2.42=17.45 [ Mbps ] f ( G 6 → G 3 ) = C 4 + C 6 + C 8 + C 9 =2.21+2.17+2.33+2.42=9.13 [ Mbps ] f ( G 6 → G 4 ) = C 3 + C 6 + C 7 + C 8 =2.13+2.17+2.33+2.42=9.05 [ Mbps ] f ( G 6 → G 5 ) = C 1 =1.91 [ Mbps ]

According to the analysis we can see that the load in link 4–6 is created by multi-service IoT traffic. This multiservice traffic also includes the real time IoT flows (IoT automated emergency call (A), monitoring temperature (B), real time IoVT (C)) for which it is necessary to provide the allowable E2E delay. The E2E delay of IoT flows passing through a link can be defined by Eq. (14).

(16) DE2E=∑(i,j)∈Edij.

We estimated the E2E delay for background multi-service IoT traffic passing through overloaded link 4-6 as follows.

DE2E(G2→G5)=d24+d46+d65=24.5+200+35.6=260.1 [ms]

DE2E(G4→G5)=d46+d65=200+35.6=235.6 [ms]

DE2E(G4→G6)=d46=200 [ms]

DE2E(G5→G2)=d56+d64+d42=35.6+200+24.5=260.1 [ms]

DE2E(G5→G4)=d56+d64=35.6+200=235.6 [ms]

DE2E(G6→G2)=d64+d42=200+24.5=224.5 [ms]

DE2E(G6→G4)=d64=200 [ms]

And we estimated the E2E delay for real time IoVT flows generate from G6 as follows.

DE2E(G6→G1)=d67+d71=34.4+35.6=70 [ms]

DE2E(G6→G2)=d67+d71+d12=34.4+35.6+27=97 [ms]

DE2E(G6→G3)=d65+d53=35.6+11.2=46.8 [ms]

DE2E(G6→G4)=d64=200 [ms]

DE2E(G6→G5)=d65=35.6 [ms]

According to the proposed routing model to prevent overload on channel 4-6, we proposed to use path 4-7-3-5 for non real-time IoT flows f(G4→G5)realtime=F2+D5+D9+D2=5.57+1.68+1.81+1.6=10.66 [Mbps] and to use path 4-6-5 for real-time flows f(G4→G5)unrealtime=C5+A2=2.15+0.19=2.34 [Mbps] generated from host G4 to G5.

The histogram of link utilization was obtained when using the proposed routing model and is depicted in Fig. 7.

We estimated the E2E delay of IoT flows passing through overloaded link 4-6 according to the proposed routing model.

The E2E delay for background multi-service IoT traffic is as follows.

DE2E(G2→G5)=d24+d46+d65=24.5+29.5+26=80 [ms]

DE2E(G4→G5)=d46+d65=29.5+26=55.5 [ms]

DE2E(G4→G6)=d46=29.5 [ms]

DE2E(G5→G2)=d56+d64+d42=26+29.5+24.5=80 [ms]

DE2E(G5→G4)=d56+d64=26+29.5=55.5 [ms]

DE2E(G6→G2)=d64+d42=29.5+24.5=54 [ms]

DE2E(G6→G4)=d64=29.5 [ms]

The E2E delay for real time IoVT flows generate from G6 is as follows.

DE2E(G6→G1)=d67+d71=34.4+35.6=70 [ms]

DE2E(G6→G2)=d67+d71+d12=34.4+35.6+27=97 [ms]

DE2E(G6→G3)=d65+d53=26+13=39 [ms]

DE2E(G6→G4)=d64=29.5 [ms]

DE2E(G6→G5)=d65=26 [ms]

Comparison of average E2E delay of the DMCQR model with our proposed routing model is depicted in Fig. 8.

In general, the above results show that the routing model proposed in this paper provides an acceptable E2E QoS guarantee for all mission-critical IoT applications, in contrast to the existing DMCQR routing model that has been developed by authors in paper [41].