Below, we follow [4] to give some formal definitions and the statement of the problem.

A time series T is a sequence of real-valued elements: T=(t1,…,tm) , ti∈R . The length of a time series is denoted by |T| .

A subsequence Ti,n of a time series T is its contiguous subset of n elements that starts at position i : Ti,n=(ti,…,ti+n−1) , 1≤n≪m , 1≤i≤m−n+1 . We denote the set of all subsequences of length n in T by STn . Let N denotes the number of subsequences in STn , i.e., N=|STn|=m−n+1 .

A distance function for any two subsequences is a nonnegative and symmetric function Rn×Rn→R .

Two subsequences Ti,n,Tj,n∈STn are non-trivial matches [6] with respect to a distance function Dist , if ∃Tp,n∈STn , i<p<j , and Dist(Ti,n,Tj,n)<Dist(Ti,n,Tp,n) . Let us denote a non-trivial match of a subsequence C∈STn by MC .

A subsequence D∈STn is said to be the most significant discord in T if the distance to its nearest non-trivial match is the largest. That is,

(1) \min \left( {{\rm Dist}\left( {C,{M_C}} \right)} \right).$$]]>∀C∈STnmin(Dist(D,MD))>min(Dist(C,MC)).

A subsequence D∈STn is called the most significant k -th discord in T if the distance to its k -th nearest non-trivial match is the largest.

Given a positive parameter r , the discord at a distance at least r from its nearest neighbor is called the range discord, i.e., for discord D min(Dist(D,MD))≥r .

DADD, the original serial disk-based algorithm [4] addresses discovering range discords, and provides researchers with a procedure to choose the r parameter. To accelerate the above-mentioned procedure, our parallel algorithm [5] for many-core accelerators can be applied, which discovers discords for the case when time series fit in the main memory.

When computing distance between subsequences, DADD demands that the arguments have been previously z-normalized to have mean zero and a standard deviation of one. Here, z -normalization of a subsequence C∈STn is defined as a subsequence C^=(c^1,…,c^n) in which

(2) c^i=ci−μσ:μ=1n∑i=1n⁡ci,σ=1n∑i=1n⁡ci2−μ2.

In the original DADD algorithm, the Euclidean distance is used as a distance measure yet the algorithm can be utilized with any distance function, which may not necessarily be a metric [4]. Given two subsequences X,Y∈STn , the Euclidean distance between them is calculated as

(3) ED(X,Y)=∑i=1n⁡(xi−yi)2.

The DADD algorithm [4] performs in two phases, namely the candidate selection and discord refinement, with each phase requiring one linear scan through the time series on disk. Algorithm 1 depicts a pseudo code of DADD (up to the replacement of the Euclidean distance by an arbitrary distance function). The algorithm takes time series T and range r as an input and outputs set of discords C . For a discord c∈C , we denote the distance to its nearest neighbor as c.dist .

At the first phase, the algorithm scans through the time series T , and for each subsequence s∈STn it validates the possibility for each candidate c already in the set C to be discord. If a candidate c fails the validation, then it is removed from this set. In the end, the new s is either added to the candidates set, if it is likely to be a discord, or it is discarded. The correctness of this procedure is proved in Yankov et al. [4].

At the second phase, the algorithm initially sets distances of all candidates to their nearest neighbors to infinity. Then, the algorithm scans through the time series T , calculating the distance between each subsequence s∈STn and each candidate c . Here, when calculating ED(s,c) , the EarlyAbandonED procedure stops the summation of ∑k=1n⁡(sk−ck)2 if it reaches k=ℓ , such that 1≤ℓ≤n for which ∑k=1ℓ⁡(sk−ck)2≥c.dist2 . If the distance is less than r then the candidate is false positive and permanently removed from C . If the above-mentioned distance is less than the current value of c.dist (and still greater than r , otherwise it would have been removed) then the current distance to the nearest neighbor is updated.

To provide parallelism at the level of the cluster nodes, we perform time series partitioning across the nodes as follows. Let P be the number of nodes in the cluster, then k -th partition ( 0≤k≤P−1 ) of the time series is defined as Tstart,len where

(4) start=k⋅⌊NP⌋+1;len={⌊NP⌋+n−1+(NmodP),k=P−1⌊NP⌋+n−1,otherwise..

This means the head part of every partition except the first overlaps with the tail part of the previous partition in n−1 data points. Such a technique prevents us from a loss of the resulting subsequences in the junctions of two neighbor partitions. To simplify the presentation of the algorithm, hereinafter in this section, we use symbol T and the above-mentioned related notions implying a partition on the current node but not the whole input time series.

The time series partition is stored as a matrix of aligned subsequences to enable computations over aligned data with as many auto-vectorizable loops as possible. We avoid the unaligned memory access since it can cause an inefficient vectorization due to time overhead for the loop peeling [22].

Let us denote the number of floats stored in the VPU (vector processing unit of the many-core accelerator) by widthVPU . If the discord length n is not a multiple of widthVPU , then each subsequence is padded with zeroes where the number of zeroes is calculated as pad=widthVPU−(nmodwidthVPU) . Thus, the aligned (and previously z-normalized) subsequence T~i,n is defined as follows:

(5) }\;0} \cr {{\hskip 40pt{\tilde T}_{i,n}},} {\hskip -50pt otherwise.} \cr } } \right..$$]]>T~i,n={(T~i,n,0,…,0⏞pad),ifnmodwidthVPU>0T~i,n,otherwise..

The subsequence matrix STn∈RN×(n+pad) is defined as

(6) STn(i,j)=t~i+j−1.

The parallel algorithm employs the data structures depicted in Fig. 1. Defining structures to store data in the main memory of a cluster node, we suppose that each structure is shared by all threads the algorithm is running on, and each thread processes its own data segment independently. Let us denote the amount of threads employing by the algorithm on a cluster node by \bip , and let \biiam ( 0≤\biiam≤\bip−1 ) denotes the number of the current thread.

Set of discords C is implemented as an object with two basic attributes, namely candidate index and candidate body, to store indices of all potential discord subsequences and their values themselves, respectively.

Let us denote a ratio of candidates selected at a cluster node and all subsequences of the time series by ξ . The exact value of the ξ parameter is a subject of an empirical choice. In our experiments, ξ=0.01 was enough to store all candidates. Thus, we denote the number of candidates as L=⌈ξ⋅N⌉ and assume that L≪N .

The candidate index is organized as a matrix C.index∈Np×L , which stores indices of candidates in the subsequence matrix STn found by each thread, i.e., i -th row keeps indices of the candidates that have been found by i -th thread. Initially, the candidate index is filled by NULL values.

To provide a fast access to the candidate index during the selection phase, it is implemented as a deque (double-ended queue) with three attributes, namely count , head , and tail . The deque count is an array C.count∈Np , which for each thread keeps the amount of non-NULL elements in the respective row of the candidate index matrix. The deque head and tail are arrays C.head and C.tail∈Np , respectively, which represent the second-level indices that for each thread keep a number of column in C.index with the most recent NULL value, and with the least recent non-NULL value, respectively.

Let H ( H<L≪N ) be the number of candidates selected at a cluster node during the algorithm’s first phase. Then the candidate body is the matrix C.cand∈RH×n , which represents the candidate subsequences itself. The candidate body is accompanied by an array C.pos∈NH , which stores starting points of candidate subsequences in the input time series.

After the selection phase, all the nodes exchange the candidates found to construct the combined candidate set, so at each cluster node the candidate body will contain potential discords from all the nodes. At the second phase, the algorithm refines the combined candidate set comparing the parameter r and distances between each element of the candidate body and each element of the subsequence matrix.

To parallelize this activity, we process rows of the subsequence matrix in the segment-wise manner and employ an additional attribute of the candidate body, namely bitmap. The bitmap is organized as a matrix C.bitmap∈Bp×H , which indicates the fact that an element of the candidate body has been successfully validated against all elements in a segment of the subsequence matrix. Thus, after the algorithm’s second phase, i -th element of the candidate body is successfully validated if ∧ℓ=1pC.bitmap(ℓ,i) is true.

In the implementation, we apply the following parallelization scheme at the level of the cluster nodes. Let the input time series T be partitioned evenly across P cluster nodes. Each node performs the selection phase on its own partition with the same threshold parameter r and produces distinct candidate set Ci .

Next, as opposed to MR-DADD [4], each node refines its own candidate set Ci with respect to the r value. Indeed, a candidate cannot be the true discord if it is pruned in the refinement phase in at least one cluster node. Thus, by the local refinement procedure, we try to reduce each candidate set Ci and, in turn, the combined candidate set CP=∪i=1PCi. In the experiments, this allows us to reduce the size of the combined candidate set at times.

Then the combined candidate set CP is constructed and transmitted to each cluster node. Next, a node refines CP over its own partition, and produces the result Ci . Finally, the true discords set is constructed as CP=∩i=1PCi .

The parallel implementation of the candidate selection and refinement phases is depicted in Algorithm 2 and Algorithm 3, respectively. To speed up the computations at a cluster node, we omit the square root calculation since this does not change the relative ranking of the candidates (indeed, the ED function is monotonic and concave).

In the selection phase, we parallelize the outer loop along the rows of the subsequence matrix while in the inner loop along the candidates, each thread processes its own segment of the candidate index. By the end of the phase, the candidates found by each thread are placed into the candidate body, and all the cluster nodes exchange the resulting candidate bodies by the MPI_Send and MPI_Recv functions to form the combined candidate set, which serves as an input for the second phase.

In the refinement phase, we also parallelize the outer loop along the rows of the subsequence matrix, and in the inner loop along the candidates, each thread processes its own segments of the candidate body and bitmap. In this implementation, we do not use the early abandoning technique for the distance calculation relying on the fact that vectorization of the square of the Euclidean distance may give us more benefits. By the end of the phase, the column-wise conjunction of the elements in the bitmap matrix will result in a set of true discords found by the current cluster node. An intersection of such sets is implemented by one of the cluster nodes where the rest nodes send their resulting sets.

In the first series of the experiments, we assessed the algorithm’s scaled speedup, which is defined as the speedup obtained when the problem size is increased linearly with the number of the nodes added to the computer cluster [25]. Being applied to our problem, the algorithm’s scaled speedup is calculated as

(7) sscaled=n⋅P⋅|C(P⋅m)|tP⋅(P⋅m),

where n is the discord length, P is the number of the cluster nodes, m is a factor of the time series length, C(P⋅m) is a set of all the candidate discords selected by the algorithm at its first phase from a time series of length P⋅m and tP⋅(P⋅m) is the algorithm’s run time when the time series is processed on P nodes.

For the evaluation, we took ECG time series [26] (see Tab. 1 for the summary of the data involved). In the experiments, we discovered discords on up to 128 cluster nodes with the time series factor m=106 , and varied the discord’s length n while the range parameter r was chosen empirically to provide the algorithm’s best performance.

The results of the experiments are depicted in Fig. 2. As can be seen, our algorithm adapts well to increasing both the time series length and number of cluster nodes, and demonstrates the linear scaled speedup. As expected, the algorithm shows a better scalability with larger values of the discord length because this provides a higher computational load.

In the second series of the experiments, we compared the performance of our algorithm against the analogs we have already considered in Section 3, namely DDD [15], MR-DADD [4], GPU-STAMP [19], and MP-HPC [21]. We omit the PDD algorithm [18] since in our previous experiments [5], PDD was substantially far behind our parallel in-memory algorithm due to the overhead caused by the message passing among the cluster nodes.

Throughout the experiments, we used the synthetic time series generated according the Random Walk model [27] as that ones were employed for the evaluation by the competitors. For comparison purposes, we used the run times reported by the authors of the respective algorithms. To perform the comparison, we ran our algorithm on Tornado SUSU with a reduced number of nodes and cores at a node to make the peak performance of our hardware platform approximately equal to that of the system on which the corresponding competitor was evaluated.

Tab. 2 summarizes the performance of the proposed algorithm compared with the analogs. We can see that our algorithm outruns its competitors. As expected, direct analogs DDD and MR-DADD are inferior to our algorithm since they do not employ parallelism within a single cluster node. Additionally, indirect analogs GPU-STAMP and MP-HPC are behind our algorithm since they initially aim to solve a computationally more complex problem of computing the matrix profile, which can also be used for discords discovery among many other time series mining problems.