Research on distributed data stream mining algorithms based on matrix weighted association rules

Abstract

In order to overcome the low efficiency of traditional data mining algorithms without considering weighted association rules, this paper proposes a distributed data flow mining algorithm based on matrix weighted association rules. According to the way of separating metadata and data flow, garbage data processing in data flow is realized. By using sliding window and data summary structure to optimize PCA algorithm, the main component decision matrix is formed in the window, and the dimension of data in sliding window is reduced by using the decision matrix. The matrix weighted association rules are used to mine the distributed data. After dimensionality reduction, the transactions in the database are clustered according to the time distribution. The weighted analysis is carried out for each aggregation to obtain the weighted frequent item set with time and output the mining results. The experimental results show that the proposed algorithm has high efficiency and the highest accuracy of 98.9%.

Keywords

Matrix weighted association rules data flow mining data dimension reduction

Get full access to this article

View all access options for this article.

References

Chao,

Lin,

Fanget al., Data-driven affinely adjustable distributionally robust unit commitment, IEEE Transactions on Power Systems33(2) (2018), 1385–1398. doi:10.1109/TPWRS.2017.2741506.

F.M.

Chen,

D.Z.

Han,

Biet al., Key technologies of distributed data stream processing based on big data, Journal of Computer Applications37(3) (2017), 620–627. doi:10.3724/SP.J.1087.2012.00620.

Dan,

Han,

Wanget al., Achieving load balance for parallel data access on distributed file systems, IEEE Transactions on Computers67(3) (2018), 388–402. doi:10.1109/TC.2017.2749229.

Ercolani and

Castelli, Variational assimilation of streamflow data in distributed flood forecasting, Water Resources Research53(1) (2017), 158–183. doi:10.1002/2016WR019208.

Jiang,

Zhang and

Hui , Towards secure data distribution systems in mobile cloud computing, IEEE Transactions on Mobile Computing16(11) (2017), 3222–3235. doi:10.1109/TMC.2017.2687931.

K.W.

Lin,

S.H.

Chung and

C.C.

Lin, A fast and distributed algorithm for mining frequent patterns in congested networks, Computing98(3) (2016), 235–256. doi:10.1007/s00607-015-0457-6.

Q.L.

Lin and

S.Z.

Yu, A distributed green networking approach for data center networks, IEEE Communications Letters21(4) (2017), 797–800. doi:10.1109/LCOMM.2016.2642188.

Lin, Research on data mining of valuable information in unstructured network, Computer Simulation34(2) (2017), 414–417.

L.L.

Lu,

Y.P.

Zhang,

H.Y.

Tan et al., Research on classification algorithm and concept drift based on big data, Journal of Frontiers of Computer Science & Technology10(12) (2016), 1683–1692.

10.

Misaki,

Tsuda,

Inoueet al., Distributed database and application architecture for big data solutions, IEEE Transactions on Semiconductor Manufacturing30(4) (2017), 328–332.

11.

D.G.

Murray,

Barham,

Barhamet al., Incremental, iterative data processing with timely dataflow, Communications of the ACM59(10) (2016), 75–83. doi:10.1145/2983551.

12.

Y.G.

Ren,

H.Z.

Qian, Lang and

H.Y.

Lag, Correlation mining method based on improved Boolean reduction and layered series for big data stream, Pattern Recognition and Artificial Intelligence29(5) (2016), 455–463.

13.

Ru and

X.Z.

He , High utility itemsets mining algorithm of data stream with reducing candidate itemsets, Application Research of Computers34(11) (2017), 3379–3383.

14.

Tang,

Zhang,

Fuet al., Distributed stable routing with adaptive power control for multi-flow and multi-hop mobile cognitive networks, IEEE Transactions on Mobile Computing18(2) (2019), 2829–2841. doi:10.1109/TMC.2018.2885762.

15.

Tomes,

E.N.

Rush and

Altiparmak, Towards adaptive parallel storage systems, IEEE Transactions on Computers67(12) (2018), 1840–1848. doi:10.1109/TC.2018.2836426.

16.

Tripathi,

Vignesh and

Tamarapalli, Optimizing green energy, cost, and availability in distributed data centers, IEEE Communications Letters21(3) (2017), 500–503. doi:10.1109/LCOMM.2016.2631466.

17.

Wang,

Huang,

Nieet al., Local differential private data aggregation for discrete distribution estimation, IEEE Transactions on Parallel and Distributed Systems30(9) (2019), 2046–2059. doi:10.1109/TPDS.2019.2899097.

18.

Wen ,

Y.Y.

Wen,

S.P.

Wang and

Zhao, The maximal regular patterns mining algorithm based on landmark window over data stream, Journal of Computer Research and Development54(1) (2017), 94–110.

19.

Zhang,

Boseet al., A distributed multi-control-center dynamic power flow algorithm based on asynchronous iteration scheme, IEEE Transactions on Power Systems33(2) (2018), 1716–1724.

20.

Zhang,

Shu,

Shanet al., Distributed bottleneck-aware coflow scheduling in data centers, IEEE Transactions on Parallel and Distributed Systems30(7) (2019), 1565–1579. doi:10.1109/TPDS.2018.2889685.

21.

Zhang,

Y.K.

Bao,

L.S.

Shaoet al., A multivariate decision tree for big data classification of distributed data streams, Acta Automatica Sinica44(6) (2018), 157–169.