Today’s large scale distributed platforms comprise thousands of resources from production, educational, and ad hoc environments including Clouds, Grids, P2P, etc. However, finding suitable resources from such a large pool to store large amounts of data and run multi-resource, long-running data processing applications (usually with few or no fault tolerance capabilities) is restricted by the dynamic availability of distributed resources. In addition to resource failures, the resources may be unavailable due to their owners’ policies for sharing their resources as well as the nature of domain they belong to (e.g. P2P systems, non-dedicated desktop Grids etc.). As a result, the availability-aware selection of distributed resources has become a challenging problem for data management, resource provisioning and job scheduling services. To this end, we present a novel resource availability characterization and prediction method for dynamic heterogeneous distributed environments. We identified 14 availability attributes that can be effectively used to model resource availability in dynamic distributed environments. Three data mining methods (particularly the neural network) are proposed to model and predict resource availability using our identified availability attributes. The availability of a resource is predicted for an instant of time as well as for a time duration. Our experiments for 28 different resources in Austrian Grid show that the predictions through the proposed approach are 18% and 31% (on average) more accurate than those by so far the best method (Naive Bayes’ Classifier) for instant and duration availability, respectively.
AcharyaA., EdjlaliG. and SaltzJ., The utility of exploiting idle workstations for parallel computation, Sigmetrics Perform Eval Rev25(1) (1997), 225–234.
2.
AlsoghayerR. and DjemameK., Resource failures risk assessment modelling in distributed environments, Journal of Systems and Software88 (2014), 42–53.
3.
AndersonD.P., Boinc: A system for public-resource computing and storage, In IEEE/ACM International Workshop on Grid Computing, Washington, DC, USA, 2004, pp. 4–10.
4.
AndrzejakA., KondoD. and AndersonD.P., Ensuring collective availability in volatile resource pools via forecasting, In Proceedings of the 19th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management: Managing Large-Scale Service Deployment, DSOM ’08, Berlin, Heidelberg, Springer–Verlag, 2008, pp. 149–161.
5.
BhagwanR., SavageS. and VoelkerG.M., Understanding availability, In Peer-to-Peer Systems II, Second International Workshop, Berkeley, CA, USA, 2003, pp. 256–267.
6.
BouyerA., MohebiE. and AbdullahA.H., Using self-announcer approach for resource availability detection in grid environment, In Proceedings of the Fourth International Multi-Conference on Computing in the Global Information Technology, ICCGI ’09, IEEE Computer Society, 2009, pp. 151–156.
7.
BrevikJ., NurmiD. and WolskiR., Automatic methods for predicting machine availability in desktop grid and peer-to-peer systems, In Proceedings of the 2004 IEEE International Symposium on Cluster Computing and the Grid, CCGRID ’04, Washington, DC, USA, IEEE Computer Society, 2004, pp. 190–199.
8.
ChunB., et al., Planetlab: An overlay testbed for broad-coverage services, Sigcomm Comput Commun Rev33(3) (2003), 3–12.
9.
De SalveA., GuidiB. and MoriP., Predicting the availability of users’ devices in decentralized online social networks, Concurrency and Computation: Practice and Experience, pp. e4390–n/a.
10.
FingerM., BezerraG.C. and CondeD.R., Resource use pattern analysis for predicting resource availability in opportunistic grids, Concurrency and Computation: Practice and Experience22(3) (2015), 295–313.
11.
HuL., HuG., TangK. and CheX., Grid resource prediction based on support vector regression and genetic algorithms, In Fifth International Conference on Natural Computation, ICNC ’09, volume 1, 2009, pp. 499–505.
12.
IosupA., JanM., SonmezO. and EpemaD.H.J., On the dynamic resources availability in grids, In Proceedings of the 8th IEEE/ACM International Conference on Grid Computing, GRID ’07, Washington, DC, USA, IEEE Computer Society, 2007, pp. 26–33.
13.
KondoD., AndrzejakA. and AndersonD.P., On correlated availability in internet-distributed systems, In Proceedings of the 2008 9th IEEE/ACM International Conference on Grid Computing, GRID ’08, Washington, DC, USA, 2008, pp. 276–283.
14.
KondoD., TauferM., BrooksC., CasanovaH. and ChienA., Characterizing and evaluating desktop grids: An empirical study, In Proceedings of Parallel and Distributed Processing Symposium2004, pp. 26–34.
15.
LeridaJ.L., SolsonaF., HernandezP., GineF., HanzichM. and CondeJ., State-based predictions with self-correction on enterprise desktop grid environments, Journal of Parallel and Distributed Computing73(6) (2013), 777–789.
16.
LongD., MuirA. and GoldingR., A longitudinal survey of internet host reliability, In Proceedings of the 14TH Symposium on Reliable Distributed Systems, SRDS ’95, Washington, DC, USA, IEEE Computer Society, 1995, pp. 2–10.
17.
MahatoD.P. and SinghR.S., Maximizing availability for task scheduling in on-demand computingŰbased transaction processing system using ant colony optimization, Concurrency and Computation: Practice and Experience, pp. e4405–n/a.
18.
NadeemF., Ranking grid-sites based on their reliability for successfully executing jobs of given durations, International Journal of Computer Network and Information Security5 (2015), 9–15.
19.
NadeemF., AlghazzawiD., MashatA., FakeehK., AlmalaiseA. and HagrasH., Modeling and predicting execution time of scientific workflows in the grid using radial basis function neural network, Cluster Computing20(3) (2017), 2805–2819.
20.
NadeemF., ProdanR. and FahringerT., Characterizing, modeling and predicting dynamic resource availability in a large scale multipurpose grid, In 8th IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2008), 2008, pp. 348–357.
21.
NadeemF., ProdanR., FahringerT. and IosupA., A framework for resource availability characterization and online prediction in the grids, In Grid Computing, Springer US, 2008, pp. 209–224.
22.
NadeemF., ProdanR., FahringerT. and KellerV., An evaluation of availability comparison and prediction for optimized resource selection in the grid. In PriolT. and VanneschiM., editors, From Grids to Service and Pervasive Computing, SpringerUS, 2008, pp. 63–76.
23.
PrakashS. and VidyarthiD.P., Maximizing availability for task scheduling in computational grid using genetic algorithm, Concurrency and Computation: Practice and Experience27(1) (2015), 193–210.
24.
QuinlanJ.R., Induction of decision trees, Machine Learning1(1) (1986), 81–106.
25.
RahmanM., HassanM.R. and BuyyaR., Jaccard index based availability prediction in enterprise grids, In Proceedings of International Conference on Computational Science, number 1 in ICCS’10, 2010, pp. 2707–2716.
26.
RamachandranK., LutfiyyaH. and PerryM., Decentralized resource availability prediction for a desktop grid, In 10th IEEE/ACM International ConferenRence on Cluster, Cloud and Grid Computing (CCGrid), 2010, pp. 643–648.
27.
RamachandranK., LutfiyyaH. and PeryM., Decentralized approach to resource availability prediction using group availability in a p2p desktop grid, Future Generation Computer Systems28(6) (2012), 854–860.
RenX., LeeS., EigenmannR. and BagchiS., Prediction of resource availability in fine-grained cycle sharing systems empirical evaluation, Journal of Grid Computing5(2) (2007), 173–195.
30.
RoodB. and LewisM.J., Multi-state grid resource availability characterization, In Proceedings of the 8th IEEE/ACM International Conference on Grid Computing, GRID ’07, Austin, TX, 2007, pp. 42–49.
31.
RubioJ.D.J., Ricardo CruzD., EliasI., OchoaG., BalcazarandR. and AguilarA., Anfis system for classification of brain signals, Journal of Intelligent & Fuzzy Systems, (Preprint) (2019), 1–9.
32.
SchroederB. and GibsonG.A., A large-scale study of failures in high-performance computing systems, In Proceedings of the International Conference on Dependable Systems and Networks, DSN ’06, Washington, DC, USA, IEEE Computer Society, 2006, pp. 249–258.
33.
ShangL., WangZ., ZhouX., HuangX. and ChengY., TM-DG: A trust model based on computer users’ daily behavior for desktop grid platform, In Proceedings of the 2007 Symposium on Component and Framework Technology in High-performance and Scientific Computing, CompFrame ’07, New York, NY, USA, 2007, pp. 59–66. ACM.
34.
ShearerC., The crisp-dm model: The new blueprint for data mining, Journal of Data Warehousing5(4) (2000).
35.
SinghM. and KaurL., Resource grouping in grid environment towards the availability and reliability of computing service, Journal of Advanced Computing1(1) (2013), 1–8.
36.
SrivastavaS. and BanicescuI., Robust resource allocations through performance modeling with stochastic process algebra, Concurrency and Computation: Practice and Experience29(7) (2017), e3894–n/a. e3894 cpe.3894.
37.
TanP.-N., SteinbachM. and KumarV., Introduction to Data Mining, (First Edition), Addison-Wesley Longman Publishing Co, Inc., Boston, MA, USA, 2005.
38.
TangD. and IyerR.K., Dependability measurement and modeling of a multicomputer system, IEEE Trans Comput42(1) (1993), 62–75.
39.
VrignatP., AvilaM., DucultyF. and KratzF., Failure event prediction using hidden markov model approaches, IEEE Transactions on Reliability64(3) (2015), 1038–1048.
40.
XiaojuanR. and RudolfE., Empirical studies on the behavior of resource availability in fine-grained cycle sharing systems, In 2006 International Conference on Parallel Processing (ICPP’06), 2006, pp. 3–11.
41.
ZhangX. and HanQ., State estimation for static neural networks with time-varying delays based on an improved reciprocally convex inequality, IEEE Transactions on Neural Networks and Learning Systems29(4) (2018), 1376–1381.
42.
ZhengY., Grid resource prediction based on support vector regression and simulated annealing algorithms, Modern Applied Science4(11) (2010), 97–104.
