With the emergence of distributed resources and grid technologies there is a need to provide higher level informatics infrastructures allowing scientists to easily create and execute meaningful data integration and analysis processes that take advantage of the distributed nature of the available resources. These resources typically include heterogeneous data sources, computational resources for task execution and various application-specific services. The effort of the high performance community has so far mainly focused on the delivery of low-level informatics infrastructures enabling the basic needs of grid applications. Such infrastructures are essential but do not directly help end-users in creating generic and re-usable applications.
In this paper, we present the Discovery Net architecture for building grid-based knowledge discovery applications. Our architecture enables the creation of high-level, re-usable and distributed application workflows that use a variety of common types of distributed resources. It is built on top of standard protocols and standard infrastructures such as Globus but also defines its own protocols such as the Discovery Process Mark-up Language for data flow management. We discuss an implementation of our architecture and evaluate it by building a real-time genome annotation environment on top.
Get full access to this article
View all access options for this article.
References
1.
Abramson, D., Sosic, R., Giddy, J., and Hall, B. 1995. Nimrod: A tool for performing parameterised simulations using distributed workstations. In HPDC, pp. 112–121.
2.
Altschul, S., Gish, W., Miller, W., Myers, E., and Lipman, D.J. 1990. Basic local alignment search tool . Journal of Molecular Biology, 215: 403–410 .
3.
Altschul, S., Madden, T.L., Schaffer, A.A., Zhang, J., Zhang, Z., Miller, W., and Lipman, D.J. 1997. Gapped basic and PSI-blast: a new generation of protein database search programs . Nucleic Acid Research, 25: 3389–3402 .
4.
Anderson, D.P., Cobb, J., Korpela, E., Lebofsky, M., and Werthimer, D. 2002. Seti@Home: an experiment in public-resource computing . Communications of the ACM, 45(11): 56–61 .
5.
Avery, P., Foster, I., Gardner, R., Newman, H., and Szalay, A. 2001. An international virtual-data grid laboratory for data intensive science. http://www.griphyn.org.
6.
Bairoch, A., and Apweiler, R. 2000. The swiss-prot protein sequence database and its supplement trembl in 2000 . Nucleic Acid Research, 28: 45–48 .
7.
Baru, C., Moore, R., Rajasekar, A., and Wan, M. 1998. The sdsc storage resource broker. In CASCON’98.
8.
Burge, C., and Apweiler, R. 1997. Prediction of complete gene structures in human genomic DNA . Journal of Molecular Biology, 268: 78–94 .
9.
Buyya, R., Abramson, D., and Giddy, J. 2000. Nimrod/g: An architecture for a resource management and scheduling system in a global computational grid. In HPC ASIA’2000.
10.
Cannataro, M., Talia, D., and Trunfio, P. 2002. Design of distributed data mining application of the knowledge grid . In Proceedings National Science Foundation Workshop on Next Generation Data Mining.
11.
Caromel, D., Klauser, W., and Vayssière, J. 1998. Towards seamless computing and metacomputing in Java . Concurrency: Practice and Experience, 10(11–13): 1043–1061 .
12.
Chapman, P., Clinton, J., Khabaza, T., Reinartz, T., and Wirth, R. March 1999. The CRISP-DM process model. http://www.crisp-dm.org/.
13.
Chattratichat, J., Darlington, J., Guo, Y., Hedvall, S., Kohler, M., and Syed, J. 1999. An architecture for distributed enterprise data mining . In Proceedings of the 7th Conference on High Performance Computing and Networking Europe.
14.
Chattratichat, J., Guo, Y., and Syed, J. 1999. A visual language for internet-based data mining and data visualisation. In IEEE Symposium on Visual Languages, Tokyo, Japan .
15.
Curbera, F., Goland, Y., Klein, J., Leymann, F., Roller, D., Thatte, S., and Weerawarana, S. 2002. Business process execution language for web services, version 1.0.
16.
Curcin, V., Ghanem, M., Guo, Y., Kohler, M., Rowe, A., Syed, J., and Wendel, P. 2002. Discovery net: Towards a grid of knowledge discovery . In Proceedings of the Eigth International Conference on Knowledge Discovery and Data Mining (KDD-2002).
17.
Eisen, M., Spellman, P., Brown, P., and Botstein, D. 1998. Cluster analysis and display of genomewide expression patterns . Proc. Natl. Acad. Sci., 95: 14863–14868 .
18.
Emanuelsson, O., Nielsen, H., Brunak, S., and von Heijne, G. 2000. Predicting subcellular localization of proteins based on their n-terminal amino acid sequence . Journal of Molecular Biology, 300: 1005–1016 .
19.
Jensen, J. et al. 2002. Ab initio prediction of human orphan protein function from post-translational modifications and localization features . Journal of Molecular Biology, 319: 1257–1265 .
20.
Foster, I., and Kesselman, C., 1999. The globus toolkit. In The Grid: Blueprint for a New Computing Infrastructure, Chap. 11, pp. 259–278. Morgan Kaufmann, San Francisco, CA .
21.
Foster, I., Kesselman, C., Nick, J.M., and Tuecke, S. 2002. The physiology of the grid an open grid services architecture for distributed systems integration. Technical report, http://www.globus.org/research/papers/ogsa.pdf.
22.
Grossman, R.L., Bailey, S.M., Sivakumar, H., and Turinsky, A.L. 1999. Papyrus: A system for data mining over local and wide-area clusters and super-clusters. In SC’99. ACM Press and IEEE Computer Society Press .
Guo, Y., and Sutiwaraphun, J. 2000. Distributed classification with knowledge probing. In H. Kargupta and P. Chan, editors, Advances in Distributed and Parallel Knowledge Discovery, chapter 4. AAAI Press .
25.
Henderson, R., and Tweten, D. 1995. Portable batch system: Requirement specification. Technical report, NASA Ames Research Center .
Knudsen, S. 1999. Promoter 2.0: for the recognition of polii promoter sequences . Bioinformatics, 15: 356–361 .
28.
Litzkow, M.J., Livny, M., and Mutka, M.W. 1988. Condor: A hunter of idle workstations . In 8th International Conference on Distributed Computing Systems, pp. 104–111 , Washington, DC, USA, June. IEEE Computer Society Press.
29.
Lowe, T.M., and Eddy, S.R. 1997. trnascan-se: A program for improved detection of transfer rna genes in genomic sequence . Nucleic Acid Research, 25: 955–964 .
30.
Maniatty, W., and Zaki, M.J. 2000. A requirements analysis for parallel KDD systems. In IPDPS Workshops, pp. 358–365.
31.
O’Donovan, C., Martin, M.J., Gattiker, A., Gasteiger, E., Bairoch, A., and Kanehisa, M. 2002. High-quality protein knowledge resource: Swiss-prot and trembl . Brief Bioinformatics, 16: 944–945 .
32.
Ogata, H., Goto, S., Sato, K., Fujibuchi, W., Bono, H., and Kanehisa, M. 1999. Kegg: Kyoto encyclopedia of genes and genomes . Nucleic Acid Research, 27: 29–34 .
33.
Rice, A.W.B.P., and Longden, I. 2000. Emboss: The European molecular biology open software suite . Trends in Genetics, 16: 276–277 .
34.
Romberg, M. 1999. The unicore architecture . In Proceeding of the 8th IEEE International Symposium on High Performance Distributed Computing.
35.
Roth, M., and Schwarz, P. 1997. Don’t scrap it, wrap it! a wrapper architecture for legacy data sources. In VLD 1997.
36.
Rutherford, K.J., Parkhill, J., Crook, J., Rice, T.H.P., Rajandream, M.-A., and Barrell, B. 2000. Artemis: sequence visualisation and annotation . Bioinformatics, 16: 944–945 .
Stein, L. 2002. Genome annotation: from sequence to biology . Nature Reviews Genetics, 2: 493–503 .
39.
Stolfo, S., Prodromidis, A.L., Tselepis, S., Lee, W., Fan, D.W., and Chan, P.K. 1997. JAM: Java agents for meta-learning over distributed databases. In D. Heckerman, H. Mannila, D. Pregibon, and R. Uthurusamy, editors, Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (KDD-97), p. 74-74. AAAI Press .
40.
Tusnady, G.E., and Green, P. 2001. The hmmtop transmembrane topology prediction server . Bioinformatics, 17: 849–850 .
41.
Wong, L. 2000. Kleislu, a functional query system . Journal of Functional Programming, 10(1): 19–56 .
42.
Zhou, S. 1992. LSF: load sharing in large-scale heterogeneous distributed systems . In Proceedings of the Workshop on Cluster Computing, Tallahassee, FL, December. Super-computing Computations Research Institute, Florida State University.
