In this paper, we present a lightweight, micro-kernel-based virtual machine monitor (VMM) for the Blue Gene/P supercomputer. Our VMM comprises a small µ-kernel with virtualization capabilities and, atop, a user-level VMM component that manages virtual Blue Gene/P cores, memory, and interconnects; we also support running native applications directly atop the µ-kernel. Our design goal is to enable compatibility to standard operating systems such as Linux on BG/P via virtualization, but to also keep the amount of kernel functionality small enough to facilitate shortening the path to applications and lowering operating system noise.
Our prototype implementation successfully virtualizes a Blue Gene/P version of Linux with support for Ethernet-based communication mapped onto Blue Gene/P’s collective and torus network devices. Our first experiences and experiments show that our VMM still shows a substantial performance hit, and that support for native application environments is a key requirement towards fully exploiting the capabilities of a supercomputer. Altogether, our approach poses an interesting operating system alternative for supercomputers, providing the convenience of a fully featured commodity software stack, while also promising to deliver the scalability and low latency of an HPC operating system.
AdigaNRBlumrichMAChenDCoteusPGaraAGiampapaMEHeidelbergerPSinghSSteinmacher-BurowBDTakkenTTsaoMVranasP (2005) Blue Gene/L torus interconnection network. IBM Journal of Research and Development49(2/3): 265–276.
2.
AgesenOGarthwaiteASheldonJSubrahmanyamP (2010) The evolution of an x86 virtual machine monitor. ACM Operating Systems Review44(4): 3–18.
3.
AppavooJUhligVStoessJWaterlandARosenburgBWisniewskiRSilvaDDvan HensbergenESteinbergU (2010) Providing a cloud network infrastructure on a supercomputer. In Proceedings of the 19th ACM international symposium on high performance distributed computing, Chicago, IL, USA, pp. 385–394.
4.
AppavooJUhligVWaterlandARosenburgBSilvaDDMoreiraJE (2009) Kittyhawk: Enabling cooperation and competition in a global, shared computational system. IBM Journal of Research and Development53(4): 1–15.
5.
BeckmanPIskraKYoshiiKCoghlanSNatarajA (2008) Benchmarking the effects of operating system interference on extreme-scale parallel machines. Cluster Computing11(1): 3–16.
6.
BhargavaRSerebrinBSpadiniFManneS (2008) Accelerating two-dimensional page walks for virtualized systems. In Proceedings of the 13th international conference on architectural support for programming languages and operating systems, Seattle, WA, USA, March 2008, pp. 48–59.
7.
EngelmannCScottSOngHValléeGNaughtonT (2007) Configurable virtualized system environments for high performance computing. In 1st workshop on system-level virtualization for high performance computing, Lisbon, Portugal, March 2007.
8.
GavrilovskaAKumarSRajHSchwanKGuptaVNathujiRNiranjanRRanadiveASaraiyaP (2007) High-performance hypervisor architectures: Virtualization in HPC systems. In 1st workshop on system-level virtualization for high performance computing, Lisbon, Portugal, March 2007.
9.
GiampapaMGoodingTInglettTWisniewskiRW (2010) Experiences with a lightweight supercomputer kernel: Lessons learned from Blue Gene’s CNK. In Proceedings of the 2010 international conference on supercomputing, New Orleans, LA, USA, November 2010, pp. 1–10.
10.
HärtigHHohmuthMFeskeNHelmuthCLackorzynskiAMehnertFPeterM (2005) The Nizza secure-system architecture. In Proceedings of the 1st international conference on collaborative computing: Networking, applications and worksharing, San Jose, CA, USA, December 2005.
11.
HärtigHHohmuthMLiedtkeJSchönbergS (1997) The performance of u-kernel based systems. In Proceedings of the 16th symposium on operating system principles, Saint Malo, France, October 1997, pp. 66–77.
12.
HeiserGLeslieB (2010) The OKL4 microvisor: Convergence point of microkernels and hypervisors. In Proceedings of the first ACM Asia-Pacific workshop on systems, August 2010, pp. 19–24.
13.
IBM Power ISA Version 2.03. IBM Corporation.
14.
IBM (2009) IBM Power ISA Version 2.06. IBM Corporation.
15.
IBM Blue Gene team (2008) Overview of the IBM Blue Gene/P Project. IBM Journal of Research and Development52(1/2): 199–220.
KellySMBrightwellR (2005) Software architecture of the light weight kernel, Catamount. In Proceedings of the 2005 Cray user group annual technical conference, Albuquerque, NM, USA, May 2005.
18.
KuzILiuYGortonIHeiserG (2007) Camkes: A component model for secure microkernel-based embedded systems. Journal of Systems and Software80(5): 687–699.
L4 Development Team (2009) L4 X.2 Reference Manual. University of Karlsruhe, Germany, May 2009.
21.
LangeJRPedrettiKDindaPBridgesPGBaeCSolteroPMerrittA (2011) Minimal-overhead virtualization of a large scale supercomputer. In Proceedings of the 7th ACM SIGPLAN/SIGOPS international conference on virtual execution environments, Newport Beach, CA, USA, March 2011, pp. 169–180.
22.
LangeJRPedrettiKTHudsonTDindaPACuiZXiaLBridgesPGGockeAJaconetteSLevenhagenMBrightwellR (2010) Palacios and Kitten: New high performance operating systems for scalable virtualized and native supercomputing. In Proceedings of the 24th IEEE international symposium on parallel and distributed processing, Atlanta, GA, USA, April 2010, pp. 1–12.
LeVasseurJUhligVStoessJGötzS (2004) Unmodified device driver reuse and improved system dependability via virtual machines. In Proceedings of the 6th symposium on operating systems design and implementation, San Fransisco, CA, USA, December 2004, pp. 17–30.
25.
LiedtkeJ (1995) On µ-kernel construction. In Proceedings of the 15th symposium on operating system principles, Copper Mountain, CO, USA, December 1995, pp. 237–250.
26.
LiedtkeJUhligVElphinstoneKJaegerTParkY (1999) How to schedule unlimited memory pinning of untrusted processes or provisional ideas about service-neutrality. In Proceedings of 7th workshop on hot topics in operating systems, Rio Rico, AZ, USA, March 1999, pp. 191–196.
27.
MergenMFUhligVKriegerOXenidisJ (2006) Virtualization for high-performance computing. ACM Operating Systems Review40: 8–11.
28.
MinnichRMcKieJ (2009) Experiences porting the Plan 9 research operating system to the IBM Blue Gene supercomputers. Computer Science – Research and Development23: 117–124.
29.
MoreiraJBrutmanMCastanosJGoodingTInglettTLieberDMcCarthyPMundyMParkerJWallenfeltBGiampapaMEngelsiepenTHaskinR (2006) Designing a highly-scalable operating system: The Blue Gene/L story. In Proceedings of the 2006 international conference on supercomputing, Tampa, FL, USA, November 2006, pp. 53–63.
PrattIFraserKHandSLimpachCWarfieldAMagenheimerDNakajimaJMalickA (2005) Xen 3.0 and the art of virtualization. In Proceedings of the 2005 Ottawa Linux symposium, Ottawa, Canada, July 2005, pp. 65–78.
32.
SteinbergJKauerB (2010) NOVA: A microhypervisor-based secure virtualization architecture. In Proceedings of the 5th ACM SIGOPS EuroSys conference, Paris, France, April 2010, pp. 209–221.
