Abstract
The Environmental Sample Processor (ESP) is an autonomous robotic instrument developed at the Monterey Bay Research Aquarium Institute (MBARI) that operates below the ocean's surface, sampling raw seawater and executing a variety of sample manipulation and analytical protocols, in situ. It uses DNA and antibody probes to identify marine planktonic organisms and substances they produce. Initial prototypes of the ESP were hosted on an Intel i486 CPU running a commercial real-time operating system (OS). The application, coded in C++, included a custom ‘macro’ language interpreter to direct biochemical analyses. To achieve greater flexibility and minimize the development effort for the 2nd generation of the ESP (2G ESP), MBARI replaced its ‘macro’ language with a general purpose, open-source scripting language, selecting Ruby for its unique combination of a succinct, English-like syntax with a seamless underlying object-oriented paradigm. The 2G ESP application, aside from custom servo control firmware, is coded entirely in Ruby, hosted on a low-power ARM9 CPU running Linux. Servo control was distributed onto a network of dedicated microcontrollers to cope with the nondeterministic delays inherent in the Linux operating system and Ruby interpreter.
Introduction
Many of the diagnostic protocols for detecting molecular signatures share an overlapping set of requirements such as collecting, concentrating, preserving, and disrupting cells. They apply a series of reagents in a timed sequence and extract particulates from the sample. By developing a low power, compact instrument that meets this core set of functional requirements, it would be possible to conduct many commonly applied molecular biological analyses remotely, in situ.
Toward that goal, a group of scientists and engineers at the Monterey Bay Aquarium Research Institute (MBARI) undertook development of the Environmental Sample Processor (ESP) 1,2,a (Scholin et al. 2005). The ESP is a robotic instrument that collects raw water samples, concentrates particulates, and applies molecular probes to identify microorganisms based on signature ribosomal ribonucleic acid (rRNA) sequences. In addition, MBARI has recently developed an antibody-based test for the detection of the algal neurotoxin, domoic acid, thus enabling the ESP to detect both specific species and the toxins they produce. It also archives particulates in discrete samples, allowing a variety of nucleic acid analyses, microscopy, and other types of laboratory procedures to be used after the instrument is recovered. To date, the ESP team has focused on detecting a suite of marine planktonic organisms. Prototypes of the ESP have been deployed in Monterey Bay and the Gulf of Maine at depths of 3–15 m. 4,5
The ESP consists of eight major robotic components: a carousel, manipulator arm, two clamps, three syringe pumps, and a CCD camera (Figs. 1 and Figs. 2). The carousel stores up to 150 puck-shaped reaction chambers, which accommodate a wide variety of 25 or 13 mm diameter filters or chemically adsorptive media. An elevator and rotary manipulator arm, with a two fingered gripper, moves pucks from the carousel to sample collection or processing stations where they are sealed in clamps, providing connections to the sample port and/or reagent valve manifolds. Each clamp contains an embedded heater pad to regulate the temperature of liquid in the puck from ambient (4–30 °C) to ∼ 100 °C. The elevator pushes pucks up to an imaging station where a CCD camera records the luminescence pattern of completed probe array assays. The instrument is usually deployed at a fixed depth on a subsurface mooring with an electromechanical cable attached to a surface float containing a radio modem and antenna.
Annotated solid model of the 2nd generation ESP mechanism with electrical and most fluidic components omitted for clarity. Overall dimensions are approximately 41 cm in diameter by 76 cm high.
2nd generation ESP being readied for its initial deployment in the Monterey Bay viewed from roughly the same point as the solid model rendering depicted in Figure 1.
The ESP uses custom DNA probe arrays for rRNA sequences that are indicative of specific species or groups of species (e.g., Figure 3). Sample collection begins by loading a puck into the collection clamp containing a filter membrane with pores smaller than the organisms being sought. A syringe pump then draws raw water through this filter until it is saturated with particulates. To develop a probe array, the ESP breaks down cell membranes, homogenizing a sample using a chaotrope and heat. The resulting filtered sample homogenate is retained. The sample collection puck is replaced with an array puck and the homogenate is applied, followed by a sequence of reagents that reveal rRNA molecules bound at specific locations on the array grid using sandwich hybridization 3,5 8 . The radio modem then transmits an image of the resulting array to a remote location for interpretation. Different arrays are tailored to specific groups of organisms such as bacterioplankton, harmful algae, planktonic microbes, harmful algae, or invertebrate larvae.
Detection of invertebrate larvae, pennate diatoms, and groups of bacteria on a single ESP probe array.
Users controlled early prototypes of the ESP via a custom “macro” language that rigorously defined the sequence of steps to be performed by the instrument. This language lacked any provision for looping, conditionals, or parameter passing, resulting in scripts that were inflexible, needlessly long, very repetitive and, consequently, difficult to maintain. Supervisory system logic and lower level control were hard coded in C++. Even trivial bug fixes or enhancements necessitated a recompilation of the application binary, requiring software engineering expertise beyond that of typical end users.
The 2nd generation
In early 2003, MBARI was awarded a grant from the National Science Foundation to make the ESP technology more accessible to outside research groups by developing a modular instrument that could be easily reconfigured to suit a wide variety of deployment and analysis scenarios. The design team soon recognized that significant improvements in the ESPs flexibility and ease of use could not be achieved without a major overhaul of its scripting capabilities. It gradually became clear that extending the existing interpreter would consume more development time than adapting an established open-source scripting language, or, if possible, adopting one unchanged. As the search for a suitable language progressed, the design team realized that the need to recompile the application for enhancements and bug fixes would diminish to the extent that the object-oriented C++ application logic was rewritten in the scripting language. This rewrite would likely be more straightforward if the chosen scripting language was also object-oriented. Furthermore, if the language interpreter could conveniently process commands entered interactively, it could also function as the instrument's “command shell”, replacing the maze of hard coded menus that characterized the existing software's user interface.
The above requirements quickly winnowed the pack of popular, open-source scripting languages. TCL, PHP and, to a lesser extent, Perl, were not sufficiently object-oriented. Javascript (or, more precisely, ECMAScript) implementations were usually tied too intimately to a web browser and lacked standardized input/output support. Python was considered, but ultimately rejected due to its rigid whitespace and punctuation requirements, which, while improving program structure and readability, tend to make the language cumbersome for interactive use. Ruby seemed to meet all the requirements. Ultimately, it was determined that Python would be a fall back if Ruby did not prove viable.
MBARIs design team chose Linux as the ESP host CPUs operating system because Python and Ruby were already ported to Linux and many vendors of low-power (Advanced Risk Machine, Ltd., ARM) CPUs boards offered Linux kernels configured for their hardware. Further, the free availability of Linux source code reduced the fear that some obscure device driver or kernel bug might significantly delay the project. However, Linux is not a replacement for a realtime operating system. The current crop of small, low-power CPUs running Linux cannot reliably process real-time servo motor control loops sampling at the 30–80 Hz required for the ESPs actuators. Even if the application level latencies inherent in Ruby's garbage collector could be eliminated, the Linux operating system running on the ESPs host CPU would still occasionally delay processing external interrupt events for over 20 ms while compact flash disk accesses were performed. Such long interrupt latencies dictate that the Linux host offload servo control onto dedicated peripheral hardware, regardless of how efficiently the application is implemented.
The 2nd generation ESPs design therefore includes six microcontrollers that relieve the host processor of any responsibility for real-time response. These microcontrollers communicate with the host and each other via a shared serial bus. b They are programmed in ANSI-’C’ (with some vendor-specific extensions) and store their code in nonvolatile flash memory. Their firmware cannot be easily modified in the field, so it must be highly configurable and implement only the most primitive operations. A unique address, set manually by switches, identifies each otherwise identical microcontroller on the bus. The host configures all microcontrollers at application startup for their specific roles according to their bus addresses.
Where ruby shines
The single language feature that has proven most useful in development of the ESP is Ruby's open classes. Ruby classes are “open” in the sense that they may be changed after objects are instantiated. c One may even add, delete, or alter methods in base classes such as String, Float, and Integer. Objects in languages, like Java, C++, and Python d , whose classes are “closed”, may not be instantiated until their class definition is completely parsed. Once the class is thus defined, it cannot be changed. Typically, one cannot alter the behavior base classes in these languages at all and even one's own classes cannot be changed once loaded.
The ESP realizes very practical benefits from Ruby's open classes. For instance, imagine that a team of molecular biologists, after spending hours working on a new biochemical protocol, discover that previously loaded code needs to be altered or enhanced before work can continue. A language with closed classes would require that the application be exited and restarted with the modified code to effect this change, losing much of the ESPs critical state in the process. However, Ruby's combination of open classes and dynamic compilation allows modified methods to be reloaded “on the fly”, as the application continues operating, uninterrupted, thus preserving the instrument's state.
There is justifiable controversy surrounding open classes because they are easily misused. Changing the behavior of existing methods in base classes is especially dangerous, as libraries and other applications typically depend on them. Base classes modified to support a specific application may also break libraries that one might try to use in the future. Therefore, programmers should adopt the convention that base class modifications be restricted to adding methods only, never deleting or altering them, despite the fact that Ruby allows this.
An unusual feature of Ruby's method inheritance is that it lets one optionally associate methods with a particular object instance rather than its class. The ESP uses such ‘singleton methods’ to good effect in a number of situations. For example, although there are a number of Clamp objects, the collection clamp is unique in that closing it also causes the ESP to prepare to collect a raw water sample. The Ruby ESP application realizes this by overriding the close instance method of the collection clamp, whereas Java or C++ would demand the creation of an extra singleton class.
Ruby's syntax is remarkably relaxed and free of punctuation. An arbitrary number of whitespace characters may be inserted anywhere to separate language tokens and there are no line indentation rules. If it would be syntacticly complete, starting a new line terminates the current expression; otherwise, the next line is appended to it. This parsing rule neatly eliminates the trailing semicolons that riddle code in languages whose syntax is derived from ’C’ e , while still allowing for multiline expressions without explicit line continuation characters. Parentheses in Ruby are needed only to alter the default operator precedence. They do not delimit boolean expressions in control structures as they do in ’C’ and its derivatives. Parentheses may even be omitted around method argument lists for simple calls, thus giving them the appearance of “commands” rather than function invocations. Of course, Ruby's free form syntax can be abused to write code that is difficult to read: a programmer intent on writing obscure code will find a way to do so in any language.
The ESP exploits Ruby's succinct, English-like syntax to make a useful subset of the language readily accessible to its intended end users. They are typically microbiologists and not expert computer programmers. Most high-level ESP commands are implemented as method calls on predefined Ruby objects so that end users may code protocols without understanding much about object-oriented programming. Here are some basic examples:
The ESP associates names with these predefined global constant objects by defining a singleton method for each that simply returns the object's canonical name. While this may sound trivial, it is an aspect of introspection missing from most object-oriented systems. This ability to derive any object's name from a reference to it (e.g., the string “Elevator”, given only a reference to the Elevator object) lets ESP log file entries take the form of valid commands that may be reentered into the system verbatim if it is desired to repeat them later.
Ruby's darker facets
Like all programming languages, Ruby has a few warts and dark corners. The fact that Ruby assigns variables no static data types allows it to forgo their declaration entirely. However, traditional variable declarations do more than determine the data type of variable (or object) identifiers. They also indicate their scope and access restrictions. Ruby resorts to prefixing all variable references with punctuation characters to indicate their scope (e.g., local, @object, @@class, or $global) and assumes that any identifier beginning with a capital letter that is the target of an assignment is a constant. f Although these conventions trip up new programmers, they are quickly mastered.
A more insidious problem with Ruby is that a context-free grammar cannot determine whether an unprefixed identifier refers to a local variable or a method name. Ruby resolves this “Variable/Method” ambiguity as each method is parsed by treating each new identifier encountered as a method name until it appears in the method's argument list or on the left side of an assignment. Thereafter, the new identifier is treated as a reference to a local variable. 9 This can lead to unexpected behavior when methods contain control structures that do not execute in the same (top-down) order in which code is parsed. Consider the following (admittedly contrived) example:
To resolve this Variable/Method ambiguity, one can explicitly force Ruby's parser to interpret an identifier as a method call by prefixing it with ‘self.’, as in ‘self.foo’. To force the parser to interpret an identifier as a variable, simply assign it some value before any other reference to it appears in the text of the method. Neither technique is particularly pretty, but they do the job. A better approach, in practice, is to try to avoid this ambiguity entirely by taking care to ensure that no local variable shares the same name as any of the object's methods.
Unit testing and simulation
Dynamically typed scripting languages provide very little information to drive static error analysis. Mistakes that would illicit trivial compiler errors in strongly typed languages, such as misspelled method or variable names, will go undetected until the offending methods execute. Programmers working in scripting languages typically make up for their lack of compile-time checking by writing extensive unit tests to exercise their code. Effective unit tests can uncover subtle logic errors in a syntactically correct program. g However, writing them is an art by itself.
The ESPs target users are not required to be experienced programmers, well versed in the art of unit testing. Nevertheless, the protocol control scripts they typically write, although quite simple logically, run for many hours and may consume irreplaceable samples and expensive reagents. Discovering a typo halfway through such a protocol's execution would be very frustrating. So, building a simulation capability into the ESP software was recognized as a requirement early in its design.
The ESPs simulator models the serial bus, its microcontrollers, and, in a very primitive way, the physical hardware to which they are normally connected. It redirects the byte stream that would normally flow between the host and the external microcontrollers to internal Ruby proxy objects that model them. Many physical constraints that would cause errors in the real hardware are not detected during simulation. However, the models are currently good enough to detect typos and most commonly occurring logic errors such as commanding a physical actuator to an undefined position or trying to heat a puck to an unattainable temperature. Planned enhancements to the ESP simulator include the ability to estimate the amount of time, power, pucks, and reagents protocols will consume.
Exceptions and continuations
The ESP is a complicated mechanism with many possible failure modes. Fortunately, objects of Ruby's Exception class allow low-level software to report errors that require unusual processing without requiring that every caller check for these conditions explicitly. Ruby's exceptions are analogous to Java's, but instead of its try … catch syntax, Ruby uses begin … rescue keywords to delineate exception handlers. Ruby exceptions are first class objects and may include associated data and methods. One way the ESP exploits this is by defining a generic SoftError exception that is passed a block of code to be retried a certain number of times before it is reported as an unrecoverable failure. Exception classes that are to be automatically retried simply inherit from this SoftError class.
A Ruby Continuation object preserves the complete state of a processing thread. Later, a thread can restore its state to that saved in the continuation, effectively resuming execution from a bookmarked point in the code. The ESP currently uses continuations (and open classes) to implement debugging breakpoints. Inserting breakpoints involves editing Ruby scripts to invoke the “breakpoint” method where desired and reloading them into the interpreter. The ESPs debugger is merely special instance of its interactive Ruby (irb) shell, where the local and object instance variables are those of the method being debugged, so they may be conveniently examined or modified. h
Work is in progress to use continuations to implement checkpoints that enable biochemical protocols to be resumed after a failure occurs. Checkpoints are similar to breakpoints except that they do not stop execution of the current thread. The thread's resumable state is merely saved in a continuation object. Currently, an unhandled exception logs a detailed error message and terminates the protocol. Users will soon have the option to resume such a failed protocol from a previously stored checkpoint. However, before the protocol could be successfully resumed from a checkpoint, a user would first need to manually clear the error and put the ESPs mechanism into the state it was in when that checkpoint was first reached.
Conclusions
Over the last 3 years of the ESPs development, Ruby has consistently shown itself to be a pleasantly intuitive and flexible programming environment. In laboratory settings, the additional simulation capabilities necessitated by Ruby's lack of static error analysis exact a small price for the increased flexibility and interactivity the language affords. Provided sufficient computing resources are available, the inherent immediacy of dynamically typed object-oriented scripting languages dramatically accelerates software development in complex instruments. Ruby's open classes, in particular, foster a programming style that encourages rapid prototyping and experimentation.
Acknowledgments
This project was funded in part by the Monterey Bay Aquarium Research Institute from funds allocated by the David and Lucille Packard Foundation and by the National Science Foundation (OCE-0314222). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Philips Semiconductors Inter-Integrated Circuit (I2C) bus.
Ruby borrows heavily from Xerox PARC's Smalltalk-80 language in this regard.
Redefining an existing Python class is allowed, however it creates a distinct new class of the same name. The behavior of existing objects is not changed. Python calls modification of the attributes of existing classes “monkeypatching”, which uses of a different syntax than class definition.
These include C++, PHP, Java, and JavaScript.
All constants in Ruby have global visibility, but they exist in the namespace of the module in which they were defined.
This is why unit testing is also recommended when working with strongly typed languages.
The GUI debuggers available for Ruby are too resource intensive for ESPs embedded target computer.