We define an approach to benchmarking, “purpose-based
benchmarks”, which explicitly and comprehensively measures the ability of
a computing system to reach a goal of human interest. This contrasts with the
traditional approach of defining a benchmark as a task to be timed, or as the rate
at which some activity is performed. Purpose-based benchmarks are more difficult to
create than traditional benchmarks, but have a profound advantage that makes them
well worth the trouble: they provide a well-defined quantitative measure of the
productivity of a computer system.
Bailey, D.
et al. 1994. The NAS Parallel Benchmarks. Technical
Report RNR-94-007,
NASA Ames Research Center, CA
.
2.
Dongarra, J.2004. Performance of various computers using standard linear
equations software in a Fortran environment. Technical Report CS-89-85,
Oak Ridge National Laboratory (updated periodically).
3.
Faulk, S.
,
Gustafson, J
,
Johnson, P.
,
Porter, A.
,
Tichy, W.
, and
Votta, L.2004.
Measuring HPC productivity
. International Journal of High Performance Computing Applications18(4).
4.
Gustafson, J.
,
Rover, D.
,
Elbert, S.
, and
Carter, M.1990.
The design of a scalable, fixed-time computer benchmark
. Journal of Parallel and Distributed Computing12(4):
388-401
.
5.
Gustafson, J.
1994.
A paradigm for grand challenge performance evaluation
. Proceedings of the Toward Teraflop Computing and New Grand
Challenge Applications Mardi Gras ’94 Conference,
Baton Rouge, LA (http://www.scl.ameslab.gov/Publications/Gus/GrandChallenge/Paradigm.html)
6.
Gustafson, J.
and
Snell, Q.
1995.
HINT: a new way to measure computer performance
. Proceedings of the 28th Annual Hawaii International Conference
on System Sciences,Vol. II,
392-401
.
7.
Hansen, E.
and
Walster, G. W.2004. Global Optimization using Interval Analysis, 2nd
edition,
Marcel Dekker, New York
.
8.
IBM
. 1986. IBM RT PC Computer Technology, IBM Form No.
SA23-1057, 81-81.
9.
Kahan, W.
1997.
The baleful effect of computer benchmarks upon applied mathematics, physics
and chemistry
. The John von Neumann Lecture at the 45th Annual Meeting of
SIAM, Stanford University, CA.
10.
Luce, R.
and
Raiffa, H.1957. Games and Decisions: Introduction and Critical Survey,
Wiley, New York
.
11.
McCalpin, J.2004. STREAM: sustainable memory bandwidth in high performance
computers, http://www.cs.virginia.edu/stream/ (updated periodically).
12.
Mullen, R.
and
Muhanna, R.1999.
Bounds of structural response for all possible loading combinations
. Journal of Structural Engineering125(1):
98-106
.
Smith, J. E.1988.
Characterizing performance with a single number
. Communications of the ACM31(10):
1202-1206
.
15.
Snelling, D.1993. A philosophical perspective on performance
measurement. Computer Benchmarks,
J. Dongarra
and
W. Gentzsch
, editors,
North-Holland, Amsterdam
, 97-103.
16.
Standard Performance Evaluation Corporation (SPEC). 2004.
http://www.specbench.org/ (updated periodically).
17.
Weingarten, F.1993.
HPCC research questioned
. Communications of the ACM36(11):
27-29
.
