In this thought-provoking article, we discuss certain myths and legends that are folklore among members of the high-performance computing community. We gathered these myths from conversations at conferences and meetings, product advertisements, papers, and other communications such as tweets, blogs, and news articles within and beyond our community. We believe they represent the zeitgeist of the current era of massive change, driven by the end of many scaling laws such as Dennard scaling and Moore’s law. While some laws end, new directions are emerging, such as algorithmic scaling or novel architecture research. Nevertheless, these myths are rarely based on scientific facts, but rather on some evidence or argumentation. In fact, we believe that this is the very reason for the existence of many myths and why they cannot be answered clearly. While it feels like there should be clear answers for each, some may remain endless philosophical debates, such as whether Beethoven was better than Mozart. We would like to see our collection of myths as a discussion of possible new directions for research and industry investment.
AbaliBEickemeyerRJFrankeH, et al. (2015) Disaggregated and optically interconnected memory: when will it be cost effective?http://arxiv.org/abs/1503.01416
2.
Allied Market Research (2020) Graphic Processing Unit (GPU) market by type (dedicated, integrated, and hybrid), device (computer, tablet, smartphone, gaming console, television, and others), industry vertical (Electronics, IT & telecommunication. Defense & Intelligence, Media & Entertainment, and Others): Global Opportunity Analysis and Industry Forecast, pp. 2020–2027. https://www.alliedmarketresearch.com/graphic-processing-unit-market
3.
Allied Market Research (2022) Field programmable gate array market by technology (EEPROM, Antifuse, SRAM, flash, and others (EPROM and PROM)), application (data processing, consumer electronics, industrial, military & aerospace, automotive, telecom, and others), and type (high-end, mid-end, and low-end FPGA): global opportunity analysis and industry forecast, pp. 2021–2030. https://www.alliedmarketresearch.com/field-programmable-gate-array-market
BellGBaileyDHDongarraJ, et al. (2017) A look back on 30 years of the Gordon Bell Prize. The International Journal of High Performance Computing Applications31(6): 469–484. DOI: 10.1177/1094342017738610
6.
Ben-NunTde Fine LichtJZiogasAN, et al. (2019) Stateful dataflow multigraphs: a data-centric model for performance portability on heterogeneous architectures. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’19. New York, NY, USA: Association for Computing Machinery, p. 14. DOI: 10.1145/3295500.3356173
7.
Ben-NunTGronerLDeconinckF, et al. (2022) Productive performance engineering for weather and climate modeling with python. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’22, Dallas, TX, USA, 13–18 November 2022. IEEE Press, p. 14. DOI: 10.5555/3571885.3571982
8.
BenderEMGebruTMcMillan-MajorA, et al. (2021) On the dangers of stochastic parrots: can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21. New York, NY, USA: Association for Computing Machinery, pp. 610–623. ISBN. DOI: 10.1145/3442188.3445922
BeverlandMEMuraliPTroyerM, et al. (2022) Assessing requirements to scale to practical quantum advantage. DOI: 10.48550/arXiv.2211.07629
11.
BiKXieLZhangH, et al. (2022) Pangu-weather: a 3D high-resolution model for fast and accurate global weather forecast. https://arxiv.org/abs/2211.02556
12.
BrenowitzNDBrethertonCS (2018) Prognostic validation of a neural network unified physics parameterization. Geophysical Research Letters45(12): 6289–6298. DOI: 10.1029/2018GL078510
13.
CalotoiuAGrafAHoeflerT, et al. (2018) Lightweight requirements engineering for exascale co-design. In: IEEE International Conference on Cluster Computing, CLUSTER 2018, Belfast, UK, September 10-13, 2018, IEEE, pp. 201–211. DOI: 10.1109/CLUSTER.2018.00038
CzechowskiKBattaglinoCMcClanahanC, et al. (2011) Balance principles for algorithm-architecture co-design. In: Proceedings of the 3rd USENIX Conference on Hot Topic in Parallelism. HotPar’11. USA: USENIX Association, p. 9.
17.
De SensiDDe MatteisTTaranovK, et al. (2022) Noise in the clouds: influence of network performance variability on application scalability. Proceedings of the ACM on Measurement and Analysis of Computing Systems6(3): 1–27. DOI: 10.1145/3570609
18.
DeelmanESinghGLivnyM, et al. (2008) The cost of doing science on the cloud: The Montage example. In: SC ’08: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, Austin, TX, USA, 15–21 November 2008, pp. 1–12. DOI: 10.1109/SC.2008.5217932
19.
DomkeJVataiEDrozdA, et al. (2021) Matrix engines for high performance computing: a paragon of performance or grasping at straws? In: 2021 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2021, Portland, OR, USA, May 17-21, 2021. Portland, OR, USA: IEEE Press, pp. 1056–1065. DOI:10.1109/IPDPS49936.2021.00114
20.
DomkeJVataiEGerofiB, et al. (2022) At the locus of performance: a case study in enhancing CPUs with copious 3D-stacked cache. https://arxiv.org/abs/2204.02235
21.
FrantarEAshkboosSHoeflerT, et al. (2022) GPTQ: accurate post-training quantization for generative pre-trained transformers. https://arxiv.org/abs/2210.17323
22.
GonzalezJG PalmaMHattinkM, et al. (2022) Optically connected memory for disaggregated data centers. Journal of Parallel and Distributed Computing163(C): 300–312. DOI: 10.1016/j.jpdc.2022.01.013
23.
GustafsonJL (1988) Reevaluating Amdahl’s law. Communications of the ACM31(5): 532–533. DOI: 10.1145/42411.42415
24.
GustafsonJLYonemotoIT (2017) Beating floating point at its own game: posit arithmetic. Supercomputing Frontiers and Innovations4(2): 71–86. DOI: 10.14529/jsfi170206
HoeflerT (2022) Benchmarking data science: 12 ways to lie with statistics and performance on parallel computers. Computer55(8): 49–56. DOI: 10.1109/MC.2022.3152681
27.
HoeflerTHaenerTTroyerM (2023) Disentangling hype from practicality: on realistically achieving quantum advantage. Communications of the ACM.
28.
HuangLHoeflerT (2022) Compressing multidimensional weather and climate data into neural networks. https://arxiv.org/abs/2210.12538
IvanovADrydenNBen-NunT, et al. (2021) Data movement is all you need: a case study on optimizing transformers. In: Proceedings of Machine Learning and Systems, 3. MLSys 2021).
32.
KasimMFWatson-ParrisDDeaconuL, et al. (2021) Building high accuracy emulators for scientific simulations with deep neural architecture search. Machine Learning: Science and Technology3(1): 015013. DOI: 10.1088/2632-2153/ac3ffa
33.
KeryellRReyesRHowesL (2015) Khronos SYCL for OpenCL: A Tutorial. In: Proceedings of the 3rd International Workshop on OpenCL, IWOCL ’15. New York, NY, USA: Association for Computing Machinery. DOI: 10.1145/2791321.2791345
KutznerCPállSFechnerM, et al. (2019) More bang for your buck: Improved use of GPU nodes for GROMACS 2018. Journal of Computational Chemistry40(27): 2418–2431. DOI: 10.1002/jcc.26011
36.
LarsenESMcAllisterD (2001) Fast matrix multiplies using graphics hardware. In: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing, SC ’01. New York, NY, USA: Association for Computing Machinery, p. 55. DOI: 10.1145/582034.582089
37.
LattnerCAdveV (2004) LLVM: a compilation framework for lifelong program analysis & transformation. In: International Symposium on Code Generation and Optimization, 2004. CGO 2004, San Jose, CA, USA, 20–24 March 2004. pp. 75–86. DOI:10.1109/CGO.2004.1281665
38.
LavinAKrakauerDZenilH, et al. (2021) Simulation intelligence: towards a new generation of scientific methods. https://arxiv.org/abs/2112.03235
39.
LeeVWKimCChhuganiJ, et al. (2010) Debunking the 100X GPU vs. CPU myth: an evaluation of throughput computing on CPU and GPU. In: Proceedings of the 37th Annual International Symposium on Computer Architecture, ISCA ’10. New York, NY, USA: Association for Computing Machinery, pp. 451–460. DOI:10.1145/1815961.1816021
40.
LiHBergerDSNovakovicS, et al. (2022) Pond: CXL-Based memory pooling systems for cloud platforms. https://arxiv.org/abs/2203.00241
MatsuokaS (2008) The rise of the commodity vectors. In: PalmaJMLMAmestoyPRDaydéM, et al. (eds), High Performance Computing for Computational Science - VECPAR 2008. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 53–62. . DOI: 10.1007/978-3-540-92859-1_7
MatsuokaSDomkeJWahibM, et al. (2023) Myths and legends in high-performance computing. DOI: 10.48550/arXiv.2301.02432
46.
MenonHLamMOOsei-KuffuorD, et al. (2018) ADAPT: algorithmic differentiation applied to floating-point precision tuning. In: SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 614–626. DOI: 10.1109/SC.2018.00051
OotomoHYokotaR (2022) Recovering single precision accuracy from tensor cores while surpassing the FP32 theoretical peak performance. The International Journal of High Performance Computing Applications36(4): 475–491. DOI: 10.1177/10943420221090256
49.
PathakJSubramanianSHarringtonP, et al. (2022) FourCastNet: a global data-driven high-resolution weather model using adaptive fourier neural operators. https://arxiv.org/abs/2202.11214
RaspSPritchardMSGentineP (2018) Deep learning to represent subgrid processes in climate models. Proceedings of the National Academy of Sciences of the United States of America115(39): 9684–9689. DOI: 10.1073/pnas.1810286115
ShanYLinWGuoZ, et al. (2022) Towards a fully disaggregated and programmable data center. In: Proceedings of the 13th ACM SIGOPS Asia-Pacific Workshop on Systems, APSys ’22. New York, NY, USA: Association for Computing Machinery, pp. 18–28. DOI: 10.1145/3546591.3547527
54.
ShawDEDeneroffMMDrorRO, et al. (2008) Anton, a special-purpose machine for molecular dynamics simulation. Communications of the ACM51(7): 91–97. DOI: 10.1145/1364782.1364802
WuCJHsiaoSTWangJY, et al. (2021) Ultra High Power Cooling Solution for 3D-ICs. In: 2021 Symposium on VLSI Technology, pp. 1–2.
60.
ZiogasANSchneiderTBen-NunT, et al. (2021) Productivity, portability, performance: data-centric python. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’21. New York, NY, USA: Association for Computing Machinery, p. 13. DOI: 10.1145/3458817.3476176
