Simplified performance evaluation of floating-point formats for implementing intelligent measurement systems

Abstract

This paper provides a simple method for efficient performance evaluation of floating-point (FP) formats, addressing the challenge of implementing DNN (deep neural network)-based sensor nodes and edge measurement devices. Since resource constraints are imposed in such scenarios, the 32-bit FP format, standardly used for DNN implementation, is unsuitable, and the alternative is found in lower-resolution FP formats which leads to certain performance degradation. Hence, an efficient mechanism for performance evaluation of different FP formats is needed to examine the influence of resolution decreasing. Existing methods utilize the analogy between FP formats and piecewise uniform quantizers (PUQs), using the signal-to-quantization noise ratio (SQNR) of the PUQ to express the performance of FP formats. However, the high complexity of SQNR calculations, involving sum with many terms (e.g. 254 terms for FP formats with an 8-bit exponent), poses a significant challenge. This paper’s main contribution is the significant simplification of the SQNR expression for Gaussian-distributed data, reducing the number of sum terms from 254 to just 5 with minimal accuracy loss, allowing for simple and efficient performance evaluation of FP formats. Major findings include an in-depth analysis of the probability distribution across PUQ segments, a closed-form expression for identifying the highest probability segment, and an evaluation of the SQNR approximation’s accuracy. These findings provide a foundational basis for implementing intelligent DNN-based measurement systems, with applications extending to computing, signal processing, and other fields utilizing FP formats.

Introduction

Due to the dominance of digital measurement systems, analyzing formats for digital representation of measurement data has become a crucial research task. The two main types of digital formats are fixed-point (FxP) and floating-point (FP) formats (Dinčić et al., 2023). FxP is simpler to implement, while FP offers a much wider variance range of consistent representation quality, making it preferable for numerous applications, including computing, signal processing (Moroz and Samotyy, 2019), and sensor data representation (Chen et al., 2024). The most commonly used is the 32-bit floating-point format (FP32) (Perić et al., 2021), providing high-quality digital representation over a very wide range of data variance, although it is complex to implement. The use of the FP32 format is especially relevant for the implementation of deep neural networks (DNNs), being used by default for representing DNN parameters (weights, activations, etc.) and input data. DNNs are increasingly used for processing of measurement signals, such as electrocardiogram (ECG) (Degachi and Ouni, 2025), electroencephalogram (EEG) (Li et al., 2024), and vibration signals (Syu and Lee, 2024), for the implementation of measurement systems (Chen and Chen, 2022; Chen et al., 2023; Liang and Huang, 2023; Liu and Yang, 2024; Lou et al., 2024), control systems (Bey and Chemachema, 2024; Xi et al., 2024), soft sensors (Luo et al., 2024; Sun and Ge, 2021), sensor networks (Kim et al., 2021; Wu et al., 2023) and Internet of Things (IoT) systems (Mohapatra et al., 2024; Mukhopadhyay et al., 2021), as well as for sensor linearization (Anandanatarajan et al., 2023), sensor drift compensation (Chaudhuri et al., 2021), and sensor fusion (Balemans et al., 2023). Initially, DNN-based measurement and control systems were implemented such that DNNs ran on powerful servers and clouds (Alamri et al., 2013), transmitting measurement data from sensors via the Internet and causing in that way latency and security issues (Sahni et al., 2022). Recently, local implementation of DNN algorithms on sensor nodes and edge measurement devices to be as close as possible to the source of data has become a key research focus, enabling the realization of intelligent and automated measurement and control systems capable of real-time data processing and decision-making with enhanced security and reliability and suitable for various applications (Al Koutayni et al., 2023; Fanariotis et al., 2023; Lai and Chang, 2023; Ronco et al., 2022; Yadav et al., 2024; Zhou et al., 2021).

However, the main challenge for DNN-based sensor nodes and edge measurement devices is their limited hardware resources (processing power, memory capacity, and especially available energy in battery-powered sensor nodes and edge devices), which makes implementation of the FP32 format very difficult due to its complexity (Syed et al., 2021). Consequently, considering lower resolution FP formats (e.g. 24-bit FP24 (Junaid et al., 2022), 16-bit bfloat16 (Agrawal et al., 2019), 8-bit FP8 (Wang et al., 2018), or some other) for implementation of intelligent sensor nodes and measurement devices becomes a topical research direction, with the aim of reducing complexity. However, lower resolution reduces performance by decreasing the quality of digital representation and narrowing the variance range, risking inadequate quality and variance range for specific applications. Thus, a critical research gap exists in developing efficient methods to evaluate the performance of various FP formats in terms of representation quality and variance range. Addressing this gap is essential for selecting the optimal FP format for each specific application and input data set, enabling resource-efficient implementation of intelligent DNN-based sensor nodes and measurement systems.

A significant advance in evaluating FP format performance was achieved in Perić et al. (2021), where an analogy between the FP format and the piecewise uniform quantizer (PUQ) was established. Quantization error is expressed by an objective measure called distortion. It is common to define distortion using the L₂ norm as the mean squared quantization error, although an alternative approach defines distortion via the L₁ norm as the mean absolute quantization error. Signal-to-quantization noise ratio (SQNR) represents another widely used objective performance measure of the quantizer. The typical approach, which will be used in this paper, calculates SQNR using the L₂ norm, as the logarithmic ratio (expressed in dB) of data variance (representing the mean squared value of the data) to distortion calculated as the mean squared quantization error. However, some studies use an alternative method to calculate SQNR with the L₁ norm, as the logarithmic ratio of the mean absolute value of the data to distortion calculated as the mean absolute quantization error (Marco and Neuhoff, 2006). In both cases, a higher quantization error results in greater distortion, leading to a lower SQNR.

The established analogy between the FP format and the PUQ allows us to express FP format performance through an objective measure such as the SQNR of the PUQ. However, the SQNR expression is very complex, containing the sum of a large number of terms (e.g. 254 for bfloat16, FP24, and FP32 formats with an 8-bit exponent (Perić and Dinčić, 2023; Perić et al., 2021)), each corresponding to one PUQ segment. In addition, the need for repeating SQNR calculation for various variances within a wide variance range (over 1500 dB for formats like bfloat16, FP24, and FP32) further increases complexity, making performance evaluation of FP formats very impractical. This paper adopts the idea of using the SQNR of the equivalent PUQ as a performance metric for FP formats, addressing in the same time the problem of high computational complexity of the exact SQNR expression and offering a simple and efficient solution for SQNR calculation, thereby enabling effective performance evaluation of FP formats. Considering that the quality of FP representation of DNN parameters, expressed by SQNR, greatly affects the prediction accuracy of DNNs, this paper will offer solutions for simplifying DNN implementations in resource-constrained environments without compromising DNNs accuracy, as a significant contribution to the development of intelligent DNN-based sensor nodes and measurement systems.

It is known that the performance of the FP format depends on the probability density function (PDF) of the measurement data (Perić et al., 2021). This paper focuses on the Gaussian PDF, which is effective for modeling stochastic measurement data (Dincic et al., 2013, 2014, 2016, 2023). Moreover, the fact that non-Gaussian signals can be transformed into Gaussian by appropriate filters (Dinčić et al., 2023) broadens the applicability of paper results.

The main novelty of this paper is a new approach to SQNR calculation, based on the analysis of the probability distribution of PUQ segments and the impact that each individual PUQ segment has on SQNR. As an important result of this new approach, it has been shown that, for any given variance, only a few segments have probabilities significantly different from zero, affecting the SQNR, while all other segments have negligible probabilities with negligible impact on SQNR. This led to the key insight that SQNR calculation can be drastically simplified by focusing just on the segment with the highest probability and a few adjacent segments. As the most probable segment of PUQ varies with changing data variance, a significant contribution of this paper is the derivation of a simple closed-form formula for determining the segment with the highest probability valid for any variance value. In addition, this paper explores different numbers of segments left and right to the most probable segment to be included in the SQNR expression, aiming to obtain a tight approximation of the original SQNR. A crucial finding achieved in this way is a very simple approximate expression for calculating SQNR involving only five terms in the sum, which drastically reduces the complexity of SQNR calculation compared to the original SQNR expression. In addition, this paper examines how different components of distortion (representing the mean squared quantization error) affect SQNR across a wide variance range, revealing that some components have negligible impact in certain variance ranges. This further simplifies the SQNR expression, as another significant finding of this paper.

This paper focuses on evaluating the performance of FP formats with an 8-bit exponent (FP32, FP24, and bfloat16) due to their particularly high complexity of calculating exact SQNR values. It provides numerical values for the exact SQNR across these formats, obtained using the complex formula involving a 254-term sum, as well as approximate SQNR values calculated by the simplified expression with only a five-term sum derived in this paper. Calculations were conducted over a broad range of variance values. Results demonstrate that the simplified SQNR expression, developed in this paper, achieves an exceptionally low error of just 0.024 dB relative to the exact SQNR formula, thereby demonstrating that the substantial reduction in computational complexity achieved by using the approximate SQNR expression does not compromise calculation accuracy. Numerical calculations show that all three considered FP formats maintain a nearly constant SQNR across an extensive variance range from approximately −750 dB to beyond 750 dB. Within this range, the SQNR values are approximately 151.9 dB for FP32, 103.75 dB for FP24, and 55.6 dB for bfloat16, indicating that the reduction of resolution for 8 bits decreases SQNR by roughly 48.15 dB. On the contrary, the reduction in resolution simplifies implementation complexity. Hence, the most suitable FP format should be determined for each specific application by balancing implementation complexity with performance. MATLAB simulations with 10 million randomly generated Gaussian numbers, as well as an experiment performed with weights of a trained MLP (Multilayer Perceptron) neural network, confirm the theoretical results.

To summarize, given the impact of SQNR in FP representation of DNN parameters on DNN accuracy, alongside the substantial effect of FP formats on DNN implementation complexity, it becomes essential to assess FP formats in the context of DNN deployment. This is particularly relevant when considering the potential application of DNNs in resource-constrained measurement systems, which could become more intelligent and automated through DNN integration. This paper provides some fundamental results related to simplified and efficient calculation of FP format performance across various resolutions and a wide range of variance, facilitating the use of lower-resolution FP formats than FP32 and reducing in that way implementation complexity and energy consumption while accelerating data processing. These findings have substantial practical applications for assessing the performance of various FP formats and selecting the most suitable format for each specific case. This is particularly valuable for developing intelligent DNN-based measurement systems with limited hardware resources and constrained available energy (such as battery-powered systems), as well as real-time measurement systems. One potential application is in vibration-based predictive maintenance measurement systems in industry, where high-frequency signal sampling generates large amounts of data, requiring efficient and low-complexity solutions. Furthermore, the results of this paper can be applied across many other fields, including computing and signal processing, due to the broad applicability of FP formats.

This paper is organized as follows. Section “Structure and performance evaluation of the FP format” explains the FP format structure and its analogy with the PUQ, from which the expression for SQNR calculation is derived. The section “Efficient performance evaluation of FP formats”, as the main part of this paper, analyzes the impact of quantizer segments on SQNR, derives the closed-form expression for determining the most probable segment, proposes simple approximate SQNR expressions, and examines the influence of various distortion components on SQNR for different variances, further simplifying the SQNR expression. This section also presents numerical results based on the theoretical analysis, as well as simulation results provided to confirm the achieved numerical results. The next section, “Experimental results,” validates the correctness of the developed theory by applying the proposed approach to the data set consisting of weights of a trained MLP neural network. The conclusion and list of references are given at the end of this paper.

Structure and performance evaluation of the FP format

A real number x is represented in an R-bit FP format as (Perić et al., 2021)

$$x=(s{a}_{e-1}...a_1{a}_0{b}_{m-1}...b_1{b}_0{)}_2$$

(1)

where one bit “s” encodes the sign of the number x, e bits $(a_{e-1}...a_1{a}_0)$ encode the exponent, and m bits $(b_{m-1}...b_1{b}_0)$ encode the significand, whereby $R=e+m+1$ . The FP format is symmetric around 0, with each positive number having a negative counterpart. FP numbers represented by equation (1) can be categorized into two groups: normal and subnormal. Normal FP numbers are calculated as

$$x=(-1{)}^s{2}^{E^{*}}(1+\frac{M}{2^m})$$

(2)

where $E=\sum _{i=0}^{e-1}{a}_i{2}^i$ is the exponent, $E^{*}=E-\mathrm{bias}$ denotes the biased exponent with bias being a predefined parameter, and $M=\sum _{i=1}^m{b}_{m-i}{2}^{m-i}$ . The exponent E can take values from 0 to $2^e-1$ , but the values 0 (all zeros) and $2^e-1$ (all ones) are reserved, leaving $2^e-2$ values (from 1 to $2^e-2$ ) for encoding numbers. This results in $2^e-2$ values of the biased exponent $E^{*}$ (from its minimum value $E_{\min }^{*}=1-\mathrm{bias}$ to its maximum value $E_{\max }^{{*}^e}=2^e-2-\mathrm{bias}$ ) usable for numbers’ representation. The parameter M can take $2^m$ values from 0 to $2^m-1$ . The smallest positive normal FP number, with the minimum values of $E^{*}$ and M ( $E^{*}=E_{\min }^{*}$ and $M=0$ ), according to equation (2), is $x_{\min }=2^{E_{\min }^{*}}$ , while the largest positive normal FP number obtained for the maximum values of $E^{*}$ and M ( $E^{*}=E_{\max }^{*}$ and $M=2^m-1$ ) is $x_{\max }=2^{E_{\max }^{*}}(1+\frac{2^m-1}{2^m})=2^{E_{\max }^{*}}(2-\frac{1}{2^m})\approx 2^{E_{\max }^{*}+1}$ . Thus, normal FP numbers defined by equation (2) are placed in the range $(-x_{\max },-x_{\min }]\cup [x_{\min },x_{\max })$ . For example, for FP32, FP24, and bfloat16 formats with e = 8, we have $E_{\min }^{*}$  = −126, $E_{\max }^{*}$  = 127, $x_{\min }=2^{-126}$ and $x_{\max }=2^{128}$ .

Very small FP numbers in the interval ( $-x_{\min }$ , $x_{\min }$ ) around 0 are called subnormal FP numbers, calculated as

$$x=(-1{)}^s·\frac{M}{2^m}·2^{1-\mathrm{bias}}=(-1{)}^s·\frac{M}{2^m}·2^{E_{\min }^{*}}$$

(3)

There are a total of $2^m$ positive subnormal FP numbers (one for each value of M), and the same number of negative ones.

Due to symmetry around 0, we can focus solely on positive FP numbers. Let us start with the positive subnormal numbers in the range (0, $x_{\min }$ ). The distance between two consecutive subnormal numbers

$${\Delta }_{\mathrm{sub}}=\frac{M+1}{2^m}·2^{E_{\min }^{*}}-\frac{M}{2^m}·2^{E_{\min }^{*}}=2^{E_{\min }^{*}-m}$$

(4)

is constant, making the subnormal FP numbers equidistant. Therefore, the structure of subnormal FP numbers is the same as the structure of a uniform quantizer with a quantization step size ${∆}_{\mathrm{sub}}$ . Next, let us examine the positive normal FP numbers in the range $[x_{\min },x_{\max })$ , defined as $x=2^{E^{*}}(1+\frac{M}{2^m})$ . Each value of the biased exponent $E^{*}$ ( $E_{\min }^{*}\le E^{*}\le E_{\max }^{*}$ ) defines a group of $2^m$ numbers belonging to the region $S_{E^{*}}=[2^{E^{*}},2^{E^{*}+1})$ , each corresponding to one value of M. The distance between two consecutive numbers within one group (for one value of $E^{*}$ )

$${∆}_{E^{*}}=2^{E^{*}}(1+\frac{M+1}{2^m})-2^{E^{*}}(1+\frac{M}{2^m})=2^{E^{*}-m}$$

(5)

is constant, but varies between different groups, making the numbers within one group equidistant. Therefore, each group of $2^m$ equidistant normal FP numbers for a given value of $E^{*}$ has the structure of a uniform quantizer with a quantization step size ${∆}_{E^{*}}$ . In total, there are $2^e-2$ groups of $2^m$ equidistant positive normal FP numbers. This pattern extends symmetrically to negative FP numbers.

It is evident that the structure of the FP format is equivalent to a symmetric PUQ with a support region $(-x_{\max },x_{\max })$ . The positive part of the PUQ consists of $2^e-1$ segments in a way that each segment undergoes uniform quantization with $2^m$ quantization levels. The first segment is the subnormal segment in the range (0, $x_{\min }$ ) with a quantization step size ${∆}_{\mathrm{sub}}$ . Also, there are $2^e-2$ segments $S_{E^{*}}=[2^{E^{*}},2^{E^{*}+1})$ , $E_{\min }^{\ast }\le E^{\ast }\le E_{\max }^{\ast }$ , representing normal FP numbers, whereby the segment $S_{E^{*}}$ corresponds to the quantization step size ${∆}_{E^{*}}$ . This PUQ, analogous to the structure of the FP format, is called the Floating-Point Quantizer (FPQ). This analogy between the FP format and the piecewise uniform FPQ is crucial, allowing us to evaluate performance of the FP format through an objective measure such as the SQNR of the FPQ.

The SQNR of the quantizer is defined as (Perić et al., 2021)

$$\mathrm{SQNR}(\sigma )=10·{\log }_{10}\frac{{\sigma }^2}{D(\sigma )}$$

(6)

where ${\sigma }^2$ is the data variance, and $D(\sigma )$ is the distortion representing the mean square error made during quantization. The distortion $D(\sigma )$ of the piecewise uniform FPQ is defined as

$$D(\sigma )=D_{\mathrm{sub}}(\sigma )+D_n(\sigma )+D_{\mathrm{ov}}(\sigma )$$

(7)

where $D_{\mathrm{sub}}(\sigma )$ represents the distortion in the range of subnormal numbers, $D_n(\sigma )$ represents the distortion in the range of normal numbers, and $D_{\mathrm{ov}}(\sigma )$ represents the overload distortion that occurs when quantizing data outside the quantizer support region $(-x_{\max },x_{\max })$ . Since $D_{\mathrm{sub}}(\sigma )$ represents the granular distortion of the uniform quantizer equivalent to subnormal numbers with the quantization step ${∆}_{\mathrm{sub}}$ , it is defined as

$$D_{\mathrm{sub}}(\sigma )=2\frac{{∆}_{\mathrm{sub}}^2}{12}{P}_{\mathrm{sub}}(\sigma )$$

(8)

Here, $P_{\mathrm{sub}}(\sigma )$ denotes the probability of the positive subnormal segment $(0,x_{\min })$ , given by $P_{\mathrm{sub}}\left(\sigma \right)={\int }_0^{x_{\min }}p(x,\sigma )\mathrm{dx}$ , where $p(x,\sigma )$ is the PDF of the input data.

Distortion $D_n(\sigma )$ is expressed through the sum of $2^e-2$ terms as

$$D_n\left(\sigma \right)=2{\sum }_{E^{*}=E_{\min }^{*}}^{E_{\max }^{*}}{D}_{E^{*}}\left(\sigma \right)$$

(9)

where each term $D_{E^{*}}(\sigma )$ corresponds to a value of $E^{*}$ ranging from $E_{\min }^{*}$ to $E_{\max }^{*}$ , representing the distortion of the uniform quantizer within the segment $S_{E^{*}}=[2^{E^{*}},2^{E^{*}+1})$ with the step size ${\Delta }_{E^{\ast }}$ . Consequently, $D_{E^{*}}(\sigma )$ is defined as

$$D_{E^{*}}(\sigma )=\frac{{∆}_{E^{*}}^2}{12}{P}_{E^{*}}(\sigma )$$

(10)

where $P_{E^{*}}(\sigma )={\int }_{2^{E^{*}}}^{2^{(E^{*}+1)}}p(x,\sigma )\mathrm{dx}$ represents the probability of the segment $S_{E^{*}}$ .

Finally, the overload distortion is defined as

$$D_{ov}\left(\sigma \right)=2\underset{x_{\max }}{\overset{+\infty }{\int }}{\left(x-x_{\max }\right)}^{2}p\left(x,\sigma \right)dx$$

(11)

Multiplying by 2 in expressions (8), (9), and (11) includes the distortion in the negative part of the quantizer.

This paper considers the Gaussian PDF, defined as (Dinčić et al., 2023)

$$p(x,\sigma )=\frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{x^2}{2{\sigma }^2})$$

(12)

which has been successfully used for statistical modeling of measurement data. For $p(x,\sigma )$ defined by equation (12), we obtain the following expressions for $P_{\mathrm{sub}}(\sigma )$ , $P_{E^{*}}\left(\sigma \right),$ and $D_{\mathrm{ov}}(\sigma )$

$$P_{\mathrm{sub}}\left(\sigma \right)=\frac{1}{2}\mathrm{erf}\left(\frac{2^{E_{\min }^{*}-1/2}}{\sigma }\right)$$

(13)

$$P_{E^{*}}(\sigma )=\frac{1}{2}(\mathrm{erf}\left(\frac{2^{E^{*}+1/2}}{\sigma }\right)-\mathrm{erf}\left(\frac{2^{E^{*}-1/2}}{\sigma }\right))$$

(14)

$$D_{ov}\left(\sigma \right)=-x_{\max }\sigma \sqrt{\frac{2}{\pi }}\exp \left(-\frac{x_{\max }^2}{2{\sigma }^2}\right)+\left(x_{\max }^2+{\sigma }^2\right)\mathrm{erfc}\left(\frac{x_{\max }}{\sigma \sqrt{2}}\right)$$

(15)

In scenarios involving a wide range of data variance, such as the case in FP formats, it is common to express variance in the logarithmic domain as $\alpha [\mathrm{dB}]=10{\log }_{10}({\sigma }^2/{\sigma }_{\mathrm{ref}}^2)$ , where ${\sigma }_{\mathrm{ref}}^2$ is a reference variance. If assume ${\sigma }_{\mathrm{ref}}^2$  = 1 without loss of generality, the definition of variance in the logarithmic domain becomes $\alpha [\mathrm{dB}]=10{\log }_{10}{\sigma }^2$ , leading to the relationship

$$\sigma ={10}^{\alpha /20}$$

(16)

Taking into account expressions from (6) to (16), we obtain the final expression for SQNR of the FPQ

$$\begin{array}{l}\mathrm{SQNR}\left(\alpha \right)=-10{\log }_{10}[\frac{2^{-2\left(R-e-1\right)}}{12\cdot {10}^{\alpha /10}}\left(2^{2E_{\min }^{*}}\mathrm{erf}\left(\frac{2^{E_{\min }^{*}-1/2}}{{10}^{\alpha /20}}\right)+\sum _{E^{*}=E_{\min }^{*}}^{E_{\max }^{*}}{2}^{2E^{*}}\left(\mathrm{erf}\left(\frac{2^{E^{*}+1/2}}{{10}^{\alpha /20}}\right)-\mathrm{erf}\left(\frac{2^{E^{*}-1/2}}{{10}^{\alpha /20}}\right)\right)\right)\\ -\frac{x_{\max }}{{10}^{\alpha /20}}\sqrt{\frac{2}{\pi }}\exp \left(-\frac{x_{\max }^2}{2\cdot {10}^{\alpha /10}}\right)+\left(\frac{x_{\max }^2}{{10}^{\alpha /10}}+1\right)\mathrm{erfc}\left(\frac{x_{\max }}{{10}^{\alpha /20}\cdot \sqrt{2}}\right)]\end{array}$$

(17)

Expression (17) is crucial as it enables us to evaluate the performance of FPQ, and consequently the FP format, for any variance α of measurement data. However, expression (17) presents a challenge due to its extensive number of terms, particularly notable in FP formats with e = 8, like bfloat16, FP24, and FP32, with 254 terms in the sum. This complexity significantly increases the computational burden of calculating SQNR. As additional difficulty, SQNR computation needs to be repeated for various values of α over a very wide range, exceeding 1500 dB for bfloat16, FP24, and FP32. Therefore, a key research objective is to simplify expression (17), aiming for a faster and more efficient method to assess FP format performance with minimal computational overhead, while maintaining accuracy. This paper focuses on addressing this critical issue.

Efficient performance evaluation of FP formats

Figure 1 shows the probabilities $P_{E^{*}}$ of segments $S_{E^{*}}$ , ( $E_{\min }^{*}\le E^{*}\le E_{\max }^{*}$ ) of the 32-bit FPQ, analogous to the FP32 format (see equation (14)), for various variance values α. We observe that for a given α, only a few FPQ segments exhibit non-negligible probabilities, while others approach zero. This pattern holds across different FP formats, not limited to FP32. An important conclusion arises from this, that only a small number of FPQ segments impact SQNR. Thus, the summation in equation (17) can be simplified by including much fewer terms without sacrificing accuracy of SQNR calculation. The rest of this paper is dedicated to implementing this idea.

Figure 1.

Probabilities of 32-bit FPQ segments for different values of the variance α.

Let $E_x^{*}$ denote the index of the FPQ segment with the highest probability for a given variance α, that is, $P_{E_x^{*}}\left(\alpha \right)=\underset{E_{min}^{*}\le E^{*}\le E_{max}^{*}}{max}\left\{P_{E^{*}}\left(\alpha \right)\right\}$ . As illustrated in Figure 1, various α values yield different values of $E_x^{*}$ . It is evident that the segment with the highest probability $S_{E_x^{*}}$ , along with several adjacent segments, has the greatest impact on SQNR. Hence, the primary objective of this paper is to efficiently determine the value of $E_x^{*}$ for any value of variance α. One way to find $E_x^{*}$ is through exhaustive numerical computation, calculating the probabilities of all segments $E^{*}$ , $(E_{\min }^{*}\le E^{*}\le E_{\max }^{*})$ , and identifying the segment with the highest probability; this procedure should be repeated for each value of α and the resulting values of $E_x^{*}$ should be stored in memory. However, due to numerous segments and the wide variance range (e.g. 254 segments and over 1500 dB wide variance range for FP32, FP24, and bfloat16 formats), this method is computationally intensive and memory-demanding, rendering it impractical. Consequently, this paper proposes an alternative approach presented below, aiming to derive a closed-form expression for simple calculation of $E_x^{*}$ .

By differentiating the expression for $P_{E^{*}}(\alpha )$ , obtained by putting equation (16) into equation (14), with respect to $E^{*}$ and solving the equation $\frac{d{P}_{E^{*}}(\alpha )}{d{E}^{*}}=0$ , as it will be explained in Appendix, we obtain

$$E_x^{*}=\frac{\ln \left(\frac{2}{3}\ln 2\right)}{2\ln 2}+\frac{\ln 10}{20·\ln 2}\alpha =-0.557+\frac{\alpha }{6.0206}$$

(18)

Note that expression (18) provides the value of $E_x^{*}$ for each specific value of variance α. However, $E_x^{*}$ obtained by equation (18) is a real number, but we need an integer value. Therefore, we initially round $E_x^{*}$ obtained by equation (18) to the nearest integer as

$$E_x^{*}=\mathrm{round}(-0.557+\frac{\alpha }{6.0206})$$

(19)

which gives accurate $E_x^{*}$ values for most values of α, but for some values of α results in an error of 1, yielding the index of a neighboring segment instead of the segment with the highest probability. To solve this problem, we introduce a constant C as a correction factor, so that equation (19) becomes

$$E_x^{*}=\mathrm{round}(-0.557+C+\frac{\alpha }{6.0206})$$

(20)

Analysis carried out over a very wide range of variance α (from −800 dB to 800 dB) shows that C = −0.048 provides accurate values for $E_x^{*}$ . Thus, the final expression for calculating $E_x^{*}$ for a specific value of variance α is

$$E_x^{*}=\mathrm{round}(-0.605+\frac{\alpha }{6.0206})$$

(21)

By introducing a small correction factor C, the rounding is finely tuned, ensuring accurate $E_x^{*}$ values. The simple calculation of $E_x^{*}$ for each value of variance α using the closed-form expression (21) represents a significant contribution of this paper.

As mentioned, only the distortion in the segment $S_{E_x^{*}}$ with the highest probability and a few adjacent segments significantly impact SQNR. This allows us to drastically reduce the number of terms in the sum of SQNR expression (17). Lets $\mathrm{SQN}{\mathrm{R}}^{\prime }$ denote the approximate SQNR value, calculated using the following simplified formula

$$\begin{array}{l}{\mathrm{SQNR}}^{\prime }\left(\alpha \right)=-10{\log }_{10}\\ [\frac{2^{-2\left(R-e-1\right)}}{12\cdot {10}^{\alpha /10}}\left(2^{2E_{\min }^{*}}\mathrm{erf}\left(\frac{2^{E_{\min }^{*}-1/2}}{{10}^{\alpha /20}}\right)+\sum _{E^{*}=E_{\mathrm{down}}^{*}}^{E_{\mathrm{up}}^{*}}{2}^{2E^{*}}\left(\mathrm{erf}\left(\frac{2^{E^{*}+1/2}}{{10}^{\alpha /20}}\right)-\mathrm{erf}\left(\frac{2^{E^{*}-1/2}}{{10}^{\alpha /20}}\right)\right)\right)\\ -\frac{x_{\max }}{{10}^{\alpha /20}}\sqrt{\frac{2}{\pi }}\exp \left(-\frac{x_{\max }^2}{2\cdot {10}^{\alpha /10}}\right)+\left(\frac{x_{\max }^2}{{10}^{\alpha /10}}+1\right)\mathrm{erfc}\left(\frac{x_{\max }}{{10}^{\alpha /20}\cdot \sqrt{2}}\right)],\end{array}$$

(22)

where

$$E_{\mathrm{down}}^{*}=\max \{E_x^{*}-k_1,E_{\min }^{*}\}$$

(23)

and

$$E_{\mathrm{up}}^{*}=\min \{E_x^{*}+k_2,E_{\max }^{*}\}$$

(24)

indicate the minimum and maximum values of the summation index. The expression (22) takes into account only $k_1$ segments to the left of the segment $S_{E_x^{*}}$ and only $k_2$ segments to the right. Definitions (23) and (24) for $E_{\mathrm{down}}^{*}$ and $E_{\mathrm{up}}^{*}$ allow summation in equation (22) to start from $E_{\min }^{*}$ if $E_x^{*}-k_1<E_{\min }^{*}$ and end up at $E_{\max }^{*}$ if $E_x^{*}+k_2>E_{\max }^{*}$ . Now, we need to determine the values of parameters $k_1$ and $k_2$ . Let $\Delta \mathrm{SQN}{\mathrm{R}}_{\max }=\max |\mathrm{SQNR}\left(\alpha \right)-\mathrm{SQN}{\mathrm{R}}^{\prime }\left(\alpha \right)|$ represent the maximum difference between the exact SQNR value calculated by equation (17) and the approximate value $\mathrm{SQNR}\prime$ calculated by equation (22), across the whole variance range of interest. Table 1 provides values of $∆\mathrm{SQN}{\mathrm{R}}_{\max }$ for different combinations of $k_1$ and $k_2$ within the variance range from −800 dB to 800 dB, covering bfloat16, FP24, and FP32 formats. It is found that the same values of $\Delta \mathrm{SQN}{\mathrm{R}}_{\max }$ are obtained for all three mentioned formats.

Table 1.

Maximal error $∆\mathrm{SQN}{\mathrm{R}}_{\max }$ of SQNR calculation for different values of $k_1$ and $k_2$ .

$k_1$	2	1	3	2	2
$k_2$	2	2	2	1	3
$\Delta {\mathrm{SQNR}}_{max}$ [dB]	0.024	0.087	0.022	2.251	0.011

From Table 1, we see that the combination ( $k_1$  = 2, $k_2$  = 1) results in an unacceptably large SQNR calculation error of 2.251 dB, so we will discard it. The remaining four-parameter combinations are excellent solutions, yielding very small SQNR calculation errors. Note that the accuracy of approximation (22) for ( $k_1$  = 3, $k_2$  = 2) is better for only 0.002 dB compared to the case ( $k_1$  = 2, $k_2$  = 2) but requires an additional term in the SQNR expression, so we will discard it as well. The choice among the remaining three combinations depends on the required SQNR calculation accuracy for specific application. The combination ( $k_1$  = 1, $k_2$  = 2) results in four terms in the SQNR sum in equation (22) but has the highest error, while ( $k_1$  = 2, $k_2$  = 3) produces six terms but the smallest error. This paper opts for ( $k_1$  = 2, $k_2$  = 2) which results in five terms in equation (22) and maximal calculation error of just 0.024 dB. One more term in the SQNR sum in equation (22) produced by ( $k_1$  = 2, $k_2$  = 2) compared to ( $k_1$  = 1, $k_2$  = 2) is justified by a reasonable SQNR error reduction of 0.063 dB; on the contrary, one extra term produced by ( $k_1$  = 2, $k_2$  = 3) compared to ( $k_1$  = 2, $k_2$  = 2) is not justified, as the accuracy of approximation (22) is increased by only 0.013 dB. The proposed method, which reduces the number of terms in the SQNR expression from a large number (254 for bfloat16, FP24, and FP32 formats) to just 5, thereby significantly decreasing the complexity of SQNR calculation while maintaining high accuracy with a negligible error of only 0.024 dB, represents a significant achievement in this paper.

The expression (22) for SQNR can be further simplified. Specifically, $D_{\mathrm{sub}}$ is significant only for extremely small variances and negligible for others, while $D_{\mathrm{ov}}$ is significant only for extremely large variances and negligible for others. Let us define the variance thresholds ${\alpha }_1$ , ${\alpha }_2$ , ${\alpha }_3$ , and ${\alpha }_4$ , which correspond to the following variance range:

• Range I, with $\alpha \le {\alpha }_1$ , is the range of extremely small variances where $D_{\mathrm{sub}}$ dominates while $D_n$ and $D_{\mathrm{ov}}$ are negligible; consequently, in expression (22), only the term originating from $D_{\mathrm{sub}}$ remains, allowing expression (22) to be simplified as follows

$${\mathrm{SQNR}}_{\mathrm{I}}^{\prime }\left(\alpha \right)=-10{\log }_{10}\left[\frac{2^{2\left(E_{\min }^{*}-R+e+1\right)}}{12\cdot {10}^{\alpha /10}}\mathrm{erf}\left(\frac{2^{E_{\min }^{*}-1/2}}{{10}^{\alpha /20}}\right)\right],\alpha \le {\alpha }_1$$

(25)

• Range II, with ${\alpha }_1<\alpha \le {\alpha }_2$ , is the range of small variances where both $D_{\mathrm{sub}}$ and $D_n$ have an influence, while $D_{\mathrm{ov}}$ is negligible; hence, terms derived from $D_{\mathrm{sub}}$ and $D_n$ remain in expression (22)

$$\begin{array}{c}{\mathrm{SQNR}}_{\mathrm{II}}^{\prime }\left(\alpha \right)=-10\mathrm{lo}{\mathrm{g}}_{10}\left[\frac{2^{-2(R-e-1)}}{12·{10}^{\alpha /10}}(2^{2E_{\min }^{*}}\mathrm{erf}\left(\frac{2^{E_{\min }^{*}-1/2}}{{10}^{\alpha /20}}\right)+\sum _{E^{*}=E_x^{*}-2}^{E_x^{*}+2}{2}^{2E^{*}}(\mathrm{erf}\left(\frac{2^{E^{*}+1/2}}{{10}^{\alpha /20}}\right)-\mathrm{erf}\left(\frac{2^{E^{*}-1/2}}{{10}^{\alpha /20}}\right)))\right],\\ {\alpha }_1<\alpha \le {\alpha }_2\end{array}$$

(26)

• Range III, with ${\alpha }_2<\alpha \le {\alpha }_3$ , is the very wide range of variances where only $D_n$ dominates, while $D_{\mathrm{sub}}$ and $D_{\mathrm{ov}}$ are negligible; therefore, only the term originating from $D_n$ remains in equation (22), reducing it as follows

$$\begin{array}{c}\mathrm{SQN}{\mathrm{R}\prime }_{\mathrm{III}}(\alpha )=-10{\log }_{10}\mathrm{[}\sum _{E^{\ast }\mathrm{=}{E}_x^{\ast }-2}^{E_x^{\ast }\mathrm{+}2}\frac{2^{2(E^{\ast }-R\mathrm{+}e\mathrm{+}1)}}{12·{10}^{\alpha /10}}\mathrm{(erf}\left(\frac{2^{E^{\ast }\mathrm{+}1/2}}{{10}^{\alpha /20}}\right)\\ -\mathrm{erf}\left(\frac{2^{E^{\ast }-1/2}}{{10}^{\alpha /20}}\right)\mathrm{)]},{\alpha }_2\mathrm{<}\alpha \le {\alpha }_3\end{array}$$

(27)

• Range IV, with ${\alpha }_3<\alpha \le {\alpha }_4$ , is the range of large variances where both $D_n$ and $D_{\mathrm{ov}}$ have an influence, while $D_{\mathrm{sub}}$ is negligible; hence, terms derived from $D_n$ and $D_{\mathrm{ov}}$ remain in equation (22), simplifying it as follows

$${\mathrm{SQNR}}_{\mathrm{IV}}^{\prime }\left(\alpha \right)=-10\mathrm{lo}{\mathrm{g}}_{10}[\sum _{E^{*}=E_x^{*}-2}^{E_x^{*}+2}\frac{2^{2(E^{*}-R+e+1)}}{12·{10}^{\alpha /10}}(\mathrm{erf}\left(\frac{2^{E^{*}+1/2}}{{10}^{\alpha /20}}\right)-\mathrm{erf}\left(\frac{2^{E^{*}-1/2}}{{10}^{\alpha /20}}\right))-\frac{x_{\max }}{{10}^{\alpha /20}}\sqrt{\frac{2}{\pi }}\exp (-\frac{x_{\max }^2}{2·{10}^{\alpha /10}})+(\frac{x_{\max }^2}{{10}^{\alpha /10}}+1)\mathrm{erfc}\left(\frac{x_{\max }}{{10}^{\alpha /20}·\sqrt{2}}\right)],{\alpha }_3<\alpha \le {\alpha }_4$$

(28)

• Range V, with $\alpha >{\alpha }_4$ , is the range of very large variances where only $D_{\mathrm{ov}}$ dominates, while $D_{\mathrm{sub}}$ and $D_n$ are negligible; consequently, only the term originating from $D_{\mathrm{ov}}$ remains in equation (22), allowing the following simplification of equation (22)

$${\mathrm{SQNR}}_{\mathrm{V}}^{\prime }\left(\alpha \right)=-10\mathrm{lo}{\mathrm{g}}_{10}[-\frac{x_{\max }}{{10}^{\frac{\alpha }{20}}}\sqrt{\frac{2}{\pi }}\exp (-\frac{x_{\max }^2}{2·{10}^{\frac{\alpha }{10}}})+(\frac{x_{\max }^2}{{10}^{\frac{\alpha }{10}}}+1)\mathrm{erfc}\left(\frac{x_{\max }}{{10}^{\frac{\alpha }{20}}·\sqrt{2}}\right)],\alpha >{\alpha }_4$$

(29)

Hence, the approximate expression (22) for SQNR can be written in a simplified way as

$$\mathrm{SQNR}\prime (\alpha )=\{\begin{array}{c}\mathrm{SQN}{\mathrm{R}\prime }_I(\alpha ),\alpha \le {\alpha }_1\hfill \\ \multicolumn{1}{c}{\mathrm{SQN}{\mathrm{R}\prime }_{\mathrm{II}}(\alpha ),{\alpha }_1<\alpha \le {\alpha }_2}\\ \multicolumn{1}{c}{\mathrm{SQN}{\mathrm{R}\prime }_{\mathrm{III}}(\alpha ),{\alpha }_2<\alpha \le {\alpha }_3}\\ \multicolumn{1}{c}{\mathrm{SQN}{\mathrm{R}\prime }_{\mathrm{IV}}(\alpha ),{\alpha }_3<\alpha \le {\alpha }_4}\\ \multicolumn{1}{c}{\mathrm{SQN}{\mathrm{R}\prime }_{\mathrm{V}}(\alpha ),\alpha >{\alpha }_4}\end{array}$$

(30)

Table 2 provides the empirically obtained values for ${\alpha }_1$ , ${\alpha }_2$ , ${\alpha }_3$ , and ${\alpha }_4$ for the bfloat16, FP24, and FP32 formats. It can be observed that across the widest range of variances ${\alpha }_2<\alpha \le {\alpha }_3$ only $D_n$ , easily calculated as the sum of just five terms, has an influence on SQNR. It should be noted that the analysis of the impact of different distortion components ( $D_{\mathrm{sub}}$ , $D_n$ , and $D_{\mathrm{ov}}$ ) on SQNR for various variance values α, along with the resulting simplifications in expression (30), is another key outcome of this paper.

Table 2.

Values of variance thresholds ${\alpha }_1$ , ${\alpha }_2$ , ${\alpha }_3$ , and ${\alpha }_4$ for different FP formats.

	${\alpha }_1$ [dB]	${\alpha }_2$ [dB]	${\alpha }_3$ [dB]	${\alpha }_4$ [dB]
bfloat16	−770	−735	756	761
FP24	−770	−735	753	757
FP32	−770	−735	752	754

Let us now comment about the accuracy of approximation (30). Namely, by applying expression (30), values of $\Delta {\mathrm{SQNR}}_{\max }$ remain the same as in Table 1, indicating that the simplifications introduced in expression (30) do not raise the SQNR calculation error.

Table 3 provides the exact SQNR values calculated by equation (17), as well as the approximate values $\mathrm{SQNR}\prime$ calculated by equation (30), for different values of α and three mentioned FP formats: bfloat16, FP24, and FP32. It is evident that the values of SQNR and $\mathrm{SQNR}\prime$ are very close, demonstrating that the approximate expression (30) significantly reduces the computational complexity of calculating SQNR compared to the exact expression (17), without compromising accuracy. To verify the theory, we simulated each case considered in Table 3 in MATLAB by generating 10 million Gaussian random numbers with the given variance. After processing generated data by FPQ corresponding to the given FP format, SQNR values (marked as ${\mathrm{SQNR}}_{\mathrm{sim}}$ ) are calculated. The simulated SQNR values very closely match the theoretical ones, proving the theory’s accuracy.

Table 3.

Values of SQNR, $\mathrm{SQNR}\prime ,$ and ${\mathrm{SQNR}}_{\mathrm{sim}}$ for different variance values and different FP formats.

α [dB]		−750	−500	−250	0	250	500	750
bfloat16	$\mathrm{SQNR}$ [dB]	55.344	55.594	55.617	55.589	55.621	55.586	55.623
	$\mathrm{SQNR}\prime$ [dB]	55.344	55.600	55.622	55.599	55.625	55.600	55.627
	${\mathrm{SQNR}}_{\mathrm{sim}}$ [dB]	55.344	55.592	55.613	55.588	55.625	55.588	55.620
FP24	$\mathrm{SQNR}$ [dB]	103.508	103.758	103.782	103.754	103.785	103.751	103.788
	$\mathrm{SQNR}\prime$ [dB]	103.508	103.765	103.787	103.764	103.79	103.766	103.792
	${\mathrm{SQNR}}_{\mathrm{sim}}$ [dB]	103.508	103.761	103.778	103.754	103.786	103.751	103.792
FP32	$\mathrm{SQNR}$ [dB]	151.673	151.923	151.946	151.919	151.95	151.916	151.952
	$\mathrm{SQNR}\prime$ [dB]	151.673	151.93	151.952	151.929	151.955	151.93	151.957
	${\mathrm{SQNR}}_{\mathrm{sim}}$ [dB]	151.676	151.923	151.943	151.918	151.951	151.914	151.947

Using SQNR as a key performance measure, we can effectively compare the three FP formats under consideration: FP32, FP24, and bfloat16. Based on the results presented in Table 3, we can conclude that all three formats maintain a nearly constant SQNR across an extensive variance range from approximately −750 dB to beyond 750 dB. This similarity of variance ranges originates from the identical value of the parameter e (e = 8) for all three formats, which predominantly affects the variance range width. Within this range, the SQNR values are approximately 151.9 dB for FP32, 103.75 dB for FP24, and 55.6 dB for bfloat16. Notably, reducing the FP format resolution by 8 bits decreases SQNR by roughly 48.15 dB. However, this reduction in resolution also simplifies implementation complexity. In practical applications, balancing implementation complexity with performance is essential.

The results of this paper impact the implementation of intelligent measurement systems by simplifying the performance evaluation of FP formats, thereby making it easier to select the most suitable FP format for each specific application. Considering that implementation complexity is a key concern for resource-constrained intelligent measurement systems, the authors recommend selecting the FP format with the lowest complexity that still meets the application’s performance requirements.

Figure 2 shows the flowchart of the applied methodology for calculating $\mathrm{SQNR}\prime$ , taking into account the conditions for variance α.

Figure 2.

The flowchart of the applied methodology for calculating $\mathrm{SQNR}\prime$ .

Finally, some limitations of the applied approach will be outlined. First, it is not recommended to use the described method for parameter values of e ≤ 3 and m ≤ 3, which occur in FP formats with very low resolutions. For e ≤ 3, the number of terms within the sum in the exact SQNR expression is not large, eliminating the need for simplifications. For m ≤ 3, the number of uniform quantization levels in each segment of the PUQ is less than 8, meaning the accuracy of formulas (8) and (10) for distortion, which were derived using asymptotic analysis (i.e. under the assumption of a sufficiently large number of quantization levels), may be insufficient. In addition, the presented method assumes that the Gaussian PDF has a zero mean. If this is not the case, the method can still be applied, but it requires subtracting the mean of the data set from each data point to center the data around zero. The method is then applied to the adjusted data set, with the mean added back at the final stage.

Experimental results

The theoretical SQNR results will be further validated by experimental findings, obtained by applying FP32, FP24, and bfloat16 formats on real data. These data comprise weights from a trained MLP neural network (Kruse et al., 2022) used for classification of images from the MNIST data set (Baldominos et al., 2019). The MLP architecture consists of input, hidden, and output layers with 784, 128, and 10 nodes, respectively. Hyperparameters were set as follows: regularization rate = 0.01, learning rate = 0.0005, and mini-batch size = 128. After 20 epochs, the MLP achieved accuracy scores of 0.9705 on the training set and 0.9686 on the test set. We used the weights between the input and hidden layers, amounting to M = 784 × 128 = 100,352. Figure 3 shows the histogram of the MLP weights, proving that they closely follow the zero-mean Gaussian distribution.

Figure 3.

Histogram of MLP weights.

Let the MLP weights be denoted as $w_i$ , $i=1,\dots ,M$ . The variance of these weights is ${\sigma }_0^2=\frac{1}{M}\sum _{i=1}^M{w}_i^2$ , giving the logarithmic variance value of ${\alpha }_0=10·{\log }_{10}{\sigma }_0^2$  = −19.725 dB. Let ${\mathrm{SQNR}}_{\exp }$ and $\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ denote the experimentally obtained SQNR values according to the exact formula (17) and approximate formula (30), respectively. Table 4 presents the theoretically calculated exact value SQNR and approximate value $\mathrm{SQNR}\prime$ for the variance ${\alpha }_0$ , alongside the experimentally obtained values ${\mathrm{SQNR}}_{\exp }$ and $\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ . The excellent agreement between theoretical and experimental results confirms the accuracy of the theoretical model.

Table 4.

Values of SQNR, $\mathrm{SQNR}\prime$ , ${\mathrm{SQNR}}_{\exp }$ , and $\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ for the variance ${\alpha }_0$ and different FP formats.

	Theory		Experiment
	$\mathrm{SQNR}$ [dB]	$\mathrm{SQNR}\prime$ [dB]	${\mathrm{SQNR}}_{\exp }$ [dB]	$\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ [dB]
bfloat16	55.619	55.622	55.615	55.597
FP24	103.784	103.787	103.764	103.748
FP32	151.949	151.952	151.939	151.925

Our next goal is to experimentally verify the accuracy of the theoretical results over a wide variance range. By scaling the values of the MLP weights with an appropriate constant c, we can generate a data set of real data $z_i=c{w}_i$ , $i=1,\dots ,M$ , with any desired variance $\alpha =10·{\log }_{10}{\sigma }^2$ , where ${\sigma }^2=\frac{1}{M}\sum _{i=1}^M{c}^2{w}_i^2=c^2{\sigma }_0^2$ . Consequently, $\alpha =10·{\log }_{10}{c}^2{\sigma }_0^2=20·{\log }_{10}c+10·{\log }_{10}{\sigma }_0^2=20·{\log }_{10}c+{\alpha }_0$ , allowing us to obtain $c={10}^{(\alpha -{\alpha }_0)/20}$ . Table 5 presents the experimentally obtained values of ${\mathrm{SQNR}}_{\exp }$ and $\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ for different values of variance α. A comparison between Tables 3 and 5 shows excellent alignment between theoretical and experimental results across all considered values of α, thereby validating the theoretical analysis over a wide range of variances.

Table 5.

Values of ${\mathrm{SQNR}}_{\exp }$ and $\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ for different variance values and different FP formats.

α [dB]		−750	−500	−250	0	250	500	750
bfloat16	${\mathrm{SQNR}}_{\exp }$ [dB]	55.345	55.583	55.636	55.575	55.621	55.585	55.605
	$\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ [dB]	55.345	55.604	55.603	55.562	55.592	55.604	55.584
FP24	${\mathrm{SQNR}}_{\exp }$ [dB]	103.521	103.720	103.801	103.809	103.824	103.737	103.822
	$\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ [dB]	103.521	103.748	103.769	103.795	103.796	103.751	103.800
FP32	${\mathrm{SQNR}}_{\exp }$ [dB]	151.657	151.902	151.973	151.911	151.958	151.889	151.954
	$\mathrm{SQN}{\mathrm{R}\prime }_{\exp }$ [dB]	151.657	151.919	151.941	151.897	151.929	151.920	151.928

Conclusion

This paper aims to develop a simple and efficient method for evaluating the performance of FP formats for Gaussian-distributed data. Starting from the analogy between FP formats and a PUQ, this paper identifies the high complexity in the SQNR expression of PUQ, which involves summing a large number of terms (e.g. 254 for bfloat16, FP24, and FP32 formats), and offers several important contributions to address this problem.

One key contribution is demonstrating that only a small number of PUQ segments significantly impact SQNR. This insight leads to the idea that the SQNR expression can be simplified by retaining only a few terms related to the segment with the highest probability and its adjacent segments. Since the segment with the highest probability depends on the variance, this paper provides a simple closed-form expression for determining this segment for any variance value. Another important result is an approximate SQNR expression with just five terms, significantly simplifying the original SQNR expression with 254 terms while maintaining accuracy with a maximum error of only 0.024 dB. The SQNR expression is further simplified by determining the ranges of variance in which certain distortion components are negligible. The theoretical results are confirmed by MATLAB simulations and an experiment conducted with the weights of a trained MLP neural network. Although the numerical results are provided for bfloat16, FP24, and FP32 formats with an 8-bit exponent due to their pronounced complexity, the findings can be applied to other FP formats, offering general significance.

As a final result, this paper presents a simple yet efficient method for calculating FP format performance, particularly important for implementing resource-constrained intelligent DNN-based sensor nodes and edge measurement devices where selecting the most suitable FP format for each specific application is crucial. The findings are also applicable in other fields, such as computing and signal processing.