A Quantization-Friendly Hardware Implementation of the H-Swish Activation Function

1. Introduction

Recently, the importance of on-device AI technology, which performs artificial intelligence computations directly on edge devices such as smartphones, autonomous vehicles, robots, and Internet of Things (IoT) devices, has been growing significantly. A key research challenge in this area is the design of lightweight neural networks, as these models must operate within the constrained computational resources and power constraints of edge devices.

MobileNet represents a typical lightweight neural network, utilizing depthwise separable convolution to reduce both the number of parameters and computational complexity while maintaining high accuracy. Therefore, MobileNet has been widely adopted across various applications, including real-time object detection and image classification, where fast inference speed is essential. MobileNet has evolved over successive versions with various architectural improvements, and advancements in activation functions have been a key contributor to its performance enhancement. The initial versions, MobileNetV1 [1] and MobileNetV2 [2], used ReLU and ReLU6 activation functions, which provide stable performance even on low-precision hardware.

Subsequently, MobileNetV3 [3] adopted the swish activation function to overcome the limitations of ReLU activation function (e.g., the Dying ReLU) and to improve accuracy. While swish effectively captures nonlinearity and enhances model performance, it suffers from a high computational cost due to its reliance on the sigmoid function. To address this, MobileNetV3 proposed the h-swish activation function, an approximation of swish that reduces complexity while maintaining accuracy. However, when implementing convolutional neural network (CNN) accelerators on field-programmable gate arrays (FPGAs), the h-swish function still poses challenges. Its formulation includes multiplication operations that, when mapped to FPGAs, require either embedded digital signal processing (DSP) blocks or a large number of look-up tables (LUTs). In CNN accelerators, DSP blocks are critical resources for executing convolution operations, which constitute the majority of the computational workload. Consequently, allocating DSP resources to an activation function, whose contribution to the overall computation is relatively small, results in inefficient hardware resource utilization.

In this paper, we propose a novel approximation method for the h-swish activation function and a hardware architecture that eliminates multiplications, thereby avoiding the use of DSP blocks. The proposed approach employs a piecewise linear (PWL) approximation by dividing the h-swish function into several segments and by replacing multiplication with hardware-friendly bit-shift and addition operations. In addition, we extend the conventional PWL approach and propose a hardware architecture optimized for integer inputs. The proposed design eliminates DSP usage for both fixed-point and integer inputs and achieves an approximate 32% reduction in LUT utilization for integer inputs. Furthermore, when the proposed h-swish activation function replaces the original activation in MobileNetV3, the top-1 accuracy loss is negligible.

The rest of this paper is organized as follows: Section 2 provides background and reviews previous work related to activation function approximation and quantization. Section 3 describes the proposed approximation method and hardware architecture. Section 4 analyzes the hardware resources of the proposed architecture and evaluates its impact on model performance, followed by the conclusion in Section 5.

2. Related Work

In this section, we review the existing approaches for the hardware implementation of activation functions and neural network quantization schemes. Section 2.1 examines various activation function approximation methods that have been developed for efficient hardware implementation. Section 2.2 discusses neural network quantization schemes for reducing computational latency and resource requirements on hardware platforms.

2.1 Activation Function Approximation Methods

Activation functions are fundamental components that determine model performance. However, those involving exponential functions or division, such as sigmoid and swish, incur high computational costs and substantial hardware resource consumption when their mathematical formulations are implemented directly without approximation. Consequently, developing hardware-friendly approximations of activation functions to reduce computational cost and resource usage has become an active area of research.

PWL approximation is one of the most prominent approaches. This approach approximates a complex nonlinear function by dividing it into multiple linear segments, which allows complex multiplication and division operations to be replaced with hardware-friendly alternatives [4-6]. Several studies have applied the PWL approximation to efficiently implement various activation functions such as sigmoid, tanh, and Gaussian error linear unit (GELU), with optimizations reducing hardware costs through the use of approximate adders [7]. As these studies demonstrate, PWL approximation offers an effective balance between hardware cost and performance, providing high accuracy with a low implementation complexity. However, many PWL approaches primarily focus on reducing the arithmetic complexity of nonlinear functions without explicitly incorporating quantization parameters into the approximation formulation. In contrast, the proposed method considers affine quantization, enabling PWL approximation to be efficiently extended to integer only arithmetic.

In addition to PWL approximation, various other approximation methods have been proposed. One approach is to approximate activation functions using hardware-friendly operations to reduce implementation complexity. Choi et al. [8] proposed a hardware-friendly approximation of the swish activation function and demonstrated improved hardware efficiency. Another approach is the LUT-based method, which stores pre-computed function values in memory and retrieves outputs by indexing, providing minimal computation latency. However, this method faces the fundamental limitation of requiring exponentially increasing memory resources to achieve high accuracy [9]. Therefore, research is being conducted to optimize memory resources through enhanced methods such as exploiting the symmetry of activation functions [10]. Polynomial approximation, which uses methods like Taylor series expansion and similar methods, can achieve high accuracy. However, it has the drawback of requiring numerous multipliers, leading to significant hardware resource consumption [11].

2.2 Quantization

On edge devices, inference is typically accelerated by applying a quantization scheme that converts floating-point weights and activations into integers. This method maps floating-point values to integers at uniform intervals, reducing the model storage and improving inference speed through the use of hardware optimized for integer operations.

Uniform quantization is fundamentally based on an affine transformation that maps floating-point real numbers to integer values. This transformation utilizes two parameters known as scale and zero-point. The scale determines the step size of the quantization, which defines the ratio for compressing the range of floating-point values into the integer range. The zero-point specifies the integer value to which the floating-point value of 0 is mapped, ensuring that the actual value zero can be represented without quantization error. This is crucial for maintaining accuracy in operations where the value 0 is significant, such as in the zero padding of convolution operations.

Jacob et al. [12] proposed an 8-bit integer quantization using the constants S (for “scale”) and Z (for “zero-point”). They introduced the method that allows for integer-arithmetic-only inference for neural networks. This is made possible by the quantization formula, as defined in Eqs. (1) and (2):

Eq. (1) defines the quantization process, converting a real value r into a quantized value q. First, the real value is divided by the constant S to adjust its range to the integer space, and then the constant Z is added to correct the offset. Conversely, Eq. (2) is the de-quantization formula that reconstructs an approximation of the original real value from the quantized value.

This bidirectional mapping formula allows a trained floating-point model to be converted into an integer model with minimal accuracy loss. This approach enables most of the complex floating-point operations during inference to be replaced with integer arithmetic operations, leading to its widespread adoption as a standard quantization scheme in deep learning frameworks.

In particular, to efficiently handle complex nonlinear activation functions in a quantized environment, Kim et al. [13] approximated the exponential function and GELU using integer bit-shifts and multiplication operations. However, these studies did not address the issue of activation functions with multiplication operations occupying hardware resources. Therefore, this paper, while following the fundamental principles of uniform quantization, focuses on optimizing h-swish for hardware resource constraints.

3. The Proposed Method

In this section, we propose a PWL approximation for the h-swish function and present its corresponding FPGA-based hardware architecture that utilizes only LUTs.

3.1 Piecewise Linear Approximation of H-Swish Function

The h-swish function is defined as the input x multiplied by the h-sigmoid function, as expressed in Eq. (3). This formulation inherently requires a multiplication operation.

The proposed method applies PWL approximation to the h-swish function within the interval (-3, 3). This interval is partitioned into eight segments of width 0.75. Each segment is approximated by the linear function connecting its start and end points. The resulting PWL formulation eliminates the division by 6 and yields coefficients that can be exactly represented in fixed-point format without additional approximation error, enabling a multiplier-free implementation using only bit-shift and addition operations. The resulting approximated function is expressed in Eq. (4).

As illustrated in Fig. 1, the proposed approximation provides an accurate representation of the h-swish function across all intervals, demonstrating excellent agreement between the theoretical curve and the approximated segments. Moreover, the elimination of the division by 6 allows all constants in Eq. (4) to be expressed in a fixed-point format with 1 integer bit and 4 fractional bits, as shown in Eq. (5):

Since multiplication operations require only 1 integer bit and 3 fractional bits, they can be efficiently computed using up to 3 right shifts.

3.2 Hardware Implementation

3.2.1 Hardware implementation with fixed-point arithmetic

We implemented the proposed hardware architecture for the approximated h-swish function. For comparison, we also implemented a reference architecture of the conventional h-swish function without PWL approximation on the FPGA.

Fig. 2(a) shows the hardware architecture of the h-swish function. Within the input range (-3,3), the h-swish can be implemented using Eq. (3). The input value x is added to the constant 3, while simultaneously the input value x is multiplied by [TeX:] $$\frac{1}{6} \text { to obtain } \frac{1}{6} x .$$ Subsequently, the two results are multiplied together, and this result is used as input to the multiplexer. The final output is selected from three candidate values [TeX:] $$\left(0, \frac{x \cdot(x+3)}{6}, x\right)$$ depending on the input range.

Fig. 2(b) illustrates the hardware architecture of the proposed h-swish function. As presented in Eq. (5), divisions by 2, 4, and 8 are implemented using right shift operations by 1, 2, and 3 bits respectively, and the final result is produced by summing these shifted values with a constant selected depending on the input range. The Shift Selector takes the input x and its inverted version as inputs. The Shift Selector outputs five values based on the input range, as defined in Eq. (5). These outputs are summed by a carry save adder (CSA) to minimize propagation delay.

The Shift Selector is composed of multiplexers, and its structure is detailed in Fig. 3. The output of each multiplexer is determined by the input range. The first output selects 0 for the input range [TeX:] $$(-\infty, 0), x \gg 1$$ for [0, 1.5), and x for [1.5, [TeX:] $$\infty$$). The second output selects 0 for the intervals (-[TeX:] $$\infty$$, -3), [-2.25, -0.75), [0, 0.75), [1.5, 2.25), and [3, [TeX:] $$\infty$$). For all other intervals, it selects either [TeX:] $$x \gg 2 \text { or } \tilde{x} \gg 2 .$$ For the third output, 0 is selected for (-[TeX:] $$\infty$$, -3) and [3, [TeX:] $$\infty$$), while [TeX:] $$\tilde{x} \gg 3$$ is selected for [-3, -1.5) and [TeX:] $$x \gg 3$$ is selected for [-1.5, 3). The fourth output generates a value to add 1 at a specific position to generate -x as a two's complement from the inverted input. Finally, the fifth output selects among the constant values 0, 0.1875, 0.5625, and 1.125 depending on the input range. These five values are then passed to the subsequent adder stage.

3.2.2 Hardware Implementation with integer only arithmetic

The proposed h-swish function is implemented as a linear function in each segment. Therefore, for integer only arithmetic, the proposed h-swish can be expressed by Eq. (6) for each segment. In this equation, [TeX:] $$R_i$$ represents each segment for the PWL approximation, as defined in Eq. (7). The [TeX:] $$I_A\left(S_x\left(x_{i n t}-z_x\right) \in R_i\right)$$ term is an indicator function that evaluates to 1 if the input [TeX:] $$s_x\left(x_{i n t}-z_x\right)$$ is within the specific interval [TeX:] $$R_i$$, and 0 otherwise, as shown in Eq. (7). This function acts as a switch that selectively activates only the term corresponding to the interval containing the current input value. The coefficients [TeX:] $$a_i \text { and } b_i$$ are predefined constants that define the linear function for each segment [TeX:] $$R_i$$.

The term [TeX:] $$a_i \cdot s_x\left(x_{i n t}-z_x\right)+b_i$$ in Eq. (6) combines integer and real number operations, resulting in computational inefficiency. To address this, we restructure the equation by factoring out the scale factor [TeX:] $$s_x$$, allowing internal operations to be performed using integer arithmetic and improving computational efficiency. Specifically, we mathematically transform the [TeX:] $$b_i$$ term into the equivalent form [TeX:] $$s_x \cdot\left(\frac{b_i}{s_x}\right),$$ then factor out [TeX:] $$s_x$$ yielding [TeX:] $$s_x \cdot\left(a_i\left(x_{i n t}-z_x\right)+\left(\frac{b_i}{s_x}\right)\right) .$$ At this stage, the term [TeX:] $$\frac{b_i}{s_x}$$ still involves a floating-point operation. Therefore, we apply the 𝑟𝑜𝑢𝑛𝑑( ) function to approximate it to the nearest integer. Since [TeX:] $$b_i \text{and} s_x$$ are known constants, the term [TeX:] $$\operatorname{round}\left(\frac{b_i}{s_x}\right)$$ can be pre-computed and stored in memory as a constant.

We rearrange Eq. (8) as follows to derive the expression for [TeX:] $$y_{\text {int }},$$ the final output to be computed in hardware. First, both sides of Eq. (8) are divided by the output scale factor [TeX:] $$s_y$$. In this process, the [TeX:] $$s_x$$ on the right-hand side is divided by [TeX:] $$s_y$$, defining a new scale factor [TeX:] $$\frac{s_x}{s_y} .$$ Since both [TeX:] $$s_x \text { and } s_y$$ are known constants, this new factor can be pre-computed and stored in memory, thus avoiding the need for a division operation. Next, [TeX:] $$z_y$$ is moved from the left to the right side of the equation, yielding the final expression for [TeX:] $$y_{\text {int }}$$ as shown in Eq. (9). By incorporating quantization parameters into the PWL formulation in this manner, the proposed method can be efficiently extended to integer-only arithmetic.

Fig. 4 depicts the hardware architecture of the proposed integer arithmetic h-swish, which is designed based on Eq. (9). The computation first subtracts [TeX:] $$z_x$$ from the integer input [TeX:] $$x_{\text {int }}$$ to obtain [TeX:] $$x_{\text {int }}-z_x .$$ The subsequent step is identical to the architecture of the proposed h-swish shown in Fig. 2(b), where the PWL operation is performed by a Shift Selector and the CSA. Finally, the summed result is forwarded to the Quantization Module. In this module, the final integer output [TeX:] $$y_{\text {int }}$$ is generated by multiplying the result by the pre-computed [TeX:] $$\frac{s_x}{s_y}$$ and then adding the output zero-point [TeX:] $$z_y$$, according to Eq. (9).

Fig. 5 illustrates the hardware architecture of the quantization module. This architecture is designed based on the constant coefficient multiplication (CCM) structure proposed by Li et al. [14]. To optimize hardware resources, this module approximates the floating-point number [TeX:] $$\frac{s_x}{s_y}$$ into the form [TeX:] $$m \cdot 2^{-n} .$$ Here, m is a fixed-point number constant, and the [TeX:] $$2^{-n}$$ is replaced by an n-bit right-shift operation. The constant m and n are pre-computed. Therefore, no additional computational overhead occurs during inference.

For comparison, we also implemented a hardware architecture for the h-swish function operating entirely with integer arithmetic. As shown in Eq. (3), the h-swish function includes a quadratic term for inputs within the interval [-3, 3), which was implemented using the method proposed by Kim et al. [13]. The integer arithmetic h-swish was constructed by connecting a quantization module to its output stage. This quantization module was implemented using the same method as that of the proposed h-swish.

4. Evaluation

In this section, we evaluate the proposed approximation in terms of MSE (mean square error), model accuracy, hardware resource utilization, and latency. Table 1 presents the results of the MSE analysis for the activation functions, computed using 4,096 samples over an input range of [-4.0, 4.0]. The MSE between the proposed h-swish and the conventional h-swish function is [TeX:] $$1.089 \times 10^{-4}$$, indicating the high approximation accuracy of the proposed method. Furthermore, the MSE of the proposed h-swish with respect to the swish ([TeX:] $$3.322 \times 10^{-3}$$) is smaller than that of the conventional h-swish ([TeX:] $$3.683 \times 10^{-3}$$). Further hardware simplification can reduce resource usage at the cost of increased approximation error.

Table 2 compares the top-1 accuracy of MobileNetV3-Large using the conventional h-swish and the proposed h-swish at FP32 and INT8 precisions. MobileNetV3-Large was trained on ImageNet [15] for 600 epochs using stochastic gradient descent (SGD) with a momentum of 0.9, a weight decay of [TeX:] $$1 \times 10^{-5}$$, and an initial learning rate of 0.05. A linear warm-up was applied for the first 5 epochs, followed by cosine annealing. Cross-entropy loss with label smoothing was used, and the batch size was set to 128. For accuracy evaluation, the arithmetic operations of the proposed hardware architecture were faithfully reproduced in software to ensure the same computational behavior as the actual hardware implementation. The experimental results demonstrate that the proposed h-swish function achieves accuracy comparable to that of the conventional h-swish in both precision settings. In the FP32 model, the proposed h-swish function recorded an accuracy of 74.92%, confirming negligible degradation compared to 75.07% of the conventional h-swish. In the INT8 model, the proposed h-swish achieved an accuracy of 73.76%, slightly higher than 73.16% of the conventional h-swish. This difference is considered to result from variations introduced during the quantization process rather than a significant performance difference between the two activation functions.

Table 3 summarizes the synthesis results for the fixed-point implementations, while Table 4 presents the results for the integer arithmetic implementations. All three designs were implemented on the xc7z020clg484-1 FPGA. In general, LUT utilization increases with bit width. However, the 8-bit conventional h-swish hardware requires more LUTs than the 12 and 16-bit implementations because the synthesis tool implements small-bit multipliers using LUTs instead of DSP blocks. The proposed h-swish hardware achieves a DSP-free implementation. For the DSP-free fixed-point implementations, the proposed method shows comparable resource utilization to Choi et al. [8] without DSP blocks, but requires fewer resources in the integer arithmetic implementation. These results demonstrate that the proposed method achieves low hardware resource utilization through its computation optimized for quantization, making it more efficient for integer arithmetic.

Table 5 compares the hardware latency of all three designs for fixed-point and integer arithmetic implementations. For the fixed-point implementations at 8, 12, and 16-bit precisions, the proposed h- swish hardware exhibits lower latency due to its simplified shift-and-add computation without multiplier operations. In the integer arithmetic implementation, the proposed h-swish hardware also achieves lower latency than the other methods. These results demonstrate that the proposed method provides efficient latency through its simplified computation and quantization-friendly design.

5. Conclusion

In this paper, we addressed the inefficiency problem of the h-swish activation function consuming DSP blocks, which are critical resources in FPGA-based CNN accelerators. To solve this problem, we proposed a hardware architecture that replaces the multiplication in the h-swish function with bit-shift and addition using PWL approximation. Furthermore, the proposed architecture is designed with a computational flow optimized for the INT8 quantization environment. FPGA synthesis results show that the proposed architecture requires no DSP blocks and, under INT8 quantization, also reduces LUT utilization. Moreover, when applied to the MobileNetV3-Large model, the proposed activation function maintains comparable Top-1 accuracy to the conventional h-swish in both FP32 and INT8 configurations. Consequently, the hardware approximation method proposed in this study offers a significant contribution to the design of efficient deep learning accelerators, enabling high-performance neural network inference on resource-constrained hardware, such as edge devices and embedded systems.

Conflict of Interest

The authors declare that they have no competing interests.

Funding

None.