In view of the flaws of low detection accuracy and high missed detection rate of small targets in PCB surface defect detection, an improved YOLOv8n PCB surface defect detection algorithm has thus been proposed. Firstly, an improvement has been made of C2f modules, followed by an introduction of ConvNeXt as well as a new design of C2XT modules so as to enhance its feature extraction capability. Next, GAM attention mechanism is introduced into the backbone network, with CARAFE upsampling technique adopted in the neck network for an enhancement of the expression of key features. In addition, a detection head for small targets is introduced into the head detection module, with an ASFF module enabling an adaptive feature fusion integrated to improve detection ability of the model for small targets. Finally, the calculation of bounding box regression in the loss function can be optimized by combining DFL with μ-IoU. Experimental results show that the improved algorithm is characterized with a significant improvement in various indicators, with a 3% improvement in efficiency and a 1.6% improvement in accuracy compared to the original YOLOv8n algorithm.