In view of the flaws of disorderly placement, intentional obstruction, and detection of small irregular items in X-ray security inspection images, as well as the requirements for real-time and fast security inspection purpose, a lightweight real-time contraband detection method (LRCD) has thus been proposed based on YOLO v5s network model to assist security personnel in rapid detection. By replacing the C3 module in the YOLO v5s backbone with the DenseOne module in the model backbone, the features can be enriched and the network’s feature expression ability can be improved; with SPPF(spatial pyramid pooling-fast) in YOLO v5s backbone replaced with SimSPPF for an improvement of the inference speed. Meanwhile, the WIoU (Wise IoU) loss function is introduced for an suppression of the influence of redundant features on the detection network, thus enhancing the network’s ability to obtain multi-scale features contained in contraband goods. In the EDS (endogenous domain shift) dataset for X-ray images of luggage and items, the mAP (mean average precision) reaches 68.96% and FPS reaches 136.9. Compared with other widely used classic object detection models in recent years, the average improvements can be as high as 6.35% and 66.7%, respectively.