In view of the flaws of micro-expression characterized with a small amplitude and low intensity, a neural network structure has thus been proposed based on the combination of convolution neural network with attention mechanism (ACNN) and bi-directional long short-term memory (Bi-LSTM). CASME II data set has been adopted in the experiment so as to reduce the risk of over-fitting, with the basic features extracted from the preprocessed feature vectors through the pre-trained VGG16 network, followed by the cropping of the output features, thus obtaining 24 micro-expression recognition blocks with local features and global feature vectors with the whole picture features. Next, based on an extraction of local features with attention information from 24 recognition blocks through local recognition block attention convolution neural network (BR-ACNN), global features with attention information are to be extracted as well from global feature vectors through global attention convolution neural network (GR-ACNN). Finally, the correlation information between the micro expression sequences can be extracted by Bi-LSTM based on the extracted local and global features. The experimental results show that the average accuracy rate of 5-fold cross validation is 0.69, UF1 is 0.638 2, and UAR is 0.675 0. The results on the CASME II data set show that the proposed algorithm model, compared with OFFApexNet model, is 0.028 1 higher in UF1, and 0.096 9 higher in UAR; while compared with ATNet model, it has increased by 0.007 2 in UF1 and by 0.032 0 in UAR.