关键词: 视频标注, 多头注意力, 语义信息

Abstract:

In the sequence-to-sequence video captioning model, the video information is greatly compressed after being encoded, resulting in the decoder side cannot fully utilized the video information. To solve this problem, a multi-head attention mechanism and semantic information are introduced into the model. The multi-head attention allows the model to focus different parts of the video information when generate different words. The semantic information is introduced by the semantic detection unit through the multi-label classification approach to generate the semantic probability information of the video, which provides additional guidance to the decoding end. The modified model is still training in end-to-end. The experimental results show that the modified model captioning effect has been significantly improved, and the modified method has a significant effect on improving the captioning ability.

Key words: video captioning, multi-head attention, semantic information