Expert Approach for Trio Awareness-Based Framework for Automatic Image Captioning Using Visual, Semantic Attention Mechanisms, and HES

Abstract

Image captioning is the process of analysing the contents of an image and changing them into a coherent Language to explain the image. According to neuroscience research human vision and language formation are connected together. Even if there are many techniques for image captioning, including template filling, and content retrieval, the deep learning-based methods are most popular in current scenario. Feature vectors are generated from an image using an image encoder through the deep learning techniques, and these feature vectors are transformed into a string of words by a language decoder. Various CNNs such as VGG16, VGG19, Inception, and ResNet are available for extracting image features. Various experiments have been performed to find out the best encoder for the purpose of image captioning. LSTM and GRU are two most recently used decoders to generate captions. GRU is the latest version of RNN. It is much faster than LSTM as compared to computational efficiency. Using the encoder-decoder approach or simple attention-based mechanisms has not provided very efficient results. So, visual attention as well as textual attention have been used for efficient image captioning. There are various visual attention mechanisms such as grid-based attention and region-based attention to specifically retrieve image features, and these features are used as additive attention to the decoder. For textual attention, word embedding have been used. In the proposed framework, a trio awareness-based mechanism has been utilized. The objective of this research is to extract Visual characteristics from the region of interest (RoI) of an image and text features using the GloVe embedding technique. The ResNet version of the Convolutional Neural Network (CNN) is utilized as an encoder. A Gated Recurrent Unit (GRU) is utilized as the decoder, and the Human Evaluation Score has been incorporated as a third parameter to further validate the model's performance. The proposed model is tested on the Flickr8k dataset, the findings show that the results attained through the proposed mechanism are more accurate than those of other models