Special video classification based on multitask learning and multimodal feature fusion

Xiao-yu WU; Chao-nan GU; Sheng-jin WANG

doi:10.3788/OPE.20202805.1177

您当前的位置：

首页 >

文章列表页 >

Special video classification based on multitask learning and multimodal feature fusion

Information Sciences | 更新时间：2020-07-09

- Special video classification based on multitask learning and multimodal feature fusion
- Optics and Precision Engineering Vol. 28, Issue 5, Pages: 1177-1186(2020)
- 作者机构：
  
  1.中国传媒大学信息与通信工程学院，北京 100024
  2.清华大学电子工程系，北京 100084
- 作者简介：
- 基金信息：
- DOI：10.3788/OPE.20202805.1177
  CLC： TP391.4
- Received：29 November 2019，
  
  Revised：08 January 2020，
  
  Accepted：08 January 2020，
  
  Published：25 May 2020
- 稿件说明：
移动端阅览
Xiao-yu WU, Chao-nan GU, Sheng-jin WANG. Special video classification based on multitask learning and multimodal feature fusion[J]. Optics and precision engineering, 2020, 28(5): 1177-1186.
DOI：

Xiao-yu WU, Chao-nan GU, Sheng-jin WANG. Special video classification based on multitask learning and multimodal feature fusion[J]. Optics and precision engineering, 2020, 28(5): 1177-1186. DOI： 10.3788/OPE.20202805.1177.

摘要

特种视频(本文特指暴力视频)的智能分类技术有助于实现网络信息内容安全的智能监控。针对现有特种视频多模态特征融合时未考虑语义一致性等问题，本文提出了一种基于音视频多模态特征融合与多任务学习的特种视频识别方法。首先，提取特种视频的表观信息和运动信息随时空变化的视觉语义特征及音频信息语义特征；然后，构建具有语义保持的共享特征子空间，以实现音视频多种模态特征的融合；最后，提出基于音视频特征的语义一致性度量和特种视频分类的多任务学习特种视频分类理论框架，设计了对应的损失函数，实现了端到端的特种视频智能识别。实验结果表明，本文提出的算法在Violent Flow和MediaEval VSD 2015两个数据集上平均精度分别为97.97%和39.76%，优于已有研究。结果证明了该算法的有效性，有助于提升特种视频监控的智能化水平。

Abstract

Classification of special videos is significant for intelligent surveillance of internet content. Existing algorithms that fuse multimodal features forclassification of special videoscannot measure multimodal audio-visual semantic correspondence.An algorithm for recognizing special videos based on multimodal audio-visual feature fusion was proposed herein over the framework of multitask learning. First

audio semantic features and spatial-temporal visual semantic cues

including appearance and motion

were extracted. A latent subspace to fuse audio and visual features whilst preserving their semantic information was learned and developed through jointly learning audio-visual semantic correspondence and special video classification. Subsequently

a multitask learning loss function was presented viacombination of the correspondence loss

obtained based on the measured audio-visual semantic information

and the cross-entropy loss of special video classification. Finally

an end-to-end intelligent system for special video recognition was implemented. Experimental results demonstrate that the accuracy of the proposed algorithm is 97.97% with respect to the Violent Flow dataset

and the average accuracy is 39.76% with respect to the Media Eval VSD 2015 dataset

where by the algorithm outperforms the other existing methods. These results show that the proposed algorithm is effective for improving the intelligence of network content surveillance.

关键词

Keywords

references

马晓晨 , 韦世奎 , 蒋翔 , 等 . 基于相机溯源的潜在不良视频通话预警 . 光学精密工程 , 2018 . 26 ( 11 ): 2785 - 2794 . http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201811020 http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201811020 .

X CH MA , SH K WEI , X JIANG , 等 . Early warning of illegal video chats based on camera source identification . Opt. Precision Eng. , 2018 . 26 ( 11 ): 2785 - 2794 . http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201811020 http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201811020 .

H D CLAIRE , P CEDRIC , S MOHAMMAD , 等 . VSD, a public dataset for the detection of violent scenes in movies: design, annotation, analysis and evaluation . Multimedia Tools and Applications , 2014 . 74 ( 17 ): 7379 - 7404 . http://cn.bing.com/academic/profile?id=257232beac94c67b5f15c46adb8eb79a&encoded=0&v=paper_preview&mkt=zh-cn http://cn.bing.com/academic/profile?id=257232beac94c67b5f15c46adb8eb79a&encoded=0&v=paper_preview&mkt=zh-cn .

D MOREIRA , S AVILA , M PEREZ , 等 . Multimodal data fusion for sensitive scene localization . Information Fusion , 2019 . ( 45 ): 307 - 323 . http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=655af91660d23b119d58299cb1d77991 http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=655af91660d23b119d58299cb1d77991 .

WANG H M, YANG L, WU X Y, et al . A review of bloody violence in video classification [C]. International Conference on the Frontiers & Advances in Data Science , 2017: 86-91.

YI Y, WANG H, ZHANG B, et al . MIC-TJU at affective impact of movies task[C]. MediaEval Workshop , 2015, 7.

LAM, LE S P, DO T, et al . Computational optimization for violent scenes detection[C]. International Conference on Computer, Control, Informatics and its Applications , 2016: 141-146.

DAI Q, ZHAO R, WU Z, et al . Fudan-Huawei at mediaeval 2015: Detecting violent scenes and affective impact in movies with deep learning[C]. MediaEval Workshop , 2015, 5.

SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[C]. NeurIPS , 2014: 568-576.

SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[C]. NeurIPS , 2014: 3104-3112.

SWATHIKIRAN S, OSWALD L. Learning to detect violent videos using convolutional long short-term memory[C]. IEEE International Conference on Advanced Video and Signal Based Surveillance , 2017: 1-6.

T BALTRUSAITIS , C AHUJA , L P MORENCY . Multimodal machine learning: a survey and taxonomy . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2019 . 41 ( 2 ): 423 - 443 . DOI: 10.1109/TPAMI.2018.2798607 http://doi.org/10.1109/TPAMI.2018.2798607 .

崔鑫 , 彭宗举 , 陈芬 . 联合多特征的未来视频快速编码 . 光学精密工程 , 2019 . 27 ( 4 ): 990 - 999 . http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201904026 http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201904026 .

X CUI , Z J PENG , F CHEN . Joint Multi-feature fast coding for future video coding . Opt. Precision Eng. , 2019 . 27 ( 4 ): 990 - 999 . http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201904026 http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201904026 .

WU Z, JIANG Y G, WANG X, et al . Multi-Stream multi-class fusion of deep networks for video classification[C]. ACM International Conference on Multimedia , 2016: 791-800.

潘仙张 , 张石清 , 郭文平 . 多模深度卷积神经网络应用于视频表情识别 . 光学精密工程 , 2019 . 27 ( 4 ): 963 - 970 . http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201904023 http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201904023 .

X ZH PAN , SH Q ZHANG , W P GUO . Video-based facial expression recognition using multimodal deep convolutional neural networks . Opt. Precision Eng. , 2019 . 27 ( 4 ): 963 - 970 . http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201904023 http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201904023 .

P K ATREY , M A HOSSAIN , A E SADDIK , 等 . Multimodal fusion for multimedia analysis: a survey . Multimedia Systems , 2010 . 16 ( 6 ): 345 - 379 .

QIU Z, YAO T, TAO M. Learning spatial-temporal representation with pseudo-3d residual networks[C]. IEEE International Conference on Computer Vision , 2017: 5534-5542.

CARREIRA J, ZISSERMAN A. Quo vadis, action recognition A new model and the kinetics dataset[C]. IEEE Conference on Computer Vision and Pattern Recognition , 2017: 6299-6308.

HERSHEY S, CHAUDHURI S, ELLIS D P W, et al . CNN architectures for large-scale audio classification[C]. International Conference on Acoustics, Speech and Signal Processing , 2017: 131-135.

WU Z, JIANG Y G, WANG J, et al . Exploring inter-feature and inter-class relationships with deep neural networks for video classification[C]. ACM International Conference on Multimedia , 2014: 167-176.

HASSNER T, ITCHER Y, KLIPER C O. Violent flows: Real-time detection of violent crowd behavior[C]. IEEE Conference on Computer Vision and Pattern Recognition , 2012: 1-6

BILINSKI P, BREMOND F. Human violence recognition and detection in surveillance videos[C]. IEEE International Conference on Advanced Video and Signal Based Surveillance , 2016: 30-36.

T ZHANG , W JIA , X HE , 等 . Discriminative dictionary learning with motion weber local descriptor for violence detection . IEEE Transactions on Circuits and Systems for Video Technology , 2017 . 27 ( 3 ): 696 - 709 . DOI: 10.1109/TCSVT.2016.2589858 http://doi.org/10.1109/TCSVT.2016.2589858 .

MATS S, YOANN B, HANLI W, et al . The mediaeval 2015 affective impact of movies task[C]. MediaEval Workshop , 2015: 1.

A ESRA , H FRANK , A SAHIN . Breaking down violence detection: combining divide-et-impera and coarse-to-fine strategies . Neurocomputing , 2016 . 208 225 - 237 . DOI: 10.1016/j.neucom.2016.05.050 http://doi.org/10.1016/j.neucom.2016.05.050 .

Views

529

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

UAV target positioning integrating multi-dimensional attitude adaptive sensing

Hyperspectral images feature extraction and classification based on fractional differentiation

Lightweight deep global-local knowledge distillation network for hyperspectral image scene classification

Dense pedestrian detection algorithm in multi-branch non-anchor frame network

Application of SENet generative adversarial network in image semantics description

Related Author

LING Zhicheng

LEI Zhongkui

HUANG Daqing

XU Cheng

HU Yanan

LIU Jing

LI Yang

LIU Yi

Related Institution

College of Astronautics， Nanjing University of Aeronautics and Astronautics

College of Electronic Information Engineering， Nanjing University of Aeronautics and Astronautics

Unmanned Aerial Vehicles Research Institute， Nanjing University of Aeronautics and Astronautics

Nanjing Changkong Technology Co.， Ltd.

School of Electronic Engineering， Xi’an University of Posts and Telecommunications

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China Postal code：130033
Tel：0431-86176855 Email：gxjmgc@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰