Publications

2023

1. Deep Multi-modal Hashing with Semantic Enhancement for Multi-label Micro-video Retrieval

   Peiguang Jing, Haoyi Sun, Liqiang Nie, Yun Li, and Yuting Su

   IEEE Transactions on Knowledge and Data Engineering, 2023

   Abstract PDF Code

   The pressing need for low storage and high efficiency has significantly propelled the advancement of deep hashing techniques in the realm of large-scale search and retrieval tasks. As one of the most prevailing forms of user-generated contents, micro-videos usually represent more complicated multi-modal behaviors that are further challenged in multi-label retrieval. Existing multi-modal hashing methods tend to prioritize the complementarity and consistency in multi-modal fusion, while neglecting the completeness problem. In this paper, we propose a deep multi-modal hashing with semantic enhancement (DMHSE) method that effectively integrates complete multi-modal representation learning with discriminative binary coding by means of collaboration between two distinct encoders, FoldCoder and HashCoder. FoldCoder translates latent multi-modal representation learning to a degradation process through mimicking data transmitting. Further, it incorporates a prompt learning paradigm to maximize the utilization of multi-label semantics for guiding representation learning. HashCoder combines pairwise and central constraints to ensure more discriminative hashing results. Pairwise constraint preserves the original local relevance structure, while central constraint tackles the problem of semantic ambiguity in multi-label data by leveraging the global label distribution. Experimental results demonstrate that DMHSE achieves superior performance in multi-label micro-video retrieval tasks.

2. Dual Preference Perception Network for Fashion Recommendation in the Social Internet of Things

   Peiguang Jing, Kai Zhang, Xianyi Liu, Yun Li, Yu Liu, and Yuting Su

   IEEE Internet of Things Journal, 2023

   Abstract PDF Code

   Nowadays, with the continuous development of information technology, the application scenarios of the Internet of Things (IoT) are progressively expanding to the social field, engendering widespread attention to the Social Internet of Things (SIoT). Personalized fashion recommendation that possesses the potential to establish social relationships between clothing and humans has substantially broadened the scope of the SIoT, particularly with the flourishing fashion industry and the ascent of smart home. Compared to conventional recommendations, fashion recommendation generally suggests a collection of items rather than individual pieces for users. Additionally, considering the public acceptance alongside the user-specific preference is reasonable for fashion recommendation, however, current methods often overlook the former. To comprehensively capture the public acceptance and the user-specific preference, we propose a dual preference perception network (DP2Net) for fashion recommendation. Firstly, a fashion corpus is constructed to facilitate the condensation of general taste, wherein adversarial learning and determinantal point process are leveraged to ensure representativeness and diversity of the corpus. Secondly, a user-general preference perception module is built based on a bottleneck transformer structure to generate aggregated representations for the corpus. Thirdly, a user-specific preference perception module is constructed to acquire collaborative representations of users and outfits by employing an attentive heterogeneous graph embedding. The final loss functions of two preference perception modules are constructed by combining the representations of users, outfits, and the corpus. Experiments on large-scale real-world datasets demonstrate the effectiveness of the proposed method.

3. VMemNet: A Deep Collaborative Spatial-Temporal Network With Attention Representation for Video Memorability Prediction

   Wei Lu, Yujia Zhai, Jiaze Han, Peiguang Jing, Yu Liu, and Yuting Su

   IEEE Transactions on Multimedia, 2023

   Abstract PDF Code

   Video memorability measures the degree to which a video is remembered by different viewers and has shown great potential in various contexts, including advertising, education, and health care. While extensive research has been conducted on image memorability, the study of video memorability is still in its early stages. Existing methods in this field primarily focus on coarse-grained spatial feature representation and decision fusion strategies, overlooking the crucial interactions between spatial and temporal domains. Therefore, we propose an end-to-end collaborative spatial-temporal network called VMemNet, which incorporates targeted attention mechanisms and intermediation fusion strategies. This enables VMemNet to capture the intricate relationships between spatial and temporal information and uncover more elements of memorability within video visual features. VMemNet integrates spatially and semantically guided attention modules into a dual-stream network architecture, allowing it to simultaneously capture static local cues and dynamic global cues in videos. Specifically, the spatial attention module is used to aggregate more memorable elements from spatial locations, and the semantically guided attention module is used to achieve semantic alignment and intermediate fusion of the local and global cues. In addition, two types of loss functions with complementary decision rules are associated with the corresponding attention modules to guide the training process of the proposed network. Experimental results obtained on a publicly available dataset verify that the proposed VMemNet approach outperforms all current single- and multi-modal methods in terms of video memorability prediction.

4. Dual Preference Perception Network for Fashion Recommendation in the Social Internet of Things

   Peiguang Jing, Kai Zhang, Xianyi Liu, Yun Li, Yu Liu, and Yuting Su

   IEEE Internet of Things Journal, 2023

   Abstract PDF Code

   Nowadays, with the continuous development of information technology, the application scenarios of the Internet of Things (IoT) are progressively expanding to the social field, engendering widespread attention to the Social Internet of Things (SIoT). Personalized fashion recommendation that possesses the potential to establish social relationships between clothing and humans has substantially broadened the scope of the SIoT, particularly with the flourishing fashion industry and the ascent of smart home. Compared to conventional recommendations, fashion recommendation generally suggests a collection of items rather than individual pieces for users. Additionally, considering the public acceptance alongside the user-specific preference is reasonable for fashion recommendation, however, current methods often overlook the former. To comprehensively capture the public acceptance and the user-specific preference, we propose a dual preference perception network (DP2Net) for fashion recommendation. Firstly, a fashion corpus is constructed to facilitate the condensation of general taste, wherein adversarial learning and determinantal point process are leveraged to ensure representativeness and diversity of the corpus. Secondly, a user-general preference perception module is built based on a bottleneck transformer structure to generate aggregated representations for the corpus. Thirdly, a user-specific preference perception module is constructed to acquire collaborative representations of users and outfits by employing an attentive heterogeneous graph embedding. The final loss functions of two preference perception modules are constructed by combining the representations of users, outfits, and the corpus. Experiments on large-scale real-world datasets demonstrate the effectiveness of the proposed method.

5. Dual-domain Aligned Deep Hierarchical Matrix Factorization Method for Micro-video Multi-label Classification

   Fugui Fan, Yuting Su, Liqiang Nie, Peiguang Jing, Daozheng Hong, and Yu Liu

   IEEE Transactions on Multimedia, 2023
6. StyleEDL: Style-Guided High-order Attention Network for Image Emotion Distribution Learning

   Peiguang Jing, Xianyi Liu, Ji Wang, Liqiang Nie, and Yuting Su

   In Proceedings of ACM International Conference on Multimedia, 2023

   Abstract PDF Code

   Emotion distribution learning has gained increasing attention with the tendency to express emotions through images. As for emotion ambiguity arising from humans’ subjectivity, substantial previous methods generally focused on learning appropriate representations from the holistic or significant part of images. However, they rarely consider establishing connections with the stylistic information although it can lead to a better understanding of images. In this paper, we propose a style-guided high-order attention network for image emotion distribution learning termed StyleEDL, which interactively learns stylistic-aware representations of images by exploring the hierarchical stylistic information of visual contents. Specifically, we consider exploring the intra- and inter-layer correlations among GRAM-based stylistic representations, and meanwhile exploit an adversary-constrained high-order attention mechanism to capture potential interactions between subtle visual parts. In addition, we introduce a stylistic graph convolutional network to dynamically generate the content-dependent emotion representations to benefit the final emotion distribution learning. Extensive experiments conducted on several benchmark datasets demonstrate the effectiveness of our proposed StyleEDL compared to state-of-the-art methods.

7. Self-supervised Deep Partial Adversarial Network for Micro-video Multimodal Classification

   Yun Li, Shuyi Liu, Xuejun Wang, and Peiguang Jing

   Information Sciences, 2023

   Abstract PDF Code

   Micro-videos have gained popularity on various social media platforms because they provide a great medium for real-time storytelling. Although micro-videos can be naturally characterized by several modalities, for situations with uncertain missing modalities, a flexible multimodal representation learning framework that integrates complementary and consistent information has been difficult to develop. To better deal with the issue regarding incomplete modalities in multimodal micro-video classification, in this paper, we propose a self-supervised deep multimodal adversarial network (SDMAN) to learn comprehensive and robust micro-video representations. Specifically, we first consider a parallel multi-head attention (MHA) encoding module that simultaneously learns the representations of complete and incomplete modality groupings. We then present a multimodal self-supervised cycle generative adversarial network module, in which multiple generative adversarial networks are explored to transfer the information obtained from the complete modality grouping to the incomplete modality groupings. As a result, complementarity and consistency are mutually promoted among the modalities. Furthermore, experiments conducted on a large-scale micro-video dataset demonstrate that the SDMAN performs better than the state-of-the-art methods.

8. Learning Dual Low-rank Representation for Multi-label Micro-video Classification

   Wei Lu, Desheng Li, Liqiang Nie, Peiguang Jing, and Yuting Su

   IEEE Transactions on Multimedia, 2023
9. Internet of Things for Diagnosis of Alzheimer’s Disease: A Multimodal Machine Learning Approach Based on Eye Movement Features

   Yunpeng Yin, Han Wang, Shuai Liu, Jinglin Sun, Peiguang Jing, and Yu Liu

   IEEE Internet of Things Journal, 2023
10. A Multimodal Aggregation Network With Serial Self-Attention Mechanism for Micro-Video Multi-Label Classification

    Wei Lu, Jiaxin Lin, Peiguang Jing, and Yuting Su

    IEEE Signal Processing Letters, 2023

    Abstract PDF Code

    Currently, micro-videos have attracted increasing attention due to their unique properties and great commercial value. Considering that micro-videos naturally incorporate multimodal information, a powerful representation method for distinct joint multimodal representations is essential for real applications. Inspired by the potential of attention neural network architectures over various tasks, we propose a multimodal aggregation network (MANET) with a serial self-attention mechanism to perform tasks of micro-video multi-label classification. Specifically, we first propose a parallel content-dependent graph neural networks (CDGNN) module, which explores category-related embeddings of micro-videos by disentangling category relations into modality-specific and modality-shared category dependency patterns. Then we introduce a serial self-attention (SSA) module to transmit the multimodal information in sequential order, in which an aggregation bottleneck is incorporated to better collect and condense the significant information. Experiments conducted on a large-scale multi-label micro-video dataset demonstrate that our proposed method has achieved competitive results compared with several state-of-the-art methods.

11. Category-aware Multimodal Attention Network for Fashion Compatibility Modeling

    Peiguang Jing, Kai Cui, Weili Guan, Liqiang Nie, and Yuting Su

    IEEE Transactions on Multimedia, 2023
12. A Novel Integrated Eye-Tracking System With Stereo Stimuli for 3-D Gaze Estimation

    Jinglin Sun, Zhipeng Wu, Han Wang, Peiguang Jing, and Yu Liu

    IEEE Transactions on Instrumentation and Measurement, 2023

2022

1. A Novel Deep Learning Approach for Diagnosing Alzheimer’s Disease Based on Eye-tracking Data

   Jinglin Sun, Yu Liu, Hao Wu, Peiguang Jing, and Yong Ji

   Frontiers in Human Neuroscience, 2022
2. Video Frame Deletion Detection Based on Time-frequency Analysis

   Xiao Jin, Yuting Su, and Peiguang Jing

   Journal of Visual Communication and Image Representation, 2022
3. Visual Sentiment Classification via Low-Rank Regularization and Label Relaxation

   Xiao Jin, Peiguang Jing, Jiesheng Wu, Jing Xu, and Yuting Su

   IEEE Transactions on Cognitive and Developmental Systems, 2022
4. Residual-Guided Multiscale Fusion Network for Bit-Depth Enhancement

   Jing Liu, Xin Wen, Weizhi Nie, Yuting Su, Peiguang Jing, and Xiaokang Yang

   IEEE Transactions on Circuits Systtem and Video Technology, 2022
5. Tripartite Graph Regularized Latent Low-Rank Representation for Fashion Compatibility Prediction

   Peiguang Jing, Jing Zhang, Liqiang Nie, Shu Ye, Jing Liu, and Yuting Su

   IEEE Transactions on Multimedia, 2022
6. Exploiting Low-rank Latent Gaussian Graphical Model Estimation for Visual Sentiment Distribution

   Yuting Su, Wei Zhao, Peiguang Jing, and Liqiang Nie

   IEEE Transactions on Multimedia, 2022

   Abstract PDF Code

   Nowadays, an increasing number of applications and services encourages users to express their emotions via images openly. Different from traditional visual sentiment classification, visual sentiment distribution learning is to explore the overall distribution to present the relative importance of sentiment labels. Considering most relevant studies that failed to completely model correlation structures or explicitly applied to unknown instances, in this paper, we proposed a low-rank latent Gaussian graphical model estimation (LGGME) method to solve visual sentiment distribution learning tasks. The main characteristics of LGGME are three folds: 1) an integrated inverse covariance matrix whose parameters characterize the latent correlation structures between and within features and sentiments is estimated with a sparse Gaussian graphical model; 2) a multivariate normal assumption is assigned on the concatenated latent feature representations and the estimated sentiment distributions instead of the original observations for a reasonable surrogate; 3) the latent feature representations are projected from a low-rank subspace which is also available for unseen instances and the estimated sentiment distributions are evaluated by KL divergence to ensure a suitable setting for distribution learning. We further developed an effective optimization algorithm based on the alternating direction method of multipliers (ADMM) for our objective function. Experiment results conduced on three publicly available datasets demonstrate the superiority of our proposed method.

7. EEG Emotion Recognition Based on Self-attention Dynamic Graph Neural Networks

   Chao Li, Yong Sheng, Haishuai Wang, Mingyue Niu, Peiguang Jing, Ziping Zhao, and 1 more author

   In In Proceedings of European Molecular Biology Conference, 2022

2021

1. Joint Nuclear- and L21-norm Regularized Heterogeneous Tensor Decomposition for Robust Classification

   Peiguang Jing, Yaxin Li, Xinhui Li, Yuting Wu, and Yuting Su

   Neurocomputing, 2021
2. Learning Robust Affinity Graph Representation for Multi-view Clustering

   Peiguang Jing, Yuting Su, Zhengnan Li, and Liqiang Nie

   Information Sciences, 2021
3. Deep Low-rank Matrix Factorization with Latent Correlation Estimation for Micro-video Multi-label Classification

   Yuting Su, Junyu Xu, Daozheng Hong, Fugui Fan, Jing Zhang, and Peiguang Jing

   Information Sciences, 2021
4. A Dual Rank-Constrained Filter Pruning Approach for Convolutional Neural Networks

   Fugui Fan, Yuting Su, Peiguang Jing, and Wei Lu

   IEEE Signal Processing Letters, 2021
5. Learning Low-Rank Sparse Representations With Robust Relationship Inference for Image Memorability Prediction

   Peiguang Jing, Yuechen Shang, Liqiang Nie, Yuting Su, Jing Liu, and Meng Wang

   IEEE Transactions on Multimedia, 2021
6. Market2Dish: Health-aware Food Recommendation

   Wenjie Wang, Ling-Yu Duan, Hao Jiang, Peiguang Jing, Xuemeng Song, and Liqiang Nie

   ACM Transactions on Multimedia Computing, Communications, and Applications, 2021
7. Joint Co-Attention And Co-Reconstruction Representation Learning For One-Shot Object Detection

   Jinghui Chu, Jiawei Feng, Peiguang Jing, and Wei Lu

   In In Proceedings of IEEE International Conference on Image Processing, 2021

2020

1. Low-Rank Regularized Multimodal Representation for Micro-Video Event Detection

   Jing Zhang, Yuting Wu, Jinghui Liu, Peiguang Jing, and Yuting Su

   IEEE Access, 2020
2. Predicting the Popularity of Micro-videos Via a Feature-Discrimination Transductive Model

   Yuting Su, Yang Li, Xu Bai, and Peiguang Jing

   Multimedia Systems, 2020
3. Single image super-resolution via low-rank tensor representation and hierarchical dictionary learning

   Peiguang Jing, Weili Guan, Xu Bai, Hongbin Guo, and Yuting Su

   Multimedia Tools and Applications, 2020
4. Low-Rank Regularized Deep Collaborative Matrix Factorization for Micro-Video Multi-Label Classification

   Yuting Su, Daozheng Hong, Yang Li, and Peiguang Jing

   IEEE Signal Processing Letters, 2020
5. Low-Rank Regularized Multi-Representation Learning for Fashion Compatibility Prediction

   Peiguang Jing, Shu Ye, Liqiang Nie, Jing Liu, and Yuting Su

   IEEE Transactions on Multimedia, 2020
6. DS-Net: Dynamic Spatiotemporal Network for Video Salient Object Detection

   Yuting Su, Weikang Wang, Jing Liu, Peiguang Jing, and Xiaokang Yang

   CoRR, 2020

2019

1. Photo-realistic image bit-depth enhancement via residual transposed convolutional neural network

   Yuting Su, Wanning Sun, Jing Liu, Guangtao Zhai, and Peiguang Jing

   Neurocomputing, 2019
2. Wearable Computing for Internet of Things: A Discriminant Approach for Human Activity Recognition

   Wei Lu, Fugui Fan, Jinghui Chu, Peiguang Jing, and Yuting Su

   IEEE Internet of Things Journal, 2019
3. Tensor-driven low-rank discriminant analysis for image set classification

   Jing Zhang, Zhengnan Li, Peiguang Jing, Ye Liu, and Yuting Su

   Multimedia Tools and Applications, 2019
4. A structure-transfer-driven temporal subspace clustering for video summarization

   Jing Zhang, Yue Shi, Peiguang Jing, Jing Liu, and Yuting Su

   Multimedia Tools and Applications, 2019
5. Low-rank regularized tensor discriminant representation for image set classification

   Peiguang Jing, Yuting Su, Zhengnan Li, Jing Liu, and Liqiang Nie

   Signal Processing, 2019
6. A Framework of Joint Low-Rank and Sparse Regression for Image Memorability Prediction

   Peiguang Jing, Yuting Su, Liqiang Nie, Huimin Gu, Jing Liu, and Meng Wang

   IEEE Transactions on Circuits and Systems for Video Technology, 2019
7. High-Order Temporal Correlation Model Learning for Time-Series Prediction

   Peiguang Jing, Yuting Su, Xiao Jin, and Chengqian Zhang

   IEEE Transations on Cybernetics, 2019
8. BE-CALF: Bit-Depth Enhancement by Concatenating All Level Features of DNN

   Jing Liu, Wanning Sun, Yuting Su, Peiguang Jing, and Xiaokang Yang

   IEEE Transactions on Image Processing, 2019
9. Spatiotemporal Symmetric Convolutional Neural Network for Video Bit-Depth Enhancement

   Jing Liu, Pingping Liu, Yuting Su, Peiguang Jing, and Xiaokang Yang

   IEEE Transactions on Mutlimedia, 2019
10. Personalized Capsule Wardrobe Creation with Garment and User Modeling

    Xue Dong, Xuemeng Song, Fuli Feng, Peiguang Jing, Xin-Shun Xu, and Liqiang Nie

    In In Proceedings of ACM International Conference on Multimedia, 2019

2018

1. Improving Bit-Depth Expansion via Context-Aware MMSE Optimization (CAMO)

   Jing Liu, Wanning Sun, Peiguang Jing, Jiexiao Yu, and Yuting Su

   IEEE Access, 2018
2. Sparsity-Based Image Inpainting Detection via Canonical Correlation Analysis With Low-Rank Constraints

   Xiao Jin, Yuting Su, Liang Zou, Yongwei Wang, Peiguang Jing, and Z. Jane Wang

   IEEE Access, 2018
3. AMFNet: An Adversarial Network for Median Filtering Detection

   Xiao Jin, Peiguang Jing, and Yuting Su

   IEEE Access, 2018
4. HyperSSR: A hypergraph based semi-supervised ranking method for visual search reranking

   Peiguang Jing, Yuting Su, Chuan-Zhong Xu, and Luming Zhang

   Neurocomputing, 2018
5. Towards a sparse low-rank regression model for memorability prediction of images

   Jinghui Chu, Huimin Gu, Yuting Su, and Peiguang Jing

   Neurocomputing, 2018
6. Graph regularized low-rank tensor representation for feature selection

   Yuting Su, Xu Bai, Wu Li, Peiguang Jing, Jing Zhang, and Jing Liu

   Journal of Visual Communication and Image Representation, 2018
7. Low-rank regularized multi-view inverse-covariance estimation for visual sentiment distribution prediction

   Anan Liu, Yingdi Shi, Peiguang Jing, Jing Liu, and Yuting Su

   Journal of Visual Communication and Image Representation , 2018
8. Video logo removal detection based on sparse representation

   Xiao Jin, Yuting Su, Liang Zou, Chengqian Zhang, Peiguang Jing, and Xuemeng Song

   Multimedia Tools and Applications, 2018
9. Structured low-rank inverse-covariance estimation for visual sentiment distribution prediction

   Anan Liu, Yingdi Shi, Peiguang Jing, Jing Liu, and Yuting Su

   Signal Processing, 2018
10. Low-Rank Regularized Heterogeneous Tensor Decomposition for Subspace Clustering

    Jing Zhang, Xinhui Li, Peiguang Jing, Jing Liu, and Yuting Su

    IEEE Signal Processing Letters, 2018
11. Low-Rank Multi-View Embedding Learning for Micro-Video Popularity Prediction

    Peiguang Jing, Yuting Su, Liqiang Nie, Xu Bai, Jing Liu, and Meng Wang

    IEEE Transaction on Knowledge and Data Engineering, 2018

2017

1. A spatial-temporal iterative tensor decomposition technique for action and gesture recognition

   Yuting Su, Haiyi Wang, Peiguang Jing, and Chuan-Zhong Xu

   Multimedia Tools and Applications, 2017
2. SnapVideo: Personalized Video Generation for a Sightseeing Trip

   Luming Zhang, Peiguang Jing, Yuting Su, Chao Zhang, and Ling Shao

   IEEE Transactions on Cybernetics, 2017
3. Predicting Image Memorability Through Adaptive Transfer Learning From External Sources

   Peiguang Jing, Yuting Su, Liqiang Nie, and Huimin Gu

   IEEE Transactions on Multimedia, 2017

2016

1. Visual search reranking with RElevant Local Discriminant Analysis

   Peiguang Jing, Zhong Ji, Yunlong Yu, and Zhongfei Zhang

   Neurocomputing, 2016
2. Quality biased multimedia data retrieval in microblogs

   Shuhan Qi, Peiguang Jing, Xuan Wang, and Liqiang Nie

   Journal of Visual Communication and Image Representation, 2016
3. A Tensor-Driven Temporal Correlation Model for Video Sequence Classification

   Jing Zhang, Chuan-Zhong Xu, Peiguang Jing, Chengqian Zhang, and Yuting Su

   IEEE Signal Processing Letters, 2016

2013

1. Ranking Fisher discriminant analysis

   Zhong Ji, Peiguang Jing, Tianshi Yu, Yuting Su, and Changshu Liu

   Neurocomputing, 2013
2. Rank canonical correlation analysis and its application in visual search reranking

   Zhong Ji, Peiguang Jing, Yuting Su, and Yanwei Pang

   Signal Processing, 2013
3. Ranking Graph Embedding for Learning to Rerank

   Yanwei Pang, Zhong Ji, Peiguang Jing, and Xuelong Li

   IEEE Transactions on Neural Networks and Learning Systems , 2013

2012

2011