Detect depression from communication: how computer vision, signal processing, and sentiment analysis join forces

Background: Depression is a common illness worldwide. Traditional procedures have generated controversy and criticism, such as accuracy and agreement on consistency of depression diagnosis and assessment among clinicians. More objective biomarkers are needed for better treatment evaluation and monitoring. Hypothesis: Depression will leave recognizable markers in a patient’s acoustic, linguistic, and facial patterns, all of which have demonstrated increasing promise for more objectively evaluating and predicting a patient’s mental state. Methods: We applied a multi-modality fusion model to combine the audio, video, and text modalities to identify the biomarkers that are predictive of depression with consideration of gender differences. Results: We identified promising biomarkers from a successive search on feature extraction analysis for each modality. We found that gender disparity in vocal and facial expressions plays an important role in detecting depression. Conclusion: Audio, video and text biomarkers provided the possibility of detecting depression in addition to traditional clinical assessments. Biomarkers detected for gender-dependent analysis were not identical, indicating that gender can affect the depression manifestations.