Abstract: Music is used to evoke emotion throughout the customer journey. This paper develops a theory-based, explainable deep learning convolutional neural network (CNN) classifier--MusicEmoCNN--to predict the time-varying emotional response to music. To develop a theory-based CNN, we first transform the raw music data into a format--mel spectrogram--that accounts for the human auditory response. Next, we design and construct novel CNN filters for higher-order music features that are based on the physics of sound waves and associated with perceptual features of music, like consonance and dissonance, which are known to impact emotion. The key advantage of our theory-based filters is that we can connect how the predicted emotional response (valence and arousal) is related to human-interpretable features of music. Our model outperforms traditional machine learning models and performs comparably to state-of-the-art black-box CNN models. Finally, we illustrate an application involving digital advertising. Motivated by YouTube’s mid-roll advertising, we first conduct a lab experiment in which we exogenously place ads at different times within content videos and find that ads placed in emotionally similar contexts are more memorable in terms of higher brand recall rates. For a given ad, we then use the model's predictions to identify emotionally similar contexts in content videos.

About the Speaker: Hortense Fong is an Assistant Professor in Quantitative Marketing at Columbia Business School. She uses machine learning, econometric, and experimental methods to study how emotions impact consumer behavior, taking advantage of the rich unstructured data (text, images, video, music) that are increasingly available. A distinguishing feature of her machine learning interests involves going beyond its use in prediction to study how to incorporate domain-specific theoretic and managerial knowledge into machine learning systems and make machine learning systems more interpretable. She also has a broader interest in questions at the interface of marketing and society (e.g., fairness) especially when it relates to the widespread adoption of artificial intelligence in various business and marketing settings.

Register in advance for this webinar: