Abstract
Many research efforts recently have aimed at understanding the phenomenon of fake news, including recognizing their common features and patterns, leading to several fake news detection models based on machine learning. Yet, the real distinct strength of those models remains uncertain: some perform well only with particular data, but others are more general. Most of the models classified the fake news as a black-box without giving any explanations to users. In this work, therefore, we conduct an exploratory investigation that evaluates and interprets fake new detection models, including looking into which important features that contribute to the models' prediction from the explainable machine learning perspective. This give us some insights on how the detection models work and their trustworthiness.