Abstract
Classification of cancer subtypes is of paramount importance for diagnosis and prognosis of cancer. In recent years, deep learning methods have gained considerable popularity for cancer subtype classification, however, the structure of the neural network is difficult to determine and the performance of the deep network depends largely on its structure. To address this problem, a flexible neural tree (FNT) may be used. FNT is a special neural network with the advantage of automatic optimization of structure and parameters which cannot be used for multi-class classification. In this paper, a deep flexible neural forest (DFNForest) model is proposed, a novel ensemble of FNT model to aid with the classification of cancer subtypes. The proposed DFNForest model differs from the conventional FNT model because it transforms a multi-classification problem into many binary classification problems for each forest. We explore the cascade structure of DFNForest to deepen the flexible neural tree model so that the depth of the model is increased without introducing additional parameters. In addition to the DFNForest model, this paper proposes a combination of fisher ratio and neighborhood rough set for dimensionality reduction of gene expression data to obtain higher classification performance. The experiments on RNA-seq gene expression data show that our gene selection method has higher accuracy with fewer genes and the proposed DFNForest model has better performance for classification of cancer subtypes as compared to the conventional methods.