Abstract
The problem of efficient modulation classification (MC) in multiple-input multiple-output systems is considered. Per-layer likelihood-based MC is proposed by employing subspace decomposition to partially decouple the transmitted streams. When detecting the modulation type of the stream of interest, a dense constellation is assumed on all remaining streams. The proposed classifier outperforms existing MC schemes at a lower complexity cost, and can be efficiently implemented in the context of joint MC and subspace data detection.