Abstract
This study investigates the multistability phenomenon and coexisting attractors in the modified Autonomous Van der Pol-Duffing (MAVPD) system and its fractional-order form. The analytical conditions for existence of periodic solutions in the integer-order system via Hopf bifurcation are discussed. In addition, conditions for approximating the solutions of the fractional version to periodic solutions are obtained via the Hopf bifurcation theory in fractional-order systems. Moreover, the technique for hidden attractors localization in the integer-order MAVPD is provided. Therefore, motivated by the previous discussion, the appearances of self-excited and hidden attractors are explained in the integer- and fractional-order MAVPD systems. Phase transition of quasi-periodic hidden attractors between the integer- and fractional-order MAVPD systems is observed. Throughout this study, the existence of complex dynamics is also justified using some effective numerical measures such as Lyapunov exponents, bifurcation diagrams and basin sets of attraction.