Abstract
This paper investigates the effect of the angle of the Fractional Fourier transform (FrFT) on encryption quality in a chaotic image cryptosystem. The adopted approach for encryption is 2D chaotic Baker map. This map is applied to the transformed images using FrFT with different angles. We inspect its application to digital images and its exhaustive security investigation. The security investigation contains statistical tests like histogram and correlation coefficient, the maximum and the irregular deviation metrics, and the processing time. Experimental results prove that the rotation angle of the FrFT does not affect the encryption quality. This indicates a wide range of angles that can be used in this transform domain for encryption in addition to increasing the scheme sensitivity and making it hard for the attacker to exactly determine the right angle used in the transform domain.