Recognizing faces is a very challenging problem in the field of image processing. Deep neural network and especially Convolutional Neural Networks are the most widely used techniques for image classification and recognition. Despite these deep neural networks efficiency, choosing their optimal architectures for a given task remains an open problem. In fact, Convolutional Neural Networks performance depends on many hyper-parameters namely the network depth, convolutional layer numbers, the number of the local receptive fields and their respective sizes, convolutional stride and dropout ratio. These parameters thoroughly affect the performance of the classifier. This paper aims to optimize these parameters and develop the optimized architecture face classification and recognition. Intensive simulated experiments and qualitative comparisons have been conducted. The achieved results show that the developed Convolutional Neural Networks configuration provided a remarkable performance improvement in in terms of the network accuracy that exceeds 94%.