Automatic recognition of mammographic images in breast cancer is a complex issue due to the confusing appearance of some perfectly normal tissues which look like masses. The existing computer-aided systems suffer from non-satisfactory accuracy of cancer detection. This paper addresses this problem and proposes two alternative techniques of mammogram recognition: the application of a variety of methods for definition of numerical image descriptors in combination with an efficient SVM classifier (so-called classical approach) and application of deep learning in the form of convolutional neural networks, enhanced with additional transformations of input mammographic images. The key point of the first approach is defining the proper numerical image descriptors and selecting the set which is the most class discriminative. To achieve better performance of the classifier, many image descriptors were defined by means of applying different characterization of the images: Hilbert curve representation, Kolmogorov-Smirnov statistics, the maximum subregion principle, percolation theory, fractal texture descriptors as well as application of wavelet and wavelet packets. Thanks to them, better description of the basic image properties has been obtained. In the case of deep learning, the features are automatically extracted as part of convolutional neural network learning. To get better quality of results, additional representations of mammograms, in the form of nonnegative matrix factorization and the self-similarity principle, have been proposed. The methods applied were evaluated based on a large database composed of 10,168 regions of interest in mammographic images taken from the DDSM database. Experimental results prove the advantage of deep learning over traditional approach to image recognition. Our best average accuracy in recognizing abnormal cases (malignant plus benign versus healthy) was 85.83%, with sensitivity of 82.82%, specificity of 86.59% and AUC = 0.919. These results are among the best for this massive database.