The influence of the silica fume (SF) addition on the basic mechanical and thermal parameters of cement concrete was presented in this paper. Numerous studies show that the specific properties of silica fume allow producing concretes with increased strength and durability properties. However, the results of the authors' own studies demonstrated that under specific circumstances, the SF addition can negatively affect the strength of cement concrete. In this paper, the concretes made from two types of coarse aggregate, 4–8 mm fraction: reference ordinary gravel aggregate (Ref) and broken basalt aggregate (Bas) were tested. Comprehensive tests (microstructural, strength and thermal) were carried out on three formulas: the first one (Ref-0) did not contain any additives or admixtures. Silica fume and superplasticizer were used in the second formula (Ref-8), introduced in the amount of 8% and 0.75% of the cement’s weight, respectively. The third formula, based on basalt aggregate (Bas-8), also contained silica fume and a superplasticizer, dosed in the same amount as in the second formula. Porosimetric studies showed that the use of silica fume in the Ref-8 formula resulted in a decrease in the number of pores smaller than 0.15 μm in comparison to the Ref-0 reference concrete without the silica fume. On the other hand, numerous additional pores with diameters ranging from 0.05 to 300 μm were found in the Bas-8 concrete. In turn, optical porosimetry proved that the basalt-based concrete had numerous pores in the range above 70 μm, which were not observed in the case of the concretes with ordinary pebble aggregate. In the case of the normal concrete, the addition of silica fume resulted in an increase in the compressive strength. However, the basalt-based concrete, due to its much higher porosity, achieved significantly lower strength values. The results showed that the observed changes in porosity, caused by the presence of silica fume, resulted in lower values of thermal conductivity and specific heat of mature composites that it had been added to. This was particularly evident for the basalt-based concrete.