Luckily the sun doesn't output that much in the wave length of gamma radiation and
those gamma rays that do travel from the sun toward Earth are diverted away from the surface by our magnetic field. This is good because these high intensity electromagnetic waves are ionizing, which means they cause cellular damage and DNA damage that, among other deleterious effect, cause cancer in humans. Wavelengths below visible light ("infrared") do hit the Earth in abundance, and are percieved as heat from the sun. These wavelengths do contain all-in-all quite a lot of energy but per square meter or square foot less than the visible light spectrum. So having photovoltaic panels that were able to convert a lot of the longer wave lengths to electricity wouldn't create a whole lot of electricity, per surface area. Remember that the shorter the wave length the more energy.
One more thing: Day and night on earth is caused by the phenomenon which is that Earth rotates around its own axis. Due to this, at any given time, one side of the sphere will face the Sun while the other side faces away from the Sun. The side facing away experiences "night" and the side facing the sun experiences "day". During "the night" not a lot of solar radiation, of any wavelength, is available to harvest.