The biological fact it exploits
The human retina carries far more rod cells, which detect brightness, than cone cells, which detect color. The practical consequence is that we resolve fine detail in brightness far better than fine detail in color.
Encoders exploit this directly. After separating an image into one luma channel and two chroma channels, they keep luma at full resolution and store chroma at half. Three quarters of the color data is discarded, and on a photograph you genuinely cannot see it.