The biological fact it exploits
The human retina carries far more rod cells, which detect brightness, than cone cells, which detect colour. The practical consequence is that we resolve fine detail in brightness far better than fine detail in colour.
Encoders exploit this directly. After separating an image into one luma channel and two chroma channels, they keep luma at full resolution and store chroma at half. Three quarters of the colour data is discarded, and on a photograph you genuinely cannot see it.