Usually I-frames are a small minority of the frames. Most frames encode motion information. So the decoder takes picture data from previous frames and moves it according to the motion vectors. Then, if the result was deemed too inaccurate by the encoder, there will be a little more picture data for the current frame. Having accurate motion vectors is more important than having perfect color reproduction.