Effectively, placing ss before the input filename seeks the file (the container?) without decoding it. Placing it after will decode the streams while skipping to the point you specify.
Seeking the container is usually much faster than decoding and then throwing away what you don't need, but it has fatal flaw: most videos use p-frames and thus require you to decode the frames before it.
So, say you want to skip to 60 seconds in. The solution is to do "-ss 50 -i input.mkv -ss 10", which is fast and should get the keyframes you need.
That trick (-ss -i -ss) should not be necessary by default (unless you use -noaccurate_seek) according to current documentation. But I haven't verified it.
-ss position (input/output)
When used as an input option (before -i), seeks in this input file to
position. Note that in most formats it is not possible to seek exactly, so
ffmpeg will seek to the closest seek point before position. When transcoding
and -accurate_seek is enabled (the default), this extra segment between the
seek point and position will be decoded and discarded. When doing stream copy
or when -noaccurate_seek is used, it will be preserved.
When used as an output option (before an output url), decodes but discards
input until the timestamps reach position.
Speed was never my concern, judging by what you said I think putting the -as after tried to cut things between p-frames and that often caused probls with the cut, while putting it before started the cut at the last p-frame? Something like that, I don't use ffmpeg enough to remember what problems I was having but I remember what fixed it.
Seeking the container is usually much faster than decoding and then throwing away what you don't need, but it has fatal flaw: most videos use p-frames and thus require you to decode the frames before it.
So, say you want to skip to 60 seconds in. The solution is to do "-ss 50 -i input.mkv -ss 10", which is fast and should get the keyframes you need.