If you use OpenGL or Metal with triple buffering (the blog post mentions MTLView which is a Metal view) I believe this doesn't matter. You'll only be using the CVDisplayLink timer events to generate frames spaced at a even interval - which is equal to the display vsync interval, depending how you configure CVDisplayLink. Then, the actual presentation of those frames will happen in sync with the actual display vsync.
If you're aiming for the lowest latency possible, you will still want to have access to the actual display refresh timing information. But these days with variable refresh rate monitors growing in popularity, this gets pretty complicated.
If you set the swap interval to 1, my recollection is that glFlush() (or -[NSOpenGLContext flushBuffers]) will block until the frame is presented. I’ve never relied on that behavior, but I’ve always wondered if the display link timer was even necessary, or if you could just use a while loop.
What's the scenario where you would want to do that? I understand there's a (theoretical?) scenario where you want to minimise latency between screen rendering (or input handling?) and presentation.
But if you use double buffering, which a while loop that block until the frame is presented, then the CPU will sit idle while the GPU is doing it's work. Instead of preparing the next frame after that - which may result in the CPU preparing the next screen too late if the scene complexity is fluctuating.
tfw you do a bunch of research you're proud of and write an article that reaches the HN front page, and then someone chimes in on Twitter that actually the research is flawed and only applies to multiple displays, and you have to update the post about how wrong you were. :/
I'm just sad about all the people that already read it and won't see the update so will go away with incorrect knowledge.
Interesting. So if I'm reading this properly, CVDisplayLink is still a timer--it's just that it tries to synchronize itself to the "vblTime" if there's only one of those to synchronize to.
If I'm not mistaken, then, it would seem you can do the same thing yourself by just blocking on CGLFlushDrawable() and setting up a timer on the refresh rate as soon as it returns. (I guess that's similar to what mstange suggested in the comments.) At least CVDisplayLink tweaks timer parameters in order to get the timer to fire more reliably.
It also says ”In the past, synchronizing your video frames with the display’s refresh rate was often a problem, especially if you also had audio. You could only make simple guesses for when to output a frame […] which didn’t take into account possible latency from user interactions, CPU loading, window compositing and so on. The Core Video display link can make intelligent estimates for when a frame needs to be output, based on display type and latencies. […] If for some reason the processing takes longer than expected (that is, the display link’s estimate is off), the video graphics card can still drop frames or otherwise compensate for the timing error as necessary.”.
So, disassembling the code wasn’t strictly necessary.
Combining this with this article, it seems they try to call your program as late as possible, possibly to decrease latency between the time you have influence on what is displayed and the time it is displayed. Could that especially if you also had audio be the reason they do that, as it might make it easier to keep audio in sync with video?
> it seems they try to call your program as late as possible
Right. The point of the API is to trigger rendering as late as possible before the next frame swap, not immmediarely after the last frame swap, as the author expected.
To do so, it has to estimate how long your app’s callback will take to render a frame. (E.g. if it expects your callback to render a frame in 10ms it might call it 11ms before the next frame swap.)
It seems the heuristic it uses to estimate the duration of the callback execution is assuming a fairly constant duration. The documentation seemed to suggest that. But as the author has inconsistent rendering duration.
So, e.g., if it estimates 10ms and therefore calls back 11ms before the frame swap but you end up taking 20ms, you’re going to drop a frame. If it then adjusts and calls back 21ms before, but you render in 1ms, you could render two frames in the space of one displayed frame.
Anyway, just thinking through it aloud.
I think the author may not want to use this API for his purpose, though perhaps by adjusting his rendering so that it always takes the “max” rendering time might let its estimates be correct and then things would smooth out.
Author here, that would make sense, except it doesn't appear to be doing that. I also don't see anything in the docs suggesting that it would. My test app takes 1ms to render yet, and the callback returns within 100us, but it's consistently calling me 7ms before vsync.
I'm also not actually working on anything that uses this right now, I was just curious. I'm just warning about the dangers if you have inconsistent rendering time.
The callback gets a timestamp as an argument, so audio sync is easy to compensate for if you’re doing it systematically. (Plus 1/60 second is well within human perception, when it comes to audio/video synchronization.)
I think the bigger concern is latency between HID input and visual feedback, which people are more sensitive to, and which you can’t schedule in advance for.
Audio synchronization is a problem if the graphics pipeline has multi-frame latency, but when you're working to reduce sub-frame input latency by delaying the start of rendering, I think audio synchronization problems are by that point a minor concern that few people could notice.
Yah I had noticed their docs never said it called you after vsync.
It's not that it calls you as late as possible though, as my experiments show it just calls you at an arbitrary but consistent point in the frame like a timer.
There’s quite a few approaches, it’s hard to figure out which one is appropriate for this one. I think that enabling setNeedsDisplay and dispatching commands asynchronously might do the trick? With asynchronous dispatch I’ve actually been getting 120 FPS. But it’s very much case by case basis and I might not be understanding the use case.
I have a lot of avenues to test but unfortunately testing each one requires setting up a sample app with signposts and recording a bunch of runs in Instruments, so there's a bit of work to do if I want to figure out the right way.
And do you have a 120fps+ monitor? If not I'm not sure what you mean by 120fps unless it's broken or not vsynced at all and painting twice per frame. If you have a proper way of waiting on vsync 120fps is just a waste of power with no benefits and you don't want that.
Haha welcome to gpu programming. You should hit up Warren Moore on Twitter (@warrenm), he’s a real life savior.
Re: 120fps, I questioned it myself but the difference is perceivable. Idk if something is broken or not I was confused too. Re power consumption, that’s why you enable setneedsdisplay.
Maybe the reason that 120fps is perceivably better is that your draw calls don't line up with vsync, so doubling the number of frames eliminates the dropped frame effect in @jordwalke's diagram!
And thanks for the Twitter recommendation, I followed him.
> IIRC this project Gets to 120 when you drag your mouse
What causes that kind of thing? I've noticed stuff like that while playing video in mpv sometimes (despite my Mac being easily fast enough to play the video) and it just baffles me.
Moving the mouse has a tendency to send various events to windows. Sometimes just the mousemotion event is enough to trigger the program to redraw. I'm not sure how cursors are implemented on osX but it might just be marking the bit under the cursor dirty, triggering a redraw.
I like the Linux DRM atomic interface, they recently added native fences so you can pass in a synchronization object from the GPU for when it's done rendering and the flip can proceed and you get back a fence that is signalled once the flip is done. Basically all asynchronous now but it's still synced to the hardware flip, no timer bullshit.
IIRC it's somewhat inspired by Android SurfaceFlinger.
Hmm. I wouldn't mind finding out how that actually works.
One question though. Is this driver-dependent in such a way that it only works on newer hardware? For example on the 10+-year old machine I'm using right now, could I play with this?
I mostly remembering it having tons of old OS update images and diagnostic tools (that were publicly released), as well as (I THINK) a few ROMs (I... think I'm misremembering that bit). Its heyday was pretty much the Classic Mac OS era; there were System 6/7 updates on here etc. I'm not sure when it finally went kaput but I remember poking it in 2008. Unfortunately I learned about it when I had absolutely no free disk space and couldn't archive it :'(
A few months ago I asked someone whose Apple ID was able to login (back in the day any .Mac ID worked, hah) and they said that it was empty. I don't clearly remember whether they said they were seeing a white page or an "Index of /" with nothing underneath, but regardless it seems pretty much gone now.
There are other nice things out there though. One fun query is "hotline kdx". (That's two separate pieces of 90s-era software that used the same P2P protocols, but googling them together is the easiest way to find KDX.) The few servers left are fun - the directories tend to be chaotic messes but patient digging finds cool stuff, like the partial NT4 and OS6 leaks. Shhh :D
Wow, this is bad. If the author is reading this, is there any chance this is a regression in recent OSes? I remember a WWDC video from quite a few years ago showing exactly why it’s important to use CVDisplayLink which is linked to when your display needs new frames, showing a similar illustration to the one shown above about how otherwise you lose frames.
Interesting turnabout from classic MacOS, where the VSync interrupt was both synced and really important. It was the way you got lots of non-graphical processing done, too - lots of event timers & the like went in there.
See Metal Triple Buffering (https://developer.apple.com/library/content/documentation/3D...) and MTLCommandBuffer presentDrawable.
For OpenGL there are similar mechanisms, eg. NSOpenGLCPSwapInterval.