In case it helps anyone else: It's something about the way that Flask handles a direct send_file that seems to be causing problems in our tests. However, we've found reports of sendfile not working properly on certain platforms and we're still investigating, so I'm not convinced at this stage that the problem is with Flask itself.
> However, we've found reports of sendfile not working properly on certain platforms and we're still investigating, so I'm not convinced at this stage that the problem is with Flask itself.
I am confused by your sendfile not working properly on certain platforms. If you set the X-Sendfile header, and the web server supports it, it should work fine. Flask send_file doesn't do much other than setting up the headers; and then either setting the X-Senfile header so that web server handles it, or sends it using WSGI file wrapper supports.
I was referring to the sendfile system call there. We've seen reports that on some versions of Linux kernel, it doesn't work, for some value of "doesn't work" that we haven't yet properly determined. We've also seen web pages that suggest Flask's send_file and send_from_directory would ultimately rely on that system call in some circumstances, though again I haven't checked through the code in detail yet so I don't know how accurate those reports are.
In any case, we haven't tried using the X-Sendfile functionality yet, and I'm making no comment about how well that works.
We do know, without any doubt at this point, that serving our video files using Flask's send_from_directory (using the default file wrapping behaviour, not X-Sendfile) is not working reliably for us with many different clients, where serving the same file statically straight from a web server like Apache works fine. We're still trying to identify exactly what it is that doesn't work, but in light of our discussion here today and what I've learned since then, I'm now hoping that we can just punt the whole set-up over to the known-working Apache implementation by using X-Sendfile and avoid the problem in the first place.
(Edit: Thanks again for the suggestions, BTW. I had a fairly long list of ideas to follow up in relation to this problem, including investigating X-Sendfile, but your mention of that here prompted me to look into it first and will probably have saved me quite a bit of time if it does work for us.)
That's environment specific; the wsgi server should define a environ['wsgi.file_wrapper'] if it needs to provide specific file handling functionality. I don't think you have an issue with Linux kernel - 2.2 and beyond will always have sendfile. It's more likely that your wsgi server isn't defining environ['wsgi.file_wrapper'] to something that uses sendfile.
If you see the flask send_file code, it's using werkzeug.wsgi.wrap_file, which in turn either uses environ['wsgi.file_wrapper'], or uses simple Python code to read the file and send the contents.
Reading file and sending the contents in your python code is a bad idea. You can either set debugging points and see if wsgi server is defining environ['wsgi.file_wrapper'] properly. Or more conveniently, you can just set 'USE_X_SENDFILE = True' in your flask settings, and then directly use send_file.
In case it helps anyone else: It's something about the way that Flask handles a direct send_file that seems to be causing problems in our tests. However, we've found reports of sendfile not working properly on certain platforms and we're still investigating, so I'm not convinced at this stage that the problem is with Flask itself.