Thanks, really appreciate the feedback! HN is actually my most frequented site, I just don't talk much!
I did see that post at the time but will read it again. Index limitation is basically just lack of servers, funds, etc.
Currently we're doing a full index search for every query and so know the exact amount of hits within our index. This might change in the future though and become "about".
No databases except for what I've written myself. It started as a hobby so everything was from scratch from the beginning and just continued that way.
Yes thanks that's a very good idea. I'll try and put something together.
The article may be interesting to go through but most of the comments were discussing fairly stock-standard "popular"/hyped software stacks, so YMMV with that specific data. Untangling all of the individual pieces into a coherent picture was also fairly involved (I gave up). I mention the link solely to reference the idea that GitLab did get lots of hits and feedback, and that you may find it an interesting idea to consider networking on here regarding server resources. It's just a thought, may not be useful.
Okay, so... a small continuous load of users is doing full index over 1 billion items within 500ms per request. That's... that needs to go into your writeup, along with what your current usage load is like. Prepare for inquiries and offers when you do your post!
Also, you're definitely going to have a lot of interest in the database system if it's homemade; your use-case (high read and query load, moderate write load) is fairly widespread, and different implementations always lean themselves toward being super-awesome at certain kinds of queries.
I'm not going to push the open-source idea myself, but you'll definitely have a bit of clamouring. That will need to be worked out; if this site is your most frequently read (cool) then you probably already have a good idea of the pros and cons of open vs closed.
Very much looking forward to hearing about this, whenever it happens. Doesn't need to be immediate by any means - comprehensive, in-depth analysis takes a while, and if it's not rushed the results are very good.
It's a very good idea and I definitely should do it anyway.
Most of the 500ms is generating the snippet, the search itself is usually much quicker. The search is effected by the load more than generating the snippet though. I'll try and get it all written down.
I did see that post at the time but will read it again. Index limitation is basically just lack of servers, funds, etc.
Currently we're doing a full index search for every query and so know the exact amount of hits within our index. This might change in the future though and become "about".
No databases except for what I've written myself. It started as a hobby so everything was from scratch from the beginning and just continued that way.
Yes thanks that's a very good idea. I'll try and put something together.
Marc