You need ollama[1][2] and hardware to run 20-70B models with quantization of Q4 at least to have similar experience to commercially hosted models. I use codestral:22b, gemma2:27b, gemma2:27b-instruct, aya:35b.
Smaller models are useless for me, because my native language is Ukrainian (it's easier to spot mistakes made by model in a language with more complex grammar rules).
As GUI, I use Page Assist[3] plugin for Firefox, or aichat[4] commandline and WebUI tool.
I have no idea what "reasonably fast" means for you. It good for performance when model fit inside memory of a graphic card. Nvidia 4090 with 24Gb will be more than enough to start learning. I use gaming notebook with Nvidia 3080Ti equipped with 16Gb of videomemory.
I have no issues with using just the CPU on smaller (<= 13b) models and it's quite fast enough for me. Even 70b models still work if you have the RAM, they're just much slower.
However, as far as I can tell, it's never actually clear what the hardware requirements are to get these to run without fussing around. Am I wrong about this?
In my experience the hardware requirements are whatever the file size is + a bit more. Cpu works, gpu is a lot faster but needs VRAM.
Was playing with them some more yesterday.
Found that the 4bit ("q4") is much worse then q8 or fp16. Llama3.1 8B is ok, internlm2 7B is more precise. And they all hallucinate a lot.
In my opinion they are not really useful.
Good for translations, to summaries some texts, and.. to ask in case you forgot some things about something.
But they lie, so for anything serious you have to do your own research.
And absolutely no good for precise or obscure topics.
If someone wants to play there's GPT4All, Msty, LM Studio.
You can give them some of your documents to process and use as "knowledge stacks".
Msty has web search, GPT4All will get it in some time.
Got more opinions, but this is long enough already.
I agree on the translation part. Llama 3.1 8B even at 4bit does a great job translating JP to EN as far as I can tell, and is often better than dedicated translation models like Argos in my experience.
I had a underwhelming experience with Llama translation, incompatable to Claude or GPT3.5+ which are very good. Kind of like Google translate but worse. I was using them through Perplexity.
Training is rather resource intensive either in time, RAM or VRAM. So it takes rather top end hardware. For the moment, nVidia's stuff works best if cost is no object.
For running them, you want a GPU. The limitation is that the model fits in VRAM or the performance will be slow.
But if you don't care about speed, there's more options.
Yeah llama3.1 is really impressive even in the small 8B size. Just don't rely on knowledge but make it interact with Google instead (really easy to do with OpenWebUI)
I personally use an uncensored version which is another huge benefit of a local model. Mainly because I have many kinky hobbies that piss off cloud models.
Yeah, I mean obviously GenX followed a similar track to Boomers and now Millennials are next in the generation line. Except this time, Boomers are in their 80's and GenX are in their 50s so when the Boomer grandparents are dying they're probably leaving inheritance to their millennial grandchildren who need it more than their grown-up GenX children.
I dunno. I've been using Full Self Driving v12.3 and it's gotten sooooo much better than v11. I used to rarely use FSD v11 but now I use v12.3 every day driving around town and on the highway. Looking forward to getting v12.4 in my car next week hopefully too.
YES. I completely 100% agree. Try to minimize the number of messages you're going to send. Just saying "Hi" and then also saying what you want to ask just makes my phone ring twice instead of once and that just gets more annoying with each subsequent message you send before I have a chance to respond.
I'd always make one for my work dev folder so I could quickly switch to it as soon as I restarted my terminal. I'd always make it just a 2 letter abbreviation. For example, my last one was an alias "s6" to take me the the correct dev folder.
