Yeah, original TurboFieldfare supports Gemma https://github.com/drumih/turbo-fieldfare

Thank you for using TurboFieldfare as a starting point for this project and thank you for mentioning it at the README. I am glad it inspired more people to explore area of on-device AI further!

Thank you for testing and sharing results!

I think I might understand your use case. You ssh the Mac and want something like `ollama run` with an interactive chat in terminal. Am I right?

There is already experimental OpenAI-compatible server in this repo:

``` swift build -c release --product TurboFieldfareServer .build/release/TurboFieldfareServer \ --model scratch/gemma4.gturbo ```

After that a small terminal client can run inside the same ssh session and talk to `/v1/chat/completions`

The client needs to keep a messages array, add each user message, send the full array with `stream:true`, print SSE chunks until `[DONE]`, then add the response back to the array. `/reset` can clear it

There is a python example in the server docs. (https://github.com/drumih/turbo-fieldfare/blob/main/docs/OPE...)

It is non-streaming, but can be used as a starting point.

The server is still experimental and I am fixing some problems currently. But you can try to vibecode a simple terminal client around it.

If not, create an issue on Github and describe desired behaviour

Yes, that is the idea. did read the paragraph about the cli server but of course that goes beyon what i know about these tools, that's why i asked. Having a streaming chat via command line would be awesome, because then multiple users could use the machine at the same time (i think 2, max 3 on a 8GB device. It would really make our old mini useful again, instead of sitting in the corner taking space)

I think there is a limit based on MoE number of active parameters and quantisation, bytes count for active experts. But I believe we will see more project like this for different models.

Uh, maybe, who knows. I am pretty bad solving leetcode, btw

Yeah, it must be exactly the same. The same weights are used, nothing skipped or pruned. But it might have differ to MLX for greedy decode because of small floating-point nums difference

Apple does something similar with their latest foundation model. They process input prompt and based on results they preload required experts. Quite neat solution for the edge devices

afaik there is some research at this area. Also the new apple foundation model uses related idea. they process the whole prompt and based on prompt load required experts and use only these experts for generation. It doesn't require fitting full model into memory or per token ssd streaming

Yeah, must be possible. Not fast, but possible if you have enough ram. I think you can search online for projects, I think I saw something related

It heavily relies on M-series Mac unified memory architecture. And shaders are written using Metal, Apple's own gpu programming technology. It cannot be ported directly to classic architecture (ram+vram)