Inference takes 30+ seconds before it starts outputting even the first token. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		dmazin on May 1, 2024 \| parent \| context \| favorite \| on: Run llama3 locally with 1M token context Inference takes 30+ seconds before it starts outputting even the first token.

danielbln on May 1, 2024 [–]

How's recall over 1M tokens?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact