The model still needs to attend to the prompt when generating the answer. Modern... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		macleginn 9 months ago \| parent \| context \| favorite \| on: Claude's system prompt is over 24k tokens with too... The model still needs to attend to the prompt when generating the answer. Modern attention techniques help here, but for lots of simple queries most of the compute still goes into taking the system prompt into account, I guess.

saagarjha 9 months ago [–]

Sure, but without the prompt you will probably have significantly "worse" queries, because you'll be starting from scratch without that context.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact