Are there significant usecases for the really small LLMs right now (<10b distill... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		myrmidon 5 days ago \| parent \| context \| favorite \| on: Raspberry Pi's New AI Hat Adds 8GB of RAM for Loca... Are there significant usecases for the really small LLMs right now (<10b distills and such)? My impression so far was that the resulting models are unusably stupid, but maybe there are some specific tasks where they still perform acceptably?

arkmm 5 days ago [–]

They're still very good for finetuned classification, often 10-100x cheaper to run at similar or higher accuracy as a large model - but I think most people just prompt the large model unless they have high volume needs or need to self host.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact