There is a really weird perception of the importance of books as if there is a important property to be preserved in every book.
This was hilighted to me by a librarian friend if mine, She said they destroy books all the time in her job, and got rather angry when people suggested that it was intrinsicly bad, because there is nothing sacred about simply being a book. It is about the replacibility. They had a program where, if they destroyed a book they would receive a replacement from the publisher for a tiny cost. If a book got damaged it would be much cheaper to get a freshly minted replacement than it would be to spend time repairing.
If there was a glitch in the matrix and 30 billion Gutenberg bibles suddenly fell in the middle of the Amazon(the wet one), most would be destroyed as an environmental hazard.
I see a lot of comparisons to book burning when the AI scanning is mentioned, but the point of book burning is a symbolic act to indicate that the information within the book should not be shared.
Destroying the book to capture the information within sits at the polar opposite reason for destroying a book.
The difference here I think is that the company (Anthropic, not OpenAI) was specifically selecting rare books for training (and subsequently destroying). I think the value proposition is a much different one in this case. In your example the books are still available in high enough amounts that replacing it is cheaper. If the book is rare enough to not have been digitized for training data yet (which is why Anthropic is interested in it), keeping that book around is much more important.
If Anthropic subsequently released the digitized version of it we’d be having a different discussion, but for obvious reasons they’re not.
I have heard allegations that they might be destroying non replaceable books, but only as a hypothetical.
Replacibility is the standard tbat matters in this which I mentioned in my post. I haven't heard of a specific instance of an occurance of destruction of an irreplaceable book, but I'm willing to look into cases if you have a link.
> I have heard allegations that they might be destroying non replaceable books, but only as a hypothetical.
> Replacibility is the standard tbat matters in this which I mentioned in my post. I haven't heard of a specific instance of an occurance of destruction of an irreplaceable book, but I'm willing to look into cases if you have a link.
I think you can assume they're destroying rare or even irreplaceable books, unless they can show they've implemented careful processes to avoid doing that.
But given they're all basically SV startups, it's very unlikely they're doing anything except the minimum effort to get what they want now.
For me it’s just as unbelievable that the second hand book stores would sell rare or irreplaceable books at bulk prices.
If they do they are really not good at what they are doing and I wouldn’t blame any startup big or small to destroy it, especially after preserving it digitally.
> If they do they are really not good at what they are doing and I wouldn’t blame any startup big or small to destroy it, especially after preserving it digitally.
1. That's fucking bizarre logic. "It's ok to do bad thing if someone didn't stop you."
2. These startups aren't "preserving" it digitally. The destroyed physical copies would have likely outlived whatever datasets they're creating.
Rarity equating monetary value is a common logical fallacy often found with stamps and other collectable cultural artifacts (details would disgress too much here, see Pokemon cards or whatever).
And rarity vs quality ir utility is even harder.
But value is 1) not determined by scarcity and 2) neither by "unique information content" (because that's by definition not measurable without a panopticon) and 3) nor by "creative value" per se 4) it is indeterminate whether some detail in a work might prove useful later, in general. Even for scientific papers.
They just can bet on all fronts and for training material it's 100% rational to do it.
Apart from that, to briefly disgress into the discussion of "cultural value": many cultural works we hold as very important today were sold for cheap, if at all, when the author died.
Music scores are good examples.
But back to the broader "value"/rarity angle:
The historical value of some odd self-help or DIY instructions book from the 80s is hard to determine! It might happen to be able to provide a secondary source for something, or even act as primary source for some other fact of the time that is tangential to its subject.
None of the qualities that make trivial literature interesting to, for example, researchers, must correlate with market value (at the time of writing or at the time of research?).
Sure, there are truckloads of "unique" books from even 200 years ago that are neither culturally valuable nor sellable right now.
But these things can change quickly.
Imagine you want research details about old retail products.
Or the usual street price of something in some city at a specific time.
Or the inhabitant of a certain rentable space at a certain time.
Or some specific detail about replacing the parts of a certain type of car that's no longer around (is this valuable then?).
Humans create endless complexities, and monetary or evolutionary value is not coupled anymore to step-changes such as "how to light a fire using X,Y,Z".
We're deep down the rabbit-holes of our own ephemeral shaping of our world.
What word was commonly used to describe a sandwich in 1925 in Hungary.
Or what problems did people commonly encounter when repairing car doors in the 1940s in England?
Whatever. I can't make up a good example right now, but market value does not imply quality; and even quality (according to some cultural norm) is not universally measurable.
They did it due to some weird copyright argument about copying and destroying the original seeming like a transfer and not copyright violation, which is insane and apparently is mot always interpreted in said way. So they are literally taking a copyright gamble they might lose.
It is fair use to train on books you buy. The anthropic case established that. The destruction of books is just cutting the spine so they have individual pages to scan.
They are not destroying any books to make any claim. They are doing it so the pages fit in the scanner.
Without an example of an actual rare book that is shown to have been destroyed, all there is is the is the claim they have the capability to do that.
The closest I have heard is the possibility of rare books purchased in bulk orders of cheap second hand books. This is a similar risk to melting down someone's wedding ring from a batch of scrap metal. It could theoretically happen, but the knowledge of the items presence is absent and it was not requested.
What other allegations have there been? I'm prepared to look at actual examples of you have them.
I think the important part of books is the knowledge in them. Not the binding or anything else physical. With some exceptions of course and sure there are things to be learned about physical attributes that don't translate to the digital version, but 95% is in the content.
> If Anthropic subsequently released the digitized version of it we’d be having a different discussion, but for obvious reasons they’re not.
I don't think you can scan books and just release a pdf. Someone correct me if I'm wrong. Even if you can, I'm sure it's a gray legal area.
Also what's worthless to you is very much not worthless to others.
A pretty obvious example is genealogical archival records. My family only cares about a few dozen pages out of millions for each US census, but every family cares about a different few dozen pages.
While I can see this being a very useful skill for many, we must also remember sometimes reading slow is desired, especially when a deep understanding and analysis of a text is required (e.g reading and analysing literature)
not too great, i submitted a photo taken of me at graduation. it got the location totally wrong and was around 50%-70% accurate on my hobbies and interests. it was able to correctly guess my sexuality and ethnicity, which is rather unsurprising.
> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state
This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a lot.
I have a lot more CPU RAM in my PC, and this would likely increase the size of models I can train locally.
To make the most of these architectures I think the key is essentially moving more of the knowledge/capabilities out of the "weights" and into the complimentary parts of the system in a way that's proportionate to the capabilities of the hardware.
In the past couple months there's been a kind of explosion in small-models that are occupying a niche in this kind of AI-transcoding space. What I'm hoping we're right on the cusp of achieving is a similar explosion in what I'd call tool-adaptation, where an LLM paired with some mostly-fixed suite of tools and problem cases can trade off some generality for a specialized (potentially hyper-specialized to the company or user) role.
The thing about more transcoding-related tasks is that they in general stay in sync with what the user of the device is actively doing, which will also typically be closely aligned with the capabilities of the user's hardware and what they want to do with their computer. So most people aren't being intentional about this kind of stuff right now, partly out of habit I think, because only just now does it make sense to think of personal computer as "stranded hardware" now that they can be steered/programmed somewhat autonomously.
I'm wondering if with the right approach to MoE on local devices (which local llms are heading towards) we could basically amortize the expensive hit from loading weights in and out of VRAM through some kind of extreme batch use case that users still find useful enough to be worth the latency. LoRa is already really useful for this but obviously sometimes you need more expertise/specialization than just a few layers' difference. Experimenting with this right now. It's the same basic principle as in the paper except less of a technical optimization and more workload optimization. Also it's literally the beginning of machine culture so that's kind of cool
> To make the most of these architectures I think the key is essentially moving more of the knowledge/capabilities out of the "weights" and into the complimentary parts of the system in a way that's proportionate to the capabilities of the hardware
I think that's only possible to limited extent. Learnt skills (RL in context of an LLM?) need to be in the weights of the model since this reflects the model's "personalized" learning of the behavioral feedback loop. Declarative knowledge (facts) can be loaded at runtime (RAG).
That's interesting. So you want to train language, linguistic reasoning, and tool use, but otherwise strip out all knowledge in lieu of a massive context? Just grade they model on how well it can access local information, perhaps also run tools?
The claims of the article assumes far more compute and far more VRAM..while the trick enables less back and forth, they don't eliminate it.
I doubt you meant 50M. Rather 50B?
You can only give it a try, but don't get your hopes high on a large context. If their technique works I would guess 8096k context limits would still OOM. 2048 maybe.
I'm extrapolating based on my experiment without this paper's trick to leverage the system memory.
> You can only give it a try, but don't get your hopes high on a large context.
You may or may not know this, but: when training off-the-shelf LLMs (i.e. ones which have a huge vocabulary) what consumes a huge amount of memory usage is calculating the cross-entropy loss (which gets worse the more tokens you stuff in your batch), so always use a fused cross-entropy kernel.
For example, for a Gemma 2 model with 2B parameters at a batch size of 8k this consumes 24GB of VRAM by default (!); you can fuse your cross-entropy loss with @torch.compile and that can cut down this memory usage to something like a few gigabytes, but with a dedicated kernel this becomes a few megabytes.
> "The integration involves modifying the TransformerDecoder module in torchtune to bypass the linear layer computation, allowing the Liger Fused Linear Cross Entropy Loss to handle the forward projection weights. "
Although this wasn't integrated into PyTorch itself (but to torchtune, which is a different thing). If you're writing your own training loop you need to use a third-party kernel, e.g. the Liger kernel mentioned in the article, or Cut Cross Entropy (which is much better than the Liger one, although IIRC it has a numeric bug in one of its kernels making the results very slightly off).
> Activation would still require gigabytes for a few kb context.
For that you use activation checkpointing, and you can also offload that to the CPU in a smart way to hide the latency. Although, yes, for long context training the activations do dominate the memory usage (and quantizing them degrades things more than just quantizing weights and/or optimizer states).
> This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a lot.
I'm on the same GPU, its intimidating to me if I even want to bother training anything at all. Do you mind sharing what kind of training you've done with that GPU? :)
Make sure you are running adaptive cooling (or just bump them up) on the top fans in your case. Also, ensure that you are undervolting appropriately using something like MSI Afterburner or GreenWithEnvy.
If you don't, you could easily toast your RAM -- especially under BF16.
Anything that can run on a AMD395+ w/128GB or whatever the apple equivelent would break things wide open. Training a model on my frameworks of choice or our business info would be awesome.
