Seems to be the case with all their models - really huge in size, no actual performance gains for the effort.
Their refined web dataset is heavily censored so maybe that has something to do with it. It’s very morally conservative - total exclusion of pornography and other topics.
So I’d not be surprised if some of the issues are they are just filtering out too much content and adding more of the same instead.
What? Falcon-7B base model is pretty much one of the only few small models that'll happily write a whole coherent fanfic all the way to the end without getting stuck in a loop right before the explicit content.
Their refined web dataset is heavily censored so maybe that has something to do with it. It’s very morally conservative - total exclusion of pornography and other topics.
So I’d not be surprised if some of the issues are they are just filtering out too much content and adding more of the same instead.