Of course we can escape set theory (and categories) when talking about the natural numbers. The concept of "two" predates the concept of a set by at least a thousand years. People were happily manipulating and investigating the natural numbers before set theory ever came along.
As I said previously, it is true that set theory provides a way to encode the natural numbers as sets (or features of a topos, etc.), so that questions about natural numbers can be stated as questions about sets. It is further true that this endeavor can be incredibly fruitful, for instance for studying the foundations of mathematics. But it does not mean that natural numbers "are" sets (or objects in a topos), any more than Quicksort "is" a piece of C++ code.
As I said previously, it is true that set theory provides a way to encode the natural numbers as sets (or features of a topos, etc.), so that questions about natural numbers can be stated as questions about sets. It is further true that this endeavor can be incredibly fruitful, for instance for studying the foundations of mathematics. But it does not mean that natural numbers "are" sets (or objects in a topos), any more than Quicksort "is" a piece of C++ code.