I'm not an expert, just a hobby language designer, but it seems to me that the advantage of starting with an interpreter first is that this allows you to more easily design a language in which compile-time expressions have some reasonable expressivity. Generally, an interpreter allows the designer to experiment more with the semantics. Typical examples are evaluation of expressions for constant definitions at compile time or having built-in access and language support (e.g. syntactic sugar) to 3rd party libraries like arbitrary precision arithmetic.

The disadvantage is that it can be a lot of work. From my experience, writing a toy VM in Ada didn't seem much easier than writing a compiler in a toolkit like LLVM. (But I've never done the latter.)