I have both an early-init.el and init.el in my .emacs. In the former I turn off package loading on startup, but also set most UI variables so menus etc are turned off before the frame is shown (stuff you might once have done with Xresources). This is also where you might do more extreme performance related tweaks (e.g. to the GC) if you really care about startup time. My init.el then does package-initialise and is mostly use-package calls.

That is your.emacs.d, I presume? I am still mostly carrying around a .emacs and not using init.el.

That said, played with this some today and I see that I can get my startup down to about 5 seconds on my old netbook. Not too bad, all told. (This computer is hilariously slow.)

esup is a great way to see where the time is going. It's not really a big deal in the grand scheme of things but it's also a good way to learn about autoloads etc.