Yes, but only because during your fork, you neglected to run the "white label" script that replaces the trademarked words and graphics. IOW, what you describe is not a problem in practice.
> So Ubuntu is open-source, but if I fork it, it comes with « Ubuntu » in package names, therefore I’m infringing on their trademark?
Yup. However, I suspect that Ubuntu gives a pretty permissive trademark license in their distribution.
This isn't a theoretical argument, the Android OS is open source, but the "Android" brand is a protected trademark. If you're a cell phone manufacturer, you can use the Android code freely, but if you want to put an Android sticker on your phone, you have to follow Google's rules.
Wouldn't affect copyright law too much, but Oracle could probably raise a trademark infringement claim too.