We can trust math because it is trivially verifiable. Its relation to physics is somewhat complicated (e.g. verifying physics is not as trivial), and yes - you can understand a lot about physics without math - by combining direct observation with trusting the "philosopher".