One particularly unpleasant job (I believe often performed by slaves) was making charcoal. More man-hours went into making charcoal than went into the actual blacksmithing, as it was so coal-intensive. Entire regions were denuded of trees. And then of course you had lumberjacks.

Honestly, following around some rich person in Rome sounds easier than the hardest manual labor jobs.