Are you anywhere near (the remains of?) a canal network? That was how bulk cargo was generally transported in the 18th century. First-mile + last-mile would be by wagon, of course, though usually pulled by oxen, not horses. Canals were economically revolutionary, for 100-150 years, until railroads largely supplanted them. They remained viable, in limited circumstances (ie, some routes for some cargos), until the mid-20th century.
Astute observation; no canal, but there is a river outflow to a bay, whereby a ship could have carried stones from the quarry, albeit a long way around a peninsula; it is possible that was a more effective way to get them close, and then use horse and cart to get them the last bit of distance.
Thinking about the logistics of such a feat at that time is wild to me for just the construction of a private residence.
