Agreed. I was at a Hackerspace years ago that worked on this principle, and wanted to start lower than that.

http://nand2tetris.org/course.php

Alas, I left before I could really get into it, and never had the peer pressure to understand a lot of core CS way over my head.