I'm not too up on the Java world, but starting to poke my toe in with my interest in JRuby and Clojure.
So forgive me if this is a dumb question: what is the nature of the Oracle closed source extensions? Are they anything I'm going to care about in playing with JVM languages, or deploying apps in a web startup (ie. non-corporate) environment?
Nothing you'll miss in those contexts. Most of what is closed source is code that 3rd parties wrote and Sun couldn't open source (code java.awt relies on mostly.)
For the actual JVM I don't think there is anything big missing, except maybe some support for SPARC. The big question will be what happens in the future when they start merging JRockit with Hotspot. JRockit has quite a few extensions for monitoring that I think Oracle will keep closed (and expensive.)
AFAIK it's rather the other way round: the original Sun JDK contained some third party components that could not be released under the GPL. OpenJDK is the result of going through everything and making sure all those parts are replaced with clean-room reimplementations.
Note that starting with Java 7, OpenJDK is the reference implementation.
So forgive me if this is a dumb question: what is the nature of the Oracle closed source extensions? Are they anything I'm going to care about in playing with JVM languages, or deploying apps in a web startup (ie. non-corporate) environment?