I have to imagine there are some costs associated with a data scientist having to pull data out of a swamp. I also have to imagine that different terms and abstractions will be created by different data scientists to understand the swamp better. Data Scientist A will say "Go over yonder to Bedem" and DS B will say "Ah, by Yumon?" and neither will have any real idea what each other is referring to.
This makes me curious, what are the costs down the road VS upfront costs? There's a side of me that just feels like it's somewhat laziness to not employ a schema -- even a very flexible one. And if data scientists were to come together and agree upon some flexible schema, would it not be at least a step in the right direction and one capable of constant iteration / improvement?
I am of the mindset that single-responsibility-principle applies here. If you could build a framework that pipes out some data in a well-tested, reliable manner, then all analysts could hook up to that single pipe -- not create their own hoses and fishing rods.
The only purpose I see in a data lake is to provide a single access point to all the stores in the organization. Since it is up to the consumers to digest this data, you have to imagine this will sprout many different solutions unless the different consumers work together to create a framework to pull out of the lake. If they did do that however, I have to imagine you'd be on your way to building a schema because it would be easier for the framework to interface with.
This makes me curious, what are the costs down the road VS upfront costs? There's a side of me that just feels like it's somewhat laziness to not employ a schema -- even a very flexible one. And if data scientists were to come together and agree upon some flexible schema, would it not be at least a step in the right direction and one capable of constant iteration / improvement?
I am of the mindset that single-responsibility-principle applies here. If you could build a framework that pipes out some data in a well-tested, reliable manner, then all analysts could hook up to that single pipe -- not create their own hoses and fishing rods.
The only purpose I see in a data lake is to provide a single access point to all the stores in the organization. Since it is up to the consumers to digest this data, you have to imagine this will sprout many different solutions unless the different consumers work together to create a framework to pull out of the lake. If they did do that however, I have to imagine you'd be on your way to building a schema because it would be easier for the framework to interface with.