I've done a lot of work using both R and Python, and a little bit using Matlab (and SAS and Stata).
Matlab is behind R and Python for both data cleaning and analysis. This is especially true if you have string variables and factor variables. Matlab's great if you are doing matrix operations of clean data (and you don't need to do anything fancy in how you report them). But, I don't find it worth using for real data analysis.
Python and R are both great, though they feel pretty similar to me. Python obviously has the big advantage if you want to do general purpose computing too. Python has been faster than R in most cases I've compared them. For my work, development speed is more important than execution speed... so this hasn't been a huge factor for me.
R has a couple of big advantages. First, the libraries. This is a big deal for me. I saw that there is a python equivalent to ggplot2 in the works. This will definitely strengthen the case for python, but the availability of libraries in R is awesome.
Second, the community and help resources in R are amazing. I rarely run into a problems in R that haven't already been addressed on stackexchange.com.
Perhaps I should be more proactive about asking python questions when I run into them, but I usually just work it out myself (which is more time consuming than looking the answer up online.)
Lastly, I'm not an expert on big data. But, spending relatively little time with both R's bigmemory and Python's PyTables, it seems easier to get up to speed on big data with R at the moment.
Though I haven't met them, my sense is that Wes, Travis Oliphant and the other relevant python developers are putting in a heroic effort to get Python up to speed. I have every expectation that Python will be my choice of the future.
Note: This reply is mostly helpful if you work with legacy Matlab code, or have colleagues who primarily know Matlab, and you have to work with string data.
In general I agree with you; Matlab's age and origins show through in some warty ways, and one of them is string processing. Whenever I have to process anything that's not simple CSV or Excel, I use Python. (For XML, there's Perl Xpath command line tool, which has come in pretty handy for simple XML extraction.)
That said, however, the Statistics Toolbox has classes dataset, nominal, and ordinal that take a huge amount of the pain out of working with string data. Dataset lets you mix column types and refer to them by name, and lets you name rows if you like. I think it's similar to a dataframe in R. Nominal and ordinal are efficient representations for string columns. They are a workaround for Matlab's lack of a runtime string pool, but also are fast and small.
This is a really detailed comparison and seems balanced. Thanks for that. Do you not use both in conjunction (R for the stats libraries and Python for the more general computing)? Having done some background reading (but without yet getting down into the weeds with them), that was my take on the relationship.
Edit: re-reading your last paragraph, I guess you're saying that Python can reach parity with R's libraries, at which point it's elegance and speed will win out. R's decades of lineage do seem to hobble it in terms of style and syntax, after all.
Even though I hear complaints about R's syntax, I don't know exactly what people dislike about it. In fact, I kind of like R's syntax.
As an example, I like the ability to use expressions on the left side of an assignment (e.g. names(df) = "stuff"). But, it sounds like you are right that the python developers are getting to learn from R's mistakes and avoid getting locked into to legacy ideas.
As far as libraries... R has a lot. So I don't expect python to totally catch up soon. But, I only use 10 or 15 R libraries, and those are really popular libraries. So, unless you do an incredible range of stuff, python probably doesn't hvae to completely catch up.
One major advantage for R is the package management system (CRAN). The uniformity of the interface... the ability to search for stuff in it... that's been really useful for me. Not sure if anything like that is in the works for python.
Lastly, there are a lot of little helper functions that I've written for myself in python that are part of base R. The first example that comes to mind is head() to view the top few lines of a data structure. It seems strange that python would be missing these little helper functions, but I never found it.
I have been programming python too long to make an objective comparison with R. I have had to use R libraries at times, and I've found rpy to be a workable bridge from R to python for this purpose. Depending on how it works under the hood, it might not be appropriate for big data, though. Also, I had to custom modify some R libraries to work with my data, so it has been useful to know a bit of both, although I mostly picked up the R as I did the mods.
I would add that an advantage of Matlab for number bashing is a much more native handling of linear algebra. Let's say you have two matrices A and B, in matlab I could write:
A*B*A'
Whereas in python it would be (approximately):
dot(A, dot(B, inverse(A)))
so matlab can evaluate everything in the right order (right to left) whereas for numpy I end up writing a little recursive function to dot a list of arguments together from right to left, which feels a bit cludgey and more of an impediment to getting your ideas down in code. Especially when your equations get very big as they often do with stats!
In [7]: A = numpy.matrix('1 2; 3 4; 5 6')
In [8]: B = numpy.matrix('1 2 3; 4 5 6')
In [9]: C = numpy.matrix('1; 2; 3')
In [10]: A * B * C
Out[10]:
matrix([[ 78],
[170],
[262]])
In [11]: (A * B) * C
Out[11]:
matrix([[ 78],
[170],
[262]])
In [12]: A * (B * C)
Out[12]:
matrix([[ 78],
[170],
[262]])
I've very briefly experimented with rpy2. It got the job done, but I thought it was tricky enough that I'd want a good reason to combine R and python. Otherwise, I'd try and do the whole project in one or the other (And I haven't used rpy2 since I first tried it out.)
It's true in the short term that social and mobile trend chasing will be detrimental to investment in other areas of tech, but markets and VC returns will correct this over the longer term.
Trend chasing and herding makes it harder to invest in the winners in a space at a reasonable valuation.
If VCs invest in 100 social/mobile start ups at $100 million valuation and only 10 reach $1 billion, then the VCs on aggregate are break even. Also, a VC has a 1 in 10 or 1 in 100 shot of getting that big hit because there are so many 'me toos' in the same space.
If a VC invests in the next spaceship, smartwatch (Pebble), or X less competitive field, there might be only 2 companies competing for that pie. So you have a 50% chance of picking the winner.
The economics are also more compelling for less crowded fields because customer acquisition costs, talent hiring costs and other costs are cheaper. They're not being driven up by the other 100 'social networking / Instagram' start ups competing for the same resources. Your market share of revenue is also higher because you are sharing it with 1 or 2 other players, instead of 100.
That makes the margins better for the the less crowded areas of technology.
If you started a social or mobile company in 2005-2008, your customer acquisition cost was close to zero and engineers were cheap so that made each VC dollar go very far. You also didn't have to compete against 10 other companies for mind share in your space.
That's not the case on social or mobile anymore.
Some smart VC firm will invest in a hardware, biotech or (insert non social / mobile) start up and make a ton of money. Then all the money will chase that area and the cycle will repeat. The VCs which invested in the 'me toos' will lose money and lose assets.
This is more likely to put mom & pop stores out of business than threaten the big players like Fastenal, Grainger, McMaster-Carr and MSC Direct.
Mom & pops focus more on retail / small business customers who are price sensitive, so this is going to hurt them.
But the big players mainly rely on their services because industrial and commercial customers care more about minimizing downtime and reliability than on price. The customer service at these companies is great and helps sort through 1 million+ skews, which is completely different from picking 1 out of 20 TVs at Best Buy.
Their customers are companies like Caterpillar who need projects to be on time and same day delivery. Fastenal even puts fasteners directly inside of bins on the assembly lines so that the customer doesn't have to haul a huge chunks of metal.
Amazon would have to seriously boost its service (ie, same day to location) delivery to gain the non-retail customer base as well as expand its sales staff expertise dramatically (which of the 1000 cutting tool bits do I need for X job, what angle does it have to be installed at etc?). What this will do is consolidate the industry with Amazon and the existing big players at the expense of the mom & pops which are over 80% of the market.
It's a $140-160bn a year market. Just goes to show how much opportunity there is in technology to expand outside of consumer-focused businesses.
GoDaddy is owned by private equity firms KKR and Silverlake. These firms aim for short term profits at the expense of creating long term value.
Silverlake was the firm behind the whole Skype compensation fiasco, which was basically a quick flip. KKR does dividend recaps where it issues debt and funnels money out of the company. The company, such as HCA or Dollar General, then has to fire people and raise prices on customers.
Do you think they'll do what's right for customers over the long term or try to extract every penny? That's why they'll flip flop on an issue like SOPA - no long term values. I moved my domains to namecheap a long time ago.
If start ups only refer to businesses where technological innovation drives competitive advantage, then New York will have a tough time. The future Googles, Intels, Apples etc fit in this category.
But if we include businesses which are technology enabled, then I think New York has a much better chance. Companies like Gilt Group, Kickstarter and Tumblr don't rely on technology to drive their competitive advantages.
If the technology component becomes commoditized due to ease of access (rails frameworks, Heroku, mobile frameworks etc), then competitive advantages will shift to community building, branding and design.
New York has the advantage of being a dense city (good for location-based mobile apps). It has a rich history of being the entrepreneurial home of fashion, media, retail and finance. New York based start ups will have an advantage in building communities around these industries.
I'm looking to start an e-learning start up. Been involved with some start ups on the business side and I can code enough to set up a web app / social app, but looking for a technical cofounder who has better chops than me.
Pretty sure the Beatles had a pretty big effect on society and the status quo. The power of art and music tends to be a second order effect: it inspires people.
Without the music of the 60's/70's, do you think Steve Jobs would have dropped acid, become a Zen Buddhist and built Apple's brand as a the challenger of the status quo?
Let's give them a taste of their own medicine. They grew off exploiting the Facebook feed and twitter.
I started #boycottZynga on twitter. Please retweet. If we get the message out and convince people to stop playing before IPO, we can hurt their valuation.
Then they'll realize they can't just bully normal people.
I hope the handmade movement really takes off. Maybe people could consume less but unique artifacts that have a real story behind them rather than accumulating mountains of mass produced, generic stuff that ends up in a trash heap a couple of years later.
There is a happy medium with mass production - the main issue is that many people (in my observation) no longer value quality equally to price, e.g. the cheapest pot from WalMart will do, and when it chips or warps next year we'll just spend $15 on another one instead of buying a single $50 version that will last. One of the greatest lessons that I've ever learned in business is that you shouldn't spend extravagantly, but you shouldn't be afraid to spend a little more for quality where it matters.
you shouldn't be afraid to spend a little more for quality where it matters
It's more likely that they don't see any reason to spend more than $15. Someone knows they can get a pot that looks OK for $15, whereas the $50 doesn't have any immediately obvious benefit.
I love tools, but I'm far from a tool snob. I have lots of cheap tools because I'm never going to wear out, e.g., a screwdriver. It's worth more to me to have five $3 screwdrivers than a single $15 one. But I just spent $200 on a wire crimping tool because I know that the $15 ones won't do the job right. But if they did, then there is no way I'd buy the more expensive one. Most people are just like that: they won't spend more than they have to unless there is a clear benefit.
> I know that the $15 ones won't do the job right. But if they did, then there is no way I'd buy the more expensive one
That's a matter of perspective. If you cook often and have had an opportunity to use the two, you'll notice a stark contrast; just like you notice the difference with crimpers. The guy spending a weekend to redo his home theater wiring is probably going to opt for the $15 option.
How many people research purchases like pans and wire crimpers in order to gain a proper perspective of where the price vs. quality lines cross, and how many just go to the store and get the one with the prettier package? I think that's where you start to see the trend I mentioned.
In many cases the $15 pot from WalMart is better than the $50 pot. It's thinner than the $50 pot, so it reacts to adjustments in temperature a lot faster. The coating is simple thin Teflon, so it wears out a lot faster than the $50 pot, but it actually has better non-stick properties than the $50 pot.
I've got expensive pots, but the skillets that I use the most are a $5 teflon/aluminum skillet I got from Loblaws that I replace every 6 months, and a $15 cast iron skillet I got from a flea market that I properly seasoned and I expect to last 100 years.
(A bit off-topic, but...) I've supported myself on the side over the years as a cook, and I can tell you there are some misconceptions there. The difference in heating time is marginal, but thicker bottoms disperse heat _far_ more evenly, preventing hotspots that can ruin whatever you're cooking. Thin pans also have a tendency to warp, but you've solved this by replacing it every six months.
As far as nonstick properties go, a hard anodized pan will be every bit as nonstick as a cheap Teflon pan, but with the added benefit of not leaching chemicals into your food when it gets scratched. Aluminum pans are also great, provided that you know what you're doing - be sure to get them up to proper heat before adding your oil and/or ingredients, and clean them properly.
That's not my experience. Aluminum is a great heat conductor and I've got a fairly nice stove, so I don't experience much in the way of hot spots. The outer ring of my pan is much cooler than the inside, but that's a good thing, IMO.
On the other hand, cast iron is not a good heat conductor, so I experience much greater heat variation with it unless I let it warm up for quite a while first. The aluminum I can use the instant I put it on the stove. I love it for that.
Teflon is pretty much the ultimate in inert. There is no way you are leaching any chemicals into your food. You could scrape the coating off and eat it and it would do nothing.
I think it is not mass production that is the issue, but the short lifespan of the things we use. If every tool, every building, every mechanism - electronic or otherwise - lasted forever, or even just ten times as long, our economy and society would undergo radical changes as have never been seen before. (Sorry for what is both an obvious observation and a complete departure from the OP, but I have had this thought stuck in my mind for a while)
You're right - I wasn't referring to all mass produced things. But I do like it when someone buys (or makes) a gift that's unique, but cheap, rather than something generic but expensive.
For example, take toys. There's this store that sells hand made toys around the block from me: http://playing-mantis.com/wooden.htm. Does it make sense for a kid to have 3 plastic toys they'll use for a couple of months and throw away or to buy a handcrafted toy they could keep for life (and even pass on)?
When I talked to the store clerk at Playing Mantis, she said most of their toys don't come from the US but from Germany, Austria and Japan because the craftsmanship is good there.
It's not a pipe dream that this could change - companies like kickstarter and etsy are enabling this move towards story-driven, experiential consumption.
I take it you don't have kids. It doesn't matter how long the toy could last, they'll still only play with it until they get bored then it gets lost or ignored.
It's the same with most things that are continuously being improved: another poster made the point about cars. Most people don't want a daily driver that's a 30 year-old gas guzzler even if it still runs fine. New cars are more comfortable, cheaper and have lower operating costs.
It's not going to change: people want stuff. The cheaper it is, the more stuff they can have or the less they have to pay for the same amount of stuff.
Lego are mass produced and they just don’t break. They are basically forever. I played with Lego through all my childhood and I broke maybe a handful of blocks.
“Mass produced” is just a bad heuristic for quality.
I don't know; plenty of people I know throw away (read: recycle) functional stuff, just because it's functionally obsolete. Even I, who never bought a cellphone, have a working Nokia 3310 on some drawer. But who'd want it? You can get a better, newer phone for the same price as the shipping fees for a package.
Besides, do we really want unsafe, gas wasting cars to last forever? Electricity wasting fridges or TVs? Isn't it better to recycle them?
Handmade / custom knives are quite popular among collectors and enthusiasts. They're expensive, you can't chuck them in the kitchen sink until the next morning or the carbon steel will start to rust, but they are just awesome to use, and I expect they'll be appreciated for generations to come.
Just for a reality check, let's say a top-quality, handmade, 8" chef's knife runs about $500. Most people consider an $80 mass-produced piece from Henckel or Wusthof to be rather extravagant...
Example: if you have one, she hires 2, those 2 hire 2 each. You have 1+2+4 = 7 = 8 - 1 = 2^3 - 1