You'd really need to see the paper to evaluate this, because pretty much every static analysis technique can generate lists of "undetected security flaws", especially if the corpus of application code is unconstrained. You'd want to know:
* Did they find a kind of security flaw that is particularly difficult to detect? Like, reliable generic authorization bypass detection result would be novel.
* Did they find 23 flaws in popular and important Rails apps? Those applications have probably been tested extensively already, and new results are interesting. Random e-commerce applications buried in Github, less so.
* How severe were the flaws? Security flaws range from remote code execution on the very damaging side to version information disclosure on the marginal side.
I've been working through the paper for almost two hours now. I'm not an expert on static analysis, so I don't know how much of the information in here is novel, but the paper does a great job of concisely conveying a lot of interesting ideas. His model for representing data exposures is beautiful in its simplicity, and I think it could almost trivially be adapted for dynamic analysis too.
> Did they find 23 flaws in popular and important Rails apps? Those applications have probably been tested extensively already, and new results are interesting. Random e-commerce applications buried in Github, less so.
Based on the paper, he performed two separate experiments.
For one of them, he chose the 50 most popular Rails projects on Github. Of these, 30 of them used a permission model that could be handled by his tools. The 23 flaws reported were among these 30 apps, with Diaspora being notable among those.
He also worked with a professor to test student submissions for an access-control assignment in a web development course at MIT. He uncovered security vulnerabilities in "over half of these projects" and "about half of those bugs were missed during manual grading."
"Derailer uses an automatic static analysis to produce a visual representation of the information flows within a Ruby on Rails web application. The visualization organizes the results of the analysis in a tree, with the goal of helping the user to quickly find the most sensitive information flows, and also allows the user to filter the set of flows based on the conditions under which they occur."
"Rubicon is a library for Ruby, Rails, and RSpec that lets you write formal specifications of the behavior of your web apps. In addition to the standard RSpec language, Rubicon gives you the quantifiers of first-order logic, so your specifications cover all possible objects of the given type, and mock objects are no longer needed. "
"SPACE is a specification-free tool for finding missing security checks in Ruby on Rails web applications using a catalog of access control patterns in which each pattern models a common access control use case. SPACE checks that for every kind of data exposure allowed by an application's code, some security pattern in our catalog also allows the exposure. The user provides a mapping from application types to the types of our catalog, and then SPACE identifies security bugs automatically."
According to the metadata in the gem, Rubicon is from 2012. We must download the gem from its site, which is weird, and I couldn't find it on github. I assume it is a dead project.
Derailer's last commit is from Jan 29 2014 so it's another dead project. Too bad because it could be interesting. However, did anybody understand what to do with its output? Example: how do I use http://people.csail.mit.edu/jnear/derailer/example1/ to assess the security of the application?
It isn't the most important aspect of this article, and I know that tech journalism by persons outside the field is prone to this sort of thing, but a little bit of me dies inside when I read or hear the phrase "The Rails Interpreter".
It is absolutely terribly wrong. No, even if that were true, _Ruby_ the language would have already had to interrupt the Raisl code to "..Rails has a ruby interpreter that loads rails".
Ruby on Rails — or Rails, as it’s called for short — has the peculiarity of defining even its most basic operations in libraries. Every addition, every assignment of a particular value to a variable, imports code from a library.
No idea, and the author didn't have any clue too. Maybe it alludes to Rails being a collection of gems. The part about assignments is hard to figure out, and I'm a Rails developer.
Tool link: https://overseer.fallible.co (I doubt people will be okay with installing our certificate, but just in case someone wants to use a hosted version)
For whose wondering, brief read of the paper's initial chapters show that it's a methodology (and toolset) to find access control issues in RoR. Seems like a smart approach, need to see it in production though. I'm not really surprised if they found various ACL related vulnerabilities with this model in web apps, it's a common issue and not easy to test & check.
Injection vulnerabilities (XSS, SQLI etc.) are out of the scope of this document/toolset, it's heavily designed for detect ACL issues in web applications.
* Did they find a kind of security flaw that is particularly difficult to detect? Like, reliable generic authorization bypass detection result would be novel.
* Did they find 23 flaws in popular and important Rails apps? Those applications have probably been tested extensively already, and new results are interesting. Random e-commerce applications buried in Github, less so.
* How severe were the flaws? Security flaws range from remote code execution on the very damaging side to version information disclosure on the marginal side.