The problem you run into is "we need some more information that is in the logs but we didn't thin to parse before." Here PL/Perl is awesome because you can write a function, index the output, and then query against the function output.
One reason I always store full source data in the db.
> The problem you run into is "we need some more information that is in the logs but we didn't thin to parse before."
Agreed, though with liblognorm rules you just shove every single variable
field into JSON field and that mostly does the job. And in the case you were
talking about logs with no matching rules, liblognorm reports all unparsed
logs, and my logdevourer sends them along the properly parsed logs, so no data
is actually omitted.
One reason I always store full source data in the db.