I love ggsql, the team behind it, ggplot and the tidyverse in general!
I've been mulling extending dbt Charts to support ggplot, and will likely reach out to the team about this in the next few weeks.
The difference is that Vega-Lite, ggsql, ggplot2 are designed to create a single chart. Yes, you can use facet_wrap() to great a series of charts based on a particular variable, but arguably that's still one chart!
What's missing that is table stakes for a BI dashboard: KPIs, text boxes, dropdowns, data sources, governance, and how the charts are arranged on the page and in relation to one another.
In the next few months we hope to ship a JSON schema that represents a dashboard in the same way that a VegaLite does for a single chart.
Oh yeah totally. I guess seeing your SQL queries embedded inside the board doc, I immediately wondered if it would make sense to embed a GGSQL program the same way, but in the charts section instead of the queries section!
I guess at the moment GGSQL itself wants to be directly connected to the source database, whereas you've got a split between queries and the charts which display the results.
It would be really cool to have SQL "all the way down" to the chart definitions as well :)
What drew me to the dbt project back in 2020 and what ultimately led me get a job at dbt Labs was this notion that when compared to software engineering teams, analytics teams had been vastly unserved by their tools.
In 2016 (and largely to this day), BI teams work in point-and-click interfaces without Software Development Lifecycle mainstays like source control, testing, and deployment environments.
There are BI engineers who do work like software engineers, but the gap b/w those who work like SWEs and those who don't is wide.
dbt Charts (like dbt before it did for data engineers) aims to empower data analysts to work with stakeholders in a more efficient and sustainable manner than was previously possible. The means to this end are: a succinct YAML DSL spec for defining charts, a powerful CLI that lets you validate, compile, and render the code into a dashboard spec the YAML.
Like data analysts being able to do PRs for changes to a dashboard is still rather unheard of especially if those PRs are in repos with other analytics code. Many BI vendors do ship some form of diffing and environment promotion, but virtually always these features fall short of what SWEs use every day.
Why go for a configuration based format over a library though? Something like Streamlit or Dash offers more control and lends itself better to interactive exploration.
A library is for coding in the classical sense. Declaring in YAML is a very light form of coding I would say, much more accessible to non-engineers. A simple dashboard in YAML is so much more readable to a human than some python code that uses a library. This is mainly a audience question.
If there's a use case in the future, we could consider also offering a library. But if you just want to vibe code dashboards in actual code, there's plenty out there already that will give you JS code (predominantly).
hey -- Anders here, I've been helping the dbt Charts team. I loved this list you've enumerated here
> There’s governance, access controls, interactivity, and connectivity to semantic layers
I'd add to that list as well: KPIs, dropdowns, text boxes, theming, shared data sources.
If you're familiar with Vega-Lite [1], I like to explain dbt Charts as the Vega-Lite but for BI, in that it is a language that specifies all of the components of BI.
> are those other pieces going to be part of dbtTran Cloud?
Certainly there are some features that are more conducive to being offered as a managed service. However, we've built this enterprise BI tool backwards in that we've started less with commercialization in mind, but rather laying the groundwork in the language and OSS project, so that we need not reinvent the wheel over and over again.
I'm hand-waving here about a future that doens't yet exist. But in theory dbt Charts could be extended in the front end to allow for not just SQL or SL metric support, but any query language. Same for the backend, we dbt Charts can render to png and html today, but other formats are also possible!
I think the advantage is simplicity. Why connect first to duckdb and attach the db when you can query it directly with ADBC which is guaranteed to be fast
Yeah for me standardization is the big win. But not just output formatting but cli commands and a guarantee that they’re as past as possible given that all the connectors use ADBC
hiya, anders from dbt here. cool project -- I especially love the branching and budgeting options you've built in. both are things that I'd love for the dbt standard to include one day. was it dbt's lack of those feature that inspired you to start this project? It also seems you have an aversion to Jinja, which, believe me, I get!
FYI dbt-fusion [1] is going GA next week (though GA for Databricks will come later) Most of it is source-available and ELv2-licensed, but there's a number of crates that are Apache 2.0, namely: dbt-xdbc, dbt-adapter, dbt-auth, dbt-jinja, dbt-agate. We also have plans to OSS more as time goes on (stay tuned).
I just wanted to call out the OSS crates in case you'd rather focus on "making your beer taste better" than have to re-build foundations. I'd love to hear if any of those crates come in handy for you (even more so if they don't work for you).
Feel free to reach out on LinkedIn or dbt community Slack if you ever want to chat more!
Hey Anders! Thanks a lot for dropping a comment and show interest in Rocky.
Yes! I won't going to lie that Jinja is one of the things that gives me some itches :). But it wasn't the major reason for start building Rocky though.
It all started with the need for auto-generating dbt models from the FiveTran connections I integrate with, then having to hot reload code location in Dagster to discover new assets. All in a zero-touch data pipeline. FiveTran connections are discovered as they're created, assets are materialized as these connections sync.
Auto-generating these dbt models and get the manifest aligned between Dagster code location reloads plus spinning up pods in EKS for each Dagster runs that need to rely on these auto-generated models have some impact on the performance overall, not only in production, but also affects DX in their local environment.
Rocky wasn't born with a "dbt replacement" in mind at all, but it was born to solve a real issue I'm facing. I made sure I can integrate well with dbt as it's in my plans to leverage the awesome work available as dbt packages for FiveTran.
I'll definitely have a look the crates you mentioned!
Thank you!
> Auto-generating these dbt models and get the manifest aligned between Dagster code location
I just added you on LinkedIn. if you accept my connection there I can DM you a private preview document that you might find very interesting related to dbt project metadata (that is way less painful than `manifest.json`)
ok, this is definitely up my alley. color me nerd-sniped and forgive the onslaught of questions.
my questions are less about the syntax, which i'm largely familiar with knowing both SQL and ggplot.
i'm more interested in the backend architecture. Looking at the Cargo.toml [1], I was surprised to not see a visualization dependency like D3 or Vega. Is this intentional?
I'm certainly going to take this for a spin and I think this could be incredible for agentic analytics. I'm mostly curious right now what "deployment" looks like both currently in a utopian future.
utopia is easier -- what if databases supported it directly?!? but even then I think I'd rather have databases spit out an intermediate representation (IR) that could be handed to a viz engine, similar to how vega works. or perhaps the SQL is the IR?!
another question that arises from the question of composability: how distinct would a ggplot IR be from a metrics layer spec? could i use ggsql to create an IR that I then use R's ggplot to render (or vise versa maybe?)
as for the deployment story today, I'll likely learn most by doing (with agents).
My experiment will be to kick off an agent to do something like: extract this dataset to S3 using dlt [2], model it using dbt [3], then use ggsql to visualize.
p.s. @thomasp85, I was a big fan of tidygraph back in the day [4]. love how small our data world is.
ggsql is modular by design. It consists of various reader modules that takes care of connecting with different data backends (currently we have a DuckDB, an SQLite, and an ODBC reader), a central plot module, and various writer modules that take care of the rendering (currently only Vegalite but I plan to write my own renderer from scratch).
As for deployment I can only talk about a utopian future since this alpha-release doesn't provide much tangible in that area. The ggsql Jupyter kernel already allows you to execute ggsql queries in Jupyter and Quarto notebooks, so deployment of reports should kinda work already, though we are still looking at making it as easy as possible to move database credentials along with the deployment. I also envision deployment of single .ggsql files that result in embeddable visualisations you can reference on websites etc. Our focus in this area will be Posit Connect in the short term
I'm afraid I don't know what IR stands for - can you elaborate?
Ah - yes, in theory you could create a "ggplot2 writer" which renders the plot object to an R file you can execute. It is not too far away from the current Vega-Lite writer we use. The other direction (ggplot2->ggsql) is not really feasible
I've been mulling extending dbt Charts to support ggplot, and will likely reach out to the team about this in the next few weeks.
The difference is that Vega-Lite, ggsql, ggplot2 are designed to create a single chart. Yes, you can use facet_wrap() to great a series of charts based on a particular variable, but arguably that's still one chart!
What's missing that is table stakes for a BI dashboard: KPIs, text boxes, dropdowns, data sources, governance, and how the charts are arranged on the page and in relation to one another.
In the next few months we hope to ship a JSON schema that represents a dashboard in the same way that a VegaLite does for a single chart.
Does that make sense?
reply