I like all the extra scanning features this provides :)
I've seen a few tools like this that try to keep things in sync. While it is better than the alternative of doing it manually, it is still a losing battle and not the greatest DX. Often you still have validation (if it exists) and types that also need to be kept in sync, and documentation is scattered throughout the codebase.
Another approach is to turn the example into a schema - and have it be included in the env loading process. This way non-sensitive values can be included directly, and it can never be out of sync. Only values that differ need to be defined in your git-ignored files or injected by the environment. This is precisely how https://varlock.dev solves this problem. The schema becomes the single source of truth, and is used to generate types as well. It also provides validation, leak prevention, and a bunch of other nice tools.
I am currently working on a turbo mono repo frontend on my work with maybe 20-25 different env variables. Here the dotenv-diff is really a game changer, but yea for smaller projects, i might be a bit overkill.
I like the projects you have linked, i will try to see if they have any need to have features that i could use
in `dotenv-diff` you also have the --compare option which will compare your .env whit your .env.example to keeping them in sync, while also have the amount of scanning features that keeps the project safe.
One really nice thing in varlock for monorepos is the import syntax. This lets you have shared config at the root, or just to break things up however you need. No need for diffing or copy pasting, as the schema validates everything - if something is required, it will yell at you.
In a team setting, it can be extremely helpful to have env/config loading logic built into the repo itself. It does not mean it has be loaded by the application process, but it can be part of the surrounding tooling that is part of your codebase.
Yes, that's indeed the right place, IMO: ephemeral tooling that leverages, or simplifies OS features.
Tooling such as xenv, a tiny bash script, a makefile etc. that devs can then replace with their own if they wish (A windows user may need something different from my zsh built-in). That isn't present at all in prod, or when running in k8s or docker compose locally.
A few years ago, I surfaced a security bug in an integrated .env loader that partly leveraged a lib and partly was DIY/NIH code. A dev built something that would traverse up and down file hierarchies to search for .env.* files and merge them runtime and reload the app if it found a new or changed one. Useful for dev. But on prod, uploading a .env.png would end up in in a temp dir that this homebuilt monstrosity would then pick up. Yes, any internet user could inject most configuration into our production app.
Because a developer built a solution to a problem that was long solved, if only he had researched the problem a bit longer.
We "fixed" it by ripping out thousands of LOCs, a dependency (with dependencies) and putting one line back in the READMe: use an env loader like ....
Turned out that not only was it a security issue, it was an inotify hogger, memory hog, and io bottleneck on boot. We could downsize some production infra afterwards.
Yes, the dev built bad software. But, again, the problem wasn't that quality, but the fact it was considered to be built in the first place.
For a more modern approach to .env files that includes built-in validation and type-safety, check out https://varlock.dev
Instead of a .env.example (which quickly gets out of date), it uses a .env.schema - which contains extra metadata as decorator comments. It also introduces a new function call syntax, to securely load values from external sources.
This is interesting. Amazing how something so fundamental is still such a pain, and we all build our own half-baked solutions for it on every new project. We've been thinking about this problem for a while now as well, and just launched another tool (https://varlock.dev) that might be interesting for you to check out. Would be very happy to collaborate or just talk about the problem space.
Our tool has similar goals, although a slightly different approach. Varlock uses decorator style comments within a .env file (usually a committed .env.schema file) to add additional metadata used for validation, type generation, docs etc. It also introduces a new "function call" syntax for values - which can hold declarative instructions about how to fetch values, and/or can hold encrypted data. We call this new DSL "env-spec" -- similar name :)
Certainly some trade-offs, but we felt meeting people where they already are (.env files) is worthwhile, and will hopefully mean the tool is applicable in more cases. Our system is also explicitly designed to handle all config, rather than just secrets, as we feel a unified system is best. Our plugin system is still in development, but we will allow you to pull specific items from different backends, or apply a set of values, like what you have done. We also have some deeper integrations with end-user code, that provide additional security features - like log redaction and leak prevention.
On most projects you end up wiring up a bunch of custom logic to handle your config, which is often injected as environment variables - think loading from a secure source, validation logic, type safety, pre-commit git scanning, etc.
It's annoying to do it right, so people often take shortcuts - skip adding validation, send files over slack, don't add docs, etc...
The common pattern of using a .env.example file leads to constant syncing problems, and we often have many sources of truth about our config (.env.example, hand-written types, validation code, comments scattered throughout the codebase)
This tool lets you express additional schema info about the config your application needs via decorators in a .env file, and optionally set values, either directly if they are not sensitive, or via calls to an external service. This shouldn't be something we need to recreate when scaffolding out every new project. There should be a single source of truth - and it should work with any framework/language.
Exactly. We will do that to stdout - and can patch JS itself too.
The goal here is to just make it dead simple to do the right thing with minimal effort. Get secrets out of plaintext, avoid the need to send them around insecurely, and help make sure you don't shoot yourself in the foot, which is surprisingly easy to do in hybrid server/client frameworks like Next.js.
Can you set up validations, syncing with various backends, and these protections all of this yourself by wiring together a bunch of tools with custom code? Of course... But here's one that will do it all with minimal effort.
This is exactly the kind of thing we are trying to replace. No more hacky custom solutions needed. With varlock you get type safety, validation, and the ability to compose together your config however you need to, without having to reinvent the wheel.
Personally I don't want to write custom code from scratch or glue together 5 tools to do this on every project. I've done it - many many many times... If your project is very simple with a few env vars, it may not be a big deal, but on large projects it quickly becomes a burden. Maybe not one you deal with every day, but regularly - when new teammates join, when new external services are added, when new deployment environments are being used.
Also a big goal here is to have a single tool that will work for all frameworks _and languages_. In monorepo projects, it's very common to have a different custom env var loading and validation setup in each child project, and sharing config across the project is very awkward.
Doing this stuff well is hard, which often means people take shortcuts. So many times dev teams gets stopped in their tracks because someone added a new env var and forgot to distribute it to everyone, or they only added the new key to staging and forgot production. We just want to make doing the right thing the easiest thing.
Imagine if this was just a standard, and when you look at a public docker container, there is a schema of the env vars it takes, rather than digging through the readme. When it boots up, if the env vars are invalid it will give you a clear error message. Imagine when you clone some template project, you get this all this validation and documentation for free, without having to trace through some custom code.
Yes - it is something new, but our hope is that it is intuitive enough to grasp even if it's new to you.
We're still working on the copy. Any specific feedback is appreciated.
Note that you can load env vars into anything via varlock run, not just javascript. The JS integration is a bit deeper, providing automatic type generation, log redaction, and leak prevention.
I've seen a few tools like this that try to keep things in sync. While it is better than the alternative of doing it manually, it is still a losing battle and not the greatest DX. Often you still have validation (if it exists) and types that also need to be kept in sync, and documentation is scattered throughout the codebase.
Another approach is to turn the example into a schema - and have it be included in the env loading process. This way non-sensitive values can be included directly, and it can never be out of sync. Only values that differ need to be defined in your git-ignored files or injected by the environment. This is precisely how https://varlock.dev solves this problem. The schema becomes the single source of truth, and is used to generate types as well. It also provides validation, leak prevention, and a bunch of other nice tools.
--
Other env-syncing libraries - https://github.com/luqmanoop/sync-dotenv - https://www.npmjs.com/package/dotenv-safe - https://www.npmjs.com/package/generate-env-example
reply