Python with a Jupyter notebook is a very solid REPL development environment:

- Prototype code in Jupyter. Write first as a few lines of code per cell. Then merge into functions. Focus is on 5-10 cells becoming a single function.

- When working, copy the functions back into the main Python file.

- Periodically reimport the main Python file back into Jupyter and recalc the cells currently in scope.

- Repeat...

- The main Python file ends up mostly like an API. And the Jupyter notebook as docs on how to call that API.

i agree that python's repl environment is nice. i worked with python and jupyter for much longer than with common lisp. actually due to my domain it is still my main language. however i was taken aback when i first saw repl development in lisp. it is just far more seemless and stable. how often do you need to do python kernel restarts in jupyter? what i found extremely surprising is the huge performance difference between sbcl and cpython. common lisp as implemented in sbcl produces some of the most performant computations out there. also writing code in s-expressions turned out to be an unexpected killer feature for me. finally something you cannot do in python that is built into lisp is being able to debug and live edit a running (production) image.

while i continue to use python the same is no longer true for jupyter. i think using org-babel in emacs is a much more superior experience, with the added benefit of having the whole development environment available to you

How does 'REPL-driven developent' work, exactly? Are you writing code in files and then testing it in the REPL? Or writing substantive code in the REPL and then copying it to files somehow?

It's how I've worked day-to-day for thirty years or so, so here's my take on it:

I generally work in Emacs. I'll have one or more source files open, plus at least one REPL buffer. The REPL is my communication channel with the running Lisp.

The object of the game is to teach the Lisp interactively how to be the program I want. It's already a working program; it just isn't the one I want to build. I'll use Lisp expressions to teach it, one expression at a time, how to be what I want.

The contents of the source files depend on the stage I'm at with a program. If I'm just starting then it's probably one file with a few snippets that I C-x C-e to send to the REPL to modify the dynamic environment of the running Lisp.

If it's a more mature project, then I have a bunch of source files that fit into an ASDF system (which defines sources to build and dependencies among them and any libraries they depend on). The system as a whole will be loaded into the memory of the running Lisp, converting it into a work-in-progress version of my app.

I'll have some subset of the project files open in buffers so that I can add or edit definitions and C-x C-e them into the running Lisp and see in the REPL whether my changes have the effect that I intend.

I continue this way indefinitely, teaching the running Lisp expression-by-expression the features that I intend for the finished product to have. When the difference between my idea and what the Lisp does shrinks to epsilon, the program is implemented. I run a build function and I have a delivered application.

When something goes wrong during my work, I land in a breakloop, which is something like a backtrace, except live. Rather than a printout of an unwound stack, it's a live REPL on the dynamic environment of the stack at the moment the error occurred. I can use the breakloop to wander up and down the stack, inspect and edit variable, type, and function definitions, and resume execution at the moment of the error when I'm ready to see the effect of my changes. The Lisp then runs the same function from where it broke, except that the variables, types, and functions I modified now have the definitions I just gave them, rather than the ones they originally had.

If I change the definition of some class that already has live instances, then the Lisp reinitializes those instances with the new class definition and continues. If I neglected to show it how to reinitialize those classes, it drops me into another breakloop and offers me the opportunity to tell it how to do so, after which it will continue, as before.

If I stumble across something curious and I want to show it to a collaborator in context, I can save a heap image of the running Lisp and give it to my colleague. That colleague can start the image on their machine and see the same dynamic environment that I saw. In some older Lisps, that dynamic state could include all of the windows on my screen, and their contents, and which one was the active window, and where the text-insertion point was.

I can deploy my work-in-progress apps to a staging machine with a repl server built in. I can let it run in a testing environment indefinitely, and use Emacs to connect to a live REPL that I can use to rummage around in the running app, inspect and edit variables, types, and functions, and generally do anything I would do if it were running in my normal dev environment. If I see something odd, I can again dump a heap image, copy it back to my development machine, and crawl around in its running state while I let the deployed test version go back to doing what it was doing.

There's quite a bit more to it, but with luck, that gives a general idea of what the workflow is like. Common Lisp environments are generally like this; some have richer sets of affordances than others. Smalltalks are like that, too, and so is Factor.

Most other Languages and repls are not so much like that. Clojurescript with figwheel gives you part of it, and the part it implements is pretty good.

> I can deploy my work-in-progress apps to a staging machine with a repl server built in. I can let it run in a testing environment indefinitely, and use Emacs to connect to a live REPL that I can use to rummage around in the running app, inspect and edit variables, types, and functions, and generally do anything I would do if it were running in my normal dev environment.

Not the GP. Thanks for the comment, I always find your comments about your experiences with Lisp insightful... But I'm curious, does/can this lead to situations where colleagues are stepping on each others toes in the live environments?

Yes, sure. Don't do that. :)

If you're going to test that way, you want to have your own staging environment, distinct from other developers', just like you don't want two different people's unit tests writing to the same test data as they run their tests (unless that kind of race condition is exactly what you're testing, of course).

Very interesting, thanks! This is indeed how I tinker with my .emacs file, though I’ve only done relatively trivial stuff there.

What do you prefer about using C-x C-e on individual sexps rather than reloading everything and keeping the entire program synced between source files and the running environment?

Generally speaking, I work by building some data that represent my understanding of the model I'm working with, and some functions to operate on that model. My models start out simple and more or less wrong, and as I work, I build improved understanding. Mostly my understanding improves incrementally, and so does my code. I build up the app piece by piece, adding and changing things bit by bit. So most of the time, changing a small piece is all I need, and that's what evaluating one expression does.

Moreover, if I recompile everything, then I'm rebuilding my in-memory state from scratch. Mostly, that's not what I want. Mostly I work like a sculptor: I make a rough approximation and then refine it stepwise. I don't want to rebuild from scratch every time I discover something new. I want to keep most of what I've built and change just what needs to be changed.

That said, sometimes a discovery does call for a larger-scale reconstruction. That's when I reload a whole file or, in more extreme cases, reload an entire ASDF system of files. Once in a while, I change enough (or make enough mistakes) that it makes sense to blow the whole thing away--kill the Lisp, restart it, and load the system from scratch. But that's fairly rare.

Mostly I want to plop down my clay and tweak it a little at a time toward my goal, discovering and refining it a little at a time.

From what I've seen, the cycle is a bit like this:

- launch the program you want to run and have it running in the background

- text editor open

- bits of code in editor loaded in the running program through text selection and sending them to the be loaded (the most common scenario is doing this in Emacs)

However my concern with this is... if you don't run the entire "code frozen" program at once, how can you guarantee the internal consistency of the program if over time you kind of haphazardly add bits of code to it? Maybe you loaded that function, maybe you didn't, did you add that dependency function in the correct order and at the correct "version" (where version is used loosely, every time you type in something, it's a new "version").

I'd really love for someone experienced with Lisp to describe their workflow and also maybe clarify how they handle this (in my opinion, huge) problem of possible program inconsistent states.

For Clojure there are libraries to refresh the program: these stop the app, reload and evaluate all the source code files, and restart the app to ensure you are working with a consistent state that reflects the source code.

Between such reloads you might end up in inconsistent states, but you benefit from fast iteration:

My workflow is to update functions in the source code, send to to the repl, quickly test them in the repl, and once done, I save, commit and push the code. CI then runs the test suite from the source files.

I guess it just doesn't really come up that much? I personally aim to have whatever I'm not currently working on written to disk, and very rarely redefine things in the REPL once I've written them to source. Other than that every time you change anything it's automatically propagated, with a few known exceptions (macros mainly). I can say I've never wound up in an unknown state, and I'll keep a slime server running for days.

Well, not the same thing but Java can hot-reload classes (to a degree depending on implementation). But what truly makes repl-driven development in lisps so good is the more functional approach to development. Hot reload gets better the smaller the replaceable units get. Lisps have it good with basically paren-level hot reload.