Setup

You normally need IPython installed into your current virtualenv for it to work properly:

pip install ipython

(See Tips section below if installing IPython is not possible)

Methods

There are basically two ways I open an IPython prompt. The first is by running it directly from a terminal:

$ ipython
Python 3.9.5 (default, Jul  1 2021, 11:45:58)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

In a Django project, ./manage.py shell can also be used if you have IPython installed, with the advantage that it will properly initialise Django for you.

This works fine if you want to explore writing some “top level” code – for example, a new bit of functionality where the entry points have not been created yet. However, most code I write is not like that. Most of the time I find myself wanting to write code when I am already 10 levels of function calls down – for example:

• I’m writing some view code in a Django application, which has a request object – an object you could not easily recreate if you started from scratch at an IPython prompt.

• or, model layer code such as inside a save() method that is itself being called by some other code you have not written, like the Django admin or some signal.

• or, inside a test, where the setup code has already created a whole bunch of things that are not available to you when you open IPython.

For these cases, I use the second method:

• Find the bit of code I want to modify, explore or debug. This will often be my own code, but could equally be a third party library. I’m always working in a virtualenv, so even with third party libraries ,“go to definition” in my editor will take me straight to a writable copy of the code (apart from code not written in Python).

• Insert the code for an IPython prompt and save the file:

  import IPython; IPython.embed()
  

  I have this bound to a function key in my editor.

  So the code might end up looking like this, if it was a Django view for example:

  def contact_us(request):
      if request.method == "POST":
          form = ContactUsForm(request.POST)
          if form.is_valid():
              import IPython; IPython.embed()
  
          # …
  

  I sometimes also might put the snippet inside a new if clause that I add to catch a particular condition, especially when using this for debugging.

• Trigger the code in the appropriate way. For the above case, it would involve first running the Django development server in a terminal, then opening the web page, filling out the form and pressing submit. For a test, it would be running the specific test from a terminal. For command line apps it would be running the app directly.

• In the terminal, I will now find myself in the IPython REPL, and I can go ahead and:

  • work out what code I need to write

  • or debug the code that I’m confused about.

Note that you can write and edit multi-line code at this REPL – it’s not quite as comfortable as an editor, but it’s OK, and has good history support. There’s much more to say about IPython and its features that I won’t write here, you can learn about it in the docs.

For those with a background in other languages, it might also be worth pointing out that a Python REPL is not a different thing from normal Python. Everything you can do in normal Python, like defining functions and classes, is possible right there in the REPL.

Once I’m done with my exploring, I can copy any useful snippets back from the REPL into my real code, using the history to scan back through what I typed.

Advantages

The advantages of this method are:

1. You can explore APIs and objects much more easily when you actually have the object, rather than docs about the object, or what your editor’s auto-complete tools believe to be true about the object. For example, what attributes and methods are available on Django’s HttpRequest? You don’t have to ensure you’ve got correct type annotations, and hope they are complete, or make assumptions about what the values are - you’ve got the object right there, you can inspect it, with extensive and correct tab completion. You can actually call functions and see what they do.

   For example, Django’s request object typically has a user attribute which is not part of the HttpRequest definition, because of how it is added later. It’s visible in a REPL though.

2. You can directly explore the state of the system. This can be a huge advantage for both exploratory programming and debugging.

   For debugging, pdb and similar debugging tools and environments will often provide you with “the state of the system”, and they are much better at being able to step through multiple layers of code. But I often find that the power and comfort of an IPython prompt is much nicer for exploring and finding solutions.

The feel of this kind of environment is not quite a smooth as REPL-driven programming in Lisp, but I still find it hugely enjoyable and productive. Compared to many other methods, like iterating on your code followed by manual or automated testing, it cuts the latency of the feedback loop from seconds or minutes to milliseconds, and that is huge.

Tips and gotchas

• IPython has tons of cool features that will help you in a REPL environment, like %autoreload (thanks haki), and many other cool magics. You should spend the time getting to know them!

• In a multi-threaded (or multi-process) environment, IPython prompts won’t play nice. Turn off multi-threading if possible, or otherwise ensure that you don’t hit that gotcha.

• If you do get messed up in a terminal, you may need to manually find the processes to kill and do reset in your terminal.

• With the Django development server:

  • It’s multi-threaded by default, so either ensure that you don’t hit the view code multiple times, or use --nothreading.

  • Beware of auto-reloading, which will mess you up if you are still in an IPython prompt when it kicks in. Either use --noreload or just ensure you exit IPython cleanly before doing anything that will trigger a reload.

• Beware of environments that capture standard input/output, that will break this technique.

• pytest captures standard input and breaks things by default. You can turn it off using -s. Also if you are using pytest-xdist you should remember to do -n0 to turn off multiple processes.

• When using IPython.embed() there’s an annoying bug involving closures and undefined names due to Python limitations. It often shows itself when using generator expressions, but at other times too. It can often be worked around by doing:

  globals().update(locals())
  

• If for some reason you can’t use IPython, but only have access to the standard library, the one-liner you need to run a (basic) REPL at any point in your code is this:

  import code; code.interact(local=locals())
  

End

That’s it, I hope you found it useful. Do you have any other tips for using this technique?

Links

• Discussion on Twitter