Tips for using Jupyter notebooks as lab books

Tip 1

Avoid rerunning and rearranging cells.

In order to function as a lab book, we want the notebook to record everything we do, not just the last thing we did! Let the notebook serve as a step by step recording of your work.

Tip 2

Run your bash commands in a notebook.

This is the whole point of using jupyter notebook as a lab book -- to record your work. Run things here, print scripts to the notebook, launch slurm jobs from here. Don't forget to put 'module purge' in your slurm scripts since jupyter notebook is run in a conda environment.

Tip 3

Make copious notes

Use markdown and/or raw cells to make notes about your actions.

Tip 4

Use the %logstart magic to log your notebooks with timestamps.

This will record all of your actions, regardless of how well you follow rule 1. It will also record your last actions if you close your notebook if you accidentally close your notebook without saving it. Remember doing something that you can't find in your notebook? Search your log.

Tip 5

Record stored variables & startup files.

If you use %store, run it at the top of each notebook so that the values of any variables are recorded in the current notebook. If you use startup scripts, print them at the start of each notebook in order to capture their current state.

Tip 6

Write protect your notebooks and log files at the end of the day.

This is the equivalent to writing in your notebook in pen. You don't want to accidentally edit your log files or notebooks. You WILL be opening old notebooks side by side with new ones, and you don't want to change the old notebooks by accident. chmod -w mynotebook.ipynb.

Tip 7

Organize and back up your notebooks

Use a good naming convention, organize them into folders, make an index. Back them up using github, uocloud, dropbox, etc.

Tip 8

Search your notebooks with nbscan.py

The tool nbscan is extremely useful for searching through your notebooks once you have a lot of them. Install it with pip if you didn't include it in your environment file:
pip install --user git+https://github.com/conery/nbscan.git
Run with:
nbscan.py --code --grep 'fmriprep' *.ipynb