Lesson 2: Blogging with nbdev

fractal

python

nbdev

Author

Chris Kroenke

Published

October 5, 2023

Making a personal blog with nbdev.

Intro

The main goal of this lesson is to set up a personal blog for this course. The blog is where you will chronicle your learnings and challenges throughout the course. From the course syllabus:

This is a public-facing course. Students should write about their journey to both cement and track their progress.

Publishing is a powerful tool for learning since it casts a piece of work under a new, critical light. Your understanding of a topic is also crystallized by seeing other’s work, highlighted from different angles.

Blogging out in the open can be scary, but here we aim to make it as easy and helpful as possible.

On the topic of blogging and publishing being scary or intimidating: try to focus on writing snippets or summaries about what you’re currently learning. Then, pretend that you’re writing the article you wish you had when first learning about the topic. The first articles are the hardest, to be sure. Once you’ve gotten into the flow with the hints above, other longer articles will flow more easily.

The `nbdev` framework

We will setup our blog using nbdev. It is a great tool for creating powerful applications. nbdev’s core idea is that a project’s code, documentation, and tests should exist together as a unified, interactive document.

nbdev is inspired by two paradigms in the coding world:

Feel free to take some time now and click on the links above to learn more about these paradigms.

Now, let’s go over these two ideas and explore why their combination is so powerful.

Literate Programming

In Literate Programming, comments and tests are woven directly into a project’s source code. The original idea was proposed by Donald Knuth in 1984. Having all of pieces together a single, interactive document makes for better, more readable code and a more cohesive project overall.

The code, documentation, and tests are all first-class citizens in Literate Programming. This is very different from the typical approach to software where tests and documentation live in a separate set of files.

In nbdev, a Jupyter Notebook is the single source of truth for code, documentation, and tests. Instead of managing three independent groups of files, everything is defined and happens in the Notebook. If the Notebook runs, then you know your code will run.

Exploratory Programming

Exploratory Programming is an open-ended approach for tackling new problems and unknown domains. It’s very useful at the start of a project, while the scope and requirements are still being fleshed out. And it can highlight different angles on a known, established approach by casting it under a new light.

Notebooks are both dynamic and interactive which makes them perfect for Exploratory Programming. The barrier for trying new things in a notebook is extremely low. And they’re downright fun!

The video below is an incredible talk by Bret Victor about the power of Exploratory Programming, with stunning examples throughout to drive the point home. If these short snippets weren’t enough to sell you on the idea, the talk certainly will.

Combining Literate and Exploratory Programming

nbdev combines these two ideas. We covered why this is a powerful combination in the Course Philosophy section of the syllabus. Please take a moment to re-read with the new context from above.

With this combo, we can interactively learn about and explore how a piece of code works. We can easily follow any hint of curiosity since iterations are fast and cheap. And we can now develop code in a much more dynamic way than usual.

These ideas can be doled out on the fly. For example, at the start of a project we could lean Exploratory to map out the problem space. Then, as the idea matures, we can pivot to Literate to refine our approach.

And the best part is that if anything breaks, we can always restart the Notebook and try again. It’s like clearing out the whiteboard and taking another pass at the problem.

We’ve talked a big game about what nbdev can do for writing great software projects. Now, to get back on track, let’s look at how nbdev can turn Notebooks into our future blog posts.

Turning Notebooks into Blog Posts

First, a high-level look at the process we will follow to setup our blog:

High-Level Steps:

Create a new nbdev project.
Make a blog inside the project.
Publish and host the blog for free on Github Pages.

nbdev relies on the Quarto framework to publish blogs. Quarto is a framework for writing beautiful technical and scientific articles. In a way, it’s a blogging platform for Literate Programming, where a series of code and comments lead the reader on a journey.

The picture below from the official Quarto site gives shows what the framework is all about.

Creating a new `nbdev` project

nbdev works on top of a Git repo. That means our first step is creating a new, empty repository. Here is a handy Github link that takes us straight to the page for creating new repos.

Note: We need a completely empty repo. Don’t include a .gitignore or README.md.

In this example the empty repo is called sample_blog, but feel free to call it anything you’d like. We’re not tied to this name. We can always make new repos with different, better names.

Clone the new, empty repo to your computer. Make sure to change the github link below so it points to your repo instead.

# clone the repo to your computer
git clone https://github.com/enzokro/sample_blog.git # <-- ! link with your repo here

Now we can move into this repo and let nbdev work its initialization magic. Run the nbdev_new command to get started. The command will prompt you for some general info, like a short description about the project.

# move into the new repo and initialize the nbdev project
cd sample_blog/
nbdev_new

Note: All of the options and configs for the project are in the settings.ini file. nbdev looks in this file when it needs any information for its commands.

After nbdev_new finishes, we will have a brand new nbdev project!

Run a git status command to see everything that was added. Then we commit and push these changes to Github.

# add, commit, and push the files created by nbdev
git add .
git commit -m'Initial nbdev project creation'
git push

As mentioned earlier, nbdev publishes notebooks using Quarto. Let’s now take a look at how to turn this project into a full Quarto blog.

Adding Quarto to the Mix

Start by activating the virtual environment from the previous Environment Setup Notebook:

# activate the environment
mamba activate llm_base

nbdev comes with a built-in command to install quarto: nbdev_install_quarto. Note that the command will ask for admin privileges.

# install quarto 
nbdev_install_quarto

You may need to refresh the terminal before it can see the newly installed quarto binary. To be safe, open up a new terminal and re-activate the environment. Then the command below will check if Quarto was installed successfully.

# shows us where quarto was installed
which quarto

Now that Quarto is installed, let’s turn our fresh nbdev project into an actual blog.

Turn an `nbdev` project into a Quarto blog

In an nbdev project the Notebooks usually live inside the nbs/ folder. These Notebooks eventually become a project’s code source, tests, and documentation.

For Quarto to instead publish Notebooks as blog posts, we need to add a few files and folders to the nbs/ folder.

Below is the structure for a nbs/ folder that’s been primed for its blog post.

Initial Structure for Quarto Blog:

sample_blog
└───nbs/
│   │   _quarto.yml
│   │   index.ipynb
│   └───blog/
│       │   index.qmd
│       └───posts/
│           └───2023-09-27-Blog-Intro/     
│               │   index.ipynb

The main change is adding a blog/ folder inside of the nbs/ directory. Inside, at its top level, we place an index.qmd file that tells Quarto about our blog. Here’s example index.qmd file that titles our blog and describes how posts will be listed:

---
title: Example Blog
subtitle: Publishing with Quarto and nbdev
listing:
  sort: "date desc"
  contents: "posts"
  sort-ui: false
  filter-ui: false
  categories: true
  feed: true
page-layout: full
---

To stay better organized, each post will get its in folder. An index.ipynb Notebook inside each folder will have the post’s actual content. We can also put photos, videos, or any other assets to enhance a post into this folder.

As a working example, we are going a bit meta and using this Notebook itself as index.ipynb in the folder called 2023-09-27-Blog-Intro/.

In other words this Notebook will also be our first blog post.

Hosting the Blog on Github Pages

The site will be hosted on Github’s Pages platform. This is a built-in feature offered by Github and we can leverage it by changing a few settings. The screenshot below shows the to configure the repo so it can be published as a blog.

Specifically, we need to set the Deploy from a branch option under the Build and deployment section. Then, we pick the gh-pages branch as the branch to deploy from. gh-pages is a special branch where nbdev parses our Notebooks into a proper Quarto website.

Next, run the quarto publish command to build and publish the blog.

# publish the blog on github pages
quarto publish

And there we go! We created and published our first blog post using nbdev. Click here for a live link to this Notebook as a blog post.

Conclusion

This Notebook covered how to create and publish a blog. We used nbdev to build the blog and Quarto to publish it. We also showed how the mix of Literate and Exploratory Programming is a powerful approach to both research and thinking in general.