Tutorial: Inheritance & Plugins

Adding to, or altering the behavior of, commands from upstream Django apps is a common use case. Doing so allows you to keep your CLI stable while adding additional behaviors by installing new apps in INSTALLED_APPS. There are three main extension patterns you may wish to employ:

1. Override the behavior of a command in an upstream app.

2. Add additional subcommands or groups to a command in an upstream app.

3. Hook implementations of custom logic into upstream command extension points. (Inversion of Control)

The django-typer plugin mechanism supports all three of these use cases in a way that respects the precedence order of apps in the INSTALLED_APPS setting. In this tutorial we walk through an example of each using a generic backup command. First we’ll see how we might use inheritance (1) to override and change the behavior of a subcommand. Then we’ll see how we can add subcommands (2) to an upstream command using plugins. Finally we’ll use pluggy to implement a hook system that allows us to add custom logic (3) to an upstream command.

A Generic Backup Command

Consider the task of backing up a Django website. State is stored in the database, in media files on disk, potentially in other files, and also in the software stack running on the server. If we want to provide a general backup command that might be used downstream we cannot know all potential state that might need backing up. We can use django-typer to define a command that can be extended with additional backup logic. On our base command we’ll only provide a database backup routine, but we anticipate our command being extended so we may also provide default behavior that will discover and run every backup routine defined on the command if no specific subroutine is invoked. We can use the context to determine if a subcommand was called in our root initializer callback and we can find subroutines added by plugins at runtime using get_subcommand(). Our command might look like this:

Django-styleTyper-style

backup/management/commands/backup.py

import inspect
import os
import typing as t
from pathlib import Path

import typer
from django.conf import settings
from django.core.management import CommandError, call_command

from django_typer.management import (
    CommandNode,
    TyperCommand,
    command,
    initialize,
)
from django_typer import completers


class Command(TyperCommand):
    """
    Backup the website! This command groups backup routines together.
    Each routine may be run individually, but if no routine is specified,
    the default run of all routines will be executed.
    """

    suppressed_base_arguments = {"verbosity", "skip_checks"}

    requires_migrations_checks = False
    requires_system_checks = []

    databases = [alias for alias in settings.DATABASES.keys()]

    output_directory: Path

    @initialize(invoke_without_command=True)
    def init_or_run_all(
        self,
        # if we add a context argument Typer will provide it
        # the context is a click object that contains additional
        # information about the broader CLI invocation
        context: typer.Context,
        output_directory: t.Annotated[
            Path,
            typer.Option(
                "-o",
                "--output",
                shell_complete=completers.complete_directory,
                help="The directory to write backup artifacts to.",
            ),
        ] = Path(os.getcwd()),
    ):
        self.output_directory = output_directory

        if not self.output_directory.exists():
            self.output_directory.mkdir(parents=True)

        if not self.output_directory.is_dir():
            raise CommandError(f"{self.output_directory} is not a directory.")

        # here we use the context to determine if a subcommand was invoked and
        # if it was not we run all the backup routines
        if not context.invoked_subcommand:
            for cmd in self.get_backup_routines():
                cmd()

    def get_backup_routines(self) -> t.List[CommandNode]:
        """
        Return the list of backup subcommands. This is every registered command
        except for the list command.
        """
        # fetch all the command names at the top level of our command tree,
        # except for list, which we know to not be a backup routine
        return [
            cmd
            for name, cmd in self.get_subcommand().children.items()
            if name != "list"
        ]

    @command()
    def list(self):
        """
        List the default backup routines in the order they will be run.
        """
        self.echo("Default backup routines:")
        for cmd in self.get_backup_routines():
            sig = {
                name: param.default
                for name, param in inspect.signature(
                    cmd.callback
                ).parameters.items()
                if not name == "self"
            }
            params = ", ".join([f"{k}={v}" for k, v in sig.items()])
            self.secho(f"  {cmd.name}({params})", fg="green")

    @command()
    def database(
        self,
        filename: t.Annotated[
            str,
            typer.Option(
                "-f",
                "--filename",
                help=(
                    "The name of the file to use for the backup fixture. The "
                    "filename may optionally contain a {database} formatting "
                    "placeholder."
                ),
            ),
        ] = "{database}.json",
        databases: t.Annotated[
            t.List[str],
            typer.Option(
                "-d",
                "--database",
                help=(
                    "The name of the database(s) to backup. If not provided, "
                    "all databases will be backed up."
                ),
                shell_complete=completers.databases,
            ),
        ] = databases,
    ):
        """
        Backup database(s) to a json fixture file.
        """
        for db in databases or self.databases:
            output = self.output_directory / filename.format(database=db)
            self.echo(f"Backing up database [{db}] to: {output}")
            call_command(
                "dumpdata",
                output=output,
                database=db,
                format="json",
            )

$> python manage.py backup list
Default backup routines:
    database(filename={database}.json, databases=['default'])

Inheritance

The first option we have is simple inheritance. Lets say the base command is defined in an app called backup. Now say we have another app that uses media files. This means we’ll want to add a media backup routine to the backup command.

Note

Inheritance also works for commands defined using the Typer-style function based interface. Import the root Typer app from the upstream command module and pass it as an argument to Typer when you create the root app in your overriding command module.

Say our app tree looks like this:

./
├── backup/
│   ├── __init__.py
│   ├── apps.py
│   ├── management/
│   │   ├── __init__.py
│   │   └── commands/
│   │       ├── __init__.py
│   │       └── backup.py
└── media/
    ├── __init__.py
    ├── apps.py
    └── management/
        ├── __init__.py
        └── commands/
            ├── __init__.py
            └── backup.py

INSTALLED_APPS = [
    'media',
    'backup',
    ...
]

Our backup.py implementation in the media app might look like this:

Django-styleTyper-style

media/management/commands/backup.py

import tarfile
import typing as t
from pathlib import Path

import typer
from django.conf import settings

from django_typer.management import command
from backup.management.commands.backup import (
    Command as Backup,
)


class Command(Backup):  # inherit from the original command
    # add a new command called media that archives the MEDIA_ROOT dir
    @command()
    def media(
        self,
        filename: t.Annotated[
            str,
            typer.Option(
                "-f",
                "--filename",
                help=("The name of the file to use for the media backup tar."),
            ),
        ] = "media.tar.gz",
    ):
        """
        Backup the media files (i.e. those files in MEDIA_ROOT).
        """
        media_root = Path(settings.MEDIA_ROOT)
        output_file = self.output_directory / filename

        # backup the media directory into the output file as a gzipped tar
        typer.echo(f"Backing up {media_root} to {output_file}")
        with tarfile.open(output_file, "w:gz") as tar:
            tar.add(media_root, arcname=media_root.name)

Now you’ll see we have another command called media available:

Now we have a media backup routine that we can run individually or part of the entire backup batch:

$> python manage.py backup list
Default backup routines:
    database(filename={database}.json, databases=['default'])
    media(filename=media.tar.gz)
# backup media only
$> python manage.py backup media
Backing up ./media to ./media.tar.gz
# or backup database and media
$> python manage.py backup
Backing up database [default] to: ./default.json
[.............................................]
Backing up ./media to ./media.tar.gz

When Does Inheritance Make Sense?

Inheritance is a good choice when you want to tweak the behavior of a specific command and do not expect other apps to also modify the same command. It’s also a good choice when you want to offer a different flavor of a command under a different name.

What if other apps want to alter the same command and we don’t know about them, but they may end up installed along with our app? This is where the plugin pattern will serve us better.

CLI Plugins

The plugin pattern allows us to add or override commands and groups on an upstream command directly without overriding it or changing its name. This allows downstream apps that know nothing about each other to add their own behavior to the same command. If there are conflicts they are resolved in INSTALLED_APPS order.

To do this we have to abandon the class based interface and place our plugins in a module other than commands. Let us suppose we are developing a site that uses the backup and media app from upstream and we’ve implemented most of our custom site functionality in a new app called my_app. Because we’re now mostly working at the level of our particular site we may want to add more custom backup logic. For instance, lets say we know our site will always run on sqlite and we prefer to just copy the file to backup our database. It is also useful for us to capture the python stack (e.g. requirements.txt) running on our server. To do that we can use the plugin pattern to add our environment backup routine and override the database routine from the upstream backup app. Our app tree now might look like this:

./
├── backup/
│   ├── __init__.py
│   ├── apps.py
│   ├── management/
│   │   ├── __init__.py
│   │   └── commands/
│   │       ├── __init__.py
│   │       └── backup.py
├── media/
│   ├── __init__.py
│   ├── apps.py
│   └── management/
│       ├── __init__.py
│       ├── commands/
│       └── plugins/
│           └── __init__.py
│           └── backup.py
└── my_app/
    ├── __init__.py
    ├── apps.py
    └── management/
        ├── __init__.py
        ├── commands/
        └── plugins/
            └── __init__.py
            └── backup.py

Note that we’ve added a plugins directory to the management directory of the media and my_app apps. This is where we’ll place our command extensions. We must register our plugins directory in the apps.py file of the media and my_app apps like this:

from django.apps import AppConfig
from django_typer.utils import register_command_plugins

class MyAppConfig(AppConfig):
    name = 'my_app'

    def ready(self):
        from .management import plugins

        register_command_plugins(plugins)

Note

Because we explicitly register our plugins we can call the package whatever we want. django-typer does not require it to be named plugins. It is also important to do this inside ready() because conflicts are resolved in the order in which the extension modules are registered and ready() methods are called in INSTALLED_APPS order.

For plugins to work, we’ll need to re-implement media from above as a composed extension like this:

Django-styleTyper-style

media/management/plugins/backup.py

import tarfile
import typing as t
from pathlib import Path

import typer
from django.conf import settings

from backup.management.commands.backup import (
    Command as Backup,
)


# instead of inheriting we add the command using the classmethod decorator
# on the backup Command class to decorate a module scoped function
@Backup.command()
def media(
    # self is optional, but if you want to access the command instance, you
    # can specify it
    self,
    filename: t.Annotated[
        str,
        typer.Option(
            "-f",
            "--filename",
            help=("The name of the file to use for the media backup tar."),
        ),
    ] = "media.tar.gz",
):
    """
    Backup the media files (i.e. those files in MEDIA_ROOT).
    """
    media_root = Path(settings.MEDIA_ROOT)
    output_file = self.output_directory / filename

    # backup the media directory into the output file as a gzipped tar
    typer.echo(f"Backing up {media_root} to {output_file}")
    with tarfile.open(output_file, "w:gz") as tar:
        tar.add(media_root, arcname=media_root.name)

And our my_app extension might look like this:

Django-styleTyper-style

my_app/management/plugins/backup.py

import datetime
import shutil
import subprocess
import typing as t

import typer
from django.conf import settings

from backup.management.commands.backup import (
    Command as Backup,
)


@Backup.command()
def environment(
    self,
    filename: t.Annotated[
        str,
        typer.Option(
            "-f",
            "--filename",
            help=("The name of the requirements file."),
        ),
    ] = "requirements.txt",
):
    """
    Capture the python environment using pip freeze.
    """

    output_file = self.output_directory / filename

    typer.echo(f"Capturing python environment to {output_file}")
    with output_file.open("w") as f:
        subprocess.run(["pip", "freeze"], stdout=f)


@Backup.command()
def database(self):
    """
    Backup the database by copying the sqlite file and tagging it with the
    current date.
    """
    db_file = self.output_directory / f"backup_{datetime.date.today()}.sqlite3"
    self.echo("Backing up database to {db_file}")
    shutil.copy(
        settings.DATABASES["default"]["NAME"],
        db_file,
    )

Note that we now have a new environment command available:

And the command line parameters to database have been removed:

Note

The extension code is lazily loaded. This means plugins are resolved on command classes the first time an instance of the class is instantiated. This avoids unnecessary code execution but does mean that if you are working directly with the typer_app attribute on a TyperCommand you will need to make sure at least one instance has been instantiated.

Overriding Groups

Some commands might have deep nesting of subcommands and groups. If you want to override a group or subcommand of a group down a chain of commands you would need to access the Typer instance of the group you want to override or extend:

Django-styleTyper-style

from somewhere.upstream.management.commands.command import Command

# add a command to grp2 which is a subgroup of grp1
@Command.grp1.grp2.command()
def my_command():  # remember self is optional
    pass

# add a subgroup to grp2 which is a subgroup of grp1
@Command.grp1.grp2.group()
def grp3():
    pass

You may even override the initializer of a predefined group:

Django-styleTyper-style

from somewhere.upstream.management.commands.command import Command

# override the initializer (typer callback) of grp1 on Command,
# this will not alter the child groups of grp1 (grp2, grp3, etc.)
@Command.grp1.initialize()
def grp1_init(self):
    pass

@Command.group()
def grp1(self):
    """
    This would override grp1 entirely and remove all subcommands
    and groups.
    """

Tip

If a group or command has not been directly defined on a Command class, django-typer will do a breadth first search of the command tree and fetch the first group or subcommand that matches the name of the attribute. This means that you do not necessarily have to walk the command hierarchy (i.e. Command.grp1.grp2.grp3.cmd), if there is only one cmd you can simply write Command.cmd. However, using the strict hierarchy will be robust to future changes.

When Do CLI Plugins Make Sense?

Plugins can be used to group like behavior together under a common root command. This can be thought of as a way to namespace CLI tools or easily share significant code between tools that have common initialization logic. Moreover it allows you to do this safely and in a way that can be deterministically controlled in settings. Most use cases are not this complex and even our backup example could probably better be implemented as a batch of commands.

Django apps are great for forcing separation of concerns on your code base. In large self contained projects its often a good idea to break your code into apps that are as self contained as possible. Plugins can be a good way to organize commands in a code base that follows this pattern. It also allows for deployments that install a subset of those apps and is therefore a good way to organize commands in code bases that serve as a framework for a particular kind of site or that support selecting the features to install by the inclusion or exclusion of specific apps.

Logic Plugins

Inversion of Control (IoC) is a design pattern that allows you to inject custom logic into a framework or library. The framework defines the general execution flow with extension points along the way that downstream applications can provide the implementations for. Django uses IoC all over the place. Extension points are often called hooks. You may use a third party library to manage hooks or implement your own mechanism but you will always need to register hook implementations. The same plugin mechanism we used in the last section provides a natural place to do this.

Some Django apps may keep state in files in places on the filesystem unknown to other parts of your code base. In this section we’ll use pluggy to define a hook for other apps to implement to backup their own files. Let’s:

1. Create a new app backup_files and inherit from our the extended media backup command we created in the inheritance section.

2. Define a pluggy interface for backing up arbitrary files

3. Add a files command to our backup command that will call all registered hooks to backup their own files.

Django-styleTyper-style

backup_files/management/commands/backup.py

import sys
import typing as t
from pathlib import Path

import typer
import pluggy

from media.management.commands.backup import (
    Command as Backup,
)


class Command(Backup):  # inherit from the extended media backup command
    plugins = pluggy.PluginManager("backup")
    hookspec = pluggy.HookspecMarker("backup")
    hookimpl = pluggy.HookimplMarker("backup")

    # add a new command called files that delegates file backups to plugins
    @Backup.command()
    def files(self):
        """
        Backup app specific non-media files.
        """
        for archive in self.plugins.hook.backup_files(command=self):
            if archive:
                typer.echo(f"Backed up files to {archive}")


@Command.hookspec
def backup_files(command: Command) -> t.Optional[Path]:
    """
    A hook for backing up app specific files.

    Must return the path to the archive file or None if no files were backed up.

    :param command: the backup command instance
    :return: The path to the archived backup file
    """


Command.plugins.add_hookspecs(sys.modules[__name__])

Now lets define two new apps, files1 and files2 that will provide and register implementations of the backup_files hook:

Django-styleTyper-style

files1/management/plugins/backup.py

import sys
import typing as t
from pathlib import Path

from backup_files.management.commands.backup import (
    Command as Backup,
)


@Backup.hookimpl
def backup_files(command: Backup) -> t.Optional[Path]:
    # this is where you would put your custom file backup logic
    return command.output_directory / "files1.tar.gz"


Backup.plugins.register(sys.modules[__name__])

Django-styleTyper-style

files2/management/plugins/backup.py

import sys
import typing as t
from pathlib import Path

from backup_files.management.commands.backup import (
    Command as Backup,
)


@Backup.hookimpl
def backup_files(command: Backup) -> t.Optional[Path]:
    # this is where you would put your custom file backup logic
    return command.output_directory / "files2.zip"


Backup.plugins.register(sys.modules[__name__])

Both files1 and files2 will need to register their plugin packages in their apps.py file:

files1/apps.py

from django.apps import AppConfig

from django_typer.utils import register_command_plugins


class Files1Config(AppConfig):
    name = "files1"
    label = name.replace(".", "_")

    def ready(self):
        from .management import plugins

        register_command_plugins(plugins)

Now when we run we see:

$> python manage.py backup
Backing up database [default] to: ./default.json
[.............................................]
Backing up ./media to ./media.tar.gz
Backed up files to ./files2.zip
Backed up files to ./files1.tar.gz

When Do Logic Plugins Make Sense?

CLI plugins make sense when you want to add additional commands or under a common namespace or to override the entire behavior of a command. Logical plugins make more sense in the weeds of a particular subroutine. Our example above has the following qualities which makes it a good candidate:

1. The logic makes sense under a common root name (e.g. ./manage.py backup files).

2. Multiple apps may need to execute their own version of the logic to complete the operation.

3. The logic is amenable to a common interface that all plugins can implement.