Tool Name: argcmdr

Status: in use & actively developed – adolescent and/or beta
Github Repo: https://github.com/dssg/argcmdr
Website: ibid
Team: Jesse London (with Tristan Crockett a reviewer)

Problem Background
As software engineers, data scientists, code monkeys and operations developers, we cannot and should not avoid the command line interface (CLI). This interface serves a diverse set of needs:

  • The CLI is a common and often necessary target for publishing our work, a valuable user interface to the software we produce
  • …As well it is often a large part of our means of developing software
  • …And sometimes the CLI even constitutes the bulk of our software

But even the most Richard Stallman of us might agree, (excepting perhaps Richard Stallman himself), that development of this interface poses fundamental challenges, problems which we must solve over and over, such as:

  • How do I parse –options again?
  • What about sub-commands?
  • How should I organize my CLI code? How will I reuse code?
  • How do I handle errors?
  • What about tab-completion?
  • …Does this need quotes?! (Yes.)

    local commit_message=$(git log -1 –pretty=format:%s)
  • …Is this good?! (Give up now before this gets worse.)

    ssh myserver “if [ \”$env\” == production ]; then mount /dev/sbd1 \“/media/our big data volume\”; fi”

These challenges hold us back – all of us.

  • Our CLIs are more frustrating and time-intensive to write – we tend to write fewer, less extensive and less useful interfaces
  • We neglect our generalized and per-project development tools, slowing down development, raising the barriers to entry for new contributors, and leaning on documentation over automation (assuming we document – I hope so!).
  • That software which itself rests heavily on the CLI is too often cumbersome, difficult to understand, to use and to extend, or…
  • …it goes the other way, using such high-level abstractions that we entirely lose the flexibility, expedience and testability of the underlying CLI elements we’re managing.

Description of Tool: How this tool solves this problem
argcmdr – “arg(ument) commander” – remediates these problems, basing itself on the Python programming language, and that language’s built-in argparse library, and then going much further.

Command interfaces are defined in argcmdr as Python classes or as decorated functions. argcmdr handles command argument processing, validation and type-casting.

Sub-command definition is as simple as nesting command class declarations or decoration.

Code reuse and organization are aided by Python itself. Error-handling is also based on Python and extended by the functionality of argparse. And, when system calls are needed, all of these concerns are further addressed by the powerful abstractions of the Plumbum library.

Command argument completion is available, and trivial to configure, thanks to integration with argcomplete.

And for development and operational tooling, argcmdr allows you to omit even the construction of any executable. In the spirit of the Makefile, (not to mention many others), you may simply write a Python file, (or a Python package, as needed), in which you define any hierarchy of argcmdr commands. argcmdr installs its own generic executable, manage, which loads the command hierarchy of any such file, (by default the file manage.py in the current working directory).

With even this obstacle removed, there is little reason to merely document operations, when they can be almost-trivially programmed, into a consistent, hierarchical, testable and self-documenting interface. And starting from such a basis, such interfaces can be all the more readily extended.

Why not X?
Much software has been written towards this problem. The motivational goals of argcmdr include:

  • Ease and clarity of command definitions – including sub-commands as a primary consideration.
  • Elegant support for inline shell scripting. argcmdr is implemented in Python, and the language yields substantial benefit to the command programmer. But with a leveled playing field, the programmer is free to choose pragmatically and appropriately any combination of Python’s standard library and the system shell.
  • Configurability … and to not needlessly reinvent the wheel. argparse, the Python language’s modern standard, is not only an internal component of argcmdr – moreover, the command’s instance of the argparse.ArgumentParser is directly exposed to the programmer to configure, as succinctly or as intricately as necessary.
  • Progressive extensibility. Commands may be trivially declared as decorated functions, and approximately as trivially as classes. But under the hood, they are in fact always classes, allowing the programmer to use their full toolset, and their own judgement, in building and extending their software.
  • CLIs everywhere! We use the CLI as developers, not only as users of our software. And so command hierarchies needn’t be deployed or installed. The same implementation serves commands tied only to a file system directory, via a manage.py.

How to use it
argcmdr is currently documented by its README.

Case Studies
argcmdr is used to manage the development, building and/or publishing of several DSaPP projects, such as DSSG Marketplace, Aequitas and Triage. It is additionally used in the definition of the Triage CLI executable, and it’s used internally at DSaPP to provide a consistent interface to its operational tasks.

Future Plans and Areas for Improvement
While argcmdr is considered stable, it is still actively developed. Actively-considered and -developed areas of improvement include:

  • Delegation: higher-level support for command delegation to other argcmdr commands, (beyond the existing, workable support of Python function calls) – (currently in branch)
  • Combinator clean-up: additional support for and/or clean-up of Plumbum commands and modifiers
  • Remote execution: additional support for execution over SSH, (beyond what is already readily-supported)
  • Distributed execution: integration with Mitogen to enable painless and efficient execution of the Python source itself on remote machines, allowing for the use of pure Python in remote contexts, and allowing for execution of “local” shell commands remotely (removing the layer of complexity of SSH).