Design Principles

immlib is designed to be useful and light-weight both to a data scientist who is using it in the interactive examination of data and to a scientific programmer who is writing interfaces for tools. It tries to follow a few principles that are intended to simplify the library’s design; these principles are described below.

For those who are looking to jump into using the library and evaluating its interface, this section is helpful but can safely be skipped.

immlib’s interface should be mostly functional.

Functional programming is a common programming paradigm that itself involves many prescriptions. The advantages and details of functional programming are described in detail elsewhere, but the important aspects in brief are:

1. Code should consist of simple functions.

2. Those functions should be pure—that is, they should not have internal state and should not edit their inputs.

Functions should expose their parameters but be smart about default values.

A common tradeoff in the design of scientific software orients around the extent to which a function exposes its parameters to the user. To explain this tradeoff, consider the following example of a function for calculating the values of a Gaussian function:

def gaussian(x):
    """Returns the Gaussian weights for the given input value or values.
    
    The Gaussian weights of the input `x` are calculated for the standard
    Gaussian function `exp( -x**2 / 2 )`.
    """
    from numpy import exp
    return exp(-x**2 / 2)

The main shortcoming of the above function is that it is extremely situational; it is useful only when you need a standard Gaussian function because it does not expose any parameters. On the other hand, this function is very easy to maintain and understand, and the more we expose parameters, the more complicated the function can become and the more work maintenance tends to be. Over-parameterized functions can be just as frustrating as underparameterized functions for different reasons, however. For example:

def gaussian(x, mu, sigma, expt, normalize):
    """Returns the Gaussian weights for the given input value or values.
    
    `gaussian(x, mu, sigma, expt, False)` returns the Gaussian weights of the
    input `x` are calculated using the generalized Gaussian function:
    `exp( -(1/2) * abs((x - mu)/sigma) ** expt )`
    
    `gaussian(x, mu, sigma, q, True)` normalizes the return value such that
    the integral of the Gaussian is 1.
    
    Note that other formulations of the generalized Gaussian function often use
    the form `exp(-(abs((x - mu)/sigma) ** exp)` instead of the form used by
    this function; however such a form does not result in a standard normal
    Gaussian `exp(-(1/2) * ((x - mu)/sigma)**2)` when `expt` is equal to 2, so
    this function uses a modified version.
    """
    from numpy import exp
    if normalize:
        from scipy.special import gamma
        u = 1 + 1/expt
        const = 1 / ((2**u) * sigma * gamma(u))
    else:
        const = 1
    return const * exp(-(1/2) * (abs(x - mu)/sigma) ** expt)

The above version of the Gaussian function is very general, but using it will almost certainly require reading the documentation and processing some math, unless one is very familiar with Gaussian distributions to begin with. A programmer who comes along a function call like gaussian(x, mu, sig, 2, True) will likely wonder what the 2 and True in the parameter list are about. A compromise between making functions overly complex and making them overly situational is the common strategy of giving some of the parameters default values that are reasonable in most situations.

def gaussian(x, mu=0, sigma=1, expt=2, normalize=False):
    """Returns the Gaussian weights for the given input value or values.
 
    `gaussian(x)` returns the standard Gaussian function `exp(-(1/2) * x**2)`.
    
    `gaussian(x, mu, sigma)` returns a Gaussian of unit height whose mean is
    `mu` and whose standard deviation is `sigma`.
    
    The optional argument `normalize` (default: `False`) may be set to `True`
    to indicate that the return value should be normalized such that the
    integral of the Gaussian is equal to 1. For a probability density function,
    this should be set to `True`. Otherwise, the Gaussian will have unit
    height instead of unit area.
    
    The shape of the returned Gaussian can further be modified with the
    optional parameters `mu`, `sigma`, and `expt`, which use the following form
    for a generalized Gaussian function:
    `exp( -(1/2) * abs((x - mu)/sigma) ** expt )`.
    
    Note that other formulations of the generalized Gaussian function often use
    the form `exp(-(abs((x - mu)/sigma) ** exp)` instead of the form used by
    this function; however such a form does not result in a standard normal
    Gaussian `exp(-(1/2) * ((x - mu)/sigma)**2)` when `expt` is equal to 2, so
    this function uses a modified version.
    """
    from numpy import exp
    if normalize:
        from scipy.special import gamma
        u = 1 + 1/expt
        const = 1 / ((2**u) * sigma * gamma(u))
    else:
        const = 1
    return const * exp(-(1/2) * (abs(x - mu)/sigma) ** expt)

This function is highly general, but it’s also easy to use. Maintaining it takes more work than the simple version, but this comes with the advantage of the code being much more general and thus more likely to be reused. Where to draw the line in an API between simplicity and generality is not a decision that immlib can prescribe a solution to, but immlib endeavors to make the more general code—code that exposes many parameters but manages them smartly—easier to write and maintian when it comes to scientific computing.

Data should be loaded and computed lazily.

immlib uses the pcollections library as a backend for immutable data structures and lazy data structures. The ldict type, inparticular, allows one to define dictionaries whose values are calculated and cached when they are first requested, and immlib builds a toolkit for declaring lazy data structures and workflows based largely on this type.

The benefits of laziness in an API that deals with data are clear when one considers how one typically interacts with data in an analysis environment like Jupyter. Ideally, one can load a record or subset of data from a dataset without waiting for the entire dataset to load, especially when a dataset is large. Simultaneously, it is often most intuitive and desirable to give users access to data structures that represent entire datasets, despite their size. If the records of a dataset are loaded lazily, then this is not a problem, as only the data that the user interacts with will be loaded.

immlib contains many tools for constructing lazily-computed data structures that behave superficially like normal data structures, including support for data structures based on computational workflows and for the caching of outputs and intermediate data of computations.