Workflows

Workflows#

One of immlib’s most powerful features is its tools for creating modular scientific workflows. Designing workflows using immlib is straightforward—managed mostly through decorators—and comes with features like automatic caching and laziness.

Introductory Example#

To explain workflows, it’s easiest to consider a concrete example. Suppose we need to write a utility that takes, as input, an array of values and that produces a variety of statistics describing them. We might write such code using immlib as follows:

import immlib as il, numpy as np

# The input data should be converted into an array:
@il.calc('data', 'n', lazy=False)
def check_data(data):
   """Ensures that the argument `data` is an array of numbers.
   
   Parameters
   ----------
   data : array-like
       The input data for which statistics should be calculated. The data must
       be a 1-dimensional vector of numbers, and it is converted into a NumPy
       array by this calculation.
   
   Outputs
   -------
   data : numpy.ndarray
       A 1-dimensional NumPy array of numbers.
   n : int
       The number of data-points in `data`.
   
   Raises
   ------
   TypeError
       If `data` is not a 1-dimensional vector of numbers.
   """
   print('Checking data...')
   data = il.to_array(data)
   if not il.is_array(data, shape=(-1,), dtype=np.number):
       raise TypeError("data must be a vector-like sequence of numbers")
   return (data, len(data))

@il.calc('sum_of_squares', 'sum')
def calc_sums(data):
    """Calculates the sum and sum of squares of the data.
    
    Outputs
    -------
    sum_of_squares : number
        The sum of squares of the absolute values of the data.
    sum : number
        The sum of the values in data.
    """
    print('Calculating sums...')
    sum_of_sq = np.sum(np.abs(data)**2)
    sum = np.sum(data)
    return (sum_of_sq, sum)

@il.calc('mean', 'var', 'std')
def calc_mean_etc(sum, sum_of_squares, n):
    """Calculates the mean, variance, and standard deviation of the data.
    
    Outputs
    -------
    mean : number
        The mean of the data.
    var : number
        The variance of the data.
    std : number
        The standard deviation of the data.
    """
    print('Calculating mean, etc...')
    mean = sum / n
    var = (sum_of_squares / n) - mean**2
    std = np.sqrt(var)
    return (mean, var, std)

# Make a plan for calculating these statistics:
stats_plan = il.plan(
    check_step=check_data,
    sums_step=calc_sums,
    mean_step=calc_mean_etc)

Once we have set up the example code above, we can create an instance of our stats plan called a plandict. When we create the plandict, the check_data function will automatically run because it is marked as not lazy (lazy=False):

# Make a plandict for a set of data:
stats = stats_plan(data=np.random.randn(100))
print("The type of stats is", type(stats))

Checking data...
The type of stats is <class 'immlib.plandict'>

The plandict has a key for each of the inputs and outputs in the entire calculation plan:

sorted(stats.keys())

['data', 'mean', 'n', 'std', 'sum', 'sum_of_squares', 'var']

The values associated with these keys are calculated according to the calculation plan, with the inputs for each calculation being derived from the outputs of other calculations. In the plan stats_plan above, there is no output named data, so it is considered an input of the plan.

stats_plan.inputs

{|'data'|}

When the values associated with keys are requested, they get calculated and cached. So in the example above, in which messages are printed as the values are calculated, the messages will be printed only the first time a key is requested.

# The message about the sums being calculated will appear immediately when this
# key is requested:
sum_sq = stats['sum_of_squares']
# But it won't appear again here:
print("The sum of squares is", stats['sum_of_squares'])

# Similarly, it won't appear again when we request the sum, because the sum was
# calculated in the same function as the sum of squares.
print("The sum is", stats['sum'])

Calculating sums...
The sum of squares is 95.4955119670381
The sum is -5.651956347797989

Downstream values such as mean and var aren’t calculated until they are required:

print("The mean is", stats['mean'])
print("The standard deviation is", stats['std'])

Calculating mean, etc...
The mean is -0.05651956347797989
The standard deviation is 0.9755822152000515

Calculation Metadata#

The @calc decorator attaches metadata to the calculation functions by creating a field calc. The calc object that manages the calculation’s integration with any plans is found here.

calc_sums.calc

<immlib.calc at 0x7f6cdff3c4c0>

Tracking Documentation#

Calculations and plans also track the documentation for their inputs and outputs. These can be accessed via the input_docs and output_docs fields.

print(calc_sums.calc.output_docs['sum'])

sum : number
    The sum of the values in data.

print(check_data.calc.input_docs['data'])

data : array-like
    The input data for which statistics should be calculated. The data must
    be a 1-dimensional vector of numbers, and it is converted into a NumPy
    array by this calculation.

Plans have a docstring that is automatically generated from their constituent calculations.

print(stats_plan.__doc__)

An ``immlib.plan`` object for a set of calculations.

This documentation was generated automatically from the docstrings
of the individual ``immlib.calc`` objects that make up this plan.

This plan contains the following calculations:
 * ``check_step``: ``__main__.check_data``  
   Inputs:  ``data``  
   Outputs: ``data``, ``n``  
 * ``sums_step``: ``__main__.calc_sums``  
   Inputs:  ``data``  
   Outputs: ``sum_of_squares``, ``sum``  
 * ``mean_step``: ``__main__.calc_mean_etc``  
   Inputs:  ``sum``, ``sum_of_squares``, ``n``  
   Outputs: ``mean``, ``var``, ``std``  

Inputs
------
data
    **``check_step``** input: ``data`` : array-like  
    The input data for which statistics should be calculated. The data must
    be a 1-dimensional vector of numbers, and it is converted into a NumPy
    array by this calculation.

Outputs
-------
data
    **``check_step``** output: ``data`` : numpy.ndarray  
    A 1-dimensional NumPy array of numbers.
n
    **``check_step``** output: ``n`` : int  
    The number of data-points in `data`.
sum_of_squares
    **``sums_step``** output: ``sum_of_squares`` : number  
    The sum of squares of the absolute values of the data.
sum
    **``sums_step``** output: ``sum`` : number  
    The sum of the values in data.
mean
    **``mean_step``** output: ``mean`` : number  
    The mean of the data.
var
    **``mean_step``** output: ``var`` : number  
    The variance of the data.
std
    **``mean_step``** output: ``std`` : number  
    The standard deviation of the data.
Contents