Nodes

class grapheval.node.EvalNode(name, parents=None, plot=False, cache=None, ignore_cache=False, cache_not_found_regenerate=True, **kwargs)

This is the base class for every node in a node chain/tree.

A node tree can be though of as a series of subsequent operations on some data. This data can come from running an experiment, reading a file,… and can be processed in an arbitrary number of small, well defined steps to yield results from this data. Every type of step is implemented as decendant of this class. Optionally, a step (=Node) can also show its results by plotting them. Also EvalNode handles caching of every step if neccessary.

The idea is to create a set of decendant classes of EvalNode for the steps you need for your evaluation. Then you create a tree of EvalNodes by creating instances and linking them in the order neccessary to achieve your desired result. This linking is achieved by the paremeter parents which can be understood as the source(s) of data needed for the respective EvalNode.

After setting up the tree you can use any EvalNode as entry point for doing the operations defined by simply calling the instance. This leads to recursively acquiring data on all parent nodes and then evaluating the called node.

At the moment modifying the nodes after calling one of them can lead to unexpected behaviour because the result of do() is cached (in RAM) and eventual modifications cannot be detected. This may change in the future to support this workflow. Until then you can deactivate the RAM cache by setting EvalNode.disable_ram_cache to True.

Every decendant class of this class should implement one (atomic) step or operation in the process of evaluating an experiment. When deriving an EvalNode the methods listed below can or should be overriden to provide the behaviour you intend.

Parameters

  • name (string) – Name of this instance

  • parents (list or dict or EvalNode) – The parents of this EvalNode. Can also be understood as the data sources. parent_data will have the same structure as this parameter with EvalNodes exchanged for the return value of EvalNode.do()

  • plot (bool or object or list) – Decide if the node should execute its plot() method. If True or False plot() is always/never called. If another object is given (preferably string but technically anything that can be compared) it defines a plot group. See plot_on for further reference.

  • cache (NodeCache) – The NodeCache instance that this EvalNode should use for caching. If None caching is disabled.

  • ignore_cache (bool) – Ignore possibly existing cached data and re-run EvalNode.do() in any case. Caution: this only prevents cache reading, not writing.

  • cache_not_found_regenerate (bool, optional) –

    If:

    • no up-to-date cache could be found,

    • children did not request data, and

    • parents haven’t updated

    then, depending on the value of this attribute either:

    • True: consider this node outdated and regenerate data after requesting parent data (which in turn forces all children to regenerate). Default.

    • False: consider this node up to date and continue without regeneration.

    True leads to the same behaviour as if caching wasn’t used at all. In many cases, False can prevent false positives (unnecessary regenerations), but it has the caveat that deleting on-disk cache files will not guarantee a regeneration of data.

    This should be most useful for “organisational” nodes that do no data processing themselves (for example, provide mappings) if their out-of- date-ness is always equal to their parent’s out-of-date-ness.

  • **kwargs – The kwargs are stored and are passed to the overridden functions and can be used for customisation of a derived class.

Derived classes can or should override the following abstract methods:

  • do() : Must be overriden. This is the entrypoint for performing the neccessary calculations etc.

  • plot() : Optional. Show the data on a specific axis object.

  • def_kwargs() : Optional. Set defaults for subclass-specific kwargs.

  • common() : Optional. Add values to a dictionary shared across the chain.

  • subclass_init() : Optional. Perform initialisation if neccessary. (Do not override __init__())

  • __contains__() : Optional. As specified in the python object model docs. Return wether a key is contained in the return value of do().

For further information on the methods to override refer to the respective documentation of the methods. Keep in mind that any one of these calls can recieve kwargs other than the ones passed on instance creation. These should be ignored.

__call__(ax=None, plot_on=None, memo=None, **kwargs)

Run the node and recursively its parents and possibly plot the results. :param ax: If not None potentially plot nodes on this Axes object :type ax: None or matplotlib.axes.Axes, optional :param plot_on: a plot group or a list of plot groups that

should be plotted. See plot_on for more information. If one of None, True or False, exactly the nodes for which plot_on are set to True are plotted.

Parameters

  • memo (list of EvalNode) – Used to prevent double plotting. Nodes that are in memo are ignored. Double evaluation is prevented by caching results in RAM.

  • **kwargs – All nodes kwargs are joined with this kwargs dict for this call. Watch out because cached nodes are not re-called even if kwargs change. (At the moment)

Returns

the memo list after running all parents. This may change in the future. Options may be: the data returned by (1) the called node (2) all nodes

Return type

list

__contains__(item)

This function always raises an exception. For more information on that see __getitem__()

__getitem__(index)

Returns a node which in turn subscripts the return value of this nodes do() call on evaluation. (and returns this subscripted value)

This may be confusing but I found it to be very useful in constructing node chains. Think of it as using the EvalNodes as if they were their generated/returned values.

This also implies that the object returned is kind of a “future”, only containing actual subscripted data when the node chain is run. This also makes the functions __contains__() and __iter__() unavailable to the base class since it is not known to the base class what subclasses’ do() call might return.

You are encouraged to override __contains__() for custom subclasses if senseful return values can be provided. Further information:

An alternative way to use subscription of this class would be to return the subscripted parents value but since this is much simpler to realize manually than the behaviour outlined above I chose it to be the way it is.

In fact, NodeGroup overrides this behaviour in exactly this way because both ways of subscription are equivalent here, i.e. the subscription of the do()-call return value is equivalent to subscripting the dictionary of parents of the NodeGroup.

Return type

SubscriptedNode

common(common, **kwargs)

Override this method to add information to the common dict that is shared across this instance and all parents and children in the evaluation chain. It is not intended to be used for data that is expensive to calculate or store.

You can either return a set of keys which are used to update the common dict or directly modify the parameter common given.

Called between do and plot.

Always be aware that unexpected kwargs may be passed to any function, so always include * **kwargs * in the call signature even if your node does not use any kwargs or you write them out.

copy(name_suffix, plot=None, ignore_cache=None, last_parents=None, last_kwargs=None, memo=None)

Create a copy of this EvalNode and all of its parents recursively.

Parameters

  • name_suffix (string) – This suffix is appended to the name of this instance to create a new name for the copy.

  • plot (bool, optional) – if not None the value of plot_on is overridden in the copy of this EvalNode and all of its parents’ copies. Default: None

  • ignore_cache – if not None the value of ignore_cache is overridden in the copy of this EvalNode and all of its parents’ copies. Default: None

  • last_parents (list or dict or EvalNode, optional) – The top-level EvalNode instances are created with the value of this parameter as their parents. Default: None

  • last_kwargs (dict, optional) – The top-level EvalNode instances are updated with this parameter where values which are itself dicts are merged into the existing dicts. The **kwargs of the EvalNode instance that is copied is deepcopied, which may lead to unexpected behaviour. (Maybe add: optionally deepcopy or copy). Other EvalNodes in the chain keep the kwargs used at the time of their creation. Default: None

  • memo (dict, optional) – If the same instance of EvalNode occurs multiple times in the evaluation tree, the first occurence is copied and this copy is used on subsequent occurences. This should preserve loops in the tree. This is realized with the memo parameter which should be left default on manual calls to this method.

It should be possible to copy multiple nodes having the same parents (resulting in to manual copy() calls) if one supplies the same dict to memo for both calls. (Start with an empty dict)

Returns

A copy of this instance

Return type

EvalNode

property data

Get the data of the node, eventually calling parent node’s do method and using RAM cache

Returns

returned object from do()

data_extra(common=None, **kwargs)

The same as data but allowing for kwargs and supplying a common dict (with gets filled eventually)

Returns

returned object from do()

def_kwargs(**kwargs)

Override this method if you have default values for optional kwargs or other things you want to do to the kwargs. Other overriden methods will see the dict returned by this method as kwargs but eventually overriden with the kwargs passed to the initial __call__() of the tree.

Always be aware that unexpected kwargs may be passed to any function, so always include **kwargs in the call signature even if your node does not use any kwargs or you write them out.

do(parent_data, common, **kwargs)

Override this method. Perform the required evaluation. parent_data is a dict containing the return values of all parent’s do calls with the keys as given at creation time.

You can skip overriding this method. If not overridden the parent data is returned.

Do not change the common dict here, because this method is not guaranteed to run at every execution of the evaluation chain because of caching.

kwargs are the kwargs that are set at creation time potentially overriden with those at call time.

Always be aware that unexpected kwargs may be passed to any function, so always include **kwargs in the call signature even if your node does not use any kwargs or you write them out.

get_color()

New bodgy way to synchronise colors. Use only in conjunction with set_color.

Colors are stored in a instance variable per node and the color of the first parent found that has this instance variable set is returned.

If no color is found, a new one is returned

get_kwargs()

Get kwargs of this instance. To modify them it is better to use set()

property global_id

An id unique to this instance counting every instance of EvalNode and its derived classes

property handles
property handles_complete_tree
property id

An id unique to this instance only counting instances of the specific class

in_plot_group(group)
Returns

Wether the node is plotted depending on the plot_on parameter passed to __call__. See plot_on for further information.

Return type

bool

is_descendant_of(possible_parent)

Return True if this instance has possible_parent anywhere in their parent chains

is_parent(possible_parent)
Returns

Returns if a EvalNode instance is a parent of this instance

Return type

bool

map_parents(callback, call_on_none=False)

Executes callback for every parent of this instance

Parameters

  • callback (Callable (EvalNode) -> object) – The Callable to be executed

  • call_on_none (bool, Default: False) – If this EvalNode has no parents call the callback with None as argument.

Returns

object with structure like parents but the nodes replaced with the objects returned by callback

Return type

same as parents

map_tree(map_callback, starts_with=None, return_values=None)

Execute map_callback for every node in this tree or for all nodes whose name starts with the given string, starting with self.

Parameters

  • map_callback (Callable (EvalNode) -> object) – The callable to execute. The only argument passed is the EvalNode instance in question.

  • starts_with (str or None, optional) – If None run the callback for every EvalNode in the tree. If not None run callback only for EvalNode instances whose name start with starts_with. Default: None

  • return_values (dict or None, optional) – The dict for collecting the return values in the recursive calls. Should be left at the default value for normal usage.

Returns

the return values of every call.

Return type

dict {EvalNode -> object}

property parent_count
property parents

The list or dict of parents of this node :rtype: list or dict

Type

return

parents_contains(key)
Returns

True if the parents iterator contains an item with the given key. For parents stored as list, this is equal to ‘key < len(parents)’ and for parents stored as dict, this is eqal to ‘key in parents’ If the parents are a single EvalNode this always returns False.

Return type

bool

property parents_iter
An iterator that can be handled uniformly

for any type of parents (list or dict or EvalNode). For lists, it is enumerate(parents), and for dict it is parents.items(). For a single parent of class EvalNode, an iterator yielding EmptyParentKey: parent is returned so that this property can be used uniformly as iterator.

Return type

iterator

Type

return

plot(data, ax, common, **kwargs)

Override this method if this eval node can plot something. It is called if requested with the return value from do(..) and an axes object to plot on.

Should return a list of handles that this Node created.

Always be aware that unexpected kwargs may be passed to any function, so always include **kwargs in the call signature even if your node does not use any kwargs or you write them out.

property plot_on

The plot property decides when a node is plotted. There are three possibilities:

  • True or False: The node is always or never plotted

  • matplotlib.axes.Axes instance: The node is always plotted on the given Axes instance

  • object or list of :py:class`object` : Defines a “plot group”, i.e. all nodes in the same group can be plotted in one call. This means this node is plotted if the same object (or one object from the list) to which plot_on is set is passed to the __call__() method of initial EvalNode.

See also: __init__() parameter plot

search_parent(query, match_partial=True)

Search the parents recursively for any nodes whose name starts with the given string and return the first match found

Parameters

  • query (str or type) – Search query. Strings are matched against the name of the node, types are matched with isinstance() builtin.

  • match_partial (boolean. Default: True) – If True, use str.startswith() to match strings. If False, only complete string matches are returned. Ignored on nodes.

search_parents_all(query, match_partial=True)

The same as search_parents(), but return a list of all matches.

search_tree(query, match_partial=True)

Search the chain of nodes for any nodes (including this node) whose name matches with the given string or is of a given type. See search_parent() for details.

search_tree_all(query, match_partial=True)

The same as search_tree(), but return a list of all matches.

set(**kwargs)

Set kwargs of this instance.

set_color(color)

See get_color()

subclass_init(parents, **kwargs)

Override this method if you need control over how the parents are attached this eval node or if you need to modify the parents in some way.

You need to set self.parents manually in this method

It is executed at instance __init__

Using this method is discouraged because customizing parent attachment can mess with copy semantics.

Always be aware that unexpected kwargs may be passed to any function, so always include **kwargs in the call signature even if your node does not use any kwargs or you write them out.

tree_kwargs(order='level')

one dict containing all of the tree’s kwargs, added to the dict in the given order. Orders available:

  • level: level-first order: most distant ancestor’s keys are added the first. (= most recent ancestor’s keys take precedence)

  • level_inverse: (inverse) level-first order: most recent ancestor’s keys are added the first. (= most distant ancestor’s keys take precedence)

  • post: post-order depth-first-search order of ancestor tree

Callback Nodes

These nodes inject a callback at some point in the evaluation chain.

class grapheval.node.KwargsCallbackNode(name, parents=None, plot=False, cache=None, ignore_cache=False, cache_not_found_regenerate=True, **kwargs)

Apply a given callback to the kwargs dictionary. The parent’s data is not accessed.

kwargs:

  • callback: callback(dict) -> Object

parents: any type

class grapheval.node.CommonCallbackNode(name, parents=None, plot=False, cache=None, ignore_cache=False, cache_not_found_regenerate=True, **kwargs)

Apply a given callback to the common dictionary. The common dictionary is shared across all nodes in a tree. The parent’s data is not accessed.

kwargs:

  • callback: callable(dict) -> Object

parents: any type

class grapheval.node.LambdaNode(name, parents=None, plot=False, cache=None, ignore_cache=False, cache_not_found_regenerate=True, **kwargs)

Apply a given callback to the parent’s data. If some collection of parents is attached, return a equally-structured collection with the callback applied to every item’s value.

kwargs:

  • callback: callable(Object) -> Object

parents: any type

Helper nodes

These nodes provided simple, but often used semantics.

class grapheval.node.SubscriptedNode(name, parents=None, plot=False, cache=None, ignore_cache=False, cache_not_found_regenerate=True, **kwargs)

Node type returned when subscripting a node. Does nothing except from returning the set subscript of the parent’s data.

kwargs:

  • subscript

parents: one parent of arbitrary type

class grapheval.node.NodeGroup(name, parents=None, plot=False, cache=None, ignore_cache=False, cache_not_found_regenerate=True, **kwargs)

Basic grouping of nodes. A NodeGroup instance returns the data of all its parents, arranged in the same data structure as the parents are stored.

Can be considered as approximate inverse to SubscriptedNode.

kwargs: none

parents: any number or type.

class grapheval.node.DebugOutNode(name, parents=None, plot=False, cache=None, ignore_cache=False, cache_not_found_regenerate=True, **kwargs)

Prints a parent’s data during the plotting stage for debugging purposes.

kwargs: none

parents: exactly one of type EvalNode.

Methods

grapheval.node.copy_to_group(name, node, count=None, last_parents=None, last_kwargs=None, memo=None)

Create copies of a node changing given properties.

This method leverages the EvalNode.copy() function to create sets of tree (segment) copies, changing the parameters of the top-most parent node for every copy.

Parameters

  • node (EvalNode) – The start node of the tree that will be copied.

  • count (int, optional) – The node is copied count times with last_parents and last_kwargs. If omitted, at least one of last_parents and last_kwargs must be passed and must be either list or dict.

  • last_parents (dict or list, optional) – If one of last_parents and last_kwargs is a dict the dict keys are used as name suffixes for the copy call, if both are list the suffixes are enumerated. If both are dicts, the keys of last_parents are used.

  • last_kwargs (dict or list, optional) – See last_parents

Returns

NodeGroup instance grouping copies of the given node.

Return type

NodeGroup