source: coopr.opt/stable/2.2/doc/opt/plugins-pca.tex @ 2196

Last change on this file since 2196 was 2196, checked in by wehart, 10 years ago

Merged revisions 2112-2195 via svnmerge from
https://software.sandia.gov/svn/public/coopr/coopr.opt/trunk

........

r2137 | wehart | 2010-01-10 12:17:40 -0700 (Sun, 10 Jan 2010) | 2 lines


Misc documentation updates.

........

r2155 | wehart | 2010-01-12 15:25:13 -0700 (Tue, 12 Jan 2010) | 2 lines


Adding a utility routine: results_attributes.

........

r2167 | jwatson | 2010-01-24 14:06:15 -0700 (Sun, 24 Jan 2010) | 3 lines


Added a _report_timing attribute to the base solver class. If true, it will print out presolve/solve/postsolve timing statistics. The derived shell-based solver will additionally report log and solution file read times.

........

r2173 | wehart | 2010-01-27 10:23:31 -0700 (Wed, 27 Jan 2010) | 2 lines


Misc edits.

........

r2174 | wehart | 2010-01-27 10:31:26 -0700 (Wed, 27 Jan 2010) | 2 lines


Adding a hook for specifying the format for reading/writing IO formats.

........

r2182 | wehart | 2010-01-27 15:39:31 -0700 (Wed, 27 Jan 2010) | 2 lines


Removing colin stuff from coopr.opt.

........

r2186 | wehart | 2010-01-27 17:39:31 -0700 (Wed, 27 Jan 2010) | 2 lines


Renaming results_attributes to results_schema.

........

File size: 18.4 KB
Line 
1
2\chapter{PyUtilib Component Architecture}
3
4\label{chap:pca}
5
6The PyUtilib Component Architecture (PCA) is an extension of the Trac
7plugin framework~\cite{Trac} that is included in the PyUtilib software
8package~\cite{PyUtilib}.  There does not appear to be a standard
9Python plugin framework, though there are some mature packages that
10support plugins including Zope~\cite{Zope}, Envisage~\cite{Envisage},
11Trac~\cite{Trac}, yapsy~\cite{yapsy} and SprinklesPy~\cite{SprinklesPy}.
12Although we discuss the design requirements for PCA later, PCA was
13initially motivated by the need to have a Trac-like plugin framework that was
14self-contained.  The core of PCA is provided by PyUtilib's
15\code{pyutilib.plugin.core} module.
16
17\section{Overview}
18
19The PCA is comprised of a small set of Python classes. A \textit{plugin}
20is a class that implements a set of related methods the context of the
21application, and a \textit{service} is an instance of a plugin class. Two
22different types of plugins are available: singleton and non-singleton
23plugins. There is at most one service for a singleton plugin, whereas
24there can be multiple services of non-singleton plugins.
25
26A software application can declare \textit{extension points} that other
27components can \textit{plug in} to. This mechanisms supports a flexible,
28modular programming paradigm that enables software applications to
29be extended in a dynamic manner. Extension points and the extensions
30contributed to them are stored in a global registry, and execution of
31these extensions is handled in a standardized manner. Thus, an application
32developer can define extension points without knowing how they will be
33implemented, and extension developers can register extensions without
34needing to know the details of how they are employed.
35
36Extension points are defined with respect to an \textit{interface} class,
37which defines the type that plugins use to register their capabilies.
38A plugin class includes declarations that denote that it implements
39one-or-more interfaces. An interface is defined by the methods and data
40that are used. However, the PCA does not enforce
41this interface or support interface conversions (see Zope~\cite{Zope}
42and Envisage~\cite{Envisage} for examples of plugin frameworks that
43support this functionality).
44
45
46\section{Core Plugin Classes}
47
48The PyUtilib plugin framework consists of the following core classes:
49\begin{description}
50\item[pyutilib.plugin.core.Interface] Subclasses of this class declare plugin interfaces that are registered in the framework
51\item[pyutilib.plugin.core.ExtensionPoint] A class used to declare extension points, which can access services with a particular interface
52\item[pyutilib.plugin.core.Plugin] Subclasses of this class declare plugins, which can be used to implement interfaces within this plugin framework.
53\item[pyutilib.plugin.core.SingletonPlugin] Subclasses of this class declare singleton plugins, for which a single service is constructed.
54\item[pyutilib.plugin.core.PluginEnvironment] A class that maintains the registries for interfaces, extension points, plugins and services.
55\item[pyutilib.plugin.core.PluginGlobals] A class that maintains global data concerning the set of environments that are currently being used.
56\item[pyutilib.plugin.core.PluginError] The exception class that is raised when errors arise in this framework.
57\end{description}
58
59\subsection{Interfaces and Extension Points}
60
61A subclass of the \code{Interface} class is used to declare extension
62points in an application. The \code{ExtensionPoint} class is used to declare
63an extension point and to retrieve information about the plugins that
64implement the specified interface. For example, the following is a
65minimal declaration of an interface and extension point:
66\begin{lstlisting}
67class MyInterface(Interface):
68   """An interface subclass"""
69
70ep = ExtensionPoint(MyInterface)
71\end{lstlisting}
72Note that the \code{MyInterface} class is not required to define the API of the
73interface. The PCA does not enforce checking of
74API conformance for plugins, and hence any declaration in the \code{MyInterface}
75class would be ignored. Additionally, note that an instance of \code{MyInterface}
76is not created; the \code{MyInterface} class is simply used to declare a type that is used to index related plugins.
77
78An instance of \code{ExtensionPoint} can be used to iterate through
79all extensions, or to search for an extension that matches a particular
80keyword. For example, the following code iterates through all extensions
81and applies the \code{pprint} method:
82\begin{lstlisting}
83for service in ep:
84  service.pprint()
85\end{lstlisting}
86If you wish to know the number of services that are registered with an
87extension point, you can call the standard \code{len} function:
88\begin{lstlisting}
89print len(ep)
90\end{lstlisting}
91Several other methods can be used to more carefully select services from
92an extension point. The \code{extensions} method returns a Python \code{set} object
93that contains the services:
94\begin{lstlisting}
95#
96# This loop iterates over all services, just the same as when an
97# the iterator method is used (see above).
98#
99for service in ep.extensions():
100  service.pprint()
101
102#
103# This loop iterates over all services, including the 'disabled'
104# services.
105#
106for service in ep.extensions(all=True):
107  service.pprint()
108\end{lstlisting}
109This example illustrates the optional argument that indicates whether
110the set returned by \code{extensions} includes all disabled services. It is
111convenient to explicitly support enabling and disabling services in many
112applications, though services are enabled by default. Disabled services
113remain in the registry, but by default they are not included in the set
114returned by an extension point.
115
116Finally, the PCA can support \textit{named services}, which requires that
117the services have a \code{name} attribute. Service names are not required
118to be unique. For example, when multiple instances of a non-singleton
119plugin are created, then these services can be accessed as follows:
120\begin{lstlisting}
121#
122# A simple plugin that implements the MyInterface interface
123#
124class MyPlugin(Plugin):
125  implements(MyInterface)
126
127  def __init__(self):
128      self.name="myname"
129
130#
131# Another simple plugin that implements the MyInterface interface
132#
133class MyOtherPlugin(Plugin):
134  implements(MyInterface)
135
136  def __init__(self):
137      self.name="myothername"
138
139#
140# Constructing services
141#
142service1 = MyPlugin()
143service2 = MyPlugin()
144service3 = MyOtherPlugin()
145
146#
147# A function that iterates over all MyInterface services, and
148# returns the MyPlugin instances (which are service1 and service2).
149#
150def get_services():
151    ep = ExtensionPoint(MyInterface)
152    return ep("myname")
153\end{lstlisting}
154In some applications, there is a one-to-one correspondence between service
155names and their instances. In this context, a simpler syntax is to use
156the \code{service} method:
157\begin{lstlisting}
158class MySingletonPlugin(SingletonPlugin):
159  implements(MyInterface)
160
161  def __init__(self):
162      self.name="mysingletonname"
163
164ep = ExtensionPoint(MyInterface)
165ep.service("mysingletonname").pprint()
166\end{lstlisting}
167The \code{service} method raises a \code{PluginError} if there is more than one service
168with a given name. Note, however, that this method returns \code{None} if no
169service has been registered with the specified name.
170
171Note that an integer cannot be used to select a service from an extension
172point. Services are not registered in an indexable array, so this option
173does not make sense.
174
175
176\subsection{Plugins and Environments}
177
178PCA plugins are subclasses of either the \code{Plugin} or \code{SingletonPlugin}
179classes. Subclasses of \code{Plugin} need to be explicitly constructed,
180but otherwise they do not need to be registered; simply constructing a
181subclass of \code{Plugin} invokes the registration of that instance. Similarly,
182simply declaring a subclass of \code{SingletonPlugin} invokes both the construction
183and registration of this plugin service.
184
185PCA plugins are registered with different interfaces using the \code{implements}
186function (which is a static method of \code{Plugin}). Note that a plugin can be
187registered with more than one interface. Plugins are applied to different
188extension points independently, but they can maintain state information
189that impacts their use across different extension points.
190
191The plugin and interfaces are organized within namespaces using the
192\code{PluginEnvironment} class. A global registry of plugin environments
193is maintained by the \code{PluginGlobals} class. This registry is a stack
194of environments, and the top of this stack defines the current
195environment. When an interface is declared, its namespace is the name
196of the current environment. For example:
197\begin{lstlisting}
198#
199# Declare an interface in the current environment
200#
201class Interface1(Interface):
202   pass
203
204#
205# Set the current environment to 'new_environ'
206#
207PluginGlobals.push_env( "new_environ" )
208
209#
210# Declare an interface in the 'new_environ' environment
211#
212
213class Interface2(Interface):
214   pass
215
216#
217# Go back to the original environment
218#
219PluginGlobals.pop_env()
220\end{lstlisting}
221The namespace that an \code{Interface} subclass is declared in defines the
222namespace where plugin services will be registered. Additionally,
223a plugin service will be registered in the namespace where it is
224declared. For example:
225\begin{lstlisting}
226#
227# Declare Interface1 in namespace env1
228#
229PluginGlobals.push_env("env1")
230
231class Interface1(Interface):
232   pass
233
234#
235# Declare Interface2 in namespace env2
236#
237PluginGlobals.push_env("env2")
238
239class Interface2(Interface):
240   pass
241
242PluginGlobals.pop_env()
243
244#
245# Declare Plugin1 in namespace env3
246#
247PluginGlobals.push_env("env3")
248
249class Plugin1(Plugin):
250
251   implements(Interface1)
252   implements(Interface2)
253   implements(Interface1,"env4")
254
255PluginGlobals.pop_env()
256\end{lstlisting}
257When \code{Plugin1} is instantiated, it's services are registered in the
258following environments:
259\begin{lstlisting}
260    env1 for Interface1
261    env2 for Interface2
262    env4 for Interface1
263    env3
264\end{lstlisting}
265The last registration is the default. A plugin service is always
266registered in the environment in which it was declared.
267
268Namespaces provide a mechanism for organizing plugin services in an
269extensible manner. Applications can define new namespaces that contain
270their services without worrying about conflicts with services defined
271in other Python libraries.
272
273\subsection{Global Plugin Data}
274
275Global plugin data in PCA is managed in the \code{PluginGlobals} class. This
276class contains a variety of static methods that are used to access
277this data:
278\begin{description}
279\item[default\_env] This method returns the default environment, which is constructed when the plugins framework is loaded.
280\item[env] This method returns the current environment if no argument is specified. Otherwise, it returns the specified environment.
281\item[push\_env,pop\_env] These methods respectively push a new environment onto the environment stack and pop the current environment from the stack.
282\item[services] This method returns the plugin services in the current environment (or the named environment if one is specified).
283\item[load\_services] Load services using IPluginLoader extension points.
284\item[pprint] This method provides a text summary of the registered interfaces, plugins and services.
285\item[clear] This method empties the environment stack and defines a new default environment. This setup then bootstraps the configuration of the \code{pyutilib.plugin.core} environment. Note that this does not clear the plugin registry; in practice that may not make sense since it is not easy to reload modules in Python.
286\end{description}
287
288
289\section{A Simple Example}
290
291Figure~\ref{fig:example1} provides a simple example that is adapted from the description of the Trac component architecture~\cite{TCA}. This example illustrates the three main steps to setting up a plugin:
292\begin{enumerate}
293\item Defining an interface
294\item Declaring extension points
295\item Defining classes that implement the interface.
296\end{enumerate}
297In this example, a singleton plugin is declared, which automatically registers the plugin service. Non-singleton plugin services need to explicitly created, but they are also automatically registered.
298
299If the script in Figure~\ref{fig:example1} 
300is in the \code{todo.py} file, then the following Python script
301illustrates how this plugin is used:
302\begin{lstlisting}
303from todo import *
304
305# Construct a TodoList object and then add several items.
306todo_list = TodoList()
307todo_list.add('Make coffee', 'Really need to make some coffee')
308todo_list.add('Bug triage',
309        'Double-check that all known issues were addressed')
310\end{lstlisting}
311This script generates the following output:
312\begin{lstlisting}
313Task: Make coffee
314      Really need to make some coffee
315Task: Bug triage
316      Double-check that all known issues were addressed
317\end{lstlisting}
318
319
320\begin{figure}
321
322\begin{lstlisting}
323# A simple example that manages a TODO list.  An observer
324# interface is used to add actions that occur when a TODO
325# item is added.
326from pyutilib.plugin.core import *
327
328# An interface class that defines the API for plugins that
329# observe when a TODO item is added.
330class ITodoObserver(Interface):
331
332        def todo_added(name, description):
333            """Called when a to-do item is added."""
334
335# The TODO application, which declares an extension point
336# for observers.  Observers are notified when a new TODO
337# item is added to the TODO list.
338class TodoList(object):
339        observers = ExtensionPoint(ITodoObserver)
340
341        def __init__(self):
342            """
343            The TodoList constructor, which initializes the list
344            """
345            self.todos = {}
346
347        def add(self, name, description):
348            """Add a TODO, and notify the observers"""
349            assert not name in self.todos, 'To-do already in list'
350            self.todos[name] = description
351            for observer in self.observers:
352                observer.todo_added(name, description)
353
354# A plugin that prints information about TODO items when they
355# are added.
356class TodoPrinter(SingletonPlugin):
357        implements(ITodoObserver)
358
359        def todo_added(self, name, description):
360            print 'Task:', name
361            print '     ', description
362\end{lstlisting}
363
364\caption{\label{fig:example1} A simple example of the Python Component Architecture}
365\end{figure}
366
367
368\section{Plugin Implementations}
369
370In addition to the core plugin framework, PCA includes implementations
371for a variety of plugins that support commonly used functionality. The
372following sections briefly describe these plugins.
373
374\subsection{Options and Configuration Files}
375
376The \code{pyutilib.plugin.config} package defines interfaces and plugins
377for managing service options. The \code{Configuration} service is used to manage
378the global configuration of all services. This class coordinates with
379\code{Option} services. Plugins can declare options with the \code{declare\_option}
380method, which registers these options with the \code{Configuration} service. This
381service reads and writes options to configuration files (using Python's
382\code{ConfigParser} package).
383
384This package also declares the \code{ManagedPlugin} and \code{ManagedSingletonPlugin}
385classes, which are plugin base classes that include options that can be
386used to enable or disable services using the \code{Configuration} service. In
387practice, most plugins will be derived from these plugin base classes.
388
389\subsection{Plugin Loaders}
390
391PCA plugins can be loaded from either Python modules or Python eggs. This
392capability supports the runtime extension of the plugins framework,
393which has proven very powerful in frameworks like Trac. The core plugin
394framework defines extension points that use these loaders, which can be
395applied as follows:
396\begin{lstlisting}
397import sys
398import os
399env = sys.environ["PATH"]
400PluginGlobals.load_services(path=env.split(os.sep))
401\end{lstlisting}
402In this example, the user's \code{PATH} environment is used to define the list
403of directories that are searched for Python modules and eggs.
404
405\subsection{Registering Executables}
406
407The \code{ExternalExecutable} plugin is used to define services that provide
408information about external executables. Services declare the executable
409name and user documentation, and then service methods indicate whether
410the executable is enabled (i.e. whether it is found, and the path of
411the executable:
412\begin{lstlisting}
413service = ExternalExecutable(name='ls',
414            doc='A utility to list file in Unix operating systems')
415
416service.enabled()     
417# Returns True if the executable is found on the user path.
418
419service.get_path()
420# Returns a string that defines the path to this executable,
421# or None if service is disabled.
422\end{lstlisting}
423
424
425\section{Related Frameworks}
426
427The general design of PCA is adapted from Trac~\cite{Trac}. The PCA
428generalizes the Trac component architecture by supporting namespace
429management of plugins, as well as non-singleton plugins. For those
430familiar with Trac, the following classes roughly correspond with
431each other:
432\begin{table}
433\begin{center}
434\begin{tabular}{|l|l|} \hline
435    Trac Class Name &   PyUtilib Class Name \\ \hline
436    Interface   & Interface \\
437    ExtensionPoint      & ExtensionPoint \\
438    Component   & SingletonPlugin \\
439    ComponentManager    & PluginEnvironment \\ \hline
440\end{tabular}
441\end{center}
442\end{table}
443The \code{PluginEnvironment} class is used to manage plugins, but unlike Trac this class does not need to be explicitly constructed.
444
445As we noted earlier, there are a variety of mature component architectures
446that support plugins.  The following requirements were motivated by our
447plugin use cases, which ultimately led to the development of PCA:
448\begin{itemize}
449
450\item \textit{Well-defined framework core}: Many component architectures
451are embedded in larger software frameworks, which makes it difficult
452to extract and use just the software packages related to the component
453architecture.
454
455\item \textit{Non-Singleton plugins}:  The computational science
456applications that motivate PCA require both singleton and non-singleton
457plugins.
458
459\item \textit{Namespaces}:  Using plugins in large software projects
460requires management across multiple libraries.  Namespaces are needed
461to effectively manage plugins in these complex software projects.
462
463\item \textit{Caching}:  PCA plugins need to be used in applications
464where plugin services are called many times.  Thus, caching of extension
465point setup is needed to minimize the overhead of the PCA infrastructure.
466
467\item \textit{Loading from EGGs}:  Support for loading EGG files is
468invaluable in dynamic applications.  Further, loading plugins from EGG
469files provides another level of modularity to the management of software
470applications.
471
472\end{itemize}
473
474
Note: See TracBrowser for help on using the repository browser.