source: coopr.opt/trunk/doc/opt/plugins-pca.tex @ 1828

Last change on this file since 1828 was 1828, checked in by wehart, 10 years ago

Update of documenation to account for PyUtilib? changes.

File size: 18.4 KB
Line 
1
2\chapter{PyUtilib Component Architecture}
3
4\label{chap:pca}
5
6The PyUtilib Component Architecture (PCA) is an extension of the Trac
7plugin framework~\cite{Trac} that is included in the PyUtilib software
8package~\cite{PyUtilib}.  There does not appear to be a standard
9Python plugin framework, though there are some mature packages that
10support plugins including Zope~\cite{Zope}, Envisage~\cite{Envisage},
11Trac~\cite{Trac}, yapsy~\cite{yapsy} and SprinklesPy~\cite{SprinklesPy}.
12Although we discuss the design requirements for PCA later, PCA was
13initially motivated by the desire to use Trac's plugin framework within
14a self-contained packages.  The core of PCA is provided by PyUtilib's
15\code{pyutilib.plugin.core} module.
16
17\section{Overview}
18
19The PCA is comprised of a small set of Python classes. A \textit{plugin}
20is a class that implements a set of related methods the context of the
21application, and a \textit{service} is an instance of a plugin class. Two
22different types of plugins are available: singleton and non-singleton
23plugins. There is at most one service for a singleton plugin, whereas
24there can be multiple services of non-singleton plugins.
25
26A software application can declare \textit{extension points} that other
27components can \textit{plug in} to. This mechanisms supports a flexible,
28modular programming paradigm that enables software applications can
29be extended in a dynamic manner. Extension points and the extensions
30contributed to them are stored in a global registry, and execution of
31these extensions is handled in a standardized manner. Thus, an application
32developer can define extension points without knowing how they will be
33implemented, and extension developers can register extensions without
34needing to know the details of how they are employed.
35
36Extension points are defined with respect to an \textit{interface} class
37that implicitly defines the registration type that is used for a
38plugin. Further, a plugin class includes a declaration that it implements
39one-or-more interfaces. An interface is defined by the methods and data
40that are used. However, the PCA does not enforce
41this interface or support interface conversions (see Zope~\cite{Zope}
42and Envisage~\cite{Envisage} for examples of plugin frameworks that
43support this functionality).
44
45
46\section{Core Plugin Classes}
47
48The PyUtilib plugin framework consists of the following core classes:
49\begin{description}
50\item[pyutilib.plugin.core.Interface] Subclasses of this class declare plugin interfaces that are registered in the framework
51\item[pyutilib.plugin.core.ExtensionPoint] A class used to declare extension points, which can access services with a particular interface
52\item[pyutilib.plugin.core.Plugin] Subclasses of this class declare plugins, which can be used to implement interfaces within this plugin framework.
53\item[pyutilib.plugin.core.SingletonPlugin] Subclasses of this class declare singleton plugins, for which a single service is constructed.
54\item[pyutilib.plugin.core.PluginEnvironment] A class that maintains the registries for interfaces, extension points, plugins and services.
55\item[pyutilib.plugin.core.PluginGlobals] A class that maintains global data concerning the set of environments that are currently being used.
56\item[pyutilib.plugin.core.PluginError] The exception class that is raised when errors arise in this framework.
57\end{description}
58
59\subsection{Interfaces and Extension Points}
60
61A subclass of the \code{Interface} class is used to declare extension
62points in an application. The \code{ExtensionPoint} class is used to declare
63an extension point and to retrieve information about the plugins that
64implement the specified interface. For example, the following is a
65minimal declaration of an interface and extension point:
66\begin{lstlisting}
67class MyInterface(Interface):
68   """An interface subclass"""
69
70ep = ExtensionPoint(MyInterface)
71\end{lstlisting}
72Note that the \code{MyInterface} class is not required to define the API of the
73interface. The PCA does not enforce checking of
74API conformance for plugins, and hence any declaration in the \code{MyInterface}
75class would be ignored. Additionally, note that an instance of \code{MyInterface}
76is not created; declaring the plugin interface simply requires the
77specification of a subclass of \code{Interface}.
78
79An instance of \code{ExtensionPoint} can be used to iterate through
80all extensions, or to search for an extension that matches a particular
81keyword. For example, the following code iterates through all extensions
82and applies the \code{pprint} method:
83\begin{lstlisting}
84for service in ep:
85  service.pprint()
86\end{lstlisting}
87If you wish to know the number of services that are registered with an
88extension point, you can call the standard \code{len} function:
89\begin{lstlisting}
90print len(ep)
91\end{lstlisting}
92Several other methods can be used to more carefully select services from
93an extension point. The \code{extensions} method returns a Python \code{set} object
94that contains the services:
95\begin{lstlisting}
96#
97# This loop iterates over all services, just the same as when an
98# the iterator method is used (see above).
99#
100for service in ep.extensions():
101  service.pprint()
102
103#
104# This loop iterates over all servicess, including the 'disabled'
105# services.
106#
107for service in ep.extensions(all=True):
108  service.pprint()
109\end{lstlisting}
110This example illustrates the optional argument that indicates whether
111the set returned by \code{extensions} includes all disabled services. It is
112convenient to explicitly support enabling and disabling services in many
113applications, though services are enabled by default. Disabled services
114remain in the registry, but by default they are not included in the set
115returned by an extension point.
116
117Finally, the PCA can support \textit{named services}, which requires that
118the services have a \code{name} attribute. Service names are not required
119to be unique. For example, when multiple instances of a non-singleton
120plugin are created, then these services can be accessed as follows:
121\begin{lstlisting}
122#
123# A simple plugin that implements the MyInterface interface
124#
125class MyPlugin(Plugin):
126  implements(MyInterface)
127
128  def __init__(self):
129      self.name="myname"
130
131#
132# Another simple plugin that implements the MyInterface interface
133#
134class MyOtherPlugin(Plugin):
135  implements(MyInterface)
136
137  def __init__(self):
138      self.name="myothername"
139
140#
141# Constructing services
142#
143service1 = MyPlugin()
144service2 = MyPlugin()
145service3 = MyOtherPlugin()
146
147#
148# A function that iterates over all MyInterface services, and
149# returns the MyPlugin instances (which are service1 and service2).
150#
151def get_services():
152    ep = ExtensionPoint(MyInterface)
153    return ep("myname")
154\end{lstlisting}
155In some applications, there is a one-to-one correspondence between service
156names and their instances. In this context, a simpler syntax is to use
157the \code{service} method:
158\begin{lstlisting}
159class MySingletonPlugin(SingletonPlugin):
160  implements(MyInterface)
161
162  def __init__(self):
163      self.name="mysingletonname"
164
165ep = ExtensionPoint(MyInterface)
166ep.service("mysingletonname").pprint()
167\end{lstlisting}
168The \code{service} method raises a \code{PluginError} if there is more than one service
169with a given name. Note, however, that this method returns \code{None} if no
170service has been registered with the specified name.
171
172Note that an integer cannot be used to select a service from an extension
173point. Services are not registered in an indexable array, so this option
174does not make sense.
175
176
177\subsection{Plugins and Environments}
178
179PCA plugins are subclasses of either the \code{Plugin} or \code{SingletonPlugin}
180classes. Subclasses of \code{Plugin} need to be explicitly constructed,
181but otherwise they do not need to be registered; simply constructing a
182subclass of \code{Plugin} invokes the registration of that instance. Similarly,
183simply declaring a subclass of \code{SingletonPlugin} invokes both the construction
184and registration of this plugin service.
185
186PCA plugins are registered with different interfaces using the \code{implements}
187function (which is a static method of \code{Plugin}). Note that a plugin can be
188registered with more than one interface. Plugins are applied to different
189extension points independently, but they can maintain state information
190that impacts their use across different extension points.
191
192The plugin and interfaces are organized within namespaces using the
193\code{PluginEnvironment} class. A global registry of plugin environments
194is maintained by the \code{PluginGlobals} class. This registry is a stack
195of environments, and the top of this stack defines the current
196environment. When an interface is declared, its namespace is the name
197of the current environment. For example:
198\begin{lstlisting}
199#
200# Declare an interface in the current environment
201#
202class Interface1(Interface):
203   pass
204
205#
206# Set the current environment to 'new_environ'
207#
208PluginGlobals.push_env( "new_environ" )
209
210#
211# Declare an interface in the 'new_environ' environment
212#
213
214class Interface2(Interface):
215   pass
216
217#
218# Go back to the original environment
219#
220PluginGlobals.pop_env()
221\end{lstlisting}
222The namespace that an \code{Interface} subclass is declared in defines the
223namespace where plugin services will be registered. Additionally,
224a plugin service will be registered in the namespace where it is
225declared. For example:
226\begin{lstlisting}
227#
228# Declare Interface1 in namespace env1
229#
230PluginGlobals.push_env("env1")
231
232class Interface1(Interface):
233   pass
234
235#
236# Declare Interface2 in namespace env2
237#
238PluginGlobals.push_env("env2")
239
240class Interface2(Interface):
241   pass
242
243PluginGlobals.pop_env()
244
245#
246# Declare Plugin1 in namespace env3
247#
248PluginGlobals.push_env("env3")
249
250class Plugin1(Plugin):
251
252   implements(Interface1)
253   implements(Interface2)
254   implements(Interface1,"env4")
255
256PluginGlobals.pop_env()
257\end{lstlisting}
258When \code{Plugin1} is instantiated, it's services are registered in the
259following environments:
260\begin{lstlisting}
261    env1 for Interface1
262    env2 for Interface2
263    env4 for Interface1
264    env3
265\end{lstlisting}
266The last registration is the default. A plugin service is always
267registered in the environment in which it was declared.
268
269Namespaces provide a mechanism for organizing plugin services in an
270extensible manner. Applications can define new namespaces that contain
271their services without worrying about conflicts with services defined
272in other Python libraries.
273
274\subsection{Global Plugin Data}
275
276Global plugin data in PCA is managed in the \code{PluginGlobals} class. This
277class contains a variety of static methods that are used to access
278this data:
279\begin{description}
280\item[default\_env] This method returns the default environment, which is constructed when the plugins framework is loaded.
281\item[env] This method returns the current environment if no argument is specified. Otherwise, it returns the specified environment.
282\item[push\_env,pop\_env] These methods respectively push a new environment onto the environment stack and pop the current environment from the stack.
283\item[services] This method returns the plugin services in the current environment (or the named environment if one is specified).
284\item[load\_services] Load services using IPluginLoader extension points.
285\item[pprint] This method provides a text summary of the registered interfaces, plugins and services.
286\item[clear] This method empties the environment stack and defines a new default environment. This setup then bootstraps the configuration of the \code{pyutilib.plugin.core} environment. Note that this does not clear the plugin registry; in practice that may not make sense since it is not easy to reload modules in Python.
287\end{description}
288
289
290\section{A Simple Example}
291
292Figure~\ref{fig:example1} provides a simple example that is adapted from the description of the Trac component architecture~\cite{TCA}. This example illustrates the three main steps to setting up a plugin:
293\begin{enumerate}
294\item Defining an interface
295\item Declaring extension points
296\item Defining classes that implement the interface.
297\end{enumerate}
298In this example, a singleton plugin is declared, which automatically registers the plugin service. Non-singleton plugin services need to explicitly created, but they are also automatically registered.
299
300If the script in Figure~\ref{fig:example1} 
301is in the \code{todo.py} file, then the following Python script
302illustrates how this plugin is used:
303\begin{lstlisting}
304from todo import *
305
306# Construct a TodoList object and then add several items.
307todo_list = TodoList()
308todo_list.add('Make coffee', 'Really need to make some coffee')
309todo_list.add('Bug triage',
310        'Double-check that all known issues were addressed')
311\end{lstlisting}
312This script generates the following output:
313\begin{lstlisting}
314Task: Make coffee
315      Really need to make some coffee
316Task: Bug triage
317      Double-check that all known issues were addressed
318\end{lstlisting}
319
320
321\begin{figure}
322
323\begin{lstlisting}
324# A simple example that manages a TODO list.  An observer
325# interface is used to add actions that occur when a TODO
326# item is added.
327from pyutilib.plugin.core import *
328
329# An interface class that defines the API for plugins that
330# observe when a TODO item is added.
331class ITodoObserver(Interface):
332
333        def todo_added(name, description):
334            """Called when a to-do item is added."""
335
336# The TODO application, which declares an extension point
337# for observers.  Observers are notified when a new TODO
338# item is added to the TODO list.
339class TodoList(object):
340        observers = ExtensionPoint(ITodoObserver)
341
342        def __init__(self):
343            """
344            The TodoList constructor, which initializes the list
345            """
346            self.todos = {}
347
348        def add(self, name, description):
349            """Add a TODO, and notify the observers"""
350            assert not name in self.todos, 'To-do already in list'
351            self.todos[name] = description
352            for observer in self.observers:
353                observer.todo_added(name, description)
354
355# A plugin that prints information about TODO items when they
356# are added.
357class TodoPrinter(SingletonPlugin):
358        implements(ITodoObserver)
359
360        def todo_added(self, name, description):
361            print 'Task:', name
362            print '     ', description
363\end{lstlisting}
364
365\caption{\label{fig:example1} A simple example of the Python Component Architecture}
366\end{figure}
367
368
369\section{Plugin Implementations}
370
371In addition to the core plugin framework, PCA includes implementations
372for a variety of plugins that support commonly used functionality. The
373following sections briefly describe these plugins.
374
375\subsection{Options and Configuration Files}
376
377The \code{pyutilib.plugin.config} package defines interfaces and plugins
378for managing service options. The \code{Configuration} service is used to manage
379the global configuration of all services. This class coordinates with
380\code{Option} services. Plugins can declare options with the \code{declare\_option}
381method, which registers these options with the \code{Configuration} service. This
382service reads and writes options to configuration files (using Python's
383\code{ConfigParser} package).
384
385This package also declares the \code{ManagedPlugin} and \code{ManagedSingletonPlugin}
386classes, which are plugin base classes that include options that can be
387used to enable or disable services using the \code{Configuration} service. In
388practice, most plugins will be derived from these plugin base classes.
389
390\subsection{Plugin Loaders}
391
392PCA plugins can be loaded from either Python modules or Python eggs. This
393capability supports the runtime extension of the plugins framework,
394which has proven very powerful in frameworks like Trac. The core plugin
395framework defines extension points that use these loaders, which can be
396applied as follows:
397\begin{lstlisting}
398import sys
399import os
400env = sys.environ["PATH"]
401PluginGlobals.load_services(path=env.split(os.sep))
402\end{lstlisting}
403In this example, the user's \code{PATH} environment is used to define the list
404of directories that are searched for Python modules and eggs.
405
406\subsection{Registering Executables}
407
408The \code{ExternalExecutable} plugin is used to define services that provide
409information about external executables. Services declare the executable
410name and user documentation, and then service methods indicate whether
411the executable is enabled (i.e. whether it is found, and the path of
412the executable:
413\begin{lstlisting}
414service = ExternalExecutable(name='ls',
415            doc='A utility to list file in Unix operating systems')
416
417service.enabled()     
418# Returns True if the executable is found on the user path.
419
420service.get_path()
421# Returns a string that defines the path to this executable,
422# or None if service is disabled.
423\end{lstlisting}
424
425
426\section{Related Frameworks}
427
428The general design of PCA is adapted from Trac~\cite{Trac}. The PCA
429generalizes the Trac component architecture by supporting namespace
430management of plugins, as well as non-singleton plugins. For those
431familiar with Trac, the following classes roughly correspond with
432each other:
433\begin{table}
434\begin{center}
435\begin{tabular}{|l|l|} \hline
436    Trac Class Name &   PyUtilib Class Name \\ \hline
437    Interface   & Interface \\
438    ExtensionPoint      & ExtensionPoint \\
439    Component   & SingletonPlugin \\
440    ComponentManager    & PluginEnvironment \\ \hline
441\end{tabular}
442\end{center}
443\end{table}
444The \code{PluginEnvironment} class is used to manage plugins, but unlike Trac this class does not need to be explicitly constructed.
445
446As we noted earlier, there are a variety of mature component architectures
447that support plugins.  The following requirements were motivated by our
448plugin use cases, which ultimately led to the development of PCA:
449\begin{itemize}
450
451\item \textit{Well-defined framework core}: Many component architectures
452are embedded in larger software frameworks, which makes it difficult
453to extract and use just the software packages related to the component
454architecture.
455
456\item \textit{Non-Singleton plugins}:  The computational science
457applications that motivate PCA require both singleton and non-singleton
458plugins.
459
460\item \textit{Namespaces}:  Using plugins in large software projects
461requires management across multiple libraries.  Namespaces are needed
462to effectively manage plugins in these complex software projects.
463
464\item \textit{Caching}:  PCA plugins need to be used in applications
465where plugin services are called many times.  Thus, caching of extension
466point setup is needed to minimize the overhead of the PCA infrastructure.
467
468\item \textit{Loading from EGGs}:  Support for loading EGG files is
469invaluable in dynamic applications.  Further, loading plugins from EGG
470files provides another level of modularity to the management of software
471applications.
472
473\end{itemize}
474
475
Note: See TracBrowser for help on using the repository browser.