|
|
|
|
|
======================
|
|
|
|
|
|
Python Source Reader
|
|
|
|
|
|
======================
|
|
|
|
|
|
:Author: David Goodger
|
|
|
|
|
|
:Contact: docutils-develop@lists.sourceforge.net
|
|
|
|
|
|
:Revision: $Revision: 7302 $
|
|
|
|
|
|
:Date: $Date: 2012-01-03 20:23:53 +0100 (Di, 03 Jan 2012) $
|
|
|
|
|
|
:Copyright: This document has been placed in the public domain.
|
|
|
|
|
|
|
|
|
|
|
|
This document explores issues around extracting and processing
|
|
|
|
|
|
docstrings from Python modules.
|
|
|
|
|
|
|
|
|
|
|
|
For definitive element hierarchy details, see the "Python Plaintext
|
|
|
|
|
|
Document Interface DTD" XML document type definition, pysource.dtd_
|
|
|
|
|
|
(which modifies the generic docutils.dtd_). Descriptions below list
|
|
|
|
|
|
'DTD elements' (XML 'generic identifiers' or tag names) corresponding
|
|
|
|
|
|
to syntax constructs.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. contents::
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Model
|
|
|
|
|
|
=====
|
|
|
|
|
|
|
|
|
|
|
|
The Python Source Reader ("PySource") model that's evolving in my mind
|
|
|
|
|
|
goes something like this:
|
|
|
|
|
|
|
|
|
|
|
|
1. Extract the docstring/namespace [#]_ tree from the module(s) and/or
|
|
|
|
|
|
package(s).
|
|
|
|
|
|
|
|
|
|
|
|
.. [#] See `Docstring Extractor`_ below.
|
|
|
|
|
|
|
|
|
|
|
|
2. Run the parser on each docstring in turn, producing a forest of
|
|
|
|
|
|
doctrees (per nodes.py).
|
|
|
|
|
|
|
|
|
|
|
|
3. Join the docstring trees together into a single tree, running
|
|
|
|
|
|
transforms:
|
|
|
|
|
|
|
|
|
|
|
|
- merge hyperlinks
|
|
|
|
|
|
- merge namespaces
|
|
|
|
|
|
- create various sections like "Module Attributes", "Functions",
|
|
|
|
|
|
"Classes", "Class Attributes", etc.; see pysource.dtd_
|
|
|
|
|
|
- convert the above special sections to ordinary doctree nodes
|
|
|
|
|
|
|
|
|
|
|
|
4. Run transforms on the combined doctree. Examples: resolving
|
|
|
|
|
|
cross-references/hyperlinks (including interpreted text on Python
|
|
|
|
|
|
identifiers); footnote auto-numbering; first field list ->
|
|
|
|
|
|
bibliographic elements.
|
|
|
|
|
|
|
|
|
|
|
|
(Or should step 4's transforms come before step 3?)
|
|
|
|
|
|
|
|
|
|
|
|
5. Pass the resulting unified tree to the writer/builder.
|
|
|
|
|
|
|
|
|
|
|
|
I've had trouble reconciling the roles of input parser and output
|
|
|
|
|
|
writer with the idea of modes ("readers" or "directors"). Does the
|
|
|
|
|
|
mode govern the tranformation of the input, the output, or both?
|
|
|
|
|
|
Perhaps the mode should be split into two.
|
|
|
|
|
|
|
|
|
|
|
|
For example, say the source of our input is a Python module. Our
|
|
|
|
|
|
"input mode" should be the "Python Source Reader". It discovers (from
|
|
|
|
|
|
``__docformat__``) that the input parser is "reStructuredText". If we
|
|
|
|
|
|
want HTML, we'll specify the "HTML" output formatter. But there's a
|
|
|
|
|
|
piece missing. What *kind* or *style* of HTML output do we want?
|
|
|
|
|
|
PyDoc-style, LibRefMan style, etc. (many people will want to specify
|
|
|
|
|
|
and control their own style). Is the output style specific to a
|
|
|
|
|
|
particular output format (XML, HTML, etc.)? Is the style specific to
|
|
|
|
|
|
the input mode? Or can/should they be independent?
|
|
|
|
|
|
|
|
|
|
|
|
I envision interaction between the input parser, an "input mode" , and
|
|
|
|
|
|
the output formatter. The same intermediate data format would be used
|
|
|
|
|
|
between each of these, being transformed as it progresses.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Docstring Extractor
|
|
|
|
|
|
===================
|
|
|
|
|
|
|
|
|
|
|
|
We need code that scans a parsed Python module, and returns an ordered
|
|
|
|
|
|
tree containing the names, docstrings (including attribute and
|
|
|
|
|
|
additional docstrings), and additional info (in parentheses below) of
|
|
|
|
|
|
all of the following objects:
|
|
|
|
|
|
|
|
|
|
|
|
- packages
|
|
|
|
|
|
- modules
|
|
|
|
|
|
- module attributes (+ values)
|
|
|
|
|
|
- classes (+ inheritance)
|
|
|
|
|
|
- class attributes (+ values)
|
|
|
|
|
|
- instance attributes (+ values)
|
|
|
|
|
|
- methods (+ formal parameters & defaults)
|
|
|
|
|
|
- functions (+ formal parameters & defaults)
|
|
|
|
|
|
|
|
|
|
|
|
(Extract comments too? For example, comments at the start of a module
|
|
|
|
|
|
would be a good place for bibliographic field lists.)
|
|
|
|
|
|
|
|
|
|
|
|
In order to evaluate interpreted text cross-references, namespaces for
|
|
|
|
|
|
each of the above will also be required.
|
|
|
|
|
|
|
|
|
|
|
|
See python-dev/docstring-develop thread "AST mining", started on
|
|
|
|
|
|
2001-08-14.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Interpreted Text
|
|
|
|
|
|
================
|
|
|
|
|
|
|
|
|
|
|
|
DTD elements: package, module, class, method, function,
|
|
|
|
|
|
module_attribute, class_attribute, instance_attribute, variable,
|
|
|
|
|
|
parameter, type, exception_class, warning_class.
|
|
|
|
|
|
|
|
|
|
|
|
To classify identifiers explicitly, the role is given along with the
|
|
|
|
|
|
identifier in either prefix or suffix form::
|
|
|
|
|
|
|
|
|
|
|
|
Use :method:`Keeper.storedata` to store the object's data in
|
|
|
|
|
|
`Keeper.data`:instance_attribute:.
|
|
|
|
|
|
|
|
|
|
|
|
The role may be one of 'package', 'module', 'class', 'method',
|
|
|
|
|
|
'function', 'module_attribute', 'class_attribute',
|
|
|
|
|
|
'instance_attribute', 'variable', 'parameter', 'type',
|
|
|
|
|
|
'exception_class', 'exception', 'warning_class', or 'warning'. Other
|
|
|
|
|
|
roles may be defined.
|
|
|
|
|
|
|
|
|
|
|
|
.. _pysource.dtd: pysource.dtd
|
|
|
|
|
|
.. _docutils.dtd: ../ref/docutils.dtd
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
..
|
|
|
|
|
|
Local Variables:
|
|
|
|
|
|
mode: indented-text
|
|
|
|
|
|
indent-tabs-mode: nil
|
|
|
|
|
|
fill-column: 70
|
|
|
|
|
|
End:
|
