MoinMoin is a popular and powerful wiki engine in Python.

The talk will give an introduction to the MoinMoin core source code, extension concepts and extension development. We will use the current development branch (1.6) to do this, but most stuff also applies to 1.5 (production releases).

Shortly, we will present the flexibility of MoinMoin and how the user base (Python, Apache, Ubuntu, ...) looks like.

We will give an architecture overview as well as explain some code and show how you can write your own.

Slideshow

MoinMoin - Introduction for developers

http://moinmoin.wikiwikiweb.de/EuroPython2006

Held by

Thomas Waldmann

Alexander Schremmer

Introduction

• MoinMoin was started as a one-man project by Jürgen Hermann

  • Currently maintained by the MoinCoreTeamGroup

  • Many contributions in form of patches, translations and documentation by our users
• Large user base

  • Grew significantly after introduction of internal unicode processing and again after the introduction of the MoinMoin Desktop Edition (MMDE)

Some historical data

_{Data was generated by sloccount}

User base

There are various groups using the MoinMoin wiki. For example:

• Python on wiki.python.org, more than 1800 pages
• The Apache Foundation for more than 50 wikis

• Ubuntu/Canonical is running most web sites (besides Launchpad etc.) on MoinMoin, even http://www.ubuntu.com/

How development is done / Communication

• IRC (Freenode)

  • #moin for user support

  • #moin-dev for core development

• Wikis

  • http://moinmoin.wikiwikiweb.de/ for general stuff

  • http://moinmaster.wikiwikiweb.de/ for i18n and documentation

• Mailinglist for users

• Repository runs on Mercurial (http://hg.thinkmo.de/ )

  • Cf. the talk "Achieving High Performance In Mercurial" held by Bryan O'Sullivan on Tuesday morning
• PEP8 coding style

  • No, we don't use "hungarian notation".

_T

Architecture Overview

MoinMoin distribution code is some (more or less modular) core code plus a set of (built-in) plugins.

Plugins

MoinMoin uses a lot of plugin types:

action

Process the request, "controller"

formatter

Define the target format (HTML, Docbook, XML, ...)

parser

Parse parts of the page (calls the formatter)

macro

Calls to Python code in the page

xmlrpc

XMLRPC plugins can be easily called remotely

filter

Make the fulltext search engine understand various attachment types

theme

Change the skin of the wiki

Other modular stuff

No plugins, but meant to be extended by using the clear interfaces.

• User authentication (LDAP, HTTP basic auth, session cookie, PHP session)

• Generic script plugin mechanism, only 1 moin shell command needed

• Converter from HTML markup (GUI editor) to e.g. wiki markup
• Request classes used by different servers (CGI, WSGI, FCGI, mod_python, Twisted, ...)
• Security policies (antispam and autoadmin currently)

Request classes

• A request object is constructed for every request
• Request class handles API differences, bugs and features of the servers
• Offers read() and write() functions for form data (later in request.form) and output to the browser
• Supported server protocols: FastCGI, Twisted, Standalone, WSGI

Page/PageEditor

• Represent pages (Page: read-only, PageEditor: read-write)

• Refactoring due

• Either use these classes, make XMLRPC calls or use PackageInstaller to create/modify pages

Directly accessing the file system is not necessarily upwards compatible.

_A

User object

• The current user is request.user

• Some attributes:
  • user.valid (True if user has logged in, False for anon users)
  • user.name
  • user.email
  • user.may.someVerb(pagename) - check for read/write/delete/... permissions

action Plugins

• The action controls the request

• Examples: show, diff, DeletePage, sitemap, ...

• Steps done for http://example.com/WikiSandBox?action=diff&rev1=23&rev2=42

  • Check user's access
  • Get parameters from the URL
  • Find the page revisions to compare
  • Call base class in order to emit the usual http headers
  • Call theme code to send the title area
  • Output the diff
  • Call theme code to send the footer area
• Control what will be done and how complete output will look like.

parser / formatter Plugins

• Page rendering

  • Normal wiki page has the mimetype text/moin-wiki

  • HTML output we usually want is text/html

  • Moin looks up the mimetype and loads the correct parser (e.g. MoinMoin.parser.text_moin_wiki) to parse the wiki markup

  • Normally, the formatter MoinMoin.formatter.text_html is loaded

  • Parser calls the formatter for every markup bit it finds
• Formatter has some generic functions (underlined, bold, make a paragraph, a headline, ...)
• Parsers included: text_rst, text/plain, text_docbook, text_csv, text_python (syntax highlighting), ...
• Formatters included: text_plain, text_docbook, text_python, ...

parser / formatter Plugins

• Example: "This is __underlined__."

  This is             # seen normal text: parser calls formatter.text
  <span class="u">    # seen __ markup: parser's _u_repl calls
                      # formatter.underline and toggles parser.is_u
  underlined          # seen normal text: parser calls formatter.text
  </span">            # seen __ markup: parser's _u_repl calls 
                      # formatter.underline again and toggles parser.is_u
  .                   # seen normal text: parser calls formatter.text

• text_python is different
  • Generates Python code that contains calls to the dynamic macros/parsers
  • Static macros are inserted as constants
  • Is compiled into a cache file
  • On page display, just the bytecode file is executed

parser / formatter Plugins

Syntax sample:

{‌{{#!rst
This is *reStructured text*!

*Really*
}‌}}

So keep in mind:

• Parsers process some lines (or even a whole page) of input data
• Parsers call some formatter to output data
• Formatters translate calls by the parser to some specific output format

_T

macro Plugins

• Used for content that can be generated with the given parameter string on just one line

• The usual syntax is [[MacroName(arg1, arg2, ...)]].

• Example: [[BR]]

     1 # BR.py
     2 def execute(macro, args):
     3     return macro.formatter.linebreak(0)
  

  • macro is the macro object, has attributes like macro.request and macro.formatter

  • args is the unprocessed argument string

macro plugins

• There are also more complex macros, like RecentChanges, WantedPages or MonthCalendar

So keep in mind that macros:

• Get a few parameters
• Render their output within a wiki page

theme Plugins

• Themes customise the way MoinMoin looks like

• Consist of:

  • theme/<themename>.py - some python code

  • wiki/<themename>/css/*.css - style sheets (colours, fonts, alignment, ...)

  • wiki/<themename>/img/*.png - icons

• E.g. there is a theme which lets MoinMoin look like Mediawiki

• Writing themes is not easy if you want to get usability, I18n and browser compatiblity right

xmlrpc Plugins

MoinMoin implements Wiki XMLRPC:

• Functions: get and put pages, get page infos, get a page list, etc.
• API v1 (the official stuff, ASCII!)
• API v2 (transmitting utf-8)
• Plus custom plugin XMLRPC code.

Usages include:

• Automatic distribution of the BadContent page that keeps the anti-spam patterns.

• Generation of group definition pages by a script.
• Wikisync will use XMLRPC as well.

_A

filter Plugins

Filters are simple plugins that help the moin search indexer to index different types of documents:

• Filter input: filename of data file
• Filter returns: file content as unicode object (doesn't need to be pretty)

We currently have filters for:

• MS Word and Excel (uses antiword and xls2csv)
• PDF (uses pdftotext)

• OpenOffice.org / Open Document XML formats (pure Python)

• JPEG EXIF header (pure Python)

• strings like fallback filter (pure Python)

auth Modules

MoinMoin uses a configurable list of auth methods and calls one auth method after the other at the beginning of request processing.

An auth module gets this as input:

• whether login or logout is happening
  • login: user name and password
  • logout or normal page request: nothing else
• user_obj from the previous auth method (or None, if there is none yet)
• request object

And returns:

• user_obj (or None)
• flag about whether to:
  • continue with auth methods (usual case)
  • or terminate ("no, never, no chance for this guy")

converter Modules

Currently there is only one converter module: text_html_text_moin_wiki.

Used together with FCKEditor:

• FCKeditor gets fed with HTML (using text_gedit HTML formatter)
• User edits the HTML (WYSIWYG)
• Converter yields wiki markup, Moin stores it

_T

Future

Moin2 / Google Summer of Code

• Three projects sponsored by Google's Summer of Code

• Next long-term goal: MoinMoin 2.0

• Expected improvements in MoinMoin 1.6:

  • Refactoring
  • Storage API
  • Advanced xapian search
  • Wiki sync

Refactoring

We want a more OO-like design of MoinMoin core:

• exploit inheritance and generalization
• make it cleaner and simpler
• have less code

Refactoring - MimeType approach

We will not have pages and attachments any more, but a hierarchy of mime-type item classes:

• One mimetype will be text/x-moin-wiki, that will replace what a page is now.
• A sub-item of mimetype image/jpeg is what an jpeg attachment is now.
• Attachments will get revisioning for free.
• Pages get up/download for free.
• Each item will have meta data (mimetype, language, etc. is stored there).
• Rendering (and other actions) will depend on the mimetype.

Storage

• We will have a storage backend API with backend plugins:
  • old storage (1.3/1.5 style) - mostly for converting existing data.

  • a new backend will support new MoinMoin 2 features.

• Implementation of Mercurial/SVN/RDBMS/etc. storage would be possible

Search

We will have a better search using the Xapian search engine:

• indexing all mimetype items supported by filters
  • items means pages + attachments + maybe even filesystem
• making use of meta data available (language, mimetypes, ...)
• having an improved search UI exploiting Xapian's features
• (will replace Lupy search code)

_A

Wiki Synchronisation

• Synchronise different wiki sites:
  • Changes will be distributed automatically,
  • concurrently changed pages will be merged.
• Use cases:
  • Keep sometimes offline notebook wiki up-to-date with some master server.
  • Mirror another site and allow the visitor to choose which site he prefers.
  • Implement load balancing by replicating the wiki contents to a few servers.

Your contribution

• Help us by:
  • contributing code, patches and bug fixes
  • translations and documentation
  • ideas with good plans
• Beginners can start with a simple plugin

• Visit our sprint here at Europython from June 6th to June 8th.

Thanks for your attention!

---

Removed

A walk through the code (optional)

• Show some macros and action
• Write some easy macro or action? E.g. we write some color macro to make coloured text ...

Core Code

Stuff like Page, PageEditor, wikiutil, wikiacl, user, etc. (most stuff located directly under the MoinMoin module) is currently considered core code, as well as some packages distributed within the distribution archive.

Other modules

i18n

translations of the system texts to different languages (we use gettext to make *.mo from *.po files) and some i18n related tools

logfile

access the log files (writing entries, parsing entries, ...)

mail

some mail processing (notify as well as mail import)

stats

making access statistics, text or graphics

support

some stuff we include for convenience, like fixed versions of Python stdlib modules, support for cgi tracebacks or Xapian wrappers

wikiutil

As the name suggests, this is a collection of misc. utility functions, which are used all over the place:

• quoting functions for URL and FS
• escaping functions
• query string parsing and generation