1.1 Why GNUN Is Being Developed

The GNU Project’s website, has become considerably large over the years. Maintaining it requires significant effort, and sometimes a new web standard is developed faster than the time required to migrate all articles to the next widely adopted one.

When it comes to internationalization, the problems are so many that it is hard to enumerate them. It has become apparent that maintaining translations up-to-date is a major undertaking, involving tedious skimming through commit logs, reviewing diffs and other medieval techniques to catch up. Some translation teams have developed their own sets of scripts, but so far there has been no universal solution.

This unpleasant situation, combined with rapid and incompatible design changes, have led some teams to neglect the important work of keeping their translations in line with the changing original articles. As a consequence, the GNU Project is facing the problem of maintaining them in suboptimal ways, in order to keep the information updated.

The reasons for developing GNUnited Nations are very similar to those that led to the inception of GNU gettext, or GNOME Documentation Utilities (gnome-doc-utils) some years later.

1.2 What GNUnited Nations Is and Should Be

The basic concept behind GNUN is that localization of HTML articles is similar to localization of computer programs¹. In articles, like in programs, not every string is considered translatable, so translatable strings must be identified first, and then collected in a file (called “PO template”) for translation. Articles, like programs, tend to change over time, but not every change in the sources calls for a translation update. Sometimes the change does not affect the translatable strings, but sometimes it does. So, translators must have means to identify those changes and apply the appropriate updates to the translation.

The GNU gettext package already provides the needed infrastructure for maintaining translations using PO files. See Introduction in GNU gettext tools, for a basic overview. GNUnited Nations fills the gaps to apply this infrastructure to articles in the https://www.gnu.org web site.²

The following diagram summarizes the relation between the files handled by GNUN. It is followed by somewhat detailed explanations, which you should read while keeping an eye on the diagram. Having a clear understanding of these interrelations will surely help translators and web maintainers.

.---<--- * Original ARTICLE.html
|
|   .---> ARTICLE.pot ---> * ARTICLE.LANG.po --->---.
`---+                                               |
    `--->---.   .------<----------------------------'
            |   |
            |   `---.
            |       +---> Translated ARTICLE.LANG.html
            `-------'

The indication ‘*’ appears in two places in this picture, and means that the corresponding file is intended to be edited by humans. The author or web maintainer edits the original article.html, and translators edit article.lang.po. All other files are regenerated by GNUN and any manual changes on them will be lost on the next run.

Arrows denote dependency relation between files, where a change in one file will affect the other. Those automatic changes will be applied by running ‘make -C server/gnun’. This is the primary way to invoke GNUN, since it is implemented as a set of recipes for GNU make.

First, GNUN extracts all translatable strings from the original English article article.html into article.pot³. The resulting file is suitable for manipulation with the various GNU ‘gettext’ utilities. It contains all original article strings and all translations are set to empty. The letter t in .pot marks this as a Template PO file, not yet oriented towards any particular language.

On the first run, though, there is no article.lang.po yet, so a translator must create article.lang.po from article.pot, where lang represents the target language. See Starting a New Translation, for details.

Then comes the initial translation of messages in article.lang.po. Translation in itself is a whole matter, whose complexity far overwhelms the level of this manual. Nevertheless, a few hints are given in another chapter of this manual.

It is possible to make GNUN get translations for common strings from dedicated PO files, the so-called compendia. See Using Compendia, for more information.

You may use any compatible PO editor to add translated messages into the PO file. See Editing in GNU gettext tools, for more information.

When the PO file actually exists (hopefully populated with initial translations), GNUN generates article.lang.html file. It takes its structure from the original article.html, but all translatable strings are replaced with the translations specified in article.lang.po.

Original articles sometimes change. A new paragraph is being added or a tiny change in the wording is introduced. Also, some articles are dynamic in nature, like the ones containing news entries or a list of other articles. If the original article changes, GNUN will automatically rebuild article.pot, and will merge the changes into article.lang.po. Any outdated translations will be marked as “fuzzy,” any new strings will be added with empty translations, waiting to be translated. In the same run article.lang.html will be rebuilt, and consequently the relevant strings in the translation will be substituted with the original English text until the translation teams update them in article.lang.po.

Those changes in the original article that do not affect the translatable strings (or just delete whole strings) will not lead to new untranslated or “fuzzy” strings in article.lang.po. Thus, no actions from translators will be needed. article.lang.html will be automatically regenerated to reflect the changes.

The POT for every article under GNUN’s control is kept in the ‘www’ repository under a special sub-directory po/ of the relevant directory in the ‘www’ tree. Thus, for ‘https://www.gnu.org/philosophy/free-sw.html’ this sub-directory is philosophy/po/. In addition to free-sw.pot, this directory holds the canonical source of every translation, such as free-sw.bg.po, free-sw.ca.po, etc. For more details, see Files and Directories.

Several additional features are implemented, like automatic update of the list of the available translations. For example, if a new free-sw.ja.po translation is added, the list of translations included in free-sw.html and all translated free-sw.lang.html is updated. This saves a lot of tedious, repetitive work and eliminates a source of mistakes. There is a basic infrastructure to “inject” general information about a translation team—like a note on how to contact the team, or how to report a bug/suggestion for improvement. Translators’ credits are also handled, as well as translators’ notes, if any.

1.3 Major Advantages of GNUN

Here is a simple list of situations where we hope this suite would prove useful.

• Automatic rebuild of all translations when the original article changes. This is the most important feature, as it prevents accumulation of seriously outdated translations.
• Global update of the whole site. Apply the previous point to the web server templates (under server/ in the ‘www’ repository). A single change to such a file will affect literally all articles, translated or not.
• Urgent notices. Sometimes an “urgent” notice is added by the webmasters, which should appear on all pages. Typically this is about an event where urgent action is needed, although often it is only relevant to a single country or even a particular city. Such a notice will propagate to all pages, and translators may choose whether to translate it or not. For example, the Urdu translation team may conclude that there are only a few Urdu speakers in Massachusetts, to participate in an event that will happen in Boston, so translating the “urgent” notice may not be very “urgent” for Urdu. However, such notice will appear in all translated pages and people who usually read gnu.org pages in their native language will see it, so they can take action as necessary. When the notice is removed, often within a week or two, it will disappear without translators’ intervention, whether they translated it or not.
• HTML validation. As a preliminary step, GNUN validates the English pages before updating the POT files, and the regenerated translations before committing them. It often detects typos and other errors in the markup.
• Simplification of the translation process—lots of errors and typos come from the fact that translators basically have to duplicate the whole HTML markup of the original. The PO files eliminate most of the basic markup, which is where most of the validation errors come from.
• Markup consistency site-wide—it will be substantially easier to update the site to a future standard, because translations will naturally follow the changes in the original articles. This also means that translation teams do not have to go through the boring process of converting their articles to the new SSI-based layout; this will be done automatically.
• Easy updates by translators. Modified paragraphs, links, etc. will appear as “fuzzy” strings in the PO files, newly added ones will appear as “untranslated”, and deleted will appear as “obsolete”. It is substantially easier to update a PO file, where a keystroke takes you to the part that needs updating, whatever it may be.
• Reporting and statistics. Since the basis is standard PO files, which are the canonical source of the translations, it is easy to manipulate them and extract useful information.

1.4 Known Bugs and Limitations

As it happens in real life, we don’t wear pink glasses and are aware of certain limitations and annoyances of this semi-automatic system.

• There is no easy way to preview the translations. The official build is invoked twice an hour, because doing it much more often is not feasible with current build server hardware. Additionally, any errors interrupt the build so they have to be fixed before the next articles are processed.
• In very rare cases the structure of the translation should diverge from the structure of the original document. So far, our system has developed no means of taking such issues into account (apart from alphabetically sorted lists).

2.1 Invoking GNUN

The central part of GNUnited Nations is a makefile; actually a GNUmakefile since it heavily relies on features and extensions available in GNU Make. Thus, invoking a build consists of typing make on the command line, or within cron. If you are deploying the software on a non-GNU machine, probably GNU Make is installed and available as gmake or gnumake. If not, you should consider installing it, since the build will fail otherwise. See GNU Make homepage for information on how to download and install GNU Make.

If you don’t specify a target, make by default builds the target all, which in this case is to rebuild all translations that are not up-to-date. However, there are special targets that do not depend on the standard all target, which can be built by make target. Some of the variables in the next section apply to them, and some do not.

Note that GNUN expects GNUmakefile, config.mk and gnun.mk to be present under server/gnun of the ‘www’ web repository, otherwise make has no way to know what to build, and how. Another file, priorities.mk, is expected to be present under server/gnun in order to define the priorities when reporting about the outdated translations. If absent, the report target will not sort the translations by priority. Since the location of the repository working copy is strictly user-specific and cannot be determined in any way, GNUmakefile and config.mk must be copied there manually after the package installation. For convenience, these files are installed in ‘$(pkgdatadadir)’ (/usr/local/share/gnun with the default ‘prefix’) so you can just create symlinks pointing to them, e.g.:

ln -s /usr/local/share/gnun/config.mk /path/to/www/server/gnun/
ln -s /usr/local/share/gnun/GNUmakefile /path/to/www/server/gnun/

If next GNUN releases are installed with the same --prefix, you will always use the latest versions without the need for any manual intervention.

If you are playing with a non-gnu.org setup, you also have to take care of gnun.mk and put a modified version under server/gnun of your hypothetical tree. (For gnu.org, that is not necessary since a proper gnun.mk is maintained in ‘www’.)

2.2 Variables to Control the Build Process

The build process has several modes of operation, and they all relate to the handling of files that are to be added to the repository or performing certain sanity checks at build time. The variables are specified on the command line, after make, in the form VARIABLE=value, e.g. make VCS=yes. In the future, additional features will be implemented in a similar fashion.

‘VCS=no’

‘…’

Do not add any files to the repository. This is the default. You may as well omit to define VCS entirely; there is no special code that expects assigning the value ‘no’.

‘VCS=yes’

Automatically add any new files in the repository (CVS, Subversion or GNU Bazaar—the repository type is auto-determined at build time, bzr being a fallback).⁴ These are any POT files, if they are generated for the first time, and the translated articles (.lang.html) in HTML format. Finally, any missing PO and their HTML counterparts of the server templates will be added, computed on the basis of the extra-templates and optional-templates variables.

‘VALIDATE=no’

‘…’

Skips validation of the HTML articles and generated translations.

‘VALIDATE=yes’

Validates all original articles before generating the POTs, to ensure that the ultimate source is valid (X)HMTL. Also, validates all generated translations in HTML format and all PO files. It is highly recommended to run the build this way, even if it is a bit tedious to fix the errors that are reported as a result of enforcing validation.

This is the default, and not defining this variable has the same effect.

‘NOTIFY=no’

‘…’

Do not send email notifications about errors. This is the default.

‘NOTIFY=yes’

If an error occurs, send a mail with a meaningful subject and the error message as body to the concerned party. The variables devel-addr, web-addr and transl-addr control the recipients; normally they should be set to the GNUN maintainers, webmasters and translators accordingly. These variables are defined in the configuration file gnun.conf and by default are set to bug-gnun@gnu.org.

‘ANNOUNCE=yes’

If defined, automatic announcements for new translations will be sent to the address defined in the ann-addr variable (in gnun.conf). The email messages contain the translated article title as Subject, and the URL of the translation as its body. For the official GNUN build, they are delivered to the trans-coord-news@gnu.org mailing list and each language has its own Mailman topic. It is possible to subscribe to the list and receive only traffic related to a specific language. See Mailing Lists in GNU Web Translators Manual.

The default behavior is not to send such announcements.

‘VERBOSE=yes’

If defined, the value of the variables templates-translated, ALL_POTS, and articles-translated will be printed to the standard output. This is off by default, but recommended in general since it will show a bug in the computation of the basic variables.

‘GRACE’

If defined, ordinary articles that have “fuzzy” strings will not be regenerated. This functionality is implemented specifically to prevent gratuitous replacement of translated strings with the English text when there are only minor formatting changes in the original. The translator should review the changes, update the translation and clear the “fuzzy” mark from the strings, while keeping the online translation intact⁵. Note that this variable has no effect on the server templates and all articles defined in the variable no-grace-articles.

‘OUTDATED-GRACE=days’

Grace period for the out-of-date notice. When the variable GRACE is defined, OUTDATED-GRACE defaults to 60 days. The out-of-date notice is a special text (server/outdated.html in the ‘www’ repository) that is inserted into every outdated translation when the period defined in this variable is over; its purpose is to inform the reader that the translation may not correspond to the original English article.

‘TEAM=lang’

The translation team whose articles need to be checked for completeness. This variable is applicable only for the report target, and is mandatory for it. See The report Target.

When validation is enabled, the original English articles are validated first, before any commands that generate the other files, and make exits with an error on the first encountered article. This is done on purpose, to prevent the propagation of an eventual error in the markup of the original article to all translations.

Validation of the translated .lang.html is performed after it is preliminarily generated as a temporary file. When no errors are found, the translation is updated; otherwise the real file is not changed (and it is not added if absent)—the build will fail and further processing of the remaining articles will not be performed. The translator has time until the next run to fix the error—usually by modifying the corresponding .lang.po file.

If notification is enabled (NOTIFY=yes), and the build system encounters errors (mostly when validating articles), email messages will be sent to the party that is expected to fix the error. The subject of the messages always includes the problematic article, for example:

Subject: [GNUN Error] gnu/gnu.fa.html is not valid HTML

2.3 Targets Specified on the Command Line

Some targets are not built by default, because they are only useful under certain circumstances. Think of them like semi-automated commands or canned command sequences that are more complicated, and more importantly, whose arguments are variables computed at the time make reads the makefiles—the filesets they affect are specific and already defined, one way or another.

• The no-grace-items Target
• The update-localized-URLs Target
• The report Target
• The triggers Target
• The validate-all Target

make report TEAM=fr

25 4,12,20 * * *  cd $HOME/projects/www; cvs -q update &>/dev/null; \
                    make -j1 -C server/gnun VCS=yes VALIDATE=yes; \
                    NOTIFY=yes VERBOSE=yes GRACE=5000; ANNOUNCE=yes; \
                    cvs commit -m \
                    "Automatic update by GNUnited Nations."; \
                    make -C server/gnun triggers

2.4 Defining Articles to Be Built

The file gnun.mk contains variable definitions, based on which almost all other important variables are computed. In other words, the variables defined in that file directly affect the overall behavior of the build process.

There are two types of variables, which are specifically separated in order to make translators’ life easier: variables that translators are free to modify and variables that are modified by the web-translators⁷, ideally after performing some local tests. A translation team leader should update only FUZZY_DIFF_LINGUAS and TEMPLATE_LINGUAS; everything else is supposed to be built automagically, without manual intervention. If not, that is a bug that should be reported and fixed.

You can also request building files separately this way:

make -C server/gnun ../../philosophy/not-ipr.bg.html

‘TEMPLATE_LINGUAS’

A space-separated list with languages. Add here your language code if and only if you have all the SSI templates translated, and have already committed all template files listed in the extra-templates and localized-ssis variables in server/gnun/gnun.mk.

‘FUZZY_DIFF_LINGUAS’

Add your language code here if you want GNUN to add differences to “previous” msgids in your PO files. See The gnun-add-fuzzy-diff Script, for more information.

‘localized-ssis’

This variable lists the templates that are not under GNUN’s control and are translated manually, like

• - server/header.lang.html
• - server/head-include-1.lang.html
• - server/banner.lang.html
• - server/footer.lang.html.

They contain HTML code and SSI directives; PO4A doesn’t extract them to any messages in PO files.

‘extra-templates’

The extra-templates variable lists templates under GNUN control; they are rebuilt from corresponding template.lang.po files; when the template.lang.po file is absent, GNUN initializes and commits a file with empty msgstrs.

‘optional-templates’

The optional-templates variable defines optional templates under GNUN control. Those are the templates of low priority items, like news lines included in some pages. They are managed like the additional templates listed in the extra-templates variable, except

• - template.lang.po is not added when absent, and
• - their POTs end in .pot.opt rather than .pot.

This way, the scripts reporting outdated translations and translations that haven’t been converted to PO files won’t complain about them unless the team decides to actually commit template.lang.po.

‘sitemap’

The sitemap variable declares the pages that are treated like sitemaps, that is, an additional externally generated compendium is used when updating them. See Building Sitemap, for more information.

‘ROOT’

Add here articles that are in the server root, like home.html and keepingup.html. Always write only the basename of the article, i.e. if you add these two articles, the value of ROOT should be home keepingup. This is true for all the variables that expect values in the form of article names.

‘ALL_DIRS’

The list of directories containing articles, like philosophy, gnu, licenses, etc.

‘gnu’

‘philosophy’

‘…directory…’

A space-separated list of basenames for articles residing in directory, for which POTs will be generated and updated when the original article changes. If an article is missing here, GNUN doesn’t maintain its translations.

2.5 The languages.txt File

The file server/gnun/languages.txt is used when generating lists of translations; those lists are subsequently included in all translations of the article. Every line in the file is either a comment (when it begins with “#”) or a definition for a language. A language is defined by three TAB-separated fields. The first field defines the language suffix; it is used in file names and in texts of the links. The second field is the name of the language in English. The third field defines the name of the language in that language itself; it is used in texts of the links, and it should be in UTF-8. For example:

de	German	Deutsch

The generated list of translations may look like this:

<!-- begin translist file -->
<div id="translations">
<p>
<span class="original"><a href="/distros/screenshot.en.html">English</a>&nbsp;[en]</span>&nbsp;&nbsp;
<span><a href="/distros/screenshot.de.html">Deutsch</a>&nbsp;[de]</span>&nbsp;&nbsp;
<span><a href="/distros/screenshot.pl.html">polski</a>&nbsp;[pl]</span>&nbsp;&nbsp;
</p>
</div> <!-- id="translations" -->
<!-- end translist file -->

2.6 Using Compendia

Compendium is a PO file including translations for common strings; it is used to fill other PO files. See Using Translation Compendia in GNU gettext tools. One example of such common strings is the footer text about reporting bugs and sending inquiries: when a webmaster updates footer texts in an article, GNUN will use compendia to automatically fill the translations for the new version of the strings.

GNUN has two types of compendia:

1. Site-Wide Compendia This type is used for all articles, and only contains common strings.
2. Per-Directory Compedia These files accumulate all strings from all articles in the directory, even if these strings are not repeated in different articles.

• Site-Wide Compendia
• Per-Directory Templates

2.7 Building Sitemap

Sitemaps differ from regular pages in two respects:

1. They are typically regenerated automatically from the texts of the articles.
2. They mostly consist of links, therefore a significant part of their text should come to translations without any changes.

Also, sitemaps contain many translated messages, so it is desirable to eliminate manual work where possible.

In order to take this into account, GNUN provides the sitemap variable in gnun.mk. All articles listed in sitemap are rebuilt using an additional compendium (when present), server/gnun/compendia/sitemap-compendium.lang.po.

Those additional compendia are supposed to be compiled outside of GNUN by the same tool that generates the sitemap. It is the responsibility of that tool to provide the compendia with reasonable translations whose msgids match those generated by PO4A when processing the English version of the sitemap.

3.1 Adapting GNUmakefile.team for a Specific Team

To adjust the makefile for your team, you need to edit the next variables.

‘TEAM’

Set this to the language code, like bg or pt-br. It is autodetected if not defined.

‘wwwdir’

The relative or absolute path to the working copy of the master ‘www’ repository. So if you have checked out your project’s Sources repository at ~/projects/www-lang and the ‘www’ Web repository at ~/projects/www, the value of wwwdir should be ../www/ or /home/user/projects/www/. Note the slash at the end, it is important.

‘TEAM_URL_PREFIX’

‘TEAM_URL_POSTFIX’

These variables are used to transform the path to a file in the team’s working copy into a URL for this file; they are only needed for automatic email notifications. It is expected that typically the URL will point to some sort of Web interface for the repository, for example,

TEAM_URL_PREFIX := http://cvs.savannah.gnu.org/viewvc/*checkout*/www-bg/
TEAM_URL_POSTFIX := ?root=www-bg

If two variants of one language share the same project and repository (such as zh-cn and zh-tw), they should maintain two directories with two GNUmakefiles and each directory having its own tree.

Some variables are specified on the command line, and alter the behavior of the build process.

‘ADD_FUZZY_DIFF’

The program to manage differences to “previous” msgids added to “fuzzy” translations when merging PO files. When undefined, the makefile tries to find gnun-add-fuzzy-diff and assigns the result to this variable. See The gnun-add-fuzzy-diff Script, for more information.

Define to empty string to disable the feature.

‘NOTIFICATION_PERIOD=days’ ¶

Default period of reminders sent by the notify target.

‘NOTIFY=yes’

Make the notify target actually send notifications.

‘NOTIFY=no’

Do not send email notifications about errors. This is the default.

‘VERBOSE=yes’

Print more information from cvs, svn and msgmerge; off by default. Note that VERBOSE can be defined to any string, it will have the same effect.

‘VCS=yes’

Update both ‘www’ and ‘www-lang’ repositories, then commit the merged PO files in the latter repository. By default, there is no VCS interaction. The VCS of the translation project repository is determined automatically; currently only CVS, Subversion, GNU Bzr, Git, Mercurial (Hg) and GNU Arch repositories are supported.

Caution: The makefile rule will commit all local changes, not only those that resulted from running msgmerge. Thus, it is better to use a separate working copy dedicated solely for this purpose.

3.2 Targets in GNUmakefile.team

update-team

Updates the working copy of team’s repository.

update-www

Updates the working copy of www.

update

A shortcut for making update-team and update-www.

sync

Merges all available PO files from the corresponding POT in www. If a POT is missing in the master repository (usually, because it was deleted when the original article was either split, renamed or deleted), a warning is printed for the corresponding file and no merging occurs for it.

report

Verifies which translations are complete, and prints a list (with statistics) of those that need to be updated. Also, it reports the translations that are not consistent with the revisions from www repository.

More fine-grained notifications are implemented as the notify target.

notify

Invokes the report target, filters its output for different translators and sends them notifications.

As an addition to the output of the report target, URLs for relevant files are written; also, when www and team revisions are not consistent, their differences generated with gnun-diff-po are optionally attached (see The gnun-diff-po Script, for more details).

The list of files to report against are configured with nottab file. The lines of this file that begin with ‘#’ are ignored; other lines should contain two fields separated by ‘:’.

The first field is an extended grep regular expression; the files are selected for the report if their names match this expression. See Regular Expressions in GNU Grep manual.

The second field is a space-separated list of single-word translators’ identifiers for which the line applies. The file may look like this:

^gnu/gnu-history: alice bob carol
^distros/(common-distros|distros|screenshot): dan alice
^(licenses|philosophy)/: eve bob frank
^licenses/: dan

Email addresses of the translators are kept in a separate file, email-aliases. Like in nottab, the lines beginning with ‘#’ are ignored; other lines contain fields separated by ‘:’.

The first field is the translator’s identifier, the second is space-separated list of translator’s email addresses. These two fields are mandatory.

The third field is the personal reminder period in days. After that period the message is sent again even though its contents don’t change. Otherwise, the notify target only sends messages when something changes. Naturally, no message is sent when no action is needed in the requested set of translations (unless the force option is used).

The team-wide default period is defined in its GNUmakefile.team using the NOTIFICATION_PERIOD variable.

The fourth field provides comma-separated options for the report. Currently recognized options are:

force

Forces sending messages even when there are no files to work on.

no-diffs

Disables sending www vs. www-lang differences as attachments.

www

Report cases that suggest actions by the team leader, that is, when the translation in the team repository is complete whereas in www it is incomplete or absent.

This is an example of email-aliases:

alice:alice@example.com abc@test.net
# Disable `dan' temporarily.
#dan:miller@test.by:5:no-diffs,www
bob:brh@invalid.tr:3
# Note that the third field is empty; the default value for period
# will be used.
carol:carol@localhost.br translator@example.bg::no-diffs
eve:eve@test.gr:2:www
frank:gnun-notifications@example.gr:1:www,no-diffs

The email-aliases file should be kept privately because it may contain personal email addresses of people. How to do this is out of scope of GNUN and this manual.

format

A convenience rule to re-wrap all files up to the standard length. Many PO editors leave the msgstr as a single long line after it has been edited, but GNUN will automatically re-wrap the file to the standard line length when it is processed. Wrapping long lines in PO files is a good practice as it avoids unnecessary revisions.

This rule filters all translations with msgcat; basically, it reformats the lines that are longer than the default page width of msgcat.

publish

The publish rule’s task is to copy all modified files to the official www repository. It depends on the format target to ensure that all files are re-wrapped to the standard line length limit, but deliberately does not depend on sync VCS=yes. Usually, one would run make publish when one or a bunch of PO files are in a satisfactory condition to be published, and this rule is just a convenience to avoid multiple manual cp invocations. As a rule of thumb, before running this rule it is sane to run sync and correct any “fuzzy” messages and other problems, if necessary.

The publish rule does not depend on sync explicitly, because that would be a nuisance for offline operations and basically unnecessary when the committer is fairly confident that there are no changes to (re-)merge. A hard dependency on sync would slow down the operation considerably.

As usual, when committing to the official repository, it is always a good practice to examine the output of cvs diff.

Invoking make publish prints warnings and does not actually copy the affected file if the sub-directory in “www” is non-existent or the corresponding .pot is missing.

Typically, after an editing session (whether it involves actual editing or just merging contributions from team members), one would do:

$ make sync VCS=yes
$ make publish
$ cd wwwdir
$ cvs up
(Add all new translations, if any.)
  $ cvs add file …
$ cvs diff
$ cvs commit -m "Some descriptive message."

clean

Deletes all backup and auto-generated files.

make VCS=yes is the recommended command to be run periodically. To check the status of the translations, run make report.

Feel free to replace all strings with equivalents in your native language and of course—do not hesitate to extend this file and modify it as much as you like. If you come up with something interesting, it would be nice to send a message to bug-gnun@gnu.org, so that GNUmakefile.team gets updated for all teams’ benefit.

3.3 Automatic Synchronization and Status Reports

It is convenient to invoke such synchronization automatically, for example once every day. If you have enabled commit notifications for the project’s repository, any new changes will be visible for subscribers. Here is an example crontab entry:

# m h  dom mon dow   command
@daily              cd $HOME/projects/www-lang; make VCS=yes

The job doesn’t have to run on the team leader’s machine, since all team members have write access to their project repository.

If desired, you could set up another job to report the status of the translations weekly or fortnightly, for example:

# m h  dom mon dow   command
@weekly             cd $HOME/projects/www-lang; \
                       make report | mail -s "Weekly statistics" \
                       www-lang-list@gnu.org

Caution: Most cron implementations do not allow the character ‘\’ as a line continuation character—the example shown is made that way for better readability.

4.1 Starting a New Translation

To start a new translation, the most simple way is to copy the existing POT as article.lang.po, where lang is your language code. The name of the POT is article.pot or, when you are translating an optional template, article.pot.opt (see optional-templates). All these files are situated in the po/ subdirectory of the directory containing the HTML file of the article. See Files and Directories, for more info on GNUN directory layout. For example, to prepare for a new translation of the essay https://www.gnu.org/philosophy/free-sw.html in Manx, you can simply cd philosophy/po; cp free-sw.pot free-sw.gv.po and then edit the latter. If free-sw.pot does not exist it is because either the article is not yet “templated” (i.e. migrated to the new style), or the GNUN maintainers have not yet added it to the value of the appropriate variable in server/gnun/gnun.mk. In that case, just ask them to do what’s needed for the POT to be generated.

You could also use the msginit utility that would populate the PO file header with the right information, provided your environment is set up correctly. See msginit Invocation in GNU gettext tools.

GNUN also provides a customized script to automatically fill more header fields. See The gnun-init-po Script.

4.2 Filling the PO File Header

The PO file header as generated usually looks like this:

# SOME DESCRIPTIVE TITLE
# Copyright (C) YEAR Free Software Foundation, Inc.
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"POT-Creation-Date: 2008-02-06 16:25-0500\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=CHARSET\n"
"Content-Transfer-Encoding: ENCODING\n"

You have to edit the header to match the already established conventions, and the rules for gnu.org translations. For reference, here is a list with all fields explained:

‘Project-Id-Version’

Add here the filename of the original article, without the sub-directory, like “body-include-1.html” or “free-sw.html”.

‘POT-Creation-Date’

Do not edit this field, it is already set when the POT is created.

‘PO-Revision-Date’

Likewise, do not edit. This field is automatically filled in when you save the file with any decent PO editor.

‘Last-Translator’

The name and email address of the last translator who has edited the translation. Pay attention that normally this is the name of a member of your team, it can be the translation team leader if he/she was the person who updated the translation. For example:

Elvis Parsley <king@grassland.com>

‘Language-Team’

This field should contain the mailing list on which the translation team can be reached—usually www-LANG-…@gnu.org. Example:

Czech <www-cs-general@gnu.org>

‘MIME-Version’

Leave it like it is.

‘Content-Type’

Usually this is text/plain; charset=UTF-8; change the charset accordingly.

‘Content-Transfer-Encoding’

Set this to 8bit. Note that the PO file header ends with this field, and it should contain a newline (‘\n’). Unfortunately, some PO editors remove the newline, which causes an unnecessary revision when the file is automatically modified by GNUN’s rules.

Here is an example of a properly edited header:

# Bulgarian translation of https://www.gnu.org/philosophy/free-sw.html
# Copyright (C) 2008 Free Software Foundation, Inc.
# This file is distributed under the same license as the gnu.org article.
# Yavor Doganov <yavor@gnu.org>, 2008.
#
msgid ""
msgstr ""
"Project-Id-Version: free-sw.html\n"
"POT-Creation-Date: 2008-02-06 16:25-0500\n"
"PO-Revision-Date: 2008-02-09 15:23+0200\n"
"Last-Translator: Yavor Doganov <yavor@gnu.org>\n"
"Language-Team: Bulgarian <www-bg-list@gnu.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"

Notice the absence of the “fuzzy” marker; you should “unfuzzy” the header after entering the necessary information (this is done by simply pressing TAB in PO mode).

It is recommended that you wrap all lines in the comments to be less than 80 characters; that looks better from a purely aesthetic point of view and improves the performance of GNUmakefile.team’s publish rule (see publish).

4.3 Special GNUN messages

There are some special messages that appear in the POT and PO:

‘*GNUN-SLOT: TRANSLATOR'S NOTES*’

This is for translator’s notes that are injected in the resulting translation. See The Special Slot for Translator’s Notes, for more information. If your translation does not have notes, you must translate this as a space:

msgid "*GNUN-SLOT: TRANSLATOR'S NOTES*"
msgstr " "

‘*GNUN-SLOT: TRANSLATOR'S CREDITS*’

This is again optional, and should contain the name (and address) of the person who made the translation. “Translate” this string as a space if you do not want your name to appear there. See The Special Slot for Translator’s Credits.

• The Special Slot for Translator’s Notes
• The Special Slot for Translator’s Credits

# type: Content of: <p>
msgid ""
"To understand the concept, you should think of <q>free</q> "
"as in <q>free speech,</q> not as in <q>free beer.</q>"
msgstr ""
"Translated message, where you want to clarify beer<sup><a "
"href=\"#TransNote1\">1</a></sup>, presumably because the "
"expression in your language is different"
…
…
# type: Content of: <div>
#. TRANSLATORS: Use space (SPC) as msgstr if you don't have notes.
msgid "*GNUN-SLOT: TRANSLATOR'S NOTES*"
msgstr ""
"<h3>Translator's notes</h3>\n"
"<ol>\n"
"<li id=\"TransNote1\">Note clarifying the text.</li>\n"
"</ol>\n"

<b>Traduction</b>: Benjamin Drieu 
<a href="mailto:foo@example.org">&lt;foo@example.org&gt;</a>,
2007, 2008.

4.4 Wrapping Long Lines

Most of the PO editors do not wrap long lines that inevitably appear in msgstrs. If that happens, long lines make reading subsequent diffs harder, and are generally annoying for most people. If this issue bothers you, you can “normalize” the already finished PO translation by executing on the command line msgcat -o file.po file.po, before installing it in the repository. Either way, the build system will treat it as a valid PO file.

For those lucky Emacs users, here is a code snippet that you can put in your .emacs; doing M-x po-wrap while in PO mode will wrap all long lines:

(defun po-wrap ()
  "Filter current po-mode buffer through `msgcat' tool to wrap all lines."
  (interactive)
  (if (eq major-mode 'po-mode)
      (let ((tmp-file (make-temp-file "po-wrap."))
	    (tmp-buf (generate-new-buffer "*temp*")))
	(unwind-protect
	    (progn
	      (write-region (point-min) (point-max) tmp-file nil 1)
	      (if (zerop
		   (call-process
		    "msgcat" nil tmp-buf t (shell-quote-argument tmp-file)))
		  (let ((saved (point))
			(inhibit-read-only t))
		    (delete-region (point-min) (point-max))
		    (insert-buffer tmp-buf)
		    (goto-char (min saved (point-max))))
		(with-current-buffer tmp-buf
		  (error (buffer-string)))))
	  (kill-buffer tmp-buf)
	  (delete-file tmp-file)))))

It is highly desirable that you check if the PO file you finished translating (or editing) is valid, before committing it. This is done by running msgfmt -cv -o /dev/null file or by simply pressing V in PO mode. The build system automatically verifies each PO file when invoked with VALIDATE=yes, but you won’t get a warm and fuzzy feeling if a stupid typo you made halts the whole update of all translations. Such things happen to everyone, so it is a good practice to check before you actually commit.

4.5 Useful Hints for Editing PO Files

This section contains additional explanations, some in the form of advice and recommendations; not all of them are strictly related to PO files editing.

• When you install a new translation of an article (that is different from a server template), all you need to do is to add your PO file in the appropriate /po sub-directory.

  In the next build, your article.lang.html will be built and the link to it will be added to the list of translations and propagate to all translations, provided that they are under GNUN’s control.

• If you don’t feel comfortable editing gnun.mk, do not worry. Someone from the GNUN maintainers will notice and will amend TEMPLATE_LINGUAS for you, as appropriate.
• Dealing with obsolete strings. Elements which are removed from the original articles appear in the PO files as “obsolete” strings—the translation is not lost, but they are marked in a special way at the end of the PO file. You don’t have to update a PO file if it contains obsolete strings—do this only if it has “fuzzy” or “untranslated”, and of course when you want to improve the existing translated ones. Sometimes these obsolete strings are useful, and they can save time. For example, if you anticipate that the deleted text may reappear some time in the future, you can preserve the string and hopefully it would be marked as “fuzzy” when this happens. Failing that, you can still copy it and yank it at the appropriate place.
• You can add comments to every message in a PO file—for example if you want to remember that you have to do something, or to remind you why this particular message is translated in a special way. These comments do not appear in the generated HTML source.
• Sometimes, especially when the original message contains many links, it is easier to copy it to msgstr and edit the latter by translating the English text. In PO mode of Emacs, this is done by C-j. This is useful also for large chunks of text in <pre> elements, which normally you would want to preserve verbatim.
• Do not use HTML entities for non-ASCII letters as in the English original. They are harder to type and read. So, if there is ‘ü’ and this is a character from the alphabet of your language, just write it as ‘ü’ directly.
• Wrapping of msgstr using M-q in Emacs (or other means) is considered harmful. It is best to leave GNUN (or more precisely, Po4a) do the wrapping—that way all generated HTML translations will have predictable results. This will help tremendously for the conversion to other formats, like Texinfo. Also, note that not all elements are wrapped by default, so deliberately wrapping the text inside the msgstr could lead to an invalid page or a page that is valid, but is rendered incorrectly by the web browser.

4.6 Transforming Existing Translation in PO Format

Migrating an existing translation to a PO file format is basically editing the header as described in the previous section, and populating each of the messages by copying the already translated text and markup from the existing translation in HTML format in the relevant message.

Typically, you will visit po/foo.lang.po (in PO mode) and foo.lang.html (in HTML mode) in another buffer. Then you can copy a paragraph or an element from the latter and yank it in the relevant message in the former. Be extra careful, since this is the time to check precisely that the translation corresponds to the original. Further changes will be reflected, but if your “initial” PO file is not a 100% match, that would not necessarily mean that it is an improvement. Since it is very easy to do this kind of check, because the relevant msgid and msgstr appear one above the other in the same buffer (or the similar concept in other PO editors), please do perform this initial sanity check even if you are confident that the translation you have been yanking strings from is a completely up-to-date translation.

There is also a semi-automatic way to produce an initial PO file. You checkout the revision of the English page, foo.html, that corresponds to the latest revision of the translation, foo.lang.html. Then you run gnun-preconvert (see The gnun-preconvert Script):

gnun-preconvert foo.lang.html foo.html

If some passages in foo.lang.html don’t match the structure of foo.html, error messages will be displayed. Check them, adjust the files and try again. When you succeed, the result will be written to foo.lang.po. After that, run gnun-merge-preconverted (see The gnun-merge-preconverted Script):

gnun-merge-preconverted -C compendium.lang.po foo.lang.po foo.pot

If you have no compendium, just omit the “-C compendium.lang.po” part.

You get foo.lang.po where all messages are marked as “fuzzy” (unless you use a compendium); you still should make sure that the translations correspond to the original and remove those “fuzzy” marks. The script adds differences against “previous” msgids to facilitate checking.

There is no need to delete the existing HTML translation, GNUN will automatically overwrite it. The only thing the translator should do is commit the PO file to the repository.

When an essay has been translated by several people through the years, it is important that this information is recorded and reflected in the PO file. In the future, special targets may be added to enable the FSF to check who translated a particular article, and when.

A recommended way to do this is as follows:

# French translation of https://www.gnu.org/philosophy/bsd.html
# Copyright (C) 2006, 2007, 2008 Free Software Foundation, Inc.
# This file is distributed under the same license as the gnu.org article.
# Cédric Corazza <cedric.corazza@wanadoo.fr>, 2006, 2008.
# Jerôme Dominguez <taz@gnu.org>, 2007.

In this example, it is clear that Cédric made the initial translation, Jerôme made some changes in 2007, and the original translator returned in 2008 and continued maintaining it.

5.1 Modifying Templates

Any significant structural diversion from boilerplate.html in a specific article may result in errors from GNUN. Any untested intrusive updates to the server templates (such as changing the entire look & feel of the site) will probably break GNUN build process. Of course, this does not mean that no changes should happen—only that they must be applied in a sandbox first, to ensure a smooth transition.

The script gnun-validate-html is useful for webmasters who want to verify if their (potentially intrusive) changes result in a valid markup. Before committing your changes, you can check whether the file is valid by running

gnun-validate-html --root . philosophy/not-ipr.html

See The gnun-validate-html Script, for more information.

5.2 Comments for Translators

If you want a comment to be visible for translators, place it inside the element, for example:

<p>
<!--TRANSLATORS: Note that foo is bar in this context.-->
The fooish bar mumbles bazzling.
</p>

This will result in:

# type: Content of: <p>
#. TRANSLATORS: Note that foo is bar in this context.
msgid "The fooish bar mumbles bazzling."
msgstr ""

As per the established convention, start the comment with TRANSLATORS: to catch their attention, and do not add a space after the beginning of the HTML comment (<!--), since this will unnecessarily indent the comment in the POT.

5.3 Localized URLs

Some articles and templates may contain diagrams or other images with English texts that call for translations. In order to make the translated versions appear in the respective translations, add a HTML comment like this to the English page:

<!-- GNUN: localize URL /philosophy/category.png,
 /licenses/template-diagram.png and /graphics/jesus-cartoon.jpg -->

The URLs are separated with spaces. One trailing comma at the end of every word is removed if present. Words without a dot, such as ‘and’, do not count as URLs; they are ignored.

Such comments will be extracted nightly and compiled into per-article lists of URLs in localized-urls.mk.

At the end of every build GNUN will check if the respective philosophy/category.lang.png and other files are present in the working copy; then GNUN will substitute the references in the HTML file of the translation.

GNUN relies on URLs being absolute, starting from the root directory as suggested in the FSF HTML Style Sheet Guidelines.

Please be sure to also commit the images in their source form (typically, in SVG format). It’s important both for the English image so that the translators could derive their localized versions from it, and for the translations so that people could maintain them.

5.4 Splitting Long Passages

GNUN splits typical text by paragraphs, and translators write the translations for the resulting parts of the text one by one. When a part is too long, it is hard to translate and proofread the translation; the likelihood of skipping some phrases is relatively high.

In order to make our translators’ life easier, it is desirable to keep the paragraphs short (no more than 350–700 characters). If rearranging the paragraphs is not an option, you can use a conventional separator, <span class="gnun-split"></span>:

<p>GNUN splits typical text by paragraphs, and translators write
the translations for the resulting parts of the text
one by one. <span class="gnun-split"></span> When a part
is too long, it is hard to translate and proofread the translation;
the likelihood of skipping some phrases is relatively high.</p>

It is important that the separator be inserted between complete sentences, because different languages may require different orders of parts of the sentence, and the translator has no control over the sequence of the strings in the translation.

5.5 Sorting Lists

Sometimes a list is sorted alphabetically to make search easier for the visitors, and this property should be preserved in translations. GNUN has specific marks for such lists.

<!-- GNUN-SORT-START -->

This HTML comment marks the begin of the list.

<!-- GNUN-SORT-STOP -->

This comment marks the end of the list.

<!-- GNUN-SORT-NEXT-ITEM -->

This mark separates list items.

The next two marks are optional, they delimit the key to sort items.

<!-- GNUN-SORT-BEGIN-KEY -->

<!-- GNUN-SORT-END-KEY -->

When extracting the keys, all HTML tags are removed, leading and trailing whitespace is trimmed.

Since the items are reordered in translations, webmasters should carefully choose places for the separators: reordering shouldn’t break the page.

The translators can modify the sorting order using HTML comments if needed. For example, let the English text say

<!-- GNUN-SORT-BEGIN-KEY -->
<h3>Afrikaans</h3>
<!-- GNUN-SORT-END-KEY -->

In the PO file, this transforms to an msgstr like

Afrikaans

The translator inserts a comment in the translation:

<!-- boer --> afrikaans

Now the item sorts like ‘boer afrikaans’.

Additional care should be taken when the items have a separator like

<!-- GNUN-SORT-START -->
A |
<!-- GNUN-SORT-NEXT-ITEM -->
B |
<!-- GNUN-SORT-NEXT-ITEM -->
...
<!-- GNUN-SORT-NEXT-ITEM -->
Y |
<!-- GNUN-SORT-NEXT-ITEM -->
Z
<!-- GNUN-SORT-STOP -->

Note that the last item has no separator (‘|’). When translating, make sure that the item to be last in your translation has no separator, and all other items have it.

6.1 Files and Directories

This is a brief diagram of www working copy; note that in this section we don’t describe the files out of GNUN control, including the old HTML-only translations.

Legend:
*   The file is maintained locally, without committing it to CVS.
#   The file is edited manually.
@   The file is regenerated automatically.
%   The file is edited both manually and automatically.
#/@ The file is generated automatically, but outside of GNUN.

Root/
  |
  +--proprietary/
  |   |
  |   +--proprietary.html #
  |   +--proprietary.lang.html @
  |   +--malware-cars.html #
  |   +--malware-cars.lang.html @
  |   +--…
  |   |
  |   +--------------------------po/
  |                               |
  +--home.html #                  +--malware-cars.lang.po %
  +--home.lang.html @             +--malware-cars.pot @
  +--planetfeeds.html #/@         +--malware-cars.proto * @
  +--planetfeeds.lang.html @      +--malware-cars.translist @
  +--po/                          |
  |  |                            +--malware-cars.lang-en.html @
  |  +--home.pot @                +--malware-cars.lang-diff.html @
  |  +--home.proto * @            +--proprietary.lang.po %
  |  +--home.translist @          +--pot @
  |  +--home.lang.po %            +--lang.po %
  |  +--home.lang-en.html @       +--…
  |  +--home.lang-diff.html @
  |  +--planetfeeds.proto * @
  |  +--planetfeeds.pot.opt @
  |  +--planetfeeds.lang.po %
  |  +--…
  |
  +--server/
  |  |
  |  +-------------------------gnun/
  |  |                           |
  |  +--sitemap.html #/@         +--GNUmakefile * #
  |  +--sitemap.lang.html @      +--config.mk * #
  |  +--po/                      +--gnun.mk #
  |  |  |                        +--priorities.mk #
  |  |  +--sitemap.pot @         +--languages.txt #
  |  |  +--sitemap.proto * @     +--compendia/
  |  |  +--sitemap.lang.po %          |
  |  |  |                             +--master.lang.po #
  |  |  +--body-include-2.pot @       +--compendium.lang.po @
  |  |  +--body-include-2.proto * @   +--compendium.pot @
  |  |  +--body-include-2.lang.po %   +--excluded.pot #
  |  |  +--…
  |  |  
  |  +--body-include-2.html #
  |  +--body-include-2.lang.html @
  |  +--…
  +--…

A description of some files follows.

…/file.html

The English files.

…/file.lang.html

The lang translation of …/file.html.

…/po/file.proto

Intermediate files generated from file.html; they are used only internally. Essentially, GNUN inserts its special slots. See Special GNUN messages, for more details.

…/po/file.translist

The list of translations for …/file.html. It is included in …/file.html and all its translations.

…/po/file.pot

PO templates for articles and included files.

…/po/file.pot.opt

POTs of optional templates. See optional-templates, for more details.

…/po/file.lang.po

Translations of file.pot or file.pot.opt committed by the translators and merged by GNUN.

…/po/file.lang-en.html

The latest revision of the English file for which a full translation into lang language was provided.

…/po/file.lang-diff.html

The highlighted differences between the current version of …file.html and …/po/file.lang-en.html.

…/po/pot

Per-directory template. It contains all strings from all articles in this directory. See Per-Directory Templates.

…/po/lang.po

Per-directory compendium. It is only used and regenerated when …/po/pot exists. See Per-Directory Templates.

planetfeeds.html

A list of news; it is automatically generated outside of GNUN. For GNUN, it is an optional template.

server/

Most template SSI files are located in this directory.

server/sitemap.html

The sitemap; it is generated automatically by a script that is not included in GNUN. See Building Sitemap.

server/gnun/

The directory for GNUN files.

server/gnun/GNUmakefile

server/gnun/config.mk

These files are maintained locally. They are used to rebuild the translations. See Invoking GNUN.

server/gnun/gnun.mk

The file that defines the list of files to build. See Defining Articles to Be Built.

server/gnun/priorities.mk

This file defines the list of translation priorities. See priorities.mk.

server/gnun/languages.txt

Canonical names for languages. See The languages.txt File.

server/gnun/compendia

The directory for compendia. See Using Compendia.

6.2 GNUN Scripts

For the time being there are several helper scripts. Some of them are used internally as commands with certain arguments in the makefile rules; other scripts were written specifically to facilitate some mechanical actions, like initially filling the headers in the PO files. They all can be invoked separately, as stand-alone programs.

• The gnun-add-fuzzy-diff Script
• The gnun-diff-po Script
• The gnun-init-po Script
• The gnun-link-diff Script
• The gnun-po4a-translate Script
• The gnun-preconvert Script
• The gnun-merge-preconverted Script
• The gnun-report Script
• The gnun-validate-html Script
• The mailfail Script
• The validate-html-notify Script

gnun-add-fuzzy-diff [OPTION...] [FILE]

gnun-diff-po [OPTION...] PO_FILE1 PO_FILE2 > diff.html

gnun-init-po [OPTION...] POT

gnun-link-diff [OPTION...] FILE

gnun-preconvert [OPTION...] TRANSLATION MASTER

gnun-merge-preconverted [OPTION...] PO POT

gnun-report --root=../www -t bg > report-bg.html

gnun-validate-html --root . philosophy/free-sw.html

mailfail [--dry-run] RCPT SUBJECT CMD [ARG ...]

mailfail translators@example.org "Bad PO" msgfmt -cv -o /dev/null bg.po

validate-html-notify [--dry-run] RCPT FILE