-
 KDE-Apps.org Applications for the KDE-Desktop 
 GTK-Apps.org Applications using the GTK Toolkit 
 GnomeFiles.org Applications for GNOME 
 MeeGo-Central.org Applications for MeeGo 
 CLI-Apps.org Command Line Applications 
 Qt-Apps.org Free Qt Applications 
 Qt-Prop.org Proprietary Qt Applications 
 Maemo-Apps.org Applications for the Maemo Plattform 
 Java-Apps.org Free Java Applications 
 eyeOS-Apps.org Free eyeOS Applications 
 Wine-Apps.org Wine Applications 
 Server-Apps.org Server Applications 
 apps.ownCloud.com ownCloud Applications 
--
-
 KDE-Look.org Artwork for the KDE-Desktop 
 GNOME-Look.org Artwork for the GNOME-Desktop 
 Xfce-Look.org Artwork for the Xfce-Desktop 
 Box-Look.org Artwork for your Windowmanager 
 E17-Stuff.org Artwork for Enlightenment 
 Beryl-Themes.org Artwork for the Beryl Windowmanager 
 Compiz-Themes.org Artwork for the Compiz Windowmanager 
 EDE-Look.org Themes for your EDE Desktop 
--
-
 Debian-Art.org Stuff for Debian 
 Gentoo-Art.org Artwork for Gentoo Linux 
 SUSE-Art.org Artwork for openSUSE 
 Ubuntu-Art.org Artwork for Ubuntu 
 Kubuntu-Art.org Artwork for Kubuntu 
 LinuxMint-Art.org Artwork for Linux Mint 
 Arch-Stuff.org Art And Stuff for Arch Linux 
 Frugalware-Art.org Themes for Frugalware 
 Fedora-Art.org Artwork for Fedora Linux 
 Mandriva-Art.org Artwork for Mandriva Linux 
--
-
 KDE-Files.org Files for KDE Applications 
 OpenTemplate.org Documents for OpenOffice.org
 GIMPStuff.org Files for GIMP
 InkscapeStuff.org Files for Inkscape
 ScribusStuff.org Files for Scribus
 BlenderStuff.org Textures and Objects for Blender
 VLC-Addons.org Themes and Extensions for VLC
--
-
 KDE-Help.org Support for your KDE Desktop 
 GNOME-Help.org Support for your GNOME Desktop 
 Xfce-Help.org Support for your Xfce Desktop 
--
openDesktop.orgopenDesktop.org:   Applications   Artwork   Linux Distributions   Documents    LinuxDaily.com    Linux42.org    OpenSkillz.com   
 
Artwork
News
Groups
Knowledge
Events
Forum
People
Jobs
Register
Login

-
- News . 
0
votes
click to vote up

Barry Warsaw: The right way to internationalize your Python app


Published Nov 22 2012 via RSS

Recently, as part of our push to ship only Python 3 on the Ubuntu 12.10 desktop, I've helped several projects update their internationalization (i18n) support.  I've seen lots of instances of suboptimal Python 2 i18n code, which leads to liberal sprinkling of cargo culted .decode() and .encode() calls simply to avoid the dreaded UnicodeErrors.  These get worse when the application or library is ported to Python 3 because then even the workarounds aren't enough to prevent nasty failures in non-ASCII environments (i.e. the non-English speaking world majority :).

Let's be honest though, the problem is not because these developers are crappy coders! In fact, far from it, the folks I've talked with are really really smart, experienced Pythonistas.  The fundamental problem is Python 2's 8-bit string type which doubles as a bytes type, and the terrible API of the built-in Python 2 gettext module, which does its utmost to sabotage your Python 2 i18n programs.  I take considerable blame for the latter, since I wrote the original version of that module.  At the time, I really didn't understand unicodes (this is probably also evident in the mess I made of the email package).  Oh, to really have access to Guido's time machine.

The good news is that we now know how to do i18n right, especially in a bilingual Python 2/3 world, and the Python 3 gettext module fixes the most egregious problems in the Python 2 version.  Hopefully this article does some measure of making up for my past sins.

Stop right here and go watch Ned Batchelder's talk from PyCon 2012 entitled Pragmatic Unicode, or How Do I Stop the Pain?  It's the single best description of the background and effective use of Unicode in Python you'll ever see.  Ned does a brilliant job of resolving all the FUD.

...

Welcome back.  Your Python application is multi-language friendly, right?  I mean, I'm as functionally monolinguistic as most Americans, but I love the diversity of languages we have in the world, and appreciate that people really want to use their desktop and applications in their native language.  Fortunately, once you know the tricks it's not that hard to write good i18n'd Python code, and there are many good FLOSS tools available for helping volunteers translate your application, such as Pootle, Launchpad translations, Translatewiki, Transifex, and Zanata.

So there really is no excuse not to i18n your Python application.  In fact, GNU Mailman has been i18n'd for many years, and pioneered the supporting code in Python's standard library, namely the gettext module.  As part of the Mailman 3 effort, I've also written a higher level library called flufl.i18n which makes it even easier to i18n your application, even in tricky multi-language contexts such as server programs, where you might need to get a German translation and a French translation in one operation, then turn around and get Japanese, Italian, and English for the next operation.

In one recent case, my colleague was having a problem with a simple command line program.  What's common about these types of applications is that you fire them up once, they run to completion then exit, and they only have to deal with one language during the entire execution of the program, specifically the language defined in the user's locale.  If you read the gettext module's documentation, you'd be inclined to do this at the very start of your application:

from gettext import gettext as _
gettext.textdomain(my_program_name)

then, you'd wrap translatable strings in code like this:

print _('Here is something I want to tell you')

What gettext does is look up the source string (i.e. the argument to the underscore function) in a translation catalog, returning the text in the appropriate language, which will then be printed.  There are some additional details regarding i18n that I won't go into here.  If you're curious, ask in the comments, and I'll try to fill things in.

Anyway, if you do write the above code, you'll be in for a heap of trouble, as my colleague soon found out.  Just running his program with --help in a French locale, he was getting the dreaded UnicodeEncodeError:

"UnicodeEncodeError: 'ascii' codec can't encode character"

I've also seen reports of such errors when trying to send translated strings to a log file (a practice which I generally discourage, since I think log messages usually shouldn't be translated).  In any case, I'm here to tell you why the above "obvious" code is wrong, and what you should do instead.

First, why is that code wrong, and why does it lead to the UnicodeEncodeErrors?  What might not be obvious from the Python 2 gettext documentation is that gettext.gettext() always returns 8-bit strings (a.k.a. byte strings in Python 3 terminology), and these 8-bit strings are encoded with the charset defined in the language's catalog file.

It's always best practice in Python to deal with human readable text using unicodes.  This is traditionally more problematic in Python 2, where English programs can cheat and use 8-bit strings and usually not crash, since their character range is compatible with ASCII and you only ever print to English locales.  As soon as your French friend uses your program though, you're probably going to run into trouble.  By using unicodes everywhere, you can generally avoid such problems, and in fact it will make your life much easier when you eventually switch to Python 3.

So the 8-bit strings that gettext.gettext() hands you have already sunk you, and to avoid the pain, you'd want to convert them back to unicodes before you use them in any way.  However, converting to unicodes makes the i18n APIs much less convenient, so no one does it until there's way too much broken code to fix.

What you really want in Python 2 is something like this:

from gettext import ugettext as _

which you'd think you should be able to do, the "u" prefix meaning "give me unicode".  But for reasons I can only describe as based on our misunderstandings of unicode and i18n at the time, you can't actually do that, because ugettext() is not exposed as a module-level function.  It is available in the class-based API, but that's a more advanced API that again almost no one uses.  Sadly, it's too late to fix this in Python 2.  The good news is that in Python 3 it is fixed, not by exposing ugettext(), but by changing the most commonly used gettext module APIs to return unicode strings directly, as it always should have done.  In Python 3, the obvious code just works:

from gettext import gettext as _

What can you do in Python 2 then?  Here's what you should use instead of the two lines of code at the beginning of this article:

_ = gettext.translation(my_program_name).ugettext

and now you can wrap all your translatable strings in _('Foo') and it should Just Work.

Perhaps more usefully, you can use the gettext.install() function to put _() into the built-in namespace, so that all your other code can just use that function without doing anything special.  Again, though we have to work around the boneheaded Python 2 API.  Here's how to write code which works correctly in both Python 2 and Python 3.

import sys, gettext
kwargs = {}
if sys.version_info[0] < 3:
    # In Python 2, ensure that the _() that gets installed into built-ins
    # always returns unicodes.  This matches the default behavior under Python
    # 3, although that keyword argument is not present in the Python 3 API.
    kwargs['unicode'] = True
gettext.install(my_program_name, **kwargs)

Or you can use the flufl.i18n API, which always uses returns unicode strings in both Python 2 and Python 3.

Also interesting was that I could never reproduce the crash when ssh'd into the French locale VM. It would only crash for me when I was logged into a terminal on the VM's graphical desktop.  The only difference between the two that I could tell was that in the desktop's terminal, locale(8) returned French values (e.g. fr_FR.UTF-8) for everything, but in the ssh console, it returned the French values for everything except the LC_CTYPE environment variable.  For the life of me, I could not get LC_CTYPE set to anything other than en_US.UTF-8 in the ssh context, so the reproducible test case would just return the English text, and not crash.  This happened even if I explicitly set that environment variable either as a separate export command in the shell, or as a prefix to the normally crashing command.  Maybe there's something in ssh that causes this, but I couldn't find it.

One last thing.  It's important to understand that Python's gettext module only handles Python strings, and other subsystems may be involved.  The classic example is GObject Introspection, the newest and recommended interface to the GNOME Object system.  If your Python-GI based project needs to translate strings too (e.g. in menus or other UI elements), you'll have to use both the gettext API for your Python strings, and set the locale for the C-based bits using locale.setlocale().  This is because Python's API does not set the locale automatically, and Python-GI exposes no other way to control the language it uses for translations.



BackRead original postSend to a friend

Add comment

Add comment
Show all posts




-
 
 
 Who we are
Contact
More about us
Frequently Asked Questions
Register
Twitter
Blog
Explore
Artwork
Jobs
Knowledge
Events
People
Updates on identi.ca
Updates on Twitter
Facebook App
Content RSS   
Events RSS   

Participate
Groups
Forum
Add Artwork
Public API
About GNOME-Look.org
Legal Notice
Spreadshirt Shop
CafePress Shop
Advertising
Sponsor us
Report Abuse
 

Copyright 2003-2014 GNOME-Look.org Team  
All rights reserved. GNOME-Look.org is not liable for any content or goods on this site.
All contributors are responsible for the lawfulness of their uploads.
GNOME and the foot logo are trademarks of the GNOME Foundation.