Richard Hughes on color management in Linux and GNOME

Color management on Linux used to be a thing for brave boys and girls in the past. Two years ago the GNOME Color Manager project led by Richard Hughes and powered by Argyll color management system made a major breakthrough to fix it once and for all. Now that GNOME 3.2 is just a week away, we decided to corner Richard and ask him some very direct questions.

Actual color management applet in GNOME

Richard, you work as full-time employee at Red Hat. Does it help a lot being backed by a Linux vendor to work on what you love?

It’s not essential, but it really helps. Red Hat are one of the cool open source companies that essentially let you hack on random stuff as long as it’s somehow related to what they would like you to do. One of the things I try to do in the Linux desktop is to identify “holes” in required desktop functionality and to try and fix it. If that means writing some code for an existing project or creating a whole new framework then it’s what I’ll do.

We tend to expect professional designers, pre-press engineers and photographers to have a good grasp of color management. We don’t need to explain them why they need it. But how does GNOME Color Manager help people who are not entirely sure if spending a hundred bucks on a Pantone Huey, the cheapest solution today, is worth the trouble?

Well, hopefully in two ways:

They talk to other people who have color calibrated devices, and they tell them to stop working and to calibrate their screens. This is pretty much what I do at conferences — find a designer, calibrate the screen and watch their jaw drop to the floor when they realize that all their previously created artwork was wrong.

New topic based help system

When users find the new “Color” control center panel, it has very a very user-visible Learn more button that takes the user to a huge help file with topics like “Why is calibrating important”, “What calibration devices are supported” and even simple things like “What is a color profile?”. This lets new users find out why color management is important, and shows established users how to test their workflow and keep things accurate.

GNOME Color Manager is part of GNOME since 2.30, so a lot of people are at least familiar with it and know that they can use it to create ICC profiles for various devices such as displays and printers to ensure consistent color reproduction. But colord is is a relatively new thing. So, in few words, what is colord and why did GNOME Color Manager need it?

GNOME Color Manager (g-c-m for short) indeed was first included in GNOME 2.30, and underwent quite a rapid cycle of growth for 2.32 and 3.0. In that time, g-c-m grew from a few standalone programs into a hastily-designed framework that was being used by other programs. This wasn’t acceptable for a few main reasons:

• Projects like CUPS needed to register devices with the color manager, which was tricky as the manager was in the session layer and run once per user and the daemon was in the system layer and run at boot time.
• We needed to have system-wide profiles, so we could do things like set a color profile for another user, on the GDM login screen, or even during boot-up.
• Other desktop environments didn’t want to duplicate the profile metadata and mapping logic between devices.

The obvious thing to do was split off the framework parts of GNOME Color Manager into a simple system component that could be used by other things in the system context, reducing g-c-m again to a simple session front-end that configured and applied policy from colord. This would let us easily create a KDE or LXDE session project to also interface with colord.

Basic metadata information

If this sounds familiar, this is quite a lot like how PackageKit and UPower work, which should be no surprise as I maintain those upstream as well.

GNOME Color Manager in GNOME 3.2 is the first version that’s going to make a full use of colord and the whole revamped printing stack. What exactly does it mean to end-users?

Well, it means that we’ve got all the foundations not only designed and built, but installed by default which is very important. Colord is a hard dependency for GNOME 3.2, which means it gets included on the LiveCD’s on distributions. This means it just works. No installing of random packages is required, and no configuring is needed. This means that display devices gets registered into colord by gnome-settings-daemon automatically, and printing devices get registered into colord directly from CUPS.

This means for the first time, a user has a simple way of saying “Use this profile for this device” and having it magically work. This means that we can profile a printer (or download a vendor-supplied profile) once and have things look correct on the page when you click File>Print. To do this the right way meant changing a lot of things underneath the hood (CUPS, Ghostscript, foomatic and Gtk+) but means it just works for users without having to teach them how to write XML files and decide on policy themselves.

While working on colord you patched quite a bit of 3rd party tools, so I guess you have a pretty good big picture. What is your impression of the existing digital imaging and printing stack on Linux? Is it up-to-date? Is it reliable for professional use? Does it need any major changes?

Well, CUPS and Ghostscript are both established projects with stable APIs. Stable means they have been used for a long time, and of course having newer things like GLib and a DBus binding available to me would have made things a lot easier. The printing stack on Linux is also quite convoluted, although hopefully in the future things will get less crazy by dropping the PS workflow and everyone switching to PDF. It’s certainly stable and documented which I guess is the most important thing for professional use.

One of the more frustrating parts of dealing with devices was with the SANE project. SANE is an old library that you can ask to do things like enumerating the connected scanners and actually scanning in images. The main libsane calls out to ‘backends’ with one back-end per manufacturer. The quality of back-ends is extremely variable. Some, like the HP backend, work correctly, but others just crash randomly. It doesn’t help that you can install proprietary back-ends too, and the quality of these is even more questionable. Of course, because libsane is in-process with colord, it causes the daemon to crash. I would estimate that 90% of all the bugs reported for colord into Fedora are really sane-backend bugs. That’s the reason we have an optional EnableSane=false line in the colord.conf file. Long term I want to move a lot of the device discovery stuff into udev probers like all the other subsystems have done, but without help from the SANE community that’s not going to happen.

Color managed printing is now available and backed by colord

One of the things I tried to deliberately ignore designing colord was actually flipping pixel values. Colord is a very high level daemon that can say to apps like Krita or GIMP “Use this profile for this device” rather than them supplying a 140Mb memory buffer and an operation list. This means we can do the conversion on the CPU using lcms2 for some programs and using the GPU using a shader in things that use 3D. By not trying to wrap lcms we can let the application do the pixel conversion in the right layer in the right way.

Of course, the downside of this is that you have to patch applications to actually do the right thing. We can make this easier by doing some framework code for Clutter and Cairo, but sooner or later the application has to know about color management in one form or another. This is going to be my main focus for GNOME 3.4; now we have all the framework installed and working and we can say to application authors “It’s already installed, so don’t worry about the additional dependency, just commit my patch and it’ll just work”.

Speaking of printing, the only way to get correct output when printing full-color images is to create profiles for particular combinations of printer, ink and paper. Even with fixed workflows that means having multiple ICC profiles around. Is it currently supported?

I think most people will have one profile for their printer, but I agree it makes sense to support multiple profiles for each device. This is actually an issue for cameras too, where you might want a “Studio lighting profile” and an “Outdoor full sun” profile.

Colord actually does the same thing that ColorSync does on OSX, so when the application (CUPS in this case) asks for a color profile for a device, it actually asks for a profile based on a qualifier. So, in the general case you press File>Print then CUPS prints on plain paper at 600dpi using color ink. In that case CUPS then asks colord for the default profile based for the qualifier “RGB.Plain.600dpi”. If you want to print on glossy paper (which has a much larger gamut, so you can print a greater number of colors) then you really need a different profile. This means you re-calibrate the printer with glossy paper and get a slightly different profile. When printing on glossy paper you select the “Glossy paper” option in the print dialog and CUPS now asks for “RGB.Glossy.600dpi” and colord will return the correct profile. Of course, if you have no glossy profile, then it falls back to the plain paper profile, as that’s probably better than no profile at all.

The only thing we’re missing now is a way of tagging profiles not created with GCM “this profile is for color ink on glossy paper at 1200dpi” — of course profiles created with the built-in calibration tool have this metadata included. If you’ve got any ideas on how this should look in the UI, jump into #gnome-design and discuss it with the designers and I’ll add it for GNOME 3.4.

How many applications knowingly make use of D-Bus interface in colord and how stable is it to rely on at the moment?

I’m not sure how many projects use the colord API as they don’t have to inform anyone if they do. Off-hand I know of CUPS, GTK+, foomatic, simple-scan, compiz-cms and of course gnome-color-manager. Alex Fiestas also has plans to make a KDE front-end.

The colord public DBus API is very stable. We broke API once in the last year, and that only affected session policy agents rather than applications. We’re aiming to keep the API stable and supported, but will be adding more properties and methods on objects as we find a need to grow the functionality. I’m not going to say we’re never going to break API to fix a critical bug, but I don’t see the need to break things just at the moment; colord is a very simple daemon with limited scope.

Until recently g-c-m used Argyll color management system by Graeme Gill for talking to measurement devices like ColorMunki, Colorvision Spyder etc. Then apparently you started working on native drivers. Then Argyll was patched to suit some of your needs. So what was the problem with Argyll, how does it work now and what’s the plan for the future?

Lots of questions! For those not color geeks, Argyll is a set of programs for getting data from color sensors and processing it into different forms, and then into a completed color profile. At this, Argyll works really well and outperforms every other tool we have. Unfortunately, Argyll has several drawbacks: It’s designed to be lots of small programs that are run in an interactive console with the user understanding everything that’s going on.

This isn’t compatible with the “just works” philosophy of GCM and colord and this means we have to interface with Argyll and try to do something sane. For 3.0 and 3.2 this meant running Argyll in a VTE widget and trying to screenscrape the output. This doesn’t work very well. I asked Graeme (the Argyll author) to make the funky terminal stuff optional, and this he did with the latest release. Unfortunately I can’t ship the latest release in Fedora (due to a modified, embedded, version of libusb1 that’s waiting for the libusb1 project upstream) which means we’re stuck screenscraping.

One of wizard’s steps explaining how to place Pantone Huey on display

So the future: Argyll is surely going to be involved in actually creating the profiles, as Graeme knows how to create great ICC profiles like nobody else. What I’m trying to take out of the equation is the actual device locking and measurement which I’m adding into colord. At the moment there is a native driver for the Huey hardware, and half a driver for the ColorMunki hardware.

This allows us to get a color sample on the screen and do standard stuff like being able to cancel a calibration, and having simple things like a progress bar with progress complete. This means we can do all the stuff that takes a lot of time just using colord, and then we just run the results file into one of the Argyll tools to generate the profile. That way we get the benefits of a great set of algorithms tested and maintained by Graeme, and we get all the async hardware stuff in colord.

Of course, in doing the hardware in colord, I found we were missing an easy way to do USB async cancellable transfers in Glib. This meant I spun out a GUsb project, which I’ll hopefully be releasing into the wild soon and then using in colord. GUsb is also being used in other random stuff like SPICE, so it’s already helping other projects.

Recently there was a thread on full screen color management (FSCM) in the list for GNOME developers. Could you please explain what FSCM means to end users?

Well, full screen color management is the next logical step for color management for the free desktops and it’s already being done on OSX. We already do a kind of full screen effect by sending the VCGT (video card gamma tables generated at calibrate time) in ICC profiles to the X server when the session starts. Windows and OSX have been loading the calibration linearization state like that for quite a long time, and do allow you to do make a lot of hardware more accurate. It can for instance make the Thinkpad LCD screens less “blue” and so is quite useful.

However the VCGT is really a crutch in some respects, as it only allows you to remap the red, green and blue channels individually to calibrate the screen. That makes it impossible to do clever things like switching the green and blue channels which although seems like a cute trick, is really useful to simulate some types of color blindness. It also obviously doesn’t actually do any gamut mapping between source and destination profiles.

The real use of full screen color management is to do automatic profile-to-profile color mapping. What this means in reality is we automatically transform colors from a known color space to the monitor color space. Usually things like work processing documents, toolkit buttons, window borders and that kind of thing are all assumed to have the sRGB space as they are not tagged as anything special, but applications like Firefox might have images from Flickr with embedded camera color profiles that actually know the capture device color space.

So called horseshoe diagram of the color gamut

What tends to happen now is that the untagged content gets sent to the display server without any kind of color management, which is why people are going to start getting headaches with the laser-like wide-gamut LED screens. CM-aware applications like the GIMP (and Firefox, if you can be bothered to set it up) run the image through the profiles on the CPU using a library like lcms2 to convert the source colors to the destination monitor colors.

This makes the image itself correct on the screen, but it does mean running every single pixel through the CPU, which takes quite a long time for a multi-megabyte image file.

So what can we do? We want to do a massively parallel operation on lots of images that will be uploaded to the display. Well, it just so happens you have a GPU in most modern machines, which is designed to do exactly this. Give a GPU a 50Mb RGBA 2D texture in graphics RAM and tell it to trilinear interpolate every single pixel against a 16x16x16 matrix and it’ll finish before you can even load the image file into system RAM.

Doing this isn’t a new concept, Kai-Uwe has been demoing this for a couple of years with an Oyranos compiz plugin and more recently Gabriel Ebner in the compiz-cms project. The latter even uses colord to get the correct profile to use, so it appears for Compiz users there exists a drop-in solution assuming you can tell all applications to render everything to sRGB.

VCGT curves plot

For GNOME 3, we use the mutter compositing manager. It’s not quite a simple as doing an assembly shader on the completed window like we can in Compiz, and we have to use clutter and cogl to make sure all the rendering and transparency is done correctly. There’s also the added complexity of defining the opt-in and opt-out regions, for instance where a program like blender just wants full control of the window and doesn’t want the compositor munging the colors, or where Firefox has already converted the image on the CPU and we don’t want to “double” color correct the image. I’m still talking to the mutter and X developers to work out a sane story for GNOME 3.4 but it’ll likely involve some changes to mutter, Gtk+, cogl and might even need some application changes.

I’ve considered using some of the existing net-color spec by Kai-Uwe, but I think a lot of the existing spec is just wrong and adding things like network transparency means we introduce a huge number of issues that I don’t think we want to support. I also want to make it super easy to opt-in or opt-out widgets (and windows) from this FSCM and the net-color spec doesn’t exactly make that easy without defining manual opt-out regions that doesn’t work so well with moving, resizing, reshaped widgets in modern toolkits. Certainly anything we do has to have a minimum possible impact of additional dependencies, UI latency, and of course power consumption so at the moment all the different ideas are all possibilities.

I’m sure people reading this probably wanted a short answer, but I couldn’t do the question justice in a few words. After GNOME 3.4, I’m sure the FSCM architecture will deserve its own article…

Did you have a chance to look at Wayland? Does it provide any benefits with regards to color management?

Wayland is a cool project run by some very smart people. Really, Wayland is just a protocol for the compositor to talk to clients and I’m sure it will do very well. For FSCM we want the client to talk to the compositor and existing protocols like the _ICC_PROFILE_IN_X atoms that are uploaded to the server are just going to become irrelevant. At the moment Wayland isn’t a project that can “replace the X server” or whatever any of the crazy news sites say, but when designing the new FSCM architecture it’s worth bearing in mind that we’re not always going to have an X Server available to us to store random blobs of data.

So, I don’t think Wayland gives us anything for free, although it’ll help make things faster.

There seemed to be some controversy regarding the need to make 95% of users happy and the need to satisfy the rest 5% of users with their requirements for flexible policy-based workflows. At this point is there a way for both colord and Oyranos to be compatible?

I would love to say yes, but I think it’s basically impossible to make things “just work” and also give users control of every policy selection nuance. I’ve always stated that we should allow 95% of people to do 99% of what they need to do, the logic being that the other 5% of people already have a custom workflow that works for them. After doing quite a bit of user research, it turns out most people just want color management to do the right thing, and not bother them with technical mumbo-jumbo. The kind of person who does care already knows how to edit configuration files like colord.conf and how to read man pages.

Plot of a profile’s color body, source code based on Oyranos based ICC Examin

Also, I don’t think it’s likely we’re ever going to see any interoperability between the colord and Oyranos projects in the future. Not for any huge ideological reason, but just because the feature overlap of the two projects is too small. Colord is of very limited scope, is installed by default and tries to make things just work. Oyranos is a project of huge scope that wraps many other libraries and tries to be involved in every part of the color workflow. It’s kind-of orthogonal to what colord is trying to be, and that’s the main reason I chose to start a new project rather than trying to fix Oyranos.

You started working on GNOME Color Manager as a fun side-project to get rid of irritating inconsistency between color outputs from two displays. Two years later, with few additional infrastructure projects, colord and GUsb, as well as patching CUPS and whatnot, is it still fun?

Very much so. It turns out (as any color expert will tell you) that the more you know about color management, the more you realize you know very little about color management! With the direction colord is taking into talking to hardware directly (for a vastly improved user experience when calibrating) it’s getting very low level and interesting for a electronics geek like me. I’m also getting a lot of help from the UI designers for GNOME, and the result is that the GNOME 3.2 calibration and preferences UI are polished and sexy. This makes it easy for end users to actually use what we’ve written.

I guess the main stimulus for me is that people are still seeing what I’m doing and offering to help. When you’re being helped by and learning from cool people on the Internet, it’s very hard to get bored and do other things. I think color is a hugely interesting topic, and I’m sure I’ll be doing framework code for a long time yet.

Chinese translation of this tutorial is available at LinuxToy.