VMD supports the use of OpenGL programmable shaders in the main view window. This offers a lot of opportunities for customization and beautiful rendering without the use of an external raytracer.

I wrote a shader for VMD that aims for a more “toon” style, similar to that of PyMol. The goal was to generate publication quality images in the view window that can be manipulated in realtime.

In the preview image, I’m highlighting a palmityol-cysteine on the beta-2 adrenergic receptor.

If you have built VMD with GLSL support (OPENGL in configure.options, or using one of the binaries with OpenGL support), and you have a GPU or integrated graphics that understands OpenGL, you can actually change the render “style” in the main window using shaders.

The official VMD documentation provides outline shaders for an old VMD version (the feature is enabled by default in current builds) and hints at some environment variables to set.

Writing shaders can be pretty difficult, and I’m not an expert or even remotely proficient at it. I used my normal approach of reading the code for the existing shaders, reading a lot of examples on the internet, and trial and error. For a more formal introduction, the Book of Shaders is a good introductory resource.

Given my complete lack of rigor developing these (the compute cluster was down one day and I wanted a way of getting out of ever making figures in PyMol), I don’t claim that these are examples to learn from, or good in any way. They do seem to run reasonably fast, and look pretty.

Usage

Download all 4 shaders here and extract to a location of your choosing. Here I put them in the new directory $HOME/.vmd/shaders.

mkdir -p $HOME/.vmd/shaders/ && cd $HOME/.vmd/shaders
wget http://robinbetz.com/files/shaders.zip
unzip shaders.zip

Set the following environment variables to have VMD use these shaders. To help debug, VMD can print out error messages when compiling shaders, so set that environment variable as well if you’re modifying them.

export VMDGLSLVERBOSE=1
export VMDOGLSHADER=$HOME/.vmd/shaders/toonoutline
export VMDOGLSPHERESHADER=$HOME/.vmd/shaders/toonoutlinesphere

This should be all you need to do!

Examples

Here are a few images I’ve made playing around with different materials. These are all screenshots of the view window – no rendering engine was used.

For the image featured at the top of the post, I’ve put a transparent MSMS or NanoShaper surface over another protein representation like VDW to provide more detail without being confusing.

The ribbons representation can also look really good when you need less detail on the protein. Here’s the same receptor with several dihydroalprenolols shown in VDW.

Caveats

I don’t actually use the “outline” property, and so when adjusting material properties, sliders relating to outline don’t do anything.

Messing around with lighting, especially moving the lights from their default positions, can make things look really bad.

I also found the default material didn’t look very good with these shaders so I redefined it in my ~/.vmdrc:

material add OpaqueCel
material change ambient OpaqueCel 0.0
material change diffuse OpaqueCel 0.25
material change specular OpaqueCel 1.0
material change shininess OpaqueCel 0.7
mol default material {OpaqueCel}

This defines a new material, OpaqueCel, that I think looks better, and sets it as the default.