The GPU programming pipeline

The shader pipeline stages

The shader programming pipeline is composed of three customizable stages:

The programming shader pipeline stages

Vertex programs

The vertex program is the first stage of the pipeline. His goal is:

Geometry programs

The geometry shaders are executed after the vertex stage and before the pixel stage. They allow to modify the mesh topology by changing the polygons or by creating new ones. The input data of the program is a geometry primitive (point, line or triangle) and it sends one or more other primitives to the next pipeline stage.


This stage is optional, if no geometry program is defined, the data will transit directly from the vertex program to the pixel program.

Pixel programs

The pixel or fragment shader program is the last step of the pipeline. It receives data of each pixel individually (position, texture coordinates, color, etc.) and its goal is to calculate the final output color of the pixel.

Shading languages

The Red engine may use any version of vertex and pixel shader program that is above the VS and PS 2.0 versions. These programs must be either ARB or GLSL compliant shader programs. A faulty shader programs will pop a warning panel once and then all geometries that must be rendered with it will be ignored.

Available later in this guide is a quick reference for the ARB assembly shader language. Due to the explosive program combinations possible with shaders, we recommend using the RED::ShaderString class to write a shader program: it packages all the most common operations for all supported RED engine platforms (namely ATI and NVIDIA cards, as of the REDsdk v1.0) and contains all the necessary code to handle all the possible shader targets. Use of the RED::ShaderString implies to play with shaders at the assembly level. We recommend using shaders at the assembly level. This may scare you a little but practically, shader assembly is easy to learn, it has the best rendering performances and is the most hardware reliable. Because these programs don't use a compiler, they have no compiler bugs...


Task: Writing a simple custom rendering shader

A minimal RED::RenderShader is composed of:

The following code sample shows the construction of such a minimal shader:

// Use RED::ShaderString for ARB shader programming.
RED::RenderShader shader;
RED::ShaderString vsh, psh;
RED::ShaderProgramID vshid, pshid;
RED::RenderCode rcode;

// Binding of the vertex channel:
rcode.BindChannel( RED_VSH_VERTEX, RED::MCL_VERTEX );

// Vertex shader program:
vsh.VertexTransform( "result.position", "state.matrix.program[2]", "vertex.attrib[0]" );

// Pixel shader program:
psh.Add( "MOV result.color, { 1.0, 0.0, 0.0, 1.0 };\n" );

// Load the shaders from strings:
RC_TEST( iresmgr->LoadShaderFromString( vshid, vsh ) );
RC_TEST( iresmgr->LoadShaderFromString( pshid, psh ) );

// Set the render code and the programs:
RC_TEST( shader.SetRenderCode( rcode, RED_L0 ) );
RC_TEST( shader.SetVertexProgramId( vshid, RED_L0, resmgr ) );
RC_TEST( shader.SetPixelProgramId( pshid, RED_L0, resmgr ) );

First the render code is used to bind the mesh vertex channel to the shader input channel.

Then the vertex and pixel shader programs are written in ARB using a RED::ShaderString helper object. The vertex program simply transforms the vertex position from the world space to the screen space. The pixel program just paints the pixels in red.

Finally all the data are registered in the RED::RenderShader.

In the previous sample we wrote the programs in ARB. The next sample shows the same programs written in GLSL:

// Use simple strings for GLSL programs:
RED::String vsh_glsl;
RED::String psh_glsl;

// Vertex shader program:
vsh_glsl  = "void main()\n";
vsh_glsl += "{\n";
vsh_glsl += "gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;\n";
vsh_glsl += "}\n";

// Pixel shader program:
psh_glsl  = "void main()\n";
psh_glsl += "{\n";
psh_glsl += "gl_FragColor = vec4( 1.0, 0.0, 0.0, 1.0 );\n";
psh_glsl += "}\n";

When a more complicated program needs to be written, one could load an external ARB or GLSL text file and fill the render shader with its content.