sm64coopdx/docs/lua/guides/shaders.md

Shaders

Resources

• Default C Shader in Lua
• Example Vertex Shader
• Example Fragment Shader

Before Starting...

You should have some basic knowledge on how writing mods in Lua works, and for more advanced shaders you should know how to work with shaders, specifically GLSL 410 or even GLSL 330. But even if you are a beginner, you may find use in this guide.

When testing shaders, it is recommended to launch your game with the terminal. While technically the game will fallback on default shaders if a shader fails to compile, there are multiple cases that are common enough to where you will appreciate running the game via a terminal since the console will be unavailable/invisible.

Types of Shaders

There are two types of shaders, post process shaders and "scene" shaders. Post process shaders are significantly easier than scene shaders, but scene shaders give you significantly more power since they run on each triangle rendered in the scene. We will mainly be going over post process shaders in this guide.

The default C scene shader converted to Lua can be found here. It does all of the boilerplate for you. The default C post process shader converted to Lua can be found here

Creating Shaders

Scene shaders can be created via the HOOK_ON_VERTEX_SHADER_CREATE and HOOK_ON_FRAGMENT_SHADER_CREATE hooks. These hooks provides a ColorCombiner, and the hooks expects you to return a shader back.

Post process shaders can be created via the HOOK_ON_POST_PROCESS_VERTEX_SHADER_CREATE and HOOK_ON_POST_PROCESS_FRAGMENT_SHADER_CREATE hooks. These hooks provide no parameters, and expect you to return a shader back.

The vertex shader allows for manipulating vertex data, whereas the fragment shader allows for manipulating color data. You do not have to provide both, but if you only use one, you are expected to send/receive data back as expected by their respective default shaders.

Shader Language

Coop uses GLSL 410 core, or GLSL 4.1 core, or #version 410 core. It uses a custom system that does a lot of work for you to ensure shaders work on DirectX, Metal, and any other render apis that get added in the future. The custom system automatically assigns input locations, uniform buffers, uniform bindings, and other backend-specific details so that the same GLSL works across OpenGL, DirectX 11, and Metal. Because of this, you should avoid manually specifying input/uniform/sampler locations or uniform blocks.

As a list, things not to do are:

• Define explicit locations
• Define custom uniform blocks

And as a list of things to always do:

• Define every input passed from C to the vertex shader (see inputs)
• For every output in the vertex shader must be a corresponding input in the fragment shader. They must have matching names

Failure in any of the above will result in either a failed to sanitize shader error or a failed to compile shader to SPIR-V cross error.

Vertex Shaders

The bread and butter of vertex shaders are inputs. Inputs are given by the game's code and can be used for getting the vertex data. A list of inputs for shaders can be found here. Something important about inputs is that a fragment shader cannot get inputs from C, so you need to pass inputs from the vertex shader as outputs for the fragment shader. An example of that would be the vertex position.

in vec4 aVtxPos;
out vec4 vVtxPos;
vVtxPos = aVtxPos;

The input aVtxPos is provided by C, and the output is created, which will be received in the fragment shader.

in vec4 vVtxPos;

The fragment shader takes in the vertex pos we exported in the vertex shader.

For a post process vertex shader example, we are going to mirror the entire world. First, grab the default C post process shader converted to Lua, specifically the vertex portion.

Note: While the format shown in the default shaders use table.insert(xShader, "line"), this is not at all required. You may use any method you'd like, including loading the shader from a different file. This method was picked for it's ease to inject any if statement checks anywhere you'd like that's outside the shader.

gl_Position = aVtxPos;

This sets the vertex position to be passed to the GPU to our vertex position provided by C. There are 4 main components to this position:

• x being the X coordinate
• y being the Y coordinate
• z being the Z coordinate
• w being the depth coordinate

This position in clip space, which means that it is already ready to be inserted into the GPU which is why we send it with no modifications. Go here for a more detailed explanation on coordinate spaces.

To get our mirror effect, we need to flip the x coordinate. So we need to create a vector 4 that flips our x component, and passes back our y, z, and w components as-is. This can be done via this syntax:

gl_Position = vec4(aVtxPos.x * -1.0, aVtxPos.yzw);

If you run the mod, you should now have a mirrored game! That's the basic rundown on vertex shaders! If something still isn't working, compare your code with the example vertex shader and try to figure out what you did wrong.

Note: Alternatively, to accomplish the same effect, you can flip the texture coordinate in the fragment shader.

Fragment Shaders

First, grab the default C post process shader converted to Lua, specifically the fragment shader portion.

For our example fragment shader, we are going to be inverting the colors. Actually doing this is quite simple, but before we get to that, let's explain what we are working with:

First, unlike the vertex shader, you have to define the output manually in a fragment shader.

out vec4 fragColor;

In the fragment shader, we create an output for the fragment color. Unlike the vertex shader, an output does not go to another shader, instead, it goes to the GPU to be used as the color for the pixel.

All the outputs in the vertex shader can be read in the fragment shader as inputs. This allows data to be carried over from one to the other. What was out in the vertex shader becomes in in the fragment shader.

For our example, inverting the colors, it's quite simple.

fragColor = texture(uPassTex, vTexCoord);

Currently, this is what the frag color is set to. Instead of setting fragColor directly, we should first take the color outputted by the texture function and insert it into it's own variable.

vec4 texColor = texture(uPassTex, vTexCoord);

Then, inverting that color becomes trivial. When we set fragColor, all we need to do is invert texColor.rgb and preserve texColor.a. This can be done with the following syntax:

fragColor = vec4(1.0 - texColor.rgb, texColor.a);

And that's it! All colors in your game should now be completely inverted! That's the basic rundown on fragment shaders! If something still isn't working, compare your code with the example fragment shader and try to figure out what you did wrong.

Uniforms

A shader may contain uniforms. As per the GLSL naming convention, uniform variables typically start with a u, for instance, uFrameCount. There are multiple different uniforms already updated by C, but you can also update custom uniforms in Lua. First, here is a list of all the uniforms updated in C. You can override these uniforms and shove in your own data, but beware, behavior is undefined.

Uniform Name	Type	Description
uTex0	sampler2D	The primary texture
uTex1	sampler2D	The secondary texture
uPassTex	sampler2D	The rendered output texture produced by the previous frame pass
uPassTexX	sampler2D	The rendered output texture from frame pass X, where X is the pass index.
uTex0Size	vec2	The width and height of the primary texture
uTex1Size	vec2	The width and height of the secondary texture
uTex0Filter	bool	true if the primary texture uses linear filtering
uTex1Filter	bool	true if the secondary texture uses linear filtering
uFilter	int	The current global filtering mode (0 = Point, 1 = Linear, 2 = Three-point)
uFrameCount	float	The current frame we are on since the lifetime of the program, acts as a timer
uLightmapColor	vec3	RGB multiplier applied to the environment/light map
uAspectRatio	float	The current viewport aspect ratio (width/height)
uXAdjustRatio	float	The horizontal clip-space scaling factor used for widescreen rendering
uModelViewProjectionMatrix	mat4	Transforms local/object space directly to clip space
uModelViewMatrix	mat4	Transforms local/object space to view space
uModelMatrix	mat4	Transforms local/object space to world space
uInverseModelMatrix	mat4	Transforms world space back to local/object space
uViewMatrix	mat4	Transforms world space to view space
uInverseViewMatrix	mat4	Transforms view space back to world space
uProjectionMatrix	mat4	Transforms view space to clip space
uInverseProjectionMatrix	mat4	Transforms clip space back to view space
uShaderFlags	int[]	Array of shader feature flags provided by the engine
uShaderFlagValues	float[]	Array of values associated with uShaderFlags

For defining a custom uniform in lua, use the HOOK_ON_SET_SHADER_UNIFORMS hook and use the appropriate gfx_shader_set_* function.

local sceneBrightness = 0.5

local function on_set_shader_uniforms()
    -- if necessary, check frame pass index using this code
    -- local framePass = gfx_shader_get_current_frame_pass()
    -- if framePass ~= FRAME_PASS_REQ then return end

    gfx_shader_set_float("uSceneBrightness", sceneBrightness)
end

hook_event(HOOK_ON_SET_SHADER_UNIFORMS, on_set_shader_uniforms)

That allows you to define your own uniforms and set your uniforms in Lua!

Inputs

Inputs are passed into the vertex shader for further use. Here is a list of inputs provided by C.

Scene Shader Inputs

Input Name	Type	Description
aVtxPos	vec4	The vertex position in clip space. Can be transformed using matrices (see TODO add coordinate spaces to guide)
aTexCoord0	vec2	UV coordinates for the primary texture
aTexCoord1	vec2	UV coordinates for the secondary texture
aLightMap	vec2	UV coordinates for the light map
aInputX	vec4	Color/alpha input from the Color Combiner, with X being the input number. There are 8 color combiner inputs, or CC_MAX_INPUTS.
aNormal	vec3	The surface normal
aBarycentric	vec3	The barycentric coordinates of the vertex within its triangle

Post Process Shader Inputs

Input Name	Type	Description
aVtxPos	vec4	The vertex position in clip space (x, y, z, w)
aTexCoord	vec2	The UV mapping for the pass texture

Dealing with the HUD

First of all, you can't filter out the HUD in a post process shader, you need to use a scene shader in order to do that. Secondly, the best option is to use the world geometry flag provided by the ColorCombiner

local worldGeometry = cc.cm.flags & CM_FLAG_WORLD_GEOMETRY ~= 0

If it is world geometry, it's not the hud, if it's not world geometry, it is the hud. This is the best solution by far. Other solutions exist, such as checking the depth of the Z coord in the clip pos, but they are a lot more convoluted and less efficent.

Multipass Shaders

Multipass shaders allow you to draw the world multiple times in a single frame. It's where custom scene shaders can do the most work. Whether you're creating a minimap, adding lighting, wanting multiple cameras, and more, you'll want to use multipass shaders.

Creating a frame pass

A frame pass can be created with gfx_shader_create_frame_pass. This function returns the frame pass index, which is anywhere from zero to the maximum number of frame passes. A frame pass may also be removed with gfx_shader_remove_frame_pass.

Frame passes can be configured with their configuration functions.

Function	Description
gfx_shader_set_frame_pass_viewport	Sets the viewport/resolution of the frame pass. A viewport width or height of 0 will cause that value to use whatever the current screen size is
gfx_shader_set_frame_pass_draw_world	Configures whether the frame pass should redraw the world or use a quad (quad uses post process shader, redrawing the world uses the scene shader)

Using frame passes

In your shader hooks, you can access which frame pass you are currently on with the gfx_shader_get_current_frame_pass. This is needed if you need to change your shader depending on the current frame pass.

Sometimes you may need to configure things before you redraw the world. For instance, on some shaders you may want to disable culling before you redraw the world. You can use HOOK_BEFORE_DRAW_GEOMETRY to achieve this. Check your current frame pass index using gfx_shader_get_current_frame_pass, and run code accordingly in this hook. This hook isn't unique to frame passes, it's called anytime the world geometry is about to be drawn.

Coordinate Spaces

The vertex position (aVtxPos) provided by C is in clip space. As shown in the uniforms section, many matrices are provided by C. There are enough matrices to get to any space you need. Each uniform explains what it transforms to, but here are some examples of the most common ones:

Getting to view space from clip space:

vec4 viewPos = uInverseProjectionMatrix * clipPos;

Getting to world space from view space:

vec4 worldPos = uInverseViewMatrix * viewPos;

Getting to local/object space from world space:

vec4 localPos = uInverseModelMatrix * worldPos;

A sweet visualtion of these matrices can be found here

Some devices are stuck on legacy renderers. That means that the NDC Z range is from -1 to 1 when normally it is from 0 to 1. Lua can detect and modify shaders accordingly with gfx_is_legacy_renderer. This is an unfortunate limitation that can't really be fixed unless we stop supporting a bunch of devices. This may change in the future, but for now this is the case.