USE TEXTURES INSIDE SHADERS

We will:
- See what a texture is and how to create and use one inside Max.
- See how to read and use a texture inside the fragment shader.
- See how to read and use a texture inside the vertex shader.

Required Level: You know how to write a basic shader in Max/MSP and how to pass uniform variables to it.
You know the basics of Jitter and OpenGL in Max.
If you don't know how to write a basic shader see my previous tutorial Introduction to Shaders.
If you don't know the basics of OpenGL in Max you can take a look at this video tutorial of mine:
Introduction to OpenGL in Max.

Introduction

Welcome to my second written tutorial about shaders. In this one we will see how to use a texture and how to access one from within the shader program.
We will use the texture data first to change the color of the pixels of our shape, and secondly to modify the position of the vertices of the shape.

We will use as a starting point the shader created in the previous tutorial Introduction to Shaders.

What is a Texture?

A texture is some data that resides on the memory of the GPU, often in the form of an image (so with 4 channels which are Red, Green, Blue and Alpha). This image gets usually used to change the color of the 3D object to which the texture is applied, by mapping the pixels in the image to the pixels of the shape (fragments).
The mapping gets performed using some 2D coordinates called texture coordinates. Those coordinates get interpolated automatically between the vertices of the shape when passed to the fragment shader, in order to map the texture smoothly to the 3D object.

Create a Texture in Max

To create and use a texture inside Max we have to create a [jit.gl.texture] object. Since it has "gl" in its name we can see that this is an OpenGL based object, so we need a valid GL context to associate it with. This is why we have to create a [jit.world] object, which we anyway need to display our 3D scene.
In the patch below you can see that we have a [jit.world] object with some basic attributes and the [jit.gl.texture] object highlighted in red.

A [jit.gl.texture] object

Load Data in the Texture
To load some data (image) inside a texture we can choose two ways:
- We load directly an existing image from our hard disk inside the [jit.gl.texture] object using the read message:

Load an existing image

- or we send a jitter matrix into the texture inlet:

Take a matrix as input

For the moment let's stick with this second method, so we load some random color values inside the texture in float32 type using the [jit.noise] object. Let's notice that the noise matrix has 4 planes and dimensions 100 x 100. This means that the texture inside [jit.gl.texture] will also have dimensions 100 x 100, and 4 planes with random values in every plane.

By default a texture has always 4 planes.

Remember to click on the [bang] button above the [jit.noise] object to transfer the matrix inside the [jit.gl.texture] object.

Attach the Texture
To use the texture however we need to assign a name (identifier) to it, in order to be able to link (attach) it to a 3D shape object container such as [jit.gl.gridshape]. We can assign a name to the texture using its @name attribute:

The @name attribute set to "texture0"

Apply the Texture to an Object

Let's now create a 3D shape using the [jit.gl.gridshape] object, and apply our noise texture to it.
In order to apply the texture to the [jit.gl.gridshape] we have to set the @texture attribute to the name of the texture we created before (in this case the name is texture0):

Create a [jit.gl.gridshape] and assign the texture

The [jit.gl.gridshape] object has by default texture coordinates stored in the fourth and fifth planes of the inner matrix (indices 3 and 4), so we don't have to assign them manually.

Our 3D sphere should look like this:

Our textured sphere

Why is it so grey? That's because the color of the image contained in the texture (our noise) is being multiplied by the color of the object. By default the [jit.gl.gridshape] has its @color attribute set to a shade of grey, so the texture values will be multiplied for a number less than 1 resulting in this not so bright final color.

We have two ways to change this:

- Change the color of our sphere to be white, so the texture values get multiplied by 1 on every channel and appear as the original. We can do that by setting the @color attribute of [jit.gl.gridshape] to 1 1 1 1:

White color for [jit.gl.gridshape]

- Or we can change the @apply attribute of [jit.gl.texture] from the default modulate to replace:

The @apply attribute of [jit.gl.texture]

In this way the texture values don't get multiplied by the color of the shape, but simply replace it.

In both cases we obtain this result:

The texture applied without modulation

Apply a Shader to our 3D Shape

First of all we need to create a [jit.gl.shader] object in our patch and give it an identifier which we use to associate it to a 3D shape. We can set this identifier using the @name attribute of the [jit.gl.shader] object (like we did for the [jit.gl.texture]).

I'm calling my [jit.gl.shader] object "shady".

Then we can give the shader object the name of the file containing the shader to use, by setting the @file attribute. Let's call this shader file "textureShader.jxs".

The [jit.gl.shader] object

I'm using as a starting point the shader from the tutorial "Introduction to Shaders", if you don't have it you can just copy the code below into a txt file and rename it "textureShader.jxs".

Here's the code for the basic shader from the previous tutorial:

<jittershader name = "myFirstShader">
<description> That is my first shader </description>
<param name = "color" type = "vec3" default = "1.0 1.0 1.0" />
<language name = "glsl" version = "1.2">
<bind param = "color" program = "fp" />
<program name = "vp" type = "vertex">
<![CDATA[

void main() {

  gl_Position = ftransform();
}

]]>
</program>
<program name = "fp" type = "fragment">
<![CDATA[

  uniform vec3 color;

  void main() {

    gl_FragColor = vec4(color, 1.0);
  }

]]>
</program>
</language>
</jittershader>

Here's the patch with the [jit.gl.shader] object applied to the [jit.gl.gridshape]:

Shader applied to the 3D shape

The shader has been applied to the 3D shape by using the shader attribute of [jit.gl.gridshape].
As you can see the sphere from the [jit.gl.gridshape] became completely white, so the texture is not applied anymore. That's because we are replacing the fixed function pipeline with our shader, which sets the color of the object to white. It's now up to us to write the code to apply the texture color to the object.

Read the Texture from inside the Fragment Program

Let's begin by passing the texture as a uniform to our fragment program. We do that by adding a new parameter to our jxs structure at the top of the file:

<jittershader name = "myFirstShader">
<description> That is my first shader </description>
<param name = "color" type = "vec3" default = "1.0 1.0 1.0" />
<param name = "tex0" type = "int" default = "0" />
<language name = "glsl" version = "1.2">
<bind param = "color" program = "fp" />
<bind param = "tex0" program = "fp" />
<program name = "vp" type = "vertex">
<![CDATA[

Interestingly the type of a texture inside the parameter declaration must be set to "int". This integer is the index of the texture assigned to the @texture attribute of the [jit.gl.gridshape], since we can assign to the object more than one.

Let's notice that we also have to bind the new parameter to the program we want to use it in (in this case the fragment program, so <bind param = "tex0" program = "fp" />).

Declaring the Uniform
We can now use the texture inside our fragment program declaring a uniform variable (like we did in the previous tutorial with the color parameter):

<program name = "fp" type = "fragment">
<![CDATA[

  uniform vec3 color;
  uniform sampler2DRect tex0;

  void main() {

    gl_FragColor = vec4(color, 1.0);
  }

]]>
</program>
</language>
</jittershader>

sampler2DRect and sampler2D
Inside the shader program the type of the texture uniform must be set to "sampler2DRect" or to "sampler2D".
The first must be used when the @rectangle attribute of [jit.gl.texture] is set to 1 (default), and the second when the attribute is set to 0.

The first difference between those two modes is that when rectangle is enabled the texture can have whatever size, whereas when rectangle is set to 0, the texture must have as a size a power of 2.

The second difference is that when rectangle is enabled, the texture coordinates are in a range that goes from 0 to the integer number relative to the texture size in both axis (width & height) - 1. For example in our case, using a matrix with dimensions 100 x 100, the texture coordinates range from [0, 99] for the width and [0, 99] for the height in integer numbers (similar to the [cell] object inside [jit.gen]).
When rectangle is disabled, the texture coordinates are normalized, and so are in the range [0, 1] floating point (similar to the [norm] object inside [jit.gen]).

Texture Coordinates
In order to be able to map the texture to the fragments of the shape, we need to access the texture coordinates that are built inside the [jit.gl.gridshape] object and transform them according to a transformation matrix provided by the pipeline.
We need to do that inside the vertex shader, since the texture coordinates are relative to each vertex. Then we have to pass the transformed coordinates to the fragment shader, so they can be interpolated and used in accessing the texture.

To pass data from the vertex shader to the fragment shader we use something called a varying variable.
We declare it before the beginning of the main function:

varying vec2 texcoord0;

void main() {

  gl_Position = ftransform();
}

]]>
</program>

Varying Variable
A varying variable is a special variable that gets passed to the programs below the vertex shader (geometry shader and fragment shader). When passed down to the fragment shader the data in the variable gets interpolated from the vertices position to cover all the pixels that form the rendered 3D object.
The variable is called varying because it varies with every vertex, whereas the uniform variables are the same for all the vertices and the fragments.

As you can see the variable texcoord0 is of type vec2, since it contains coordinates for the width and the height of the texture. When talking about texture coordinates we don't use the letters x and y when referring to the axis, but the letters s and t. So the horizontal coordinates are called s coordinates, and the vertical are called t coordinates.

Transforming the Texture Coordinates
Let's now define the values for the texcoord0 variable:

<program name = "vp" type = "vertex">
<![CDATA[

varying vec2 texcoord0;

void main() {

  texcoord0 = vec2(gl_TextureMatrix[0] * gl_MultiTexCoord0);
  gl_Position = ftransform();
}

]]>
</program>

The matrix contained into gl_TextureMatrix[0] is the transformation matrix for the texture coordinates. We need that because for example we could scale the coordinates in our patch, and in order to get them scaled inside the GLSL shader we need to transform them for that matrix. As you can see this is an array of matrices, which length depend on how many textures we assign to the 3D object (in our case just one). Since we have only one texture we get it out from the array using the index 0.

Then we multiply that matrix for the texture coordinates of our 3D shape provided automatically by the [jit.gl.gridshape] object. We can access those by using the built-in variable gl_MultiTexCoord0.
This multiplication gives us a vec4, but we only need the first two components of that vector, so we strip it down to a vec2 using the vec2 data cast. Finally, we have our texture coordinates ready to be used inside the fragment shader.

Remember that matrix multiplications are not commutative, so there is a difference if we multiply the vector for the matrix or the matrix for the vector. The order presented in the above code is the correct one in that case.

Pass the Texture Coordinates
Let's now go inside the fragment shader and declare again the varying variable texcoord0 before the main function, so we can use it:

<program name = "fp" type = "fragment">
<![CDATA[

  uniform vec3 color;
  uniform sampler2DRect tex0;

  varying vec2 texcoord0;

  void main() {

    gl_FragColor = vec4(color, 1.0);
  }

]]>
</program>
</language>
</jittershader>

Now the texcoord0 variable contains the transformed texture coordinates of every vertex, interpolated through all the fragments of the shape.

Read the Texture Values
To read the texture values we have to sample the texture image using our texture coordinates. We can do that using the texture2DRect function:

<program name = "fp" type = "fragment">
<![CDATA[

  uniform vec3 color;
  uniform sampler2DRect tex0;

  varying vec2 texcoord0;

  void main() {
    vec4 textureColor = texture2DRect(tex0, texcoord0);
    gl_FragColor = vec4(color, 1.0);
  }

]]>
</program>
</language>
</jittershader>

In case we want to disable the @rectangle attribute for [jit.gl.texture] we have to use here the function texture2D instead of texture2DRect and sampler2D instead of sampler2DRect.

Apply the Texture Values
Now is the time to apply the color from the texture to our 3D shape. We can do that by multiplying the red, green and blue channels of the texture for the color uniform variable:

<program name = "fp" type = "fragment">
<![CDATA[

  uniform vec3 color;
  uniform sampler2DRect tex0;

  varying vec2 texcoord0;

  void main() {
    vec4 textureColor = texture2DRect(tex0, texcoord0);
    gl_FragColor = vec4(color * textureColor.rgb, 1.0);
  }

]]>
</program>
</language>
</jittershader>

By default the color uniform contains the values 1.0 1.0 1.0, so the texture will be reproduced without modifications.

In our window we can now see the texture applied to the object through the shader:

Texure applied to the 3D shape via shader

Since we are now using a shader, the @apply attribute of [jit.gl.texture] is not used anymore.

We made it! Let's now see how we can use the values from a texture to modify the shape of our 3D object.

Read the Texture from inside the Vertex Program

Let's add to our patch a [jit.bfg] object, with 3 planes, type float32 and dimensions 100 x 100. With those dimensions every vertex can be mapped to a cell of the bfg matrix.
We can start by setting the bfg @basis attribute to noise.simplex and the @scale attribute to 4 4 for both dimensions.

Then let's pass the bfg matrix inside a new [jit.gl.texture] object sending a bang to the [jit.bfg] object. We are passing the texture to the [jit.gl.gridshape] adding the name of the new texture (texture1) to its texture attribute:

[jit.bfg] created and new texture passed to [jit.gl.gridshape]

I also added a [jit.gl.handle] object to be able to move the 3D shape around.

Pass the Second Texture as a Uniform
We have to pass also the new texture as a uniform to our jxs structure as we did for the previous texture:

<jittershader name = "myFirstShader">
<description> That is my first shader </description>
<param name = "color" type = "vec3" default = "1.0 1.0 1.0" />
<param name = "tex0" type = "int" default = "0" />
<param name = "tex1" type = "int" default = "1" />
<language name = "glsl" version = "1.2">
<bind param = "color" program = "fp" />
<bind param = "tex0" program = "fp" />
<bind param = "tex1" program = "vp" />
<program name = "vp" type = "vertex">
<![CDATA[

uniform sampler2DRect tex1;

This time we are linking the new texture with the vertex program, so we bind it to the "vp" program instead of the "fp".
We also declare the new uniform tex1 in the vertex program before the main function as a sampler2DRect type.

Create new Texture Coordinates
Now we have to go inside our fragment program and fetch the cells of the texture to modify the position of the vertices.
To do that we need to create new texture coordinates for our new texture:

<program name = "vp" type = "vertex">
<![CDATA[

uniform sampler2DRect tex1;

varying vec2 texcoord0;

void main() {

  texcoord0 = vec2(gl_TextureMatrix[0] * gl_MultiTexCoord0);

  vec2 texcoord1 = vec2(gl_TextureMatrix[1] * gl_MultiTexCoord1);

  gl_Position = ftransform();
}

]]>
</program>

Note that we don't have to create the variable texcoord1 as a varying variable, because we are not going to pass it to the fragment shader. We can therefore create it directly inside the main function.

Let's also note that we are using this time the index 1 for the gl_TextureMatrix array, and also we are using the gl_MultiTexCoord1 variable to access the texture coordinates from [jit.gl.gridshape], since we are now accessing the second texture linked to the object.

Access the Second Texture Data
Now we are going to get the values from the noise.simplex texture and use them to modify the position of the vertices.
To do that we need to sample the new matrix as we did in the fragment shader:

<program name = "vp" type = "vertex">
<![CDATA[

uniform sampler2DRect tex1;

varying vec2 texcoord0;

void main() {

  texcoord0 = vec2(gl_TextureMatrix[0] * gl_MultiTexCoord0);

  vec2 texcoord1 = vec2(gl_TextureMatrix[1] * gl_MultiTexCoord1);
  vec3 textureNoise = texture2DRect(tex1, texcoord1).xyz;

  gl_Position = ftransform();
}

]]>
</program>

When we passed the bfg matrix inside the [jit.gl.texture] the matrix has been automatically converted into a 4 planes texture, but actually we just want the first 3 planes inside our shader, since we are interested in those random values only as x,y,z coordinates for our displacement.
This is why we swiz the first 3 planes of the texture before passing the result into the textureNoise variable using the ".xyz" syntax.

Add the Noise to the Vertices
Let's now add the noise inside the textureNoise variable.
We will do the addition in model space. If you don't remember what the model space is, I suggest you to revise the previous tutorial Introduction to Shaders. Basically it means that we are working with the values of the vertices of the 3D shape before they are being translated, scaled, or rotated.

Let's create a new vec3 variable and store into it the value of the vertex plus the noise:

<program name = "vp" type = "vertex">
<![CDATA[

uniform sampler2DRect tex1;

varying vec2 texcoord0;

void main() {

  texcoord0 = vec2(gl_TextureMatrix[0] * gl_MultiTexCoord0);

  vec2 texcoord1 = vec2(gl_TextureMatrix[1] * gl_MultiTexCoord1);
  vec3 textureNoise = texture2DRect(tex1, texcoord1).xyz;

  vec3 displacedPos = gl_Vertex.xyz + textureNoise*0.1;

  gl_Position = ftransform();
}

]]>
</program>

gl_Vertex is a vec4 built-in variable, so to add the vec3 noise to it we have to swiz the xyz components with the .xyz syntax.
Before the addition we multiply the noise for a small value like 0.1, in order to reduce the displacement of the vertices.

Let's now pass the new vertex position to the built-in gl_Position output variable:

<program name = "vp" type = "vertex">
<![CDATA[

uniform sampler2DRect tex1;

varying vec2 texcoord0;

void main() {

  texcoord0 = vec2(gl_TextureMatrix[0] * gl_MultiTexCoord0);

  vec2 texcoord1 = vec2(gl_TextureMatrix[1] * gl_MultiTexCoord1);
  vec3 textureNoise = texture2DRect(tex1, texcoord1).xyz;

  vec3 displacedPos = gl_Vertex.xyz + textureNoise*0.1;

  gl_Position = gl_ModelViewProjectionMatrix * vec4(displacedPos, 1.0);
}

]]>
</program>

Instead of using the ftransform() function we now have to esplicitely transform the vertex using the gl_ModelViewProjectionMatrix matrix, since we want to use a custom vertex position.
Notice that to multiply the new vertex for the transformation matrix we have to transform it into a vec4. We simply put a 1.0 in the fourth component to accomplish that.

When we save the jxs program, the shape in our window should look like that:

The shape with the displaced vertices

Improve Appearance
Let's now improve a bit the appearance of our object by working a bit on the shader code.
First of all let's multiply the values of the color texture by the absolute values of the distortion texture. We need the absolute values because it doesn't make sense to have negative numbers for the color.

Remember that the values of the matrix from [jit.bfg] noise.simplex go in a range a bit bigger than [-1, 1]. So it has also negative numbers.

To multiply the two textures values we need to pass the values from the distortion texture down to the fragment program. To do that we need to declare the textureNoise variable as a varying variable:

<program name = "vp" type = "vertex">
<![CDATA[

uniform sampler2DRect tex1;

varying vec2 texcoord0;
varying vec3 textureNoise;

void main() {

  texcoord0 = vec2(gl_TextureMatrix[0] * gl_MultiTexCoord0);

  vec2 texcoord1 = vec2(gl_TextureMatrix[1] * gl_MultiTexCoord1);

  textureNoise = texture2DRect(tex1, texcoord1).xyz;

  vec3 displacedPos = gl_Vertex.xyz + textureNoise*0.1;

  gl_Position = gl_ModelViewProjectionMatrix * vec4(displacedPos, 1.0);
}

]]>
</program>

We can now declare and use it inside the fragment shader:

<program name = "fp" type = "fragment">
<![CDATA[

  uniform vec3 color;
  uniform sampler2DRect tex0;

  varying vec2 texcoord0;
  varying vec3 textureNoise;

  void main() {
    vec4 textureColor = texture2DRect(tex0, texcoord0);
    gl_FragColor = vec4(color * textureColor.rgb * abs(textureNoise), 1.0);
  }

]]>
</program>
</language>
</jittershader>

If we save the file the shape in our window should look like that:

The shape with the two textures multiplied

I also changed the dimensions of the [jit.noise] matrix and rescaled the [jit.bfg] function using the @scale attribute followed by a bang:

Changed the dimensions of [jit.noise] and @scale attribute

If you change the float number above the @scale attribute you can see the shape change in real time.

Fixing the Artifact
As you can see there is something wrong with our shape: it looks like it is cut on one axis. That's because our distortion and color texture start and end with different values, and when wrapped together those values don't correspond.

In order to solve that we can use the @wrap mirroredrepeat attribute of [jit.gl.texture], which will repeat the texture mirroring it when the texture coordinates go above 1 or below 0.
In order for this attribute to work though, we have to set the @rectangle attribute of both [jit.gl.texture] objects to 0, and modify consequently our shader code:

The [jit.gl.texture] objects with the attributes modified

And now the shader code:
First the vertex program:

<program name = "vp" type = "vertex">
<![CDATA[

uniform sampler2D tex1;

varying vec2 texcoord0;
varying vec3 textureNoise;

void main() {

  texcoord0 = vec2(gl_TextureMatrix[0] * gl_MultiTexCoord0);

  vec2 texcoord1 = vec2(gl_TextureMatrix[1] * gl_MultiTexCoord1);

  textureNoise = texture2D(tex1, texcoord1 * 2.0).xyz;

  vec3 displacedPos = gl_Vertex.xyz + textureNoise*0.1;

  gl_Position = gl_ModelViewProjectionMatrix * vec4(displacedPos, 1.0);
}

]]>
</program>

We changed the sampler2DRect into sampler2D since we are not using rectangle textures anymore.
We also changed the texture2DRect function into texture2D to access the texture.
Since we are using @rectangle 0 the texture coordinates are now normalized.
We multiply the texcoord1 variable by two in order to have the texture repeated and mirrored.

We make the same modifications into the fragment program:

<program name = "fp" type = "fragment">
<![CDATA[

  uniform vec3 color;
  uniform sampler2D tex0;

  varying vec2 texcoord0;
  varying vec3 textureNoise;

  void main() {
    vec4 textureColor = texture2D(tex0, texcoord0 * 2.0);
    gl_FragColor = vec4(color * textureColor.rgb * abs(textureNoise), 1.0);
  }

]]>
</program>
</language>
</jittershader>

We can change the @dim attribute of the [jit.gl.gridshape] object to 200 200, or more, in order to have more resolution:

The [jit.gl.gridshape] with the dim attribute modified

The final result should look something like that:

The [jit.gl.gridshape] with the shader applied

We can make the shape continuously change by using the @offset attribute for the dimensions of the [jit.bfg]:

The [jit.gl.gridshape] with the shader applied

Conclusion

That was it for this second introductive tutorial about shaders. We saw how to use textures inside Max and how to read and use them inside a shader program. We unlocked a new tool that allows us to achieve endless graphical effect, some of which we will see in further tutorials.
Thank you for following!

Patreon
Writing this tutorial was a pleasure, but it required quite some time. I couldn't do that if it wasn't for the Patreon friends that support me.
You can support me with a small donation becoming a Patron at this link:
Amazing Max Stuff Patreon.
This will also give you access to all my Patron-only patches.

You can download the patch and the shaders used in this tutorial for free also on my Patreon: Patch Download.

Donation
If you don't want to create a Patreon account you can also send me a one-time donation:

Thank you guys, this helps me so much!

Facebook
For all the updates on tutorials or patch sharing follow my facebook page: Amazing Max Stuff FB.