Vuforia and LibGDX 3D model renderer

Making simple, yet interesting AR/VR apps tends to get a little overcomplicated, especially when 3D is involved. Vuforia nicely handles the VR/AR part, but if you need to render some non-trivial 3D models, you usually have to rely on Unity, which can get expensive and complicated really fast (And usually requires some quality time with C++ and NDK). Therefore in this text I’d like to describe how to use LibGDX – a nice, simple game engine – to render 3D models together with Vuforia.

If you aren’t interested in the how-s, you can also skip right to the example github repo.

3D Model

Before you start coding (or after, I don’t care, I’m not your boss), you need to prepare the model you’d like to render. In general, LibGDX is quite powerful, however, there are some limitations.

First, you are going to need your model as FBX file and convert it into the G3DB/G3DJ format (See LibGDX tutorial on how to handle this). There is a ton of free models on the internet and most of them can be used without an issue, however, make sure that:

• The model isn’t too big: On modern devices, you can usually afford to go as far as 20-30MB models. However, if you aren’t targeting the latest hardware, it’s usually safer to stick to something in the <10MB category.
• The G3DB format supports only 32k vertices per mesh. If you have a big complex model, it usually has to be split into multiple meshes.
• The textures have to be PNG or JPEG. A lot of models come with TGA textures. You have to convert these to something LibGDX can read.
• Make sure that there are no corrupted/missing textures. Best way to do this is to convert the model to G3DJ format, open it is some text editor and at the end of the file check the materials section for any weird stuff.
• The conversion utility is sometimes little buggy, if it’s not working as you expect (especially with materials), try fiddling with the export settings in your 3D editor (not really my area of expertise).
• Usually you are going to need to rotate/scale the model to fit the coordinate system of LibGDX and your recognition target size. You can do this in code (good for “debugging” models), however it has to be done for each frame, so it might be a good idea to do it beforehand in some editor in order not to sacrifice performance.

If everything goes well, you should end up with a G3DB model and possibly some texture JPEG/PNG files in your assets folder.

For this example, we are going to use this model of a combat jet.

Integration

We would like to separate Vuforia and LibGDX as much as possible. We can’t make them completely independent, since they have to synchronise on frame rendering and use the same View after all. As far as I can tell, it is also kind of hard to interfere with the LibGDX rendering loop. Therefore we are going to leave the main render loop in hands of LibGDX and render Vuforia content from there.

First, let’s look at the Vuforia related code.

It is located in the ar.vuforia package in the example project and contains pretty much standard Vuforia integration. The fun stuff happens in the VuforiaRenderer class. Standard Vuforia renderers usually implement GLSurfaceView.Renderer. We are not going to use this interface, because the renderer will be called from LibGDX. Instead, we define a processFrame method that is going to render the camera background and return the recognised targets back to LibGDX.

Next we have a look at the LibGDX code.

It is located in the ar.libgdx package and is also based on one of the examples. The key change is that both Display and Engine classes have access to the VuforiaRenderer and are responsible for calling the appropriate methods such as initRendering and surfaceChanged when needed.

Apart from this, the only thing happening in Engine and Display is model loading. It happens on the main thread, which of course will freeze your app. In real life, you will have to load the models on background using some dedicated loading screen. We don’t bother with this here, for simplicity. It is also a good idea to specify in the manifest that your app requires larger heap:

android:largeHeap="true"

A 10MB model can easily take up as much as 100MB of RAM while standard Android heap sizes are usually just around 64MB.

Finally, the place where all of this comes together is the LibGDX Renderer class. In the constructor, we prepare the render environment – lights, camera and model renderer. More about this stuff can be found in any good LibGDX tutorial.

With the environment ready, we can start rendering frames. First, Vuforia renderer is called to render camera background and return recognised targets. We then use these targets to update the camera position. The idea is that the model stays at it’s initial position while the camera moves around it and creates the AR illusion (if no target is recognised, the camera is pointed away from the model).

The only problem is that LibGDX and Vuforia use different matrix representations for position (and it also depends on which physical camera is being used – front or back). We have to swap the first two columns and then in order to obtain correct camera position perform a transposition and inversion of the matrix. Finally we can use this data to update camera location.

Here we can also update the model scale/rotation if necessary.

All of this is then integrated together in the ArActivity, which is again a pretty standard merge of the LibGDX and Vuforia example activities.

And that’s it. The example code can use some polishing (feel free to make pull requests), but should provide you with a good starting point when creating simple yet powerful AR/VR apps. If you have any question, you can hopefully catch me in the comments.

Troubleshooting

If your model is not visible, first check that it has the right scale and orientation.

If the model is too big, the camera will get inside of it and nothing will be visible. Similarly, if the model is too small, you won’t be able to see it. Experiment with different scaling factors.

If your model is only partially visible, try changing the visibility distance (camera.far property). Depending on scale, sometimes your model will be cut out because it is too far.

In case of weird/missing materials, check the G3DJ file and look if the materials there correspond to what you would expect. Also make sure there is a proper lighting in the scene.

Some references:
forum1, forum2 and stack overflow

Chroma Keying with Vuforia (a.k.a Transparent Video)

Ok, so you want to use transparent video in your augmented reality Android app. Sorcery? Magic? No. OpenGL.

You just need to change few lines of code in original Vuforia VideoPlayback sample app to make it work.

In the past, you had to use the NDK to compile some nasty pieces of C++ to nicely render transparency as you wanted. With new version of Vuforia, you don’t need to bother yourself with such things. You can write everything using only Java. In case you are interested in the old good native way, Pilcrow Pipe wrote a great article about it.

The Video

To make things transparent, we are going to use Chroma Keying. This process makes transparent all pixels with color that falls into specified range. It is good to avoid smooth transitions between transparent and solid area and shadows. They can alter color of transparent areas enough to make them visible again. Therefore creating colorful “ghosts”.

In this example, we will use this video, which should do well, since all transitions are sharp and there are no shadows involved.

The Code

You know the drill. Download samples from Vuforia site, import VideoPlayback sample into your IDE/workspace/notepad/… and make sure you can build it and run it. Or you can download example Android Studio project from Github.

Replace original videos in assets with your own video. (If you want to have matching video preview, you will have to replace also the .png files) If you changed the file name, make sure to update it correspondingly in the VideoPlayback Activity.

Note: We are talking only video transparency here. Sample app can render transparent png images without special shader, so you don’t have to bother with OpenGL magic, you just have to enable blending. (more on that later)

Now the important part! Open VideoPlaybackShaders and replace VIDEO_PLAYBACK_FRAGMENT_SHADER with this code:

public static final String VIDEO_PLAYBACK_FRAGMENT_SHADER =
"#extension GL_OES_EGL_image_external : require \n" +
"precision mediump float; \n" +
"uniform samplerExternalOES texSamplerOES; \n" +
" varying vec2 texCoord;\n" +
" varying vec2 texdim0;\n" +
" void main()\n\n" +
" {\n" +
" vec3 keying_color = vec3(0.647, 0.941, 0.29);\n" +
" float thresh = 0.45; // [0, 1.732]\n" +
" float slope = 0.1; // [0, 1]\n" +
" vec3 input_color = texture2D(texSamplerOES, texCoord).rgb;\n" +
" float d = abs(length(abs(keying_color.rgb - input_color.rgb)));\n" +
" float edge0 = thresh * (1.0 - slope);\n" +
" float alpha = smoothstep(edge0, thresh, d);\n" +
" gl_FragColor = vec4(input_color,alpha);\n" +
" }";

In the keying_color variable is stored the actual color we want to replace. It is using classic RGB model, but intensity is not expressed as 0-255 integer. It is a float value in range 0-1. (So 0 = 0, 255 = 0, 122 = 0.478…) In our case, the green color has value (0.647, 0.941, 0.29), but if you are using different video, measure the color yourself.

Note: Make sure you have the right color. Some color measurement software automatically converts colors to slightly different formats, such as AdobeRGB.

So where’s the magic?

We load current pixel color in the input_color, then calculate difference between input and keying color. Based on this difference, alpha value is calculated and used for specific pixel.

You can control how strict the comparison is using the slope and threshold values. It is a bit more complicated, but the most basic rule is: The more threshold you have, the bigger tolerance.

So, we are done, right? Nope.

Blending

To enable transparency, you need to also enable blending. Without blending, every time you write a new pixel color value, transparent or not, the old one gets deleted. So you get nice video playback, but you destroy the camera image behind it.

To enable blending, insert following two lines of code into VideoPlaybackRenderer class, renderFrame method, before each of these two glUseProgram calls:

GLES20.glUseProgram(keyframeShaderID);
GLES20.glUseProgram(videoPlaybackShaderID);.

Code to insert:

GLES20.glEnable(GLES20.GL_BLEND);
GLES20.glBlendFunc(GLES20.GL_SRC_ALPHA, GLES20.GL_ONE_MINUS_SRC_ALPHA);

First glUseProgram call is responsible for rendering png video previews. So if you don’t use transparency in you png files, you can skip that. Second call is used when rendering actual video.

Regarding those two lines you have to insert: They enable blending and set it to blend pixels based on their alpha values. So the colors of pixels are not affected and only alpha values matter.

Done?

Not exactly. On most devices, this will do. But on some, you will still get black background instead of camera image. That’s because we turned the blending on, but we never turned it off.

To turn blending off, you have to add this line after all glUseProgram(0) calls:

GLES20.glDisable(GLES20.GL_BLEND);

There should be three glUseProgram(0) calls, the last one is already followed by such line, so that one shouldn’t concern you.

And that’s it? Yes. That’s it.

Useful links and resources:

Github project with example app

Pilcrow Pipe article about doing this in older Vuforia

Some forum threads: link1 link2