Should I Render Three.js to Video on the Client or Server?

I’m working on an exciting new geegaw with a business partner and old friend in Seattle. It’s a 3D music visualizer built in HTML5 / WebGL using PureMVC, React, and Three.js. We already have a solid, extendable client framework, and some great stuff going on in the rendered scene, which the user can customize and then save.

Now we want to generate a video, so we can upload it to YouTube or share it on Facebook.

Our original plan was to render the saved project on the server with Node.js, render-farm stylee. That would allow us to ensure it played smooth and didn’t stutter due to unknown client load and/or capabilities. Should be a snap, right? Er, no.

If you’ve researched this topic to any degree, you probably already know that there’s currently no out-of-box, miracle solution to make this happen, and certainly no example lash-togethers that will absolutely give you the same rendering on the server as you see on the client. If there is, and I’ve missed it (or if one comes along months from now), please let me know in the comments. For anyone else who may be trodding this path, her face sullen, set, and grim.

What’s wrong with rendering on the server?

  • There is an npm package for Three.js, however, there are a lot of document.createElement calls, and so it needs a DOM. The example on the npm page won’t work in node.js at all for this reason. I assume it is there for use with browserify and not specifically intended for server-side rendering.
  • You easily can add a DOM to your project with a package like jsdom, however you won’t be able to do more than get a CanvasRenderer. For a WebGLRenderer, you need a WebGL implementation, which jsdom doesn’t give you.
  • Assuming it’s possible to get a WebGLRenderer, you’ll most likely need to implement a requestAnimationFrame() method depending on what library you chose for adding your DOM. No biggie, it’s just in charge of calling the passed in callback when a new frame is needed. You can just use setInterval() to call it at the desired frame rate.
  • There are several options for out-of-browser WebGL, including:
  • The above link for headless-gl takes you straight to the list of reasons why you might choose it over node-webgl or electron. I was moved to chose headless-gl.
  • Finally, here is the beginning of several hero posts in an issue thread about getting Three.js and WebGLRenderer to work in Node.js. The gist is in CoffeeScript. But Texturing and AntiAliasing both required special attention after getting a basic PNG of a cube drawn.

Wait a minute. Am I certain this is the only way to go about this?

It is at this point in my research that I begin to question all of the dependencies and potential hacks required to pull this off, and whether it truly is The Right Way(tm). I don’t take on project dependencies lightly. My little test project for figuring this part out had only the following top-level dependencies:

"dependencies": {
  "gl": "^4.0.2",
  "jsdom": "^9.5.0",
  "three": "^0.81.2"
}

But a quick peek into node_modules shows (…OMG…):

Destiny:node_modules cliff$ ls -al |wc -l
187

Yikes! A hundred and eighty seven packages already? That’s a lot of independently maintained places for this to fail. A lot of roads to dependency hell.

What are the pros and cons of trying to do this client-side?

  • One obvious win is that we don’t pay our cloud provider for the CPU cycles necessary to render the frames.
  • We were worried about potential load or capabilities issues on the client, but actually, our particular target demographic (music producers) probably have pretty hefty machines anyway. And if they can get a scene they want to share working on their screen, then why not just hit record right then?
  • The frame rates we’re currently achieving on relatively ordinary hardware (my 2011 MacBook Air, for instance) are around 60 FPS. We’d only need about 30 FPS for video anyway, so we can probably spare a few cycles to grab each frame as an image. But can we turn those images into a movie?
  • Amazingly, the popular ffmpeg has been translated to JS and runs in the browser. However the discussion thread where I discovered that indicates that it may not be ready for production. Hmmm. It’s an avenue for exploration at least.
  • Here’s a different approach: Render on the client, send the rendered frames to the server, and create the movie there.
  • Creating a movie from a sequence of files is going to be a much more well-trodden server-side path than rendering WebGL. Probably a less dodgy proposition.
  • In the above linked example, the developer is A) writing frames to a node server on his local machine, B) doing it with synchronous XHR requests, and C) making those requests within the render loop. I don’t think this is feasible over the Interwebz, even given the fact that we’d be doing 30 FPS instead of 60.
  • I believe the concept is still sound, if we 1) write the frames to the browser’s local storage as a buffer within the render loop, 2) have a separate web worker process that sends those frames to the server, and 3) keeps a socket open and dumps in frame after frame until we’re done, eliminating the chatty overhead associated with all the individual XHR requests.
  • The audio should be on the server already, and should be trivial to include when creating the videos from the images.

Alright. I’m convinced that exploring the path of client-side frame rendering with server-side video encoding is worth giving a try. I’ll get back to you in another post with what I find. In the meantime, if you have anything to weigh in with, please hit me up in the comments section below!

Addendum

A good point was brought up on the Dzone comment thread: Am I not worried that I’d be trading CPU cycles for bandwidth in the charges from our cloud provider by going to a server side assembly of images with client side rendering. For us, this would be an issue if we’d stuck with our initial intended provider of AWS. However, after recent great experiences with Google’s Cloud Platform we looked into it and discovered they offer free ingress traffic.

So, in short, we’re still fully committed to exploring this architecture and have already had some success with the microservice. I’ll be writing about that shortly.


Author’s Note: This article is part of a series, wherein my partner and I are trying out the development of our a product ‘out in the open’.

This is diametrically opposed to my typical ‘skunk works’ approach, holing myself up for months on end in a pointless attempt to keep secret something that will eventually become public anyway. We’re not building anything earth-shattering, paradigm-shifting, or empire-building. Just something cool that serves a niche we know and are interested in helping. It’s a 3D music visualizer built in HTML5 / WebGL using Three.js., PureMVC, React, and Node.js. When we’re done, you’ll be able to create a cool video for your audio track and upload it to YouTube.

The benefit of blogging about it as we go is that we get a chance to pass on some of our thought processes as we navigate the hurdles and potholes strewn along our path. Getting those thoughts down while they’re still fresh in mind might guide someone else following a similar path. If we fail owing of these decisions, maybe it’ll help you avoid your own smoking crater. Either way, later on, we’ll be  busy chasing different squirrels in some other park.

The next article in this series is: Creating Video on the Server in Node.js

This article has been reblogged at the following sites:

DZone: http://bit.ly/threejs-rendering-client-or-server


8 thoughts on “Should I Render Three.js to Video on the Client or Server?”

  1. Congrats for this interesting article! I am also in the process of searching for a solution to render video from webgl on the server.

    With your technique you have to upload every frame to the server, so reasonable bandwidth is required, and then the user needs to have a machine that is not too old.

    I see that it is possible to rent servers with dual GTX 970 video cards for about 200$ per month, so webgl rendering on these would be very fast.

    It depends on your target audience, but it might be a good idea to have server-side webgl rendering as a fallback. But your points are valid that it seems a pain to get it to work headless, so might not be worth the hassle…

    1. Hi Antoine,

      It would be good to have server side rendering as a fallback, but as I’ve found in this article it’s not going to be easy, nor will I be sure that the output is the same as it would be in the browser.

      I’ve just completed some testing with rendering in the browser and pushing the frames across a socket to be saved and later turned into a video. (http://bit.ly/a-three-js-socket-io-node-js-render-and-transmit-test). In this experiment, I found that it is possible, but doing so in realtime is impossible. Even rendering offscreen and using a web worker to offload the socket I/O, the problem is that the canvas.toDataURL() call takes ~116ms. For realtime rendering at 30fps, that number needs to be less than 33.33ms. If I don’t do the canvas.toDataURL() call it only takes ~20ms per frame.

      The problem for me is that in my actual app, I am also doing audio processing, such that scene objects can be tied to activity in the audio spectrum and overall volume. So, that means things are going to get out of sync. I even precreated an array of WebGLRenderTargets and render to the next one each time rather than try to extract the data inside the render loop. Even adding a new target to the third param of the WebGLRenderer’s render() method causes the time per frame takes ~20ms longer than just rendering to the same target each time, leaving me at >33.33ms per frame.

      That means I can’t render audio-affected scenes in realtime and capture each frame, period. I will need to pre-compute the audio sample data, and use the appropriate values for the frame being rendered. That will free me from the shackles of realtime rendering, and the frames and the audio should line up appropriately in the final video.

    2. Also, Antoine, please, please please, let me know if you find a solid approach to server side rendering of Three.js scenes.

      Cheers,
      -=Cliff>

  2. There is a Linux video editor named Shotcut that can import webgl. Here is an interesting demo:

    https://www.youtube.com/watch?v=xs3bv1TzkYw

    It turns out that Shotcut is just a GUI on top of the MLT framework which is designed to be remote controlled via scripting : https://www.mltframework.org/docs/melt/

    I asked the creator if MLT could be used to render webgl on a remote server and export it to video, and here is what he answered : https://github.com/mltframework/mlt/issues/138

    1. Hi Antoine. I searched the mlt site and repo and didn’t find any mentions of ‘webgl’. Also, at least for my app, I have the constraint of needing to sync scene elements to audio. I’m not sure if this would do that, particularly if it can’t render in realtime, which is an issue for my app. Headless or not, I’m going to have to go to an approach of extracting the audio levels prior to rendering, in order to keep sync.

  3. I guess if you can control the speed of your threejs scene then you could play it at 20%, then you would capture every frame and then play it back at 5x server side with ffmpeg, overlay audio and it should be in sync?

    1. Something similar is what I eventually came up with. Over the next four articles, (culminating with http://cliffordhall.com/2016/11/breaking-free-realtime-jail-creating-audio-modulated-hd-video-three-js/) I figured out how to create the video ont the server, communicate with it via sockets to send over rendered frames, and finally render frames – not in real time – at 48fps and then layer the audio over it on the server. Works pretty well. You just have to toss the idea of being able to do this in real time.

Leave a Reply to Antoine Cancel reply

Your email address will not be published. Required fields are marked *