Should I Render Three.js to Video on the Client or Server?

I’m working on an exciting new geegaw with a business partner and old friend in Seattle. It’s a 3D music visualizer built in HTML5 / WebGL using PureMVC, React, and Three.js. We already have a solid, extendable client framework, and some great stuff going on in the rendered scene, which the user can customize and then save.

Now we want to generate a video, so we can upload it to YouTube or share it on Facebook.

Our original plan was to render the saved project on the server with Node.js, render-farm stylee. That would allow us to ensure it played smooth and didn’t stutter due to unknown client load and/or capabilities. Should be a snap, right? Er, no.

If you’ve researched this topic to any degree, you probably already know that there’s currently no out-of-box, miracle solution to make this happen, and certainly no example lash-togethers that will absolutely give you the same rendering on the server as you see on the client. If there is, and I’ve missed it (or if one comes along months from now), please let me know in the comments. For anyone else who may be trodding this path, her face sullen, set, and grim.

What’s wrong with rendering on the server?

• There is an npm package for Three.js, however, there are a lot of document.createElement calls, and so it needs a DOM. The example on the npm page won’t work in node.js at all for this reason. I assume it is there for use with browserify and not specifically intended for server-side rendering.
• You easily can add a DOM to your project with a package like jsdom, however you won’t be able to do more than get a CanvasRenderer. For a WebGLRenderer, you need a WebGL implementation, which jsdom doesn’t give you.
• Assuming it’s possible to get a WebGLRenderer, you’ll most likely need to implement a requestAnimationFrame() method depending on what library you chose for adding your DOM. No biggie, it’s just in charge of calling the passed in callback when a new frame is needed. You can just use setInterval() to call it at the desired frame rate.
• There are several options for out-of-browser WebGL, including:
  • node-webgl
  • electron
  • headless-gl (confusingly, just gl on npm, but headless-gl in docs and on GitHub).
• The above link for headless-gl takes you straight to the list of reasons why you might choose it over node-webgl or electron. I was moved to chose headless-gl.
• Finally, here is the beginning of several hero posts in an issue thread about getting Three.js and WebGLRenderer to work in Node.js. The gist is in CoffeeScript. But Texturing and AntiAliasing both required special attention after getting a basic PNG of a cube drawn.

Wait a minute. Am I certain this is the only way to go about this?

It is at this point in my research that I begin to question all of the dependencies and potential hacks required to pull this off, and whether it truly is The Right Way^(tm). I don’t take on project dependencies lightly. My little test project for figuring this part out had only the following top-level dependencies:

"dependencies": {
  "gl": "^4.0.2",
  "jsdom": "^9.5.0",
  "three": "^0.81.2"
}

But a quick peek into node_modules shows (…OMG…):

Destiny:node_modules cliff$ ls -al |wc -l
187

Yikes! A hundred and eighty seven packages already? That’s a lot of independently maintained places for this to fail. A lot of roads to dependency hell.

What are the pros and cons of trying to do this client-side?

• One obvious win is that we don’t pay our cloud provider for the CPU cycles necessary to render the frames.
• We were worried about potential load or capabilities issues on the client, but actually, our particular target demographic (music producers) probably have pretty hefty machines anyway. And if they can get a scene they want to share working on their screen, then why not just hit record right then?
• The frame rates we’re currently achieving on relatively ordinary hardware (my 2011 MacBook Air, for instance) are around 60 FPS. We’d only need about 30 FPS for video anyway, so we can probably spare a few cycles to grab each frame as an image. But can we turn those images into a movie?
• Amazingly, the popular ffmpeg has been translated to JS and runs in the browser. However the discussion thread where I discovered that indicates that it may not be ready for production. Hmmm. It’s an avenue for exploration at least.
• Here’s a different approach: Render on the client, send the rendered frames to the server, and create the movie there.
• Creating a movie from a sequence of files is going to be a much more well-trodden server-side path than rendering WebGL. Probably a less dodgy proposition.
• In the above linked example, the developer is A) writing frames to a node server on his local machine, B) doing it with synchronous XHR requests, and C) making those requests within the render loop. I don’t think this is feasible over the Interwebz, even given the fact that we’d be doing 30 FPS instead of 60.
• I believe the concept is still sound, if we 1) write the frames to the browser’s local storage as a buffer within the render loop, 2) have a separate web worker process that sends those frames to the server, and 3) keeps a socket open and dumps in frame after frame until we’re done, eliminating the chatty overhead associated with all the individual XHR requests.
• The audio should be on the server already, and should be trivial to include when creating the videos from the images.

Alright. I’m convinced that exploring the path of client-side frame rendering with server-side video encoding is worth giving a try. I’ll get back to you in another post with what I find. In the meantime, if you have anything to weigh in with, please hit me up in the comments section below!

Addendum

A good point was brought up on the Dzone comment thread: Am I not worried that I’d be trading CPU cycles for bandwidth in the charges from our cloud provider by going to a server side assembly of images with client side rendering. For us, this would be an issue if we’d stuck with our initial intended provider of AWS. However, after recent great experiences with Google’s Cloud Platform we looked into it and discovered they offer free ingress traffic.

So, in short, we’re still fully committed to exploring this architecture and have already had some success with the microservice. I’ll be writing about that shortly.

Author’s Note: This article is part of a series, wherein my partner and I are trying out the development of our a product ‘out in the open’.

This is diametrically opposed to my typical ‘skunk works’ approach, holing myself up for months on end in a pointless attempt to keep secret something that will eventually become public anyway. We’re not building anything earth-shattering, paradigm-shifting, or empire-building. Just something cool that serves a niche we know and are interested in helping. It’s a 3D music visualizer built in HTML5 / WebGL using Three.js., PureMVC, React, and Node.js. When we’re done, you’ll be able to create a cool video for your audio track and upload it to YouTube.

The benefit of blogging about it as we go is that we get a chance to pass on some of our thought processes as we navigate the hurdles and potholes strewn along our path. Getting those thoughts down while they’re still fresh in mind might guide someone else following a similar path. If we fail owing of these decisions, maybe it’ll help you avoid your own smoking crater. Either way, later on, we’ll be  busy chasing different squirrels in some other park.

The next article in this series is: Creating Video on the Server in Node.js

This article has been reblogged at the following sites:

DZone: http://bit.ly/threejs-rendering-client-or-server