Taking a HoloLens visualization from the screen to a phone

I really wanted to understand the Vuforia AR kit, and had a couple of ideas that would prompt me to actually sit down and get it working. I found that the online tutorials are a bit confusing, and nothing seemed to work ‘out of the box’ (at least for me).

Who knows, it could be my lack of understanding or my personal machine configuration. However, once I got things working, it turned out to be an absolute delight.

The process of registering an image with the database and a project is actually pretty simple. The idea was this: have an image target embedded into a spreadsheet. When recognized, the visualization would appear. Simple enough. However, the target image could exist anywhere — on a computer screen, on a phone, printed on a piece of paper. This gave me the idea that you could simulate the process of taking your 3D visuals with you, and use any device to rotate, tilt or zoom in and out. The video that I captured here illustrates the process on an iPhone and an iPad.

For the moment it is just conceptual. The image target is static (I just used a QR code that I generated and downloaded it to the device), which displays a simple 3D scatterplot of the same data in the spreadsheet. I also put a translucent copy of the QR code behind the data. However, I can imagine a process whereby the data (a spreadsheet, SQL query... anything) updates the visuals, and creates a unique image target marker (which is then uploaded to the Vuforia image target database).

Ideally, you could encapsulate the data to be visualized into a visual target. It’s not very realistic right now, but I foresee the technology progressing to the point that a high resolution, multi-color image can actually encode multi-megabytes of data. This would clearly need very high resolution cameras and ultra-fast processing to decode the image and re-interpret the source data.

The video, illustrating both iPhone and iPad use can be seen

The Black Lodge - HoloLens edition

After watching the new episodes of Twin Peaks, I felt inspired to try to re-create the elements of the space with Unity and the HoloLens. I thought it might be fun to see what it would be like to wander around the four-curtained walled lodge, and see what poor Agent Cooper has been stuck in, lo these may years.

What I thought would be a simple exercise, turned out to actually be much more difficult than expected. The Venus de Milo model that I found turned out to be much larger than the device could render, and I saw considerable tearing as I panned past the model. The solution was to bring it into Blender, and decimate it considerably (by about 90%). I had to do the same thing with the leather chairs, and the faceting can be seen on the backrest.

The curtains were another matter altogether. The Unity 'cloth renderer' is pretty flexible, but it's a delicate balance trying to get as many faces of the surface to look realistic, but without killing the rendering performance.

Global Poplulation Visualizer

This is a slick global population growth visualization that was created by Abdullah Aldandarawy. I found it on the Unity store, and, with a little bit of effort, ported it to the HoloLens. The actual asset did allow for some interactivity, but this version simply shows off the beauty of its execution.

The Unity store asset can be found here.

3D Trendalyzer

Often, in time-series visualizations, it is effective to give an indication of trend or direction. I was hoping that the Unity 'trail renderer' would be useful to show where a visual element was, and therefore add meaning to where it currently is, at that point in time, as one is stepping through a time-series.

This example also illustrates some subtle animation between subsequent dates the way Gapminder does so elegantly.

3D Surface visualization

For whatever reason, and for every technology, I start with some variation of this simple, hypnotic curve. It looks pretty and organic, and I could watch it for hours.

It also is a bit of a quick cheat. I can use it for testing positive and negative values, and there aren’t any nasty spikes, or awkward edge cases, and is naturally well-suited for animating.

This exercise gave me my first chance to dive into Unity vertex shaders. In the example above, I adapted a simple red to green function, based on the Y value (height) of the mesh. Moving forward, I then plugged in my favourite test data sets, US stocks, and a US crime data.

Since this data set did not have negative values, I decided to change the colour scale (low values green, high values red). It also made me realize that organizing data in this way doesn’t exactly reveal any insights. However, I would consider this a successful test of the surface visual, done completely from scratch. They are both fairly small, however, with only 20 by 20 ‘cells’ displayed. I was a bit reluctant to push too much geometry on the device, especially as it was my first attempt.

The biggest thing missing is the ‘tooltip’ drill-down, which gives the actual numeric values in the form of a floating text grid as the viewer gazes at a ‘cell’ in the visualization. The visualization affords a basic level of interaction (sort of) in the form of cycling through different metrics as you air-tap/click on the X or Z axes.

A holographic CAVE

This is an experimental holographic CAVE created for the HoloLens. The idea is to immerse yourself within the visuals. In a typical CAVE, the visuals are presented using rear-projection screens, or panel displays. Presenting them holographically obviously requires much less hardware, and can be used wherever there is enough space. In this experiment, I used five different visualizations, all using the same S&P dataset.
1. Scønes view. This is a thought experiment using scatterplot of inverted cone-like objects, with a mini
sparkline graph displayed on it's inverted base. The idea here was to be able to show the current state of the element, but also it's historical value.
2. Grid Array view. This is a simple 3D bar chart with row and column layout that allows the rows and columns to be sorted.
3. 3D scatter plot. This is the view on the right wall. Each data row is represented in X, Y, Z space as a spheroid.
4. Stalactites view. This is the view that is mapped on the CAVE ceiling (hence the name) This uses the TubeRenderer object and creates a spiralized tube shape. For now, this is just using random data.
5. Ring view. This can be seen on the left wall. I wanted to re-purpose the rings used for the Këpler visualization, and realized that there are many additional visual features that can be used with the ring. Inner, and outer thicknesses can set to create interesting shapes. How useful it is remains to be seen.

MultiViz: Four different visualizations

Four different visual representations of stock market data.
1. The Këpler view. Each row is represented as a 'planet' with a ring. Different metrics are applied to planet and ring size, planet and ring color, orbit speed, rotation speed etc.
2. Grid Array view. This is a simple 3D bar chart with row and column layout that allows the rows and columns to be sorted.
3. 3D scatter plot. Each data row is represented in X, Y, Z space as a spheroid.
4. Tubes view. This is something that is still at the concept stage. I was thinking that it would be interesting to present data in a sorted grid view, but allow the visualization to show where that element was over time. Imagine a stock market dataset, showing end of day prices. However, instead of a bar, it would be presented as a tube, with the opening price at the base of the 'tube' and the closing price at the top. This would highlight how much (or little) the stock had changed over the course of the day. The data repreented in the video is simulated random data.

Scattrpløt: An interactive holographic scatter plot visualization for the HoloLens

A more customary scatter plot visualization of the S&P 500 data set. Once again, drilldown details are given in the form of a tooltip as the user gazes at each spheroid. It actually started it's life as a scatter plot of cubes, but when I changed the shape prefab in Unity to a sphere with a metallic material, it looked so glossy and shiny I couldn't resist. This implementation has clickable axis labels so that the viewer can cycle through the metrics in the data set by 'clicking' on them (I'm using the clicker peripheral instead of air tapping).

The interactivity of this one makes it far more useful (and really speeds up the testing process). How the user will select the colour still elludes me... Unless there is some sort of Color Picker UI object out there (or if I need to build one from scratch)

The Këpler Visualization: 500 Stocks in 8 dimensions.

A visualization of all stocks in the S&P 500, using the planetary visualization construct that I have been experimenting with. Drilldown details are given in the form of a tooltip as the user gazes at each planetary element. Unfortunately, the video does not do it justice. The text in the tooltip is very crisp, and highly readable, and does not compete with the visuals. Noteably missing is a way of dynamically setting the visual features and applying them to the data dimensions. I'm hoping that voice command will not be too arduous to implement, as this would give the most amount of control without losing the immersiveness of the visualization.

Planetary scatter plot.

This is an experimental scatter plot visualization that represents many dimensions simulatneously, in what I'm calling, Project Këpler. This uses visual features like planet color, ring color, size, ring arc, orbit speed and direction, rotation speed (among others).

In this example, random data is used, and the lines were created with the very useful Vectrosity Unity asset. This started purely as a thought experiment, to see how many visual dimensions I could add to a scatter plot. The hardest part, amazingly, was building the mesh for a partial 3D ring. There wasn't anything built into Unity, and I had to build it from scratch, including the 'end caps' of each partial ring.

Space Invaded!

I wanted to try my hand at a simple game for the HoloLens. This is a loving tribute to one of my favourite games of all time, Space Invaders. However, I wanted to explore some of the features of the HoloLens, and see how integration of the XBox One controller would work. There are some interesting twists with this treatment, namely the addition of 'shields' to some classes of the invaders, as well as a 'fire-and-forget' seeking missle weapon for the player. I can't take credit for the particle effects used for the laser and explosions. Those were created using the Hyperbit Arsenal Particle Systems Unity Store asset.

Zombie Command

This is a prototype of a game I created to test out the XBox Bluetooth controller and the HoloLens. The idea here was to have a shooter type game, with the option of three air-based missle launchers, similar to the game Missle Command. Missles are launched with the X, A, and B buttons on the controller, and a laser is fired with the trigger. This is also my first real attempt at integrating particle effects and explosions. The video was captured in a nearby park (Saint Andrew's playground) on a bright fall day, and I was impressed at how well the HoloLens performed.

Strange Park

I wanted to extend my experiments with the HoloLens in an outdoor setting. This creation was made using a TubeRenderer Unity construct, with textures by Toronto artist Mike Parsons. The idea here was to create some organic-looking structures that could then be overlaid onto a park setting, and see how well they looked. I was blown away by the ability of the device to render the visual elements on a bright (but overcast) day. Interestingly, some passerby was pacing through the park, on his cell phone and oblivious to my HoloLens recording. It really does look like he his moving between the 'creatures'.

Star Castle Prototype

This in an homage to an old Cinematronics video arcade game called Star Castle. The premise of the original game was to blast away at an enemy that was protected by three layers of counter rotating shields. I thought it might be interesting to re-create a HoloLens version that did a similar thing, but in 3D (of course). It took quite a bit of time to locate a hexagonal sphere type of geometry - something that wasn't too heavyweight, and could be customized and textured.
The missle logic was implemented using a very inexpensive Unity Asset Store package called '3D Homing Missiles'. I thought it might be interesting to have the 'base' launch defensive missiles at the player, who would then have to shoot them down to avoid being destroyed.

The Whiteclown Mosh

I wanted to experiment with animation controllers in Unity, and for my first stab at it, decided to go with the excellent (and free... for now) Mixamo models and animations. These are professionally rigged 3D models that can be specifically targetted for Unity development. I picked one (he's called Whiteclown) that looked like it might be fun, and added two sets of animations. The cubes were added just to experiment with the collision detection settings.
Once I recorded the videos, it looked so rythmic and chotic, I thought it might go nicely with the music of Yusuke Hasegawa, who performs these amazing percussion + didgeridoo pieces here in Toronto.

Lunar Lander

My first attempt at a game (more of a simulation, really) for the HoloLens. This is a 3D representation of the old school Lunar Lander video arcade game. There's no point system here, and just a very basic win/lose kind of situation. The challenge here was trying to get meaningful input with the HoloLens, give (at the time) all there was available was the clicker peripheral. The application takes click events from the device and applies a small amount of vertical thrust for each click. Too much vertical speed on touchdown means a crash. Just the right amount earns you a polite round of applause.
The ground textures were done by Mike Parsons

Force-directed Graph

This is a simple experiment with a Force-directed graph using the HoloLens. Additional nodes are added randomly, with the layout handled by the EpForceDirected project on CodeProject. Adding the lines with Unity was actually the most difficult part. I ended up using the Vectrosity asset on the Asset Store, which ended up saving quite a bit of time (and avoid a lot of headaches).

Cube Scatter Plot

This is another scatterplot that uses size and color, as well as a rudimentary force-directed algorithm. The real experiment here was to use semi-transparent 'nodes' to limit occlusion. Rather than use straight-up transparency, I created some geometry with Blender that had holes in it.
Not the most elegant looking thing, but interesting to look at!

GIS HoloLens Visualization

This is an experimental use of the HoloLens for a GIS visualization, displaying oil platforms and pipelines in the Gulf of Mexico using a simulated data set. The idea here was to present a large network of data for the purposes of monitoring or security, using layers to offset key elements
Because of the density of the circular elements, it was a problem to display all of them at once - there are just too many of them. Offsetting from the base plane allowed many more dots to be displayed at a greater size.
This was the first time I really bumped into the limitations of the HoloLens, and needed a way to cap the geometry that was being sent to the device.
The background map is nothing fancy. It's simply an image of the region for visual context.