Tag Archives: gaming

Can I Have A Show of Hands?

21 Apr

Advances in sensors computer vision algorithms have introduced interesting new ways to interact with technology. Body- and gesture-tracking for game consoles like Kinect, FOVE’s eye tracking, detailed hand-tracking like Leap Motion and Nimble VR. Most recently I was pretty blown away by Microsoft Research’s HandPose, demonstrating a low-latency, highly accurate and robust full hand and finger sensing from a single depth camera.

I think this is cool tech and hope folks keep researching it super hard. But I’ll tell you what: I don’t think hand and finger tracking is the main way we’ll be interacting with computers and especially in Virtual or Augmented Reality.

On the one hand, much of our interaction with the world involves hands and fingers. Humans have highly evolved hands and fingers wired with enormously complicated, nerve-dense, small-muscle-dense, fine-motor-control abilities to manipulate tools with our hands and fingers. We also have a highly evolved sense of proprioception – the knowledge of the positions of our body, and especially of our hands and fingers within arms-reach, even in the dark or with our eyes closed, even when we can’t see our body or hands. Our strongest sense of 3D and depth-perception given our binocular vision is within reach of our arms. It’s absolutely the case that our hands will be involved in input in VR and AR. But the reality is we manipulate our tools very, very subtly with learned tool-specific haptic feedback, very often out of our own sight using our proprioception. Our hands are rarely the tools themselves.

Forks, knives, spoons, bowls, dishes, stoves, faucets and sponges. Our hands manipulate these tools to prepare, eat, and clean up food. Learning to use most of these tools is pretty easy, and there are both fine- and gross-motor skills involved (but professional chefs can accurately chop hundreds of 0.5mm slices of vegetables).

Pianos, cellos, flutes. Our hands (and sometimes our mouths) manipulate these tools to create music, and there is an exceptionally fine level of arm, hand, and finger control and lengthy training involved to become skilled at making music – there may be a difference of 0.5mm or less in finger position and a few hundredths of Newtons of force difference between the correct and incorrect note played on a flute.

Keyboards, mice, touchpads, game controllers. These tools capture a tremendous amount of information from very small motions of our fingers, hands, wrists, and forearms. A keyboard keypress may involve your fingertip traveling 1-2mm, and typing whole sentences rarely involves fingers moving more than a few cm in any direction. Mice and touchpads allow very small pressures and sub-millimeter motions to accurately translate to complicated 2D interfaces. Gamepads are designed to accurately register joystick and trigger motions of less than 0.1mm radially or linearly at 60Hz or higher and skilled gamers can trigger multiple 1-2mm actuated buttons 30 times per second in brief bursts.

Mobile phone touchscreens and user interactions based on touch are fascinating tools combining the eye-hand coordination of mice with very fine spatial horizontal and vertical finger motor control and direct manipulation visual feedback. As a direct manipulation tool, touchscreens rarely take advantage of proprioception.

I think there will be some uses in technology and even in VR/AR for gesture-based, non-haptic interaction of the type we saw with Kinect and as we’re seeing with more accuracy and better hand models with LeapMotion and HandPose. But we are tool users. We hold our tools in our hands or we rest our hands on them or we lay our hands across our tools. And we move our tools around, sometimes out of sight behind us. We do this so that our tools amplify the millions of years of evolution that went into our brains, eyes, hands, fingers, and our sense of proprioception.

So far I’ve run across two tools that make sense in VR (and I’ve tried… all of them): The first is the game controller – the XBox controller and the Playstation DualShock4 controller both work well to give you a well-understood, input-dense tool which can be mapped to in-VR manipulations easily. I slightly prefer the DualShock4 since it also provides accelerometer and gyroscope sensor data, giving even more input. For gamers, it’s a positive that game controllers have an established haptic and proprioceptive model (gamers know the controller’s feel and layout without seeing it), but several negatives in my mind, primarily that gamers expect their left thumb to control motion, but also that your hands feel stuck together by this tool.

The second tool is the Valve/HTC Vive lighthouse wands (prototypes shown to the left, the actual devices are somewhat different). These controllers are, in a nutshell, unbelievably great. A highly accurate cross between a joystick and a touchpad beneath each thumb, a trigger beneath each pointer finger, and a gentle squeeze-actuated button on the barrel. Very clear haptic sense just in its shape. I have only used them a few times so I don’t yet have a sense of how dense the input can be, but it definitely feels right in VR to have both hands free and in a neutral position, to be able to use the fine motor control of your wrists, thumbs, and fingers to manipulate the environment, and to do so from any position you choose to put them (above your head, behind your back, etc). When you get a chance to try these, I think you’ll agree that it takes VR to a different level.

Apple TV – What I’d Buy, What I’d Sell

19 Aug

The noise1 about Apple introducing a new TV product, finally getting the little Apple TV “right”, is getting more and more frenzied and less and less thoughtful as we head towards a likely fall release. I haven’t found a single tech analyst (or even multiple analysts I could blend together) who project a compelling and consistent set of features for something I would like to buy while at the same time being something that Apple could possibly build and sell. I personally think we can make an accurate prediction about the basic features of a product that would delight consumers by just thinking in these terms, so I thought I would try.

Before I start, it’s maybe worth a bit of background as to why I might have an armchair opinion worth caring about. I haven’t bought any of the earlier Apple TV’s but I’m not a hater – they just haven’t fit an urgent or painful need for me since I have a Mac Mini already connected to my TV which we use for photos, home movies, Netflix, and DVDs (movie night with my famous buttered popcorn is very important to my kids). I was one of the founders of the xBox project at Microsoft, a project we got funding and approval for based on a long-term vision of entertainment in the living room as well as the financial modeling of game studios, movie-/tv-studios, and cable companies. I have consumed TV, movies, and music (and photos) on computers since about forever: my CD collection was online via cdda2wav in the early 90’s and MP3 (l3enc ftw!) soon thereafter,2 and in the days after it was released I used DeCSS to make copies of all my DVD’s so I would never need to swap discs again. Oh, and I’ve used Plex and MythTV and SageTV extensively (I even contributed minorly to some of the hardware tuner drivers for these), I have owned a TiVO (and tried most other DVR’s) since 1999, I have cable and DirectTV, and lastly, I was the CTO of (hard failed) ’04/’05 startup called C.A.C. Media, building and licensing a China-made-hardware + linux-based-software set-top-box with a subscription video-on-demand service.

You might think, then, that I’d require every bell and whistle feature from a new Apple TV, but the honest truth is I wouldn’t — I’m still just searching for something that eases the following simple pain-points:

  • With antenna/terrestrial service plus cable or a satellite plus a DVD player plus a video-game console, it’s simply too hard to just turn on the TV to the show/game/DVD you want, change channels, and adjust the volume.
  • The many different on-screen guides are slow and clunky, search is impossible, and managing recordings is horrendous (yes, even you, my precious TiVO, have failed to make this much better since 1999)
  • Personal content is hard to find and play on my TV or on the stereo elsewhere in my house — DVDs, music in iTunes, pictures and home-videos in iPhoto/iMovie are all hard to uncover
  • I have too many remotes, each of which is lousy, and they only work when their battery isn’t dead and when I point them properly
  • Trying to have more than one TV is painful and can be expensive.

You’ll notice I don’t list needing “a cloud DVR for every possible show” or “a-la-carte purchasing of channels” or any number of other complicated to implement and difficult to negotiate features which people think would “revolutionize TV” or “disrupt cable and content companies”.  If you think about it, I bet the majority of you don’t either, I bet you and your family fight these same minor irritations with your TVs and media. And I think you would pay $99 or even more or even a small recurring subscription in order to make these problems go away. If you got some of these additional features out of the product, it would be gravy — you would be compelled to pay just to resolve 2 or more of these time- and life-sucking, irritating problems.

So, based on what I would buy and based on the reality of how cable companies operate and license content, here’s what I would minimally sell to resolve consumer pain if I were an Apple-ite: A small device with:

  • multiple audio+video inputs (let’s say 3 HDMI + maybe 1 component) to accomodate my cable or satellite set-top-box, xBox, and DVD/Blu-Ray player, and one HDMI output for the TV. During setup I would recognize your location based on your IP address (or prompt you for your zipcode) so that I could present and search your time-based guide, and I would recognize your connected set-top-box and game console(s) based on their HDMI EDID‘s so that I could wrap their output and overlay program-guide and source-choosing information. I would try to include IR “mouse”  universal-remote connectivity so it could control the other devices and users could throw away their other crappy remotes. I would detect when you turn on your DVD and when you turn on your xBox and bring up the right choice automagically 90% of the time — you would think I was magical when in reality I was simply watching the signals that for some reason your TV doesn’t.
  • use of any iOS device as a unified remote, and possibly a new lower-cost iOS device that is remote-only (a touch-sensitive screen running only the Remote App, $99 or less) since I would want to target users without iOS devices but keep the price under $199.
  • a unified search and guide UI which overlays all sources and backfills with iTunesYour cable operator only offers the current season of The Killing but Netflix offers prior seasons as well as the original Danish show: when I search for “Killing” the results show upcoming episodes from my cable subscription as well as back episodes on Netflix and iTunes as well as a reference to the original Danish show. I don’t really care where content comes from: I just want one place to search for it and a way of paying for it — the oatmeal’s comic on this subject is spot-on. 
  • Better search for programs and for finding and controlling recordings. I would be very satisfied with good UI based on the keyboard on my iPhone (finding and managing shows on TiVO and other DVRs or on my cable box via remote is beyond painful), but voice-control would also be compelling. Siri isn’t perfect or even great for everything, but she’s perfect for narrow-context situations and constrained vocabularies like “record this show for me,” or “find cop shows”. It wouldn’t surprise me if Siri comes later — the iOS touch UI and keyboard is enough of a game changer for search and control versus your crappy remotes with up/down that it would be enough to get most people (and me!) to buy.
  • don’t fix and standardize existing cable content capabilities or negotiate new content rights, just create end-user consistent UI: UI for finding and recording, pause/rewind/fast-forward self-consistency, purchase things for a low price that aren’t available from the given provider (back-fill with iTunes content). You see, consumers are either (a) already aware and used to the idiocy of their particular content provider’s capabilities — one can record four shows at once, another can only record two; one can play all old episodes of some show, others can only play prior episodes of the current season, some support pause and go-back, others don’t — or (b) they have no clue that their set-top-box can do those things at all because the UI sucks and they’re lucky to be able to turn it on and change channels. By delivering a consistent UI around any particular content’s current constraints you either (a) make the UI simpler, in which case consumers are happier and have neither lost nor gained features, or (b) you make the UI discoverable, in which case consumers are wildly delighted because they believe you’ve delivered a feature (like pause, record, or go-back) to them. As part of this, obviously, give cloud-based “virtual” DVR (playback based on content in iTunes) to anybody whose cable operator has those rights.
  • expanding on the point above about unified search/guide let users back-fill any content with simple purchasing from iTunes. My cable operator lets me watch all 5 episodes of Breaking Bad on-demand, so I can do so without needing to purchase anything. Your operator only lets you watch the current season: you have to pay $0.99/episode for prior seasons, and the UI on our Apple TV’s is simply clear about the difference to both of us. This is clearly just part of my cable package, not Apple’s fault that it’s different between us: it works for me. If it really ticks me off, I’ll buy a different package or switch providers.
  • related to the above: let consumers keep paying their cable company and let them switch cable companies and packages at will. the existing set-top-box subsidy structure easily gives Apple their 40% margin on a $60 bill-of-material (BOM) device ($99 MSRP).
  • support content apps (netflix, hulu, airvideo, etc, and give them access to programming and recording data so that app builders can innovate on the platform for choosing/offering, recommending, discovering, and recording programs and even for providing a better price than the cable company for some packages) as well as games. If I don’t get back seasons of Game of Thrones from my cable operator but I’m a Netflix subscriber and I get them there, offer them to me in the guide/search-results for free with a Netflix icon instead of purchase through iTunes — backfill any guides or recommendations with app- or operator-provided content in all cases, and fall back on iTunes. Supporting iOS games will as a useful competitive side-effect effectively destroy the Microsoft/xBox, Nintendo, and Sony games businesses within a few years, simply because iOS is a better development and distribution platform than their consoles.3
  • stream sources between multiple Apple TV devices and to and from any devices in the house, presumably by releasing AirPlay client/servers software updates for all platforms (iOS, Mac, PC4) to support AirVideo-like streaming from all the AppleTV sources to all my screens. Any source to any device without moving wires or clicking on multiple remotes — this is actually the true Holy Grail of home-theater. Because the current home-theater world is so full of high-end audio- and video-philes, you often see crazy constraints about streaming Dolby 7.1 sound or pure Hi-Def video or other crazy requirements — they think that perfect quality is the Grail. But it’s not — that is the false, gold-plated grail: choose that and you will Choose Poorly. The simple, wooden, true Grail is just letting your kids watch Phineas and Ferb on an iPad on the couch while you watch the news on the TV.
  • Cost to consumers: $0 recurring. Just sell the devices and have users buy content which their cable operator doesn’t have, both apps and content which I get a 30% cut of. If any of the above capabilities were hard for me to negotiate with cable operators, I’ve got some content and app/subscription revenue that I can share with the operators to assuage them. The plan is to sell so many millions of these devices that the 40% margin on the device and the app revenue is dominant to the 30% cut of the iTunes content revenue in the short-term — any expense there is really just customer acquisition. In fact, sharing the 30% with the operator could be a hedged bet: if the operators continue to control most content and I don’t sell a lot through iTunes, I don’t lose much to them. If content begins to break out of cable operators and I can get it in iTunes, consumers will start to buy their content from me and will start dropping their operator, in which case I can stop sharing revenue with them.

So, I could end up being completely wrong or missing something major, but this is my $0.02 for what I’m looking for as a consumer and it is what I would build if it were my product and I was paying attention to what is wrong with home television for consumers while trying to avoid the land-mines of content owners and cable operators.

1 The WSJ and GigOM articles were what really got me frustrated with their lack of analysis, but they aren’t alone, just two that I pulled out quickly.

2 perhaps for another post, my story of being verbally abused by Nathan Myhrvold for pushing to license MP3 and MPEG2 for Windows rather than pursuing WMA+DRM. Oh, the glorious irony given his “Intellectual” Ventures.

3 IMHO there are several ways that Microsoft in particular could redefine xBox in order to defend and even win in this category, but it doesn’t look to me like on its current path it’s going to be anything but roadkill to a halfway decent Apple TV running games; its market will be 10x the hardcore console gaming market. Nintendo and Sony have no long-term chance.

4 I would support both PCs and Macs, though I think there is a good chance that Apple would eschew PCs — iTunes is the exception that proves the rule these days.