Developer interview - Part 2

Prototype party

Now I’d like to dig a bit deeper into the Nintendo Labo development process. I heard that Sakaguchi-san and Kawamoto-san weren’t on the same team when the development process began, but that your teams were later merged. Can you talk a little about how you all came to work on Nintendo Labo?

Mr. Kawamoto: Sakaguchi-san was actually already prototyping before we started working together.

Mr. Sakaguchi: That’s right. At the beginning, I was working on it by myself. At some point four other people joined me and we decided to do something with our ideas.

And you wanted that “something” to be built around the Joy-Con controllers?

Mr. Sakaguchi: All we had been tasked with at the time was to do something with Nintendo Switch.

So you wanted to do “something,” but didn’t know what it would be? Sounds like a very uncertain start!

Mr. Sakaguchi: One of the ideas from our first brainstorming sessions was to make scissors using two Joy-Con. We figured they could move like this—you know, like scissors. We mentioned this earlier when we were talking about physical feedback, too, but it was the scissors idea that made us realize that constricting the movement of the Joy-Con, so they could only move in particular ways, improved the data we got from them. We could then write a program assuming that the Joy-Con could only move in these ways, which really simplified the process. The gyro sensors hadn’t been used this way before. There were a lot of ideas from these early sessions, but I was mostly focused on trying to make new attachments for the hardware.

What did people think about that?

Mr. Sakaguchi: Most people, including myself, thought it was a unique idea that would never get off the ground.

Mr. Kawamoto: I heard you were prototyping attachments with a 3D printer...

Mr. Sakaguchi: I was! But it turned out that making attachments out of plastic would be too costly, so at a certain point I had to set that idea aside.

Just to be clear, when you say “attachments,” you’re referring to hardware accessories for the Nintendo Switch system, right? That was your focus at the time?

Mr. Sakaguchi: Correct. Once we joined up with Kawamoto-san’s group we agreed that everyone would bring in their ideas and we’d start prototyping games that worked with them.

Mr. Kawamoto: The whole group was focused on making something unique with the Joy-Con, and coming up with ideas for intuitive games. We called our efforts “prototype parties.”

So you started out just wanting to gather as many ideas as you could?

Mr. Kawamoto: Pretty much. We agreed to ignore concerns about the cost and just solicited ideas that were interesting and intuitive. We didn’t provide any more detailed direction than that. Even though I was the one soliciting ideas, I have to admit that it must have been hard for the team to work with such a vague task. (Laughs.)

Mr. Sakaguchi: Yeah, we didn’t want to worry about being realistic at that point, we just wanted ideas. I decided to explore the idea for attachments again. There was just one thing that I kept thinking about … The right Joy-Con has an IR Motion Camera in it, but the camera was a little hard to use in a normal game.

How so?

Mr. Sakaguchi: It’s a little hard to explain. The camera is plenty powerful, but it’s hard to control. Consider the gyro sensor: if we want to make a game where a character transforms into a different shape when the Joy-Con is tilted, that’s easy enough to implement. But the IR Motion Camera isn’t as simple—and yet we knew that if we didn’t use the camera in addition to the controller differential controls, we wouldn’t be fully utilizing the Nintendo Switch hardware, and we wouldn’t be creating something intuitive.

Zoom in photograph of IR camera

What was the hardest part about working with the IR Motion Camera?

Mr. Sakaguchi: Well, it’s a camera, so the data you get from it isn’t stable.

What does that mean?

Mr. Kawamoto: This gets a little technical, but imagine spinning a gyro sensor through the air. In that case the data received is also difficult to parse.

Hand waving a Joy-Con

Mr. Kawamoto: Remember when we talked about how we can’t know how the consumer will hold or move the Joy-Con, which is why constricting the movement of the Joy-Con makes it easier to get reliable data from the sensors? It’s the same issue with the IR Motion Camera. Moving it through the air results in unreliable data. We don’t know how the consumer will move, either. They could step out of the camera’s field of view, or they could stand too far away.

Mr. Sakaguchi: Eventually we realized that we could get the performance we wanted if we enclosed the camera in a box. If the position is fixed then the camera will know where to look for a reflective marker sticker and being enclosed means it won’t erroneously pick up other light sources.

Mr. Kawamoto: We realized pretty early on that we’d have to put it in a box!

Mr. Sakaguchi: Here is the first project we made with that idea.

Nose picking prototype

What the ... nose picking?!

Mr. Ogasawara: We built a small enclosure around the Joy-Con’s IR motion camera. This allowed us to reliably read the motion of objects inside the enclosure. This is the first prototype we made to test the concept.

I don’t even know what to say. That’s ridiculous. (Laughs.)

Mr. Sakaguchi: It might look ridiculous at first, but we were testing an important idea. You see, we wanted to start with the smallest enclosure we could think of…

You mean... A nostril?

Mr. Sakaguchi: In this case, the IR Motion Camera inside the box responds to movement and translates it to the finger on the screen. This was just a demo, but I remember thinking that if it worked, it would help us pin down whether the box trick would be effective or not.

One funny thing that happened during a test play session: a team member playing with it checked his finger afterward to see if anything had stuck to it. I remember thinking we were onto something there―with the technology working together seamlessly we’d achieved a real sense of immersion.

Mr. Kawamoto: Of course there was nothing to get stuck on his finger! (Laughs.)

Mr. Sakaguchi: When we finished our tests with this prototype, we wanted to try making a really large box next. We had ten people from the prototype team working on it.

What did you make the big box out of?

Mr. Sakaguchi: For the nose-picking game we used a 3D printer, but even when the printing process goes smoothly it still takes a whole day to produce one prototype.

Since our last project was about discovering how small we could make something, we wanted to see what the maximum size we could work with would be. We decided to go to the packaging material storage room and look for materials. We came back with a bunch of cardboard and used it to make this tank thing.

It was like a tank

So this is where cardboard makes its appearance! What’s the broom-like thing on the top?

Mr. Sakaguchi: It’s a floor sweeper. The foot pedals are hole punchers.

How were the sensors used in this prototype?

Mr. Sakaguchi: There is a Joy-Con controller inserted in the bottom, and there are reflective balls suspended beneath the pedals that move up and down. The IR Motion Camera just responds to the motion of the balls.

Mr. Kawamoto: The tank project showed us how much fun the mechanisms behind the projects themselves can be.