How a robot got Super Mario 64 and Portal “running” on an SNES

Can you really, playably emulate games like Super Mario 64 and Portal on a stock standard SNES only by hacking in through the controller ports? The answer is still no, but for a brief moment at this week's Awesome Games Done Quick (AGDQ) speedrunning marathon, it certainly looked like the impossible finally became possible.

For years now, AGDQ has featured a block where TASBot (the Tool-Assisted Speedrun Robot) performs literally superhuman feats on classic consoles simply by sending data through the controller ports thousands of times per second. This year's block (viewable above) started off simply enough, with some show-offy perfect play of Galaga and Gradius on the new NES Classic hardware using a device made by TASBot team member Peter Greenwood (who goes by the name micro500). TASBot organizer dwangoAC Allan Cecil (dwangoAC) described the NES Classic as "absolutely horrible" when it comes to automation.

After that, TASBot moved on to a few "total control runs," exploiting known glitches in Super Mario Bros. 3 and Mega Man to insert arbitrary code on the NES. This is nothing new for the computer-driven TASBot—the basics of the tricks vary by game, but they generally involve using buffer overflows to get into memory, then bootstrapping a loader that starts reading and executing a stream of controller inputs as raw assembly level opcodes. The method was taken to ridiculous extremes last year, when TASbot managed to "beat" Super Mario Bros. 3 in less than a second with a very specific total control glitch.

With those out of the way, TASBot moved on to a similar total control run of The Legend of Zelda: A Link to the Past. After a few minutes of setup, the Zelda screen faded out, then faded back in on a bordered window with an ersatz logo for the "Super N64." Without any forthcoming explanation from the runners on stage, TASBot started apparently playing through a glitch-filled speedrun of Super Mario 64 on the Super NES, following it up with a similar glitch-filled speedrun through Valve's PC classic Portal. After that, the scene somehow transitioned to a Skype video call with a number of speedrunners speaking live from the AGDQ event through the SNES.

No one on the AGDQ stage acknowledged how weird this all was, leaving hundreds in the Herndon, VA ballroom and nearly 200,000 people watching live on Twitch temporarily guessing at what, exactly, was going on.

Streaming audio to the NES

To unwind how the TASBot team "played" relatively modern games through 25-year-old SNES hardware, we need to go back to China's Geekpwn hacking conference a few months ago. That's where Cecil first presented a method for streaming high-quality audio through the NES sound chip after total control has been achieved.

Despite the NES' reputation for retro-style "bleep bloop" game audio, the audio processor inside the system is actually perfectly capable of outputting high-quality, PCM-encoded audio. Some games take advantage of this through extremely short samples, such as the steel drum effect that runs in the background of some Super Mario Bros. 3 stages.

The NES can play back much lengthier and more complex sound samples, too, but "you could just never store that on a cartridge," Cecil told Ars. "No one ever really used the audio chip to its full capability because you couldn't fit anything [in an NES cartridge]."

But with TASBot taking total control of the system (and bootstrapping a Commodore 64-style loader payload), complex sound files from the computer could be converted to a 7-bit PCM and sent almost directly to the APU as 52 khz mono audio via the controller ports. This method uses what Cecil calls "a bizarre encoding mechanism," designed by TASbot team members total and Ilari. This mechanism uses "every ounce of computational power available to shove that data at the sound processor as fast as we possibly could... We are literally streaming audio into the NES."

At Geekpwn, Cecil says a stream of a popular Chinese folk song "totally blew people away. They had no idea the Nintendo was capable of producing that high end quality of audio." At AGDQ, the TASBot team would expand on this by taking control of two NES systems at once (remember those SMB3 and Mega Man takeovers mentioned above?) and sending carefully synchronized mono streams to each one in order to simulate stereo sound.

Using those two NES systems for audio helped free up valuable controller bandwidth for the main event: streaming video directly to an SNES.

Turning an SNES into a monitor

After taking total control of the SNES through a known Link to the Past glitch, TASBot essentially turns the SNES into a dumb pipe for video data. By polling four simulated multitap inputs at 300 times per frame using a method devised by p4plus2 (Jordan Potter), the TASBot team is able to effectively send data to the SNES at 1.15 Mbps, Cecil said.

That actually exceeds the rate that can be handled by the RAM on the stock Link to the Past cartridge, which has to be used as a staging area for all that data before it gets sent to the SNES GPU. The solution? "We simply overclocked the cartridge," Cecil said. The team creates a fast RAM system that works perfectly well, even though the game wasn't designed to use it.

At that data rate, it takes TASBot roughly six frames to send an image of 960 SNES "tiles" to the PPU. That's enough to display a 128x112, Super Game Boy-style bordered window in the center of the screen roughly ten times a second. That creates a weird aspect ratio of about 8:7, which ends up cutting off the corners of a 4:3 game like Super Mario 64, but the effect isn't much worse than on an old, misaligned CRT TV.

Compression could theoretically provide better-looking video in that same bitstream, but even very simple compression would take too many precious SNES CPU cycles to decode at an acceptable frame rate. Instead, TASBot pre-processes the video through ffmpeg and pushes it through Linux pipes. That lets the robot "write the data in exactly the format SNES is expecting it in memory," Cecil said, passing it directly through the overclocked cartridge RAM to the GPU.

While the SNES can display over 32,000 different colors, only 256 of those can be shown on screen at any one time. To make those individual frames look better, the TASBot pre-processor uses an adaptive palette that best approximates which colors will best capture the full color depth of the source frame on the computer.

Changing those color palettes every frame would takes additional CPU time, though, which would make an already choppy frame rate about 10 percent choppier, Cecil said. As a compromise, the TASBot system only changes palettes every five frames or so, using an algorithm to determine which colors are going to be needed in the near future and loading them into the less used pieces of the most recent color palette.

Can you play it?

While the AGDQ SM64 and Portal TASBot runs were really just pre-recorded video and audio streaming through the SNES, Cecil said it would be theoretically possible to let a speedrunner actually play those games live on the SNES (or, more accurately, through a PC with TASBot-encoded video piped to a TV through the SNES). Using a fixed, "good enough" palette of 256 colors and no pre-processing, TASBot would be able to pipe through 10 fps gameplay with about 100 ms of latency, he said.

That's actually comparable to the lag on some modern LCD TVs, and it would be perfectly playable casually. At a speedrunning marathon like AGDQ, though, that might not be acceptable. "If you gave any kind of speedrunner six frames of lag, they would be kind of upset," Cecil said. "It's not a very polite thing to do to a professional speedrunner."

We suppose you could argue that sending 1.2 million inputs a second to three consoles isn't very polite to hardware designed for more human input speeds, either. But that hasn't stopped TASBot yet.