It is essentially on-the-fly converting of your words to a different language, and with your own voice to boot. It is not perfect yet, and gets one in eight words wrong on average, but just think about never having to learn a foreign language, and still practice your best sarcastic remarks on the locals when they serve you food that still looks like it's moving.
The software has to be provided first with hours of recordings, so as it can analyze the subtleties of your vocal cords, and then learn to speak Cantonese with them, just like the first voice recognition efforts required you to talk to the computer for quite a while until it gets your brand of crazy accent. Rick Rashid, the head of Microsoft Research who demonstrated the technology, clarified how this became possible:
If anyone is still wondering why do we need desktop-class processors in our phones and tablets, it's features like these that will evidently require all the horsepower your Windows Phone can muster, and we hope this direct translation tech will arrive soon to our handsets. The fun starts about 5:30 into the video below, but it is worth watching in its entirety.