By default, S Voice can be launched by double-pressing the home button. This means that when you press the button once, it waits for a few moments to see if a second press is coming. If it isn't, it decides that you just want to go to your home screen and sends you there. This can be the cause for a small delay, which can be reduced by simply disabling the S Voice shortcut.
In order to do so, you need to launch S Voice, then go to settings and uncheck "Open via the home key." Once this is done, you'll be enjoying a faster home button response on devices that have S Voice (including the Galaxy S4, S III and Note II), but you'll also have to launch S Voice using a screen icon. No big deal, if you ask us.