Your OP is one of the least "pro" of any lately posted. 🙄 You don't say what version of Retrospect Windows you are using, or what version of Windows your "backup server" is running, or what version of what OS your "remote machine" is running. You probably should get someone who knows more about IT to help you with future posts to these Forums.
Nevertheless, although I'm a Retrospect Mac administrator, I'll try to give you an answer based on no provided information. When you say "create a new backup set on a remote machine connected via a site to site VPN", you must mean the destination is a NAS share on your VPN. Watch this video 3 times before you go any further. Don't create a Storage Group unless you really want to. As the video implies, you shouldn't put the Catalog for the backup set on the NAS; instead the Catalog should be in the default location on your "backup server"'s C:\ drive. Be especially sure you are following the procedure from video minute 0:36 to 0:48, and also from minute 2:04 to the end; maybe your problem is that you didn't configure automatic login per minute 2:04.
If that doesn't solve your problem, and you are using a Retrospect version earlier than 17, consider doing at least a trial upgrade—AFAIK free for 45 days. The cumulative Release Notes for Retrospect Windows lists a fix, under 126.96.36.199, that may also apply to creating a backup set on a NAS share: