markboss Posted March 19, 2008 Report Share Posted March 19, 2008 I think I have narrowed down this issue and wanted to start a new thread on it. I am getting a -116 error message when trying to scan a folder with more than 37,000 items in it. It errors during the scan.. not the backup. I believe the issue might have to do with our firewall, so bear with me and I'll try to explain this the best that I can The backup will error out when scanning. I created a test folder with 40,000 folders in it and that fails as well. So it is not the size of the folder (the real folder is 30GB), just the amount of folders. It is not the retrospect server. I have tried setting up Retro server on another machine and it also fails. Between each of our sites, we use a monowall firewall to create an IPSec VPN. If I put the 40K folders on the same network as the Retro server, it backs up fine. If I put the folder at another site (PA or Baltimore), it backs up fine (thru the IPSec connection). It only happens at our ISP. Even if I use a connection that is not thru the IPSec VPN, it still fails. Since I have 2 servers at the ISP, I tired the folder on both servers and I get the same error, which makes me think it's the firewall that's causing the issue. A third server is up at our ISP which is behind a different monowall and that backs up fine. All other backups run fine via the ISP. It is only this one folder that is causing the issue. And it directly has to do with the amount of folders. I have port 497 TCP/UDP open on both the OS X firewall and the Monowall (tho it shouldn't matter over the VPN) Would anyone have any idea what could be going on here? If there's any other info I can provide, let me know... Quote Link to comment Share on other sites More sharing options...
Mayoff Posted March 19, 2008 Report Share Posted March 19, 2008 It is true that a firewall could kill the connection after X amount of data is transfered. I have experienced the "black ice" firewall killing a network backup after about 2 GB of data has been copied. It drove me crazy. The backup would run great until about 2 GB into copying and black ice would decide to kill the network connection. If you are using the Retrospect Client software (macintosh client) in this process, I would download and try the beta version of the Universal Client. It has a totally new codebase and may respond differently in this environment. Quote Link to comment Share on other sites More sharing options...
markboss Posted March 19, 2008 Author Report Share Posted March 19, 2008 It is true that a firewall could kill the connection after X amount of data is transfered. I have experienced the "black ice" firewall killing a network backup after about 2 GB of data has been copied. It drove me crazy. The backup would run great until about 2 GB into copying and black ice would decide to kill the network connection. If you are using the Retrospect Client software (macintosh client) in this process, I would download and try the beta version of the Universal Client. It has a totally new codebase and may respond differently in this environment. I wish I could even get to the backup part! I cant even get past the scanning. I just find it weird that it's only one of our firewalls and the rest of the monowalls (with the same config) runs the same test fine. But this isn't even a size issue. The folders I'm testing with are empty and equate to about 20 megs... I'll try the new beta and see if that works. Thanks!! Quote Link to comment Share on other sites More sharing options...
rhwalker Posted March 19, 2008 Report Share Posted March 19, 2008 Here are several thoughts. I haven't fought your same issue with Retrospect over VPN (I shudder at the performance hit), but I've fought similar issues with our SonicWALL firewall with file downloads that you might want to explore: (1) perhaps it is a fragmentation issue. The path to that ISP might have a lower MTU, causing fragmentation that causes, somewhere along the line, stuff to get dropped. You might want to play with MTU and the DF (don't fragment) bit. (2) perhaps, oddly enough, it is a false positive on a virus signature somewhere down the line on one of the many packets. You might try disabling security services (intrusion prevention, virus checking, etc.) on what is coming out of / going in to the tunnel. (3) perhaps it is a dropped packet issue, whether because of fragmentation or some other issue, causing clogging of the network queue, causing lots of mbuf() allocations that aren't quickly released, causing memory starvation. As a middle ground, it sounds like this ISP has multiple firewalls. Perhaps you could get the problematic server(s) moved to the different firewall that doesn't exhibit this issue. It's hard for me to figure out how this could cause a -116 error unless the networking services are gobbling up oodles of memory for mbuf() allocation, or perhaps there are gobs of dropped packets on which the server is awaiting replies, that take a while to get released. Russ Quote Link to comment Share on other sites More sharing options...
markboss Posted March 20, 2008 Author Report Share Posted March 20, 2008 It is true that a firewall could kill the connection after X amount of data is transfered. I have experienced the "black ice" firewall killing a network backup after about 2 GB of data has been copied. It drove me crazy. The backup would run great until about 2 GB into copying and black ice would decide to kill the network connection. If you are using the Retrospect Client software (macintosh client) in this process, I would download and try the beta version of the Universal Client. It has a totally new codebase and may respond differently in this environment. I attempted the new beta and it did not work. Still getting a -116 error when the back up runs. Quote Link to comment Share on other sites More sharing options...
markboss Posted March 20, 2008 Author Report Share Posted March 20, 2008 Here are several thoughts. I haven't fought your same issue with Retrospect over VPN (I shudder at the performance hit), but I've fought similar issues with our SonicWALL firewall with file downloads that you might want to explore: (1) perhaps it is a fragmentation issue. The path to that ISP might have a lower MTU, causing fragmentation that causes, somewhere along the line, stuff to get dropped. You might want to play with MTU and the DF (don't fragment) bit. (2) perhaps, oddly enough, it is a false positive on a virus signature somewhere down the line on one of the many packets. You might try disabling security services (intrusion prevention, virus checking, etc.) on what is coming out of / going in to the tunnel. (3) perhaps it is a dropped packet issue, whether because of fragmentation or some other issue, causing clogging of the network queue, causing lots of mbuf() allocations that aren't quickly released, causing memory starvation. As a middle ground, it sounds like this ISP has multiple firewalls. Perhaps you could get the problematic server(s) moved to the different firewall that doesn't exhibit this issue. It's hard for me to figure out how this could cause a -116 error unless the networking services are gobbling up oodles of memory for mbuf() allocation, or perhaps there are gobs of dropped packets on which the server is awaiting replies, that take a while to get released. Russ I checked all of these and this doesn't seem to be the case. Our Monowalls are set up with the default config and we do not change that. The only difference between the Monowalls in all of our sites are NAT configs & Rules. I dont se how either would be causing this issue. Whats really odd is that we had no issues running this backup as of 2 months ago. There were no updates performed on the Monowall, Server OS or anything. I can back up 35,000 folders just fine. But when I hit 37,000 I get the -116 error... Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.