Amazon S3 maximum member size

[erostratus]

Hello,

I just wanted to give a tip to anyone who comes across this problem (Retrospect Mac 17.5.1). I copy a number of local backups to an S3 account, which has always worked perfectly. The other day, I created a new cloud media set, using the same Amazon S3 account I use for my other media sets. However, whenever I initiated a backup, I'd very quickly get an "error -1021 (data overflowed expected amount)" and the backup would terminate. For the life of me, I could not figure out what was wrong, since my credentials to S3 were correct, and my other S3 backups were running without problem. 

As it turns out, when I created the member set, under the section that says, "Use at most ________ GB," I wrote in 16384 (16 TB). This was the problem. I reviewed the Retrospect 17 manual, and under chapter 8, in the section "Information for Retrospect," the manual states that the maximum cloud member size is 8 TB. It appears that if you specify the maximum size of the member set as being able to be larger than that, the backup fails, even if the consumed space in the member set is below 8 TB. I did not know that until yesterday, and curiously, all of my other S3 media sets have 10 TB set as their maximum size, and they back up correctly. So I'm not sure 8 TB really is the max, but 16 TB is clearly too much. I created a new media set with a max of 8 TB, and the backup runs smoothly. It seems like the Retrospect app could enforce a maximum number of 8192 in that field, when you create the media set, but it lets you set a number that is too high. Just FYI.

[DavidHertzberg]

23 hours ago, erostratus said:

Hello,

....

As it turns out, when I created the member set, under the section that says, "Use at most ________ GB," I wrote in 16384 (16 TB). This was the problem. I reviewed the Retrospect 17 manual, and under chapter 8, in the section "Information for Retrospect," the manual states that the maximum cloud member size is 8 TB. It appears that if you specify the maximum size of the member set as being able to be larger than that, the backup fails, even if the consumed space in the member set is below 8 TB. I did not know that until yesterday, and curiously, all of my other S3 media sets have 10 TB set as their maximum size, and they back up correctly. So I'm not sure 8 TB really is the max, but 16 TB is clearly too much. I created a new media set with a max of 8 TB, and the backup runs smoothly. It seems like the Retrospect app could enforce a maximum number of 8192 in that field, when you create the media set, but it lets you set a number that is too high. Just FYI.

erostratus,

IMHO you've encountered a deficiency in the Retrospect interface "Adding Cloud Storage in Retrospect" Add dialog.  The dialogs shown on page 149 of the Retrospect Mac 17 User's Guide are identical to those shown under "Adding Cloud Storage in Retrospect" in the Retrospect Knowledge Base articles "... How to Set Up Amazon S3 for Cloud Backup", "... How to Set Up Minio for Cloud Backup , and "... How to Set Up Backblaze B2 for Cloud Backup".  These Add dialogs start with a Member Type pop-up;  the pop-up value shown for the Mac dialog in all 4 documents is "Cloud Storage"—and the pop-up value shown for the Windows dialog in all 4 documents is either "Amazon S3 compatible" or "Backblaze B2".  Jeff Bezos' Amazon S3 has a stated space limit of 8TB, whereas Minio S3 has no stated space limit, but the Member Type pop-up value "Amazon S3 compatible" does not permit the code associated with the dialog to determine which space limit should be enforced for an "Amazon S3 compatible" Media Set.  That would not be true if code had the—unlikely—capability of analyzing credentials as separately defined per "Setup" of a storage "bucket" in those documents.

Thus IMHO, to allow the code associated with the Add dialog to determine which space limit should be enforced, there would have to be separate Member Type pop-up values for "Amazon S3" and "Minio S3".  Here's why and how to create a Support Case for such an enhancement.  Note: When I today tested (I don't actually use cloud storage) adding a Media Set "TestCloudStorage" on my Retrospect Mac 16.6 Console, the Member Type pop-up choices in the Add dialog were "Amazon S3 Compatible" and "Dropbox" and "Backblaze B2" and "Local Storage".  The outdated Mac screenshot Member Type pop-up choice "Cloud Storage" in the 4 documents linked in the preceding paragraph in this post is from Mac version 13 UG page 9.

BTW, this 17 October 2019 OP said you were actually considering using Retrospect to back up via Minio to a Synology NAS.  Did you actually implement that local backup strategy, via Minio or directly to the Synology—as Nigel Smith suggested in that thread?

Edited November 14, 2020 by DavidHertzberg
Expanded second paragraph with results of my Mac 16.6 test, showing Member Type pop-up choice "Cloud Storage" _displayed_ in Add Mac screenshots is outdated.

[erostratus]

I did use Minio for a year, and it went smoothly. I had some local backups, some Minio, and some S3. I no longer use Minio, but no regrets. I appreciate all of the help available here on the forums!