only scan specific folders

[j-polasektamu.edu]

Howdy,

I am running Retrospect Multi Server 12.6.1.101 on Windows Server 2012R2 running on a 6 core - 16GB VM. .  We are backing up data from a large NAS.  On an existing share point, We would like to backup certain folders to their own backup sets (sue to size) and allow smaller folders to backup to a single backup set.  For example

Folder 1 ~10TB,  Folder 2 ~10TB, Folder 3 <1TB, Folder 4 ~500GB

I can create scripts to backup Folder1 and Folder 2.  Folder 3 and Folder 4 can be backed up with a single script, but when a short term project comes up a new folder is created (Folder 5), I need to be sure Folder 5 is backed up with no intervention.  These short term projects are from 1 month to 1 year in length.

It appears that Retrospect scans the complete volume before processing the exclusions.  So, If I select the volume to backup,  retrospect will scan the + 20TB  of files in the folders being backed up independently of the volume, extending the scan time to over 18 hours, to backup the remaining small folders.  The number of large ~10TB + will be increasing soon, causing the small folder backups to take more than 24 hours.

Is there any way to have retrospect omit folders from a scan? Or have retrospect scan the NAS faster?

Thanks

Jeff Polasek

Sr IT Pro I

College of Engineering - Texas A&M University

[DavidHertzberg]

2 hours ago, j-polasektamu.edu said:

Howdy,

I am running Retrospect Multi Server 12.6.1.101 on Windows Server 2012R2 running on a 6 core - 16GB VM. .  We are backing up data from a large NAS.  On an existing share point, We would like to backup certain folders to their own backup sets (sue to size) and allow smaller folders to backup to a single backup set.  For example

Folder 1 ~10TB,  Folder 2 ~10TB, Folder 3 <1TB, Folder 4 ~500GB

I can create scripts to backup Folder1 and Folder 2.  Folder 3 and Folder 4 can be backed up with a single script, but when a short term project comes up a new folder is created (Folder 5), I need to be sure Folder 5 is backed up with no intervention.  These short term projects are from 1 month to 1 year in length.

It appears that Retrospect scans the complete volume before processing the exclusions.  So, If I select the volume to backup,  retrospect will scan the + 20TB  of files in the folders being backed up independently of the volume, extending the scan time to over 18 hours, to backup the remaining small folders.  The number of large ~10TB + will be increasing soon, causing the small folder backups to take more than 24 hours.

Is there any way to have retrospect omit folders from a scan? Or have retrospect scan the NAS faster?

Thanks

Jeff Polasek

Sr IT Pro I

College of Engineering - Texas A&M University

First, when you say "On an existing share point", are you referring to Microsoft Sharepoint?  If so, the top of page 431 in the Retrospect Windows 12 User's Guide says "Retrospect does not support clusters or SharePoints."

Second, assuming that your NAS is instead mounted as a shared volume on your Retrospect Windows backup server, I would think that you could enable Instant Scan for the backup server.  This is discussed on pages 571-576 in the UG, with the setting for the installation in general—which I think applies to the backup server—discussed on page 573.  I have Instant Scan enabled for my Mac backup server, as well as my one Mac client computer that is modern enough that its Client software can do Instant Scan.  The Instant Scan time is measured in minutes, not hours.

Third, Instant Scan is enabled for a backup server or client computer as a whole, not to individual volumes connected to that computer.  However it operates volume-by-volume, but not AFAIK Subvolume-by-Subvolume.  When you refer to Folder1 etc. I hope you are referring to Subvolumes, which are described on  pages 444-446 of the UG.  I fear that you may not be, because you say "It appears that Retrospect scans the complete volume before processing the exclusions."  Page 444 of the UG says "If you only want to back up files in a single folder, specifying a Subvolume (instead of specifying a volume and using a custom selector [my emphasis]) reduces the file scanning time, minimizes the number of files displayed in a browser, and reduces the amount of memory needed."  When you mention "the exclusions", that sounds as if you are using Selectors instead of Subvolumes, which would explain why your scan is taking over 18 hours.  Maybe if you use Subvolumes, Instant Scan wouldn't be so important—but enabling it can't hurt.

Fourth, I hope you noticed my latest post to your previous thread, in which I pointed out that Retrospect Windows 12.6 has "New: Support for concurrent [my emphasis] backups from different subvolumes of the same source".

[jotrago]

AFAIK Instant Scan works only on NTFS volumes as it relies on the NTFS File System Journal. Therefore it is unlikely to work on a NAS.

 

For your situation I would recommend the following:-

Create your scripts for the specific volumes to be backed up to their own BackupSets as you have already done.

For the Project folders create a Projects script targeting the whole volume, and then create Selectors to EXCLUDE the regular folders catered for by other scripts.

This script will then backup ANYTHING that appears in the volume EXCEPT the regular folders.
Selectors provide a very powerful method to control exactly what gets backed up. See the docs and my note here

[Lennart_T]

The scan time is not dependent of the amount of data, it depends only on the amount of files. So if folder 1 contains 10 files of 1TB each, it would scan very fast.

I would do something like this:

1) Have 5 folders at the root level, folders 1-5.

2) Set the permissions so that no new folders can be created at the root level of the volume, only inside the existing folders. Short term projects would have to be created inside folder 5 (as they can't be created as a folder 6 at root level).

3) Create 5 subvolumes, one for each root folder.

4) Create scripts for each subvolume. When running the scripts, only the subvolume will be scanned.

 

[DavidHertzberg]

5 hours ago, jotrago said:

AFAIK Instant Scan works only on NTFS volumes as it relies on the NTFS File System Journal. Therefore it is unlikely to work on a NAS.

 

For your situation I would recommend the following:-

Create your scripts for the specific volumes to be backed up to their own BackupSets as you have already done.

For the Project folders create a Projects script targeting the whole volume, and then create Selectors to EXCLUDE the regular folders catered for by other scripts.

This script will then backup ANYTHING that appears in the volume EXCEPT the regular folders.
Selectors provide a very powerful method to control exactly what gets backed up. See the docs and my note here

So enlighten me, please.  In order for a NAS to be both readable and writable from a Windows computer, wouldn't it have to have an implementation of NTFS?  I see that there is NTFS-3G, "an open source cross-platform implementation of the Microsoft Windows NTFS file system with read-write support."  Given that "NTFS-3G supports partial NTFS journaling", does that mean that it supports the USN Journal well enough to for Instant Scan to work?

In any case, it seems to me that jotrago's approach—using Selectors—would recreate the same problem of long entire-volume scans that j-polasektamu.edu now has.  So IMHO he should use Lennart_T's approach—using Subvolumes—instead.

P.S.: According to this Knowledge Base article, Instant Scan "does not work with NAS devices or Linux clients."  However that article was written in 2012, so I wonder if those limitations still exist.

Edited November 28, 2017 by DavidHertzberg
P.S. KB article says Instant Scan doesn't work with NAS, but that was written in 2012

[j-polasektamu.edu]

Thanks for the information.  The NAS is being mounted as a Shared Volume, not from a sharepoint server.

It is my understanding that the source system needs to have the instant scan client (part of the retrospect client) installed and running to work.  Instant scan keeps track of file changes locally, then tells retrospect what needs to be backed up.  Unfortunately, there is no client for our Isilon NAS.

My best bet is implementing Lennart_T's approach, but this will require convincing the powers-that-be to change a well estcabilished policy/workflow.

 

[DavidHertzberg]

1 hour ago, j-polasektamu.edu said:

....

It is my understanding that the source system needs to have the instant scan client (part of the retrospect client) installed and running to work.  Instant scan keeps track of file changes locally, then tells retrospect what needs to be backed up.  Unfortunately, there is no client for our Isilon NAS.

....

 

No, you can also activate Instant Scan on your backup server.  I have Instant Scan activated on my Mac Pro backup server, and it scans each of the two locally-mounted HDDs (which I backup only once a week; their contents seldom changes) in around a minute.  However, as jotrago stated here and I stated after that, your NAS volume's presumably-NTFS-compatible file system would have to support the USN Journal feature.  I would suggest checking with your NAS vendor, and maybe also with Retrospect Tech Support to see if the Knowledge Base article I linked to is still valid.

If the NAS doesn't support USN Journal, I agree that your best approach is Lennart_T's.  Tell the powers-that-be that the alternative is not having their temporary projects backed up.  You all work in a College of Engineering; it shouldn't be too difficult to get them to understand why you need to implement this.

[Scillonian]

9 hours ago, DavidHertzberg said:

So enlighten me, please.  In order for a NAS to be both readable and writable from a Windows computer, wouldn't it have to have an implementation of NTFS?

For a Linux or BSD based NAS the NTFS file system that a Windows PCs would see when accessing a share through SMB/CIFS is an emulation provided by Samba. This emulation translates the NTFS file attributes and metadata to their nearest equivalents on the underlying Linux or BSD file system (e.g. ext4, XFS, UFS, ZFS, etc).

[DavidHertzberg]

1 hour ago, Scillonian said:

For a Linux or BSD based NAS the NTFS file system that a Windows PCs would see when accessing a share through SMB/CIFS is an emulation provided by Samba. This emulation translates the NTFS file attributes and metadata to their nearest equivalents on the underlying Linux or BSD file system (e.g. ext4, XFS, UFS, ZFS, etc).

As of Samba 4.3.0, it has a "new FileChangeNotify subsystem" that sounds to me (I was never a system programmer) as though it might implement the equivalent of the USN Journal.  Could Retrospect Inc. use that to implement Instant Scan for Linux/BSD-based NASes?

[Scillonian]

My interpretation of the description of FileChangeNotify in the link you provided is that is related to the synchronisation of file data between the native file system and emulated file system(s). On a NAS it is quite possible that an individual file can be changed from the host Linux OS, a Windows machine through SMB/CIFS, an OS X machine through AFP, and a Linux machine through NFS. Any change from one access method has to be propagated to the other access methods.

However there still remains the problem of installing the Retrospect Client on a Linux based NAS. Most Linux based NAS devices use a custom Embedded Linux where it can range from difficult to impossible to install any third party system level applications such as the Retrospect Client.

Instant Scan would ne nice to see on Linux but I suspect development of it would be a low priority because of the low, relative to Windows and OS X, user base and the added complication of testing on the many more file systems native to Linux compared to Windows and OS X.

[j-polasektamu.edu]

I discovered the Isilon uses ifs, a proprietary distributed filesystem. It presents an smb filesystem to Windows clients.

[DavidHertzberg]

Just to make it clear, I am talking about jpolasektamu.edu's accessing his NAS using SMB as a Shared Volume from his backup server, not as a client.  This Knowledge Base article, as I stated above, says Instant Scan "does not work with NAS devices or Linux clients"; I see that the example of my two HDDs is not applicable because they are locally mounted on the desktop of my Mac Pro.

My point was that, if SMB now contains a journaling facility that can emulate the Windows USN Journal, it would be nice if Retrospect Inc. enabled Instant Scan to work with that emulated facility.  They would not have to test on all the native filesystems; if the SMB USN Journal emulation didn't work with a particular NAS's filesystem, that would be a problem for that NAS manufacturer to straighten out with its filesystem provider.

I get the impression that a lot of Retrospect installations are now using NASes.

P.S.: If any administrator using a NAS wants Instant Scan on it, I suggest that he/she submit a Support Case requesting the enhancement.  The worst thing that could happen is Retrospect T.S. replying that SMB can't emulate the Windows USN Journal.    My next post will be the boilerplate one telling how to submit a Support Case.  I won't submit one because I don't have a NAS.

Edited December 1, 2017 by DavidHertzberg
P.S. suggesting someone submit a Support Case request for NAS Instant Scan via SMB

[DavidHertzberg]

If you think this is a enhancement that should be made by Retrospect Inc., you will have to submit it as a Support Case.  For English speakers, that is done by going here http://www.retrospect.com/en/support/contact, and filling out the form (sorry, I don't know what the equivalent addresses are for non-English speakers, but they can figure it out from their appropriate Retrospect website address).  IMHO this is quite reasonable; obliging you to fill out the form provides Retrospect Inc. with useful details about your Retrospect installation that they would otherwise have to query you for.

 

As a result, Retrospect Inc. will pay no attention to your post in this forum.  On 12 December 2016, in response to a letter I snail-mailed to Mayoff,  I received an e-mail through a Mayoff account that was signed by JG Heithcock, CEO, Retrospect, Inc. http://www.retrospect.com/en/about#exec.  In it he says "From reading your letter, I think the main issue is that you view the forums as a good place to talk to us, Retrospect, Inc. But we view the audience of the forums as restricted to our customers [my emphasis]. The one caveat we have made on that is for feature requests, largely as we would like to see if other customers also agree on the desirability and feature set for these requests."

 

That means that the only audience for "Product suggestions" in this forum will be other administrators of Retrospect.  Nevertheless, by posting in this forum you are providing a useful service to us fellow administrator peasants.  Thank you.

 

Please be aware that the "description of your issue" in the Support Case form is IME limited to about 2000 characters by the Support Case software.  If you go over that limit your "description" will be broken up into a "description" plus one or more "additional notes".  The same is true for any additional notes you may later post yourself.  I suggest that, to avoid the appearance of choppiness in your Support Case, you create your case in a post in this forum and then copy it paragraph-by-paragraph to your Support Case. 

 

Note that, despite the new dialogs in the Retrospect Inc. Support Case system urging you to sign up for Annual Support and Maintenance, Mayoff has verbally assured me that you don't need to be signed up for ASM to report a bug—only to get personal assistance with coping with it.

 

If this post sounds formulaic, that's because I intend it to be.  I intend to link to it in every new thread that appears in this forum, unless the OP either is merely asking for help or indicates that he/she has or will open a Support Case for the enhancement that the thread requests.  Of course, Mayoff could take 5 minutes of his time to post a slightly-more-polite version of this post as a  "sticky thread" that will always appear at the top of the forum.  I don't intend to hold my breath until that happens .

 

 

Edited January 7, 2019 by DavidHertzberg
Tried to change entire post to 12-point, then changed .ODT original to 12-point and re-pasted; changed last prgf. to indicate I won't _link_ to it if OP is merely asking for help; added smiley

[Mayoff]

> I intend to post it in every new thread that appears in this forum

David, if you decide to keep posting the above document to every thread, it is going to be deleted.  It isn't appropriate to spam the forum with repeated content. 

[jotrago]

Perhaps I should have more correctly stated that Instant Scan only works on Windows Hosts with NTFS volumes (and NTFS Journals), and the Retrospect client installed directly on the host.

NAS boxes share files use the open source SAMBA suite to present files using the SMB / CIFS protocol independently of the underlying file system. Most NAS Units run a variant of Linux and use EXT, ZFS, or BTRFS file systems under the hood, all of which are journaling file systems, but not NTFS, and hence would not work with Instant Scan.

Even Windows NAS Servers ( Windows Storage Server) present their NTFS files as SMB/CIFS shares. (But of course, being Windows, you could install the Retrospect client on them and take advantage of Instant Scan because they DO use the NTFS file system under the hood)

I do also admit that my approach would not reduce Volume Scan times, whereas Lennart's approach would.