Jump to content

davidduff

Members
  • Content count

    116
  • Joined

  • Last visited

  • Days Won

    1

davidduff last won the day on December 22 2012

davidduff had the most liked content!

Community Reputation

1 Neutral

About davidduff

  • Rank
    Occasional Forum Poster
  1. quick summary of my experiments to date with grooming under retrospect 10: a few times, i had media sets that seemed otherwise healthy, but when i tried to do a groom, problems were found and i had to fall back and do a full catalog rebuild. hopefully, that's not a recurring issue but instead is either a one-time v9-v10 transition issue or something. i will monitor ongoing behavior and see. after rebuild, i can now get groom to complete successfully on a few of my media sets. here are some performance numbers: mediaset A: ~13M files. 1.5TB. catalog size: ~22GB. FW800 drobo storage (slow) mediaset B: ~4M files. 1TB. catalog size: 7.5GB. separate FW800-connected single hard drive. mediaset C: ~13M files. 3TB. catalog size: 53GB. iscsi drobopro storage (somewhat fast). results for mediaset A: groom took 21.5hrs - "success" (but see note below) rebuild took 19:02* results for mediaset B: groom took: (can't find the number here, but i think it was ~8-10hrs - i will edit this next time i groom this set) rebuild took 7:18* results for mediaset C: groom took: <not yet attempted> rebuild took 18:44 results marked with * above were done on a slower CPU with less memory so are not directly comparable. "*" indicates ops done on core2duo 6GB ram. no "*" indicates ops done on quad-core i7 with 16GB of ram. all groom operations above are "groom to retrospect defined policy", which (perhaps stupidly/stubbornly) i am still continue to try to get to work -- for some definition of "work". here's a big question: after full rebuild, followed by a lengthy groom (on mediaset A above) logs indicated that a successful groom had completed. a total of 1.6GB was groomed (about .1% of the total size of the mediaset). WTF!? (so in about 78000 cpu-intensive wall-clock seconds, i freed up 1.6 GB - roughly dial-up modem speed - not good). the real question or issue is: how is this supposed to work? when i initiate a groom, i don't get to tell retrospect how much it should try to cull from the media set - so does it have some default target, e.g. 10%? apparently not based on the results above. does it merely figure out the minimum amount of stuff it needs to cull to fit the existing space? well that might make sense for an "in-line" groom operation (my term), where retrospect knows (maybe) how much it wants to add to the mediaset as part of the current backup operation, but that can't be the way a manual groom works, since obviously there was enough space before, there's going to be enough space afterwards, even if nothing gets culled. does a manual groom merely cull snapshots on a schedule basis and then delete whatever files from the mediaset that may no longer be accessed from snapshots? this seems to match the data. here's the retrospect (v8) manual on the topic: "When the backup drive fills up, or when you run a scripted or manual groom operation, Retrospect uses its own grooming policy to delete old backups. At a minimum, Retrospect’s policy retains two backups for each source, saving the last backup of the day for each source from the two most recent days on which each source was backed up. Given enough space in the Media Set, Retrospect keeps a backup of each source for every day in the last week, a backup for each week in the last month, and a backup for each previous month." so i think this means that if i am using groom to retrospect-defined policy (GtRDP), that a manual groom may generally accomplish little or nothing in terms of recovering space. it allows me to assess performance and confirms the health of my media sets, but is otherwise mostly useless. the only way using GtRDP can accomplish anything useful is if it invoked in-line with a backup. so for example, i'm trying to backup a user's machine over a network, and if one of them happens to be the unlucky one, their backup will trigger an inline groom, which may take 10+ hours to complete, after which their backup will finish if they've bothered to stay connected for that long. and at the end of this process, enough stuff will have been deleted from the mediaset to fit the new backup, but there's no guaranty that the next user to come along won't have the same experience. in other words, GtRDP, unless i'm missing something GtRDP is really not useful for what i am trying to accomplish. it does not provide basic TimeMachine-like behavior of a simple "rolling backup set", or at least it does it so slowly as to be non-useful from an operational perspective.
  2. davidduff

    New Forum for Retrospect 10?

    agree completely. past history shows that v10 issues will get reported and questions asked in this forum. then eventually/inevitably a separate retro10 forum will be opened up and past inputs/questions will be orphaned in the old forum. i hate to complain (not really ) but dantz/emc/roxio/retrospect has a bad track record in this regard. with no user manual for for v9 or v10, i think retrospect inc needs to foster and respect user-driven forums like this. honestly, there's no way a normal user can muddle through all the issues, warts, complexities, bugs, performance problems, etc. associated with retrospect w/o this forum.
  3. i'm confused. george76, we have a mac (A) and a windows box ( B ). to which machine is the storage actually attached? which machine is running the retrospect engine? it sounds to me like you are saying that the storage is attached to B but is accessed over the network from A via SMB protocol. retrospect is running on A and you want to use the storage on B for backups. lennart, is this your understanding? if my understanding is correct, i don't see what retrospect client has to do with this... you first add a network source (in retrospect's confusing terminology) either by browsing for it or by using the "add source directly..." button. i've done this for mac (afp) network shares, but have not had occasion recently to try it with windows (smb), but i think it works the same. once added, retrospect sees the remote volume and the interface treats it pretty much the same as any other (attached) drive. that is, it can be used as a backup destination either by creating a new mediaset or adding new member(s) to an existing media set that resides on that drive. i think it is easy to get confused by the requirement that the drive be added as a source. intuitively, adding it as a "source" implies that you are going to back it up rather than that you are going use it for storing a backup (i.e. a media set). that intuition is reenforced by the fact that all of the other drives and clients that you backup appear in the list of sources. however that's not how retrospect uses the term. to retrospect a "source" is just a volume that can be something you backup or something you use to store a backup.
  4. thanks for the input. very interesting, mastercam and maser. when you get a chance, here are some additional details that would be useful: do the details you provide above related to retrospect 9 or 10? (i assume 9) have you had a chance to test retrospect 10 yet? if so, have you noticed any changes? i.e., any have you seen any improvements to performance (specifically wrt grooming) under v10? anything interesting to report about your system config? what type of high-volume storage are you using for your media sets? for your catalog? are you using SSD or RAID? you mention size of your media sets in terms of bytes. what about # of files? regarding "grooming has been fine"... can you elaborate? are you using "groom to retrospect defined policy" or are you using grooming to keep around a fixed number of backups for each source? do you let grooming happen "on demand" or do you invoke it as a separate operation? how long do grooming ops take to complete?
  5. catalog file was on separate disk. disk space was not an issue on the drive containing the catalog file. i have done a full rebuild. media set is functional again. i notice last few backups show "using instant scan" in the logs. i don't see evidence in the logs that an (auto-)groom has taken place since the rebuild. i'll initiate one manually and see how it goes.
  6. davidduff

    how to stop a groom operation

    moving the catalog accomplished what i needed in this case. thanks for the suggestion.
  7. need help! i have a groom operation that's causing my retroengine to crash. when the engine restarts, it tries to continue on the grooming task and it crashes again. i tried stopping the task, but that did not work. so now essentially i'm stuck. anytime the server is running, it keeps trying to execute that groom task. i'm pretty sure that i need to do a full rebuild on the media set in question. is there something i can do to break this infinite crash loop and force the groom task to stop?
  8. hitting "stop" in the activities view with a grooming operation selected does not stop the selected operation. i have a runaway groom operation that is crashing retroengine. every time it crashes, it restarts itself and comes back trying to do the same groom. hitting the stop button seeming accomplishes nothing. the status of the task switches to "please wait...", then nothing happens for a while. then the engine eventually crashes again.
  9. ok - thanks. i eventually reached that conclusion myself "the hard way" - i.e., everything else i tried failed badly. fyi, i tried doing a repair on the media set a couple of times and it crashed the engine and the console. not good. also, for my own understanding, in the case where a catalog is corrupt, shouldn't repair do essentially the same as rebuild? if not, why not? and why should repair crash both engine and console?
  10. davidduff

    retrosect10: console instability

    well, it looks like i misread the situation, and thus the title of this post is misleading. my engine is crashing every five minutes. i don't know the actual root cause, but it seems likely that whatever the problem is, it is likely that is not the fault of the console or engine-console comms as i incorrectly assumed. refer to a separate forum thread i've opened under the heading "retrospect10: retroengine assertion failure at grx.cpp-1098".
  11. it appears to me that my retroengine process is dying each time the above message is written to the logs. i didn't realize it at first, because one of the running activities (a groom operation) seems to survive the engine crashes. so this is a more serious problem than i originally thought.
  12. i'm seeing assertion failures from retrospect engine. oddly, they are happening at 5 minute intervals. they seem to be related to an instability i observed with client communications (reported under a separate thread). "Assertion failure at "grx.cpp-1089", on threadID <hexID>"
  13. i'm running retro v10 console on my server, connected to the local retro engine and roughly every 5 mins, the console "blanks out", acting as though it has just restarted itself. it jumps up to the top level default display. from the log view within the retro console app, it appears that this is happening every 5 mins give or take a few seconds, with occasional "skips" where the interval is roughly 10 mins. in the console log view, i also see one odd msg of the form: "StucOpen: IOCreatePlugInInterfaceForService(kIOMMCDeviceUserClientTypeID) (error -536870210)". also, i had been away for a while and came back to the machine to find the console app had crashed. i have the crashdump and can post if it is useful. in this situation i'm running the latest retrospect v10 console and engine (installed ~4 days ago). the retro engine is running locally on the same box as the retro console. my system is running mac os x 10.6.8 server. in the system console on the machine, i see bunches of msgs at 5 minute intervals of the form "Server::updateScriptsCache exception: MacRequestor::Request: connection closed by remote endpoint" or " and other similar msgs involving updateSetsChangeCounter, updateSelectorsCache, etc. i'll post these log records as an attachment - it actually looks like there might be a lot of different things going on there that don't look familiar, including some scary-looking assertion failures - hmmm... RetroEngine or Retrospect.txt
  14. retrospect engine process in a new V10 upgrade is generating multiple pairs of log lines per second of the form: TVol::GetInfo: UGetVolumeInformation failed, /private/var/folders/Sw/blahblahblah MapError: unknown Mac error 22. i reported this some time back (under either v8 or v9). iirc, i was told that these are not a sign of anything wrong and can be ignored. even so, i consider it a significant bug and i'm not happy that a previously reported problem survived into v10. i don't like my log files getting spammed like this - for several reasons.
  15. the process of doing a rebuild is awkward in several ways and has been so through versions 8, 9, and now 10 of retrospect. some observations: under the media sets view in the retrospect client, there are a set of icons in the toolbar. some of them apply to the selected item in the list. others are just general actions that you can perform independent of the selection. even as an experienced user, i find this slightly confusing and it sometimes throws off my expectation of how things will work. if my understanding is correct, {add, locate, rebuild} are independent of selection while {remove, copy, verify, unlock, and repair} are not. i don't really have a problem with add/remove. they are pretty understandable and use clear icons that follow a somewhat standard UI pattern. "locate" actually means to find a catalog file that exists on disk but does not exist in the list and add it to the list. in my most common use of this, however, i point at a media set in the list that can't find its catalog and then click "locate" to fix it, so it feels like it is an operation on the selected item even though it is not. perhaps this doesn't matter. "rebuild" suffers from the same issue. it feels intuitively like "rebuild" is an operation you apply to an existing media set. in fact, when you select "rebuild" (from disk), you are starting from scratch each time, specifying a set of member folders. ask yourself: is the user supposed to intuitively grock that "repair" is an operation on the selected media set while "rebuild" is not? "rebuild" is non-optimal in several ways: first, in the years i have been using retrospect, by far the most common scenario where i'm using rebuild is to simply reconstruct the catalog of an existing media set with the same set of members. thus, forcing me to locate the directory for each member is an unnecessary step and it introduces the possibility of potentially damaging user error. second, the dialog for selecting members to use in a rebuild has several problems. once you have located a member folder, there is an extra step after you have identified the member folder where you are asked to select an identified member from a list. in my experience, there is only ever one item in this list. so i have to manually select this item, then click "next" or whatever the button says. this is a modal window that pops up on top of an other modal window - not ideal from an interface standpoint in terms of making the process intuitive. also, it seems to me that in the (common) case of there being only one candidate member in the directory that i select, the program should just skip this unnecessary "choose from a list of one item" step and go directly back to the higher-level window where i can add another member or proceed with rebuild. third, after hitting the button on the aforementioned sub-dialog in the rebuild process, there is (at least potentially) a very long delay. beachball spins, the program becomes unresponsive. the program doesn't give any indication of what it is doing. this is bad. i think there are two possible improvements here. either take my input and save whatever "work" needs to be done based on those inputs for the end of the process or at least put up some indicator of what's happening - for example, if this process is enumerating files in the directory and processing them, then put up some kind of progress indicator that shows this. merely spinning the beachball and freezing the interface is annoying and makes it impossible to tell the whether the program is completely hosed vs. when it is just working on a very big task. fourth, in the years i have been using retrospect, by far the most common scenario where i'm using rebuild is to simply overwrite the existing catalog file. thus, forcing me to locate the directory to save the catalog is an unnecessary step and introduces the possibility of potentially damaging user error. it is extra-annoying because (for reasons i understand) this is not a standard mac file dialog. a suggestion: at least add a button in this dialog for "use default catalog folder". fifth, building on the first and fourth points above, i argue that there should be something like a "rebuild in-place" operation. unlike normal rebuild, this operation would use the selected media set. the "select members" part of the process could be skipped, since the user is requesting that you just use the existing set of member directories associated with the media set. similarly the "select catalog location" part of the process can be skipped - you just assume you will overwrite the existing catalog file. if this operation existed, it would be a time saver for me. i gather that this may be what the "repair" operation is possibly supposed to do - i.e., it looks at the media set and the catalog and if it determines that the catalog is messed up somehow and needs to be rebuilt, it does essentially the kind of "rebuild in place" operation i describe above. if the catalog only needs minor repairs, it does something else (not sure i understand the details there). so "rebuild" is the right idea, in my opinion. however, i have encountered numerous situations where rebuild completes quickly (and thus obviously decides that a full rebuild is not necessary), however certain operations (like grooming) don't work, but if i manually/tediously go through a full rebuild, it works. thus, while rebuild seems like the right concept, there's something lacking there. perhaps holding option key down could invoke a "force repair" or something?
×