Need to see disk backup set utilization by client - reportswatcher?

[blm14]

Ok, here is the problem I am having. I back up 120+ individual machines. When I set up my retrospect pro server 3 years ago most people had a 50-60gb drive max, and now people have much larger drives. Because of reasons I dont need to get into I cannot use grooming, so when my RAIDs fill up I transfer from my disk backup sets to tapes.

 

In older versions of retrospect (before disk backup sets) I had one backup set per client and so it was very easy for me to tell who were the bad people who were taking up a lot of space with large nightly backups. Now with disk backup sets, all I know is how many FILES each user stored which is completely not helpful when my RAIDs are filling up in 3 weeks. Normally I would find out who the bad people are and make them pay for new RAIDs but now I don't have that option.

 

Does anyone know of a way to be able to figure out how much TOTAL SPACE each client is taking up in a disk backup set? I cannot parse the logs because the log files will include entries from before I dumped to tape and overreport total disk usage.

 

There is this lovely little tool called reportswatcher which appears to do 99% of what I need - I would just need to add a column to the "successful backups" page which shows the total amount of space used by each client that has at least one successful backup. Can I get the source code for this tool and/or edit it myself without decompiling it?

 

Has anyone else solved this problem?

[Mayoff]

The source code for reports watcher is already included in the external scripts\reports watcher folder.

[blm14]

I found it easier to write in java.

 

import java.io.*;
import java.util.*;
import java.math.BigDecimal;

public class RetroSpace{
   public static void main(String argv[]){
RetroSpace r = new RetroSpace();
try{
    r.go(argv[0]);
}catch(Exception e){
    e.printStackTrace();
}
   }

   public void go(String filename) throws Exception{
BufferedReader r = new BufferedReader(new FileReader(new File(filename)));
String oneLine = r.readLine();
Hashtable clients = new Hashtable();
String client = null;
String space = null;
int count =0;
while ( oneLine!= null){
    //if(count>=100) System.exit(0);
    //System.out.println(oneLine);
    oneLine = oneLine.toUpperCase();


    if(oneLine.indexOf("COPYING") != -1){//found client specification
	int begin = oneLine.indexOf("$[3]");
	int end = oneLine.indexOf("$[4]");
	client = oneLine.substring(begin+8,end);
	//System.out.println(count + "-------Found new client: " + client);
	count++;
	if(!clients.containsKey(client))
	    clients.put(client, new BigDecimal(0));
    }
    if(oneLine.indexOf("COMPLETED") != -1 && oneLine.indexOf("COMPRESSION")!= -1){//found amount saved at client
	int begin = oneLine.indexOf("FILES,")+6;
	int end = oneLine.indexOf(", WITH");
	if(begin != -1 && end != -1){
	    space = oneLine.substring(begin,end);
	    //System.out.println(oneLine);
	    BigDecimal value = new BigDecimal(1);
	    double d = 0;
	    String sizetype = "";
	    if(space.indexOf("KB") != -1 && space.indexOf("ZERO") == -1){
		sizetype = "KB";
		d = Double.parseDouble(space.replaceAll(",","").substring(0,space.indexOf("KB")-1));
		value = value.multiply(new BigDecimal(d));
	    }else if (space.indexOf("MB") != -1){
		sizetype = "MB";
		d = Double.parseDouble(space.substring(0,space.indexOf("MB")));
		value = value.multiply(new BigDecimal(d)).multiply(new BigDecimal(1024));
	    }else if (space.indexOf("GB") != -1){
		sizetype = "GB";
		d = Double.parseDouble(space.substring(0,space.indexOf("GB")));
		value = value.multiply(new BigDecimal(d)).
		    multiply(new BigDecimal(1024)).multiply(new BigDecimal(1024));
	    }else{
		//System.out.println(oneLine);
	    }


	    BigDecimal oldVal = (BigDecimal)clients.get(client);
	    if(client.indexOf("GOTTESMAN") != -1){
		System.out.println("old: " + oldVal + "new: " + oldVal.add(value));
	    }
	    clients.put(client, oldVal.add(value));
	}
    }
    oneLine = r.readLine();
}
Enumeration keys = clients.keys();
while(keys.hasMoreElements()){
    String key = (String)keys.nextElement();
    BigDecimal myspace = (BigDecimal)clients.get(key);
    System.out.println(key + "|" + myspace);
}
   }
}

This program works with the results of exporting the entire contents of the "backup report" report, and then removing any entries you do not wish to count. So, if my entire logfile goes from 2/8/2008 to 8/29/2008, I would export the "backup report," find the last entry for 8/28/2008 and remove it and everything above it, compile this program, and do:

java RetroSpace 

 

and it outputs pipe-delimited tuples of client name and total amount of data stored, not counting the size of the snapshots which are usually pretty minimal. I may modify it to include the snapshots - should be pretty easy. Hopefully this is of some use to the retrospect user community.

Edited August 29, 2008 by Guest

[blm14]

Ok, this one now includes utilization from snapshots. Enjoy!

import java.io.*;
import java.util.*;
import java.math.BigDecimal;

public class RetroSpace{
   public static void main(String argv[]){
RetroSpace r = new RetroSpace();
try{
    r.go(argv[0]);
}catch(Exception e){
    e.printStackTrace();
}
   }

   public void go(String filename) throws Exception{
BufferedReader r = new BufferedReader(new FileReader(new File(filename)));
String oneLine = r.readLine();
Hashtable clients = new Hashtable();
String client = null;
String space = null;
int count =0;
while ( oneLine!= null){
    //if(count>=100) System.exit(0);
    //System.out.println(oneLine);
    oneLine = oneLine.toUpperCase();


    if(oneLine.indexOf("COPYING") != -1){//found client specification
	int begin = oneLine.indexOf("$[3]");
	int end = oneLine.indexOf("$[4]");
	client = oneLine.substring(begin+8,end);
	//System.out.println(count + "-------Found new client: " + client);
	count++;
	if(!clients.containsKey(client))
	    clients.put(client, new BigDecimal(0));
    }
    if(oneLine.indexOf("COMPLETED") != -1 && oneLine.indexOf("COMPRESSION")!= -1){//found amount saved at client
	int begin = oneLine.indexOf("FILES,")+6;
	int end = oneLine.indexOf(", WITH");
	if(begin != -1 && end != -1){
	    space = oneLine.substring(begin,end);
	    //System.out.println(oneLine);
	    BigDecimal value = new BigDecimal(1);
	    double d = 0;
	    String sizetype = "";
	    if(space.indexOf("KB") != -1 && space.indexOf("ZERO") == -1){
		sizetype = "KB";
		d = Double.parseDouble(space.replaceAll(",","").substring(0,space.indexOf("KB")-1));
		value = value.multiply(new BigDecimal(d));
	    }else if (space.indexOf("MB") != -1){
		sizetype = "MB";
		d = Double.parseDouble(space.substring(0,space.indexOf("MB")));
		value = value.multiply(new BigDecimal(d)).multiply(new BigDecimal(1024));
	    }else if (space.indexOf("GB") != -1){
		sizetype = "GB";
		d = Double.parseDouble(space.substring(0,space.indexOf("GB")));
		value = value.multiply(new BigDecimal(d)).
		    multiply(new BigDecimal(1024)).multiply(new BigDecimal(1024));
	    }else{
		//System.out.println(oneLine);
	    }


	    BigDecimal oldVal = (BigDecimal)clients.get(client);
	    if(client.indexOf("GOTTESMAN") != -1){
		System.out.println("old: " + oldVal + "new: " + oldVal.add(value));
	    }
	    clients.put(client, oldVal.add(value));
	}
    }
    if(oneLine.indexOf("SNAPSHOT") != -1){
	int begin = oneLine.indexOf("SNAPSHOT STORED")+16;
	int end = oneLine.length();
	space = oneLine.substring(begin,end);
	BigDecimal value = new BigDecimal(1);
	double d = 0;
	String sizetype = "";
	if(space.indexOf("KB") != -1 && space.indexOf("ZERO") == -1){
	    sizetype = "KB";
	    d = Double.parseDouble(space.replaceAll(",","").substring(0,space.indexOf("KB")-1));
	    value = value.multiply(new BigDecimal(d));
	}else if (space.indexOf("MB") != -1){
	    sizetype = "MB";
	    d = Double.parseDouble(space.substring(0,space.indexOf("MB")));
		value = value.multiply(new BigDecimal(d)).multiply(new BigDecimal(1024));
	}else if (space.indexOf("GB") != -1){
	    sizetype = "GB";
	    d = Double.parseDouble(space.substring(0,space.indexOf("GB")));
	    value = value.multiply(new BigDecimal(d)).
		multiply(new BigDecimal(1024)).multiply(new BigDecimal(1024));
	}else{
	    //System.out.println(oneLine);
	}


	BigDecimal oldVal = (BigDecimal)clients.get(client);
	if(client.indexOf("GOTTESMAN") != -1){
	    System.out.println("old: " + oldVal + "new: " + oldVal.add(value));
	}
	clients.put(client, oldVal.add(value));
    }
    oneLine = r.readLine();
}
Enumeration keys = clients.keys();
while(keys.hasMoreElements()){
    String key = (String)keys.nextElement();
    BigDecimal myspace = (BigDecimal)clients.get(key);
    System.out.println(key + "|" + myspace);
}
   }
}

Edited August 29, 2008 by Guest

[MRIS]

unfortunately this code doesn't work very well because:

 

Sometimes there will be a line in the report such as:

-	16/08/2008 2:42:04 AM: Copying $[*!20612,,14,+3]system (C:)

 

Which although it does have the word copying in it, there is no "$[3]" or "$[4]" in it, causing execution to halt on such lines with error:

 

-       16/08/2008 2:42:04 AM: COPYING $[*!20612,,14,+3]SYSTEM (C:)
java.lang.StringIndexOutOfBoundsException: String index out of range: -8
       at java.lang.String.substring(Unknown Source)
       at RetroSpace.go(RetroSpace.java:32)
       at RetroSpace.main(RetroSpace.java:9)

 

So this probably needs a little more work before it's useful.

[blm14]

You must have scripts that only back up particular drives, not whole clients. The pattern matching I used assumes that each time you're only backing up the whole client (since that's how I do it ).

 

It could be pretty easily modified to fit other patterns assuming that you can find out what they are. It's easy to handle them, but I never came up against that problem because I always back up all drives on each of my clients...

 

Alternatively you could change the delimiters to look for instances of the string "COPYING" and not the $[3] and $[4] stuff and that would probably work too...