I'm trying to figure out what is going on here. I have a Suse Linux client that I'm backing up using Accelerator AND client side dedupe. What I am seeing is odd however. It appears when accelerator is turned on, the nightly diffs send the entire data set over the wire to the storage (or media?) server, which takes several hours over the 1gb link, and result in 0% optimization, then the client side dedupe runs and the job finishes in minutes:
03/25/2013 22:26:06 - Info nbjm (pid=1008) starting backup job (jobid=933064) for client sasserver4, policy sasserver4, schedule differential-inc
03/25/2013 22:26:06 - Info nbjm (pid=1008) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=933064, request id:{E5274B1E-95C4-11E2-87F9-00144FBB31A9})
03/25/2013 22:26:06 - requesting resource sasserver4_disk
03/25/2013 22:26:06 - requesting resource netbackup1.NBU_CLIENT.MAXJOBS.sasserver4
03/25/2013 22:26:06 - requesting resource netbackup1.NBU_POLICY.MAXJOBS.sasserver4
03/25/2013 22:26:07 - awaiting resource sasserver4_disk. Maximum job count has been reached for the storage unit.
03/25/2013 22:28:35 - granted resource netbackup1.NBU_CLIENT.MAXJOBS.sasserver4
03/25/2013 22:28:35 - granted resource netbackup1.NBU_POLICY.MAXJOBS.sasserver4
03/25/2013 22:28:35 - granted resource MediaID=@aaaad;DiskVolume=PureDiskVolume;DiskPool=nbapp1-dedupe-pool-1;Path=PureDiskVolume;StorageServer=nbapp1;MediaServer=nbsrv1
03/25/2013 22:28:35 - granted resource sasserver4_disk
03/25/2013 22:28:35 - estimated 1957092319 kbytes needed
03/25/2013 22:28:35 - Info nbjm (pid=1008) started backup (backupid=sasserver4_1364268515) job for client sasserver4, policy sasserver4, schedule differential-inc on storage unit sasserver4_disk
03/25/2013 22:28:36 - started process bpbrm (pid=14651)
03/25/2013 22:28:38 - Info bpbrm (pid=14651) sasserver4 is the host to backup data from
03/25/2013 22:28:38 - Info bpbrm (pid=14651) reading file list from client
03/25/2013 22:28:39 - Info bpbrm (pid=14651) accelerator enabled
03/25/2013 22:28:59 - connecting
03/25/2013 22:29:00 - Info bpbrm (pid=14651) starting bpbkar on client
03/25/2013 22:29:00 - Info bpbkar (pid=30683) Backup started
03/25/2013 22:29:00 - Info bpbrm (pid=14651) bptm pid: 14689
03/25/2013 22:29:00 - connected; connect time: 0:00:00
03/25/2013 22:29:01 - Info bptm (pid=14689) start
03/25/2013 22:29:01 - Info bptm (pid=14689) using 262144 data buffer size
03/25/2013 22:29:01 - Info bptm (pid=14689) using 30 data buffers
03/25/2013 22:29:01 - Info nbsrv1 (pid=14689) Using OpenStorage client direct to backup from client sasserver4 to nbapp1
03/25/2013 22:29:04 - begin writing
03/25/2013 22:30:15 - Info bpbkar (pid=30683) 4999 entries sent to bpdbm
03/25/2013 22:34:03 - Info bpbkar (pid=30683) 9999 entries sent to bpdbm
03/25/2013 22:39:13 - Info bpbkar (pid=30683) 14999 entries sent to bpdbm
03/25/2013 22:45:03 - Info bpbkar (pid=30683) 19999 entries sent to bpdbm
03/25/2013 22:45:22 - Info bpbkar (pid=30683) 24999 entries sent to bpdbm
03/25/2013 23:04:41 - Info bpbkar (pid=30683) 29999 entries sent to bpdbm
03/25/2013 23:14:36 - Info bpbkar (pid=30683) 34999 entries sent to bpdbm
03/25/2013 23:21:29 - Info bpbkar (pid=30683) 39999 entries sent to bpdbm
03/26/2013 04:20:29 - Info bpbkar (pid=30683) accelerator sent 1467964042240 bytes out of 1467171781632 bytes to server, optimization 0.0%
03/26/2013 04:20:29 - Info bpbkar (pid=30683) bpbkar waited 53233 times for empty buffer, delayed 788487 times
03/26/2013 04:20:30 - Info nbsrv1 (pid=14689) StorageServer=PureDisk:nbapp1; Report=PDDO Stats for (nbapp1): scanned: 1432788229 KB, CR sent: 26891771 KB, CR sent over FC: 0 KB, dedup: 98.1%, cache hits: 0 (0.0%)
03/26/2013 04:20:30 - Info nbsrv1 (pid=14689) Using the media server to write NBU data for backup sasserver4_1364268515 to nbapp1
03/26/2013 04:20:36 - Info bptm (pid=14689) EXITING with status 0 <----------
03/26/2013 04:20:36 - Info nbsrv1 (pid=14689) StorageServer=PureDisk:nbapp1; Report=PDDO Stats for (nbapp1): scanned: 6989 KB, CR sent: 2261 KB, CR sent over FC: 0 KB, dedup: 67.6%
03/26/2013 04:20:36 - Info bpbrm (pid=14651) validating image for client sasserver4
03/26/2013 04:20:37 - Info bpbkar (pid=30683) done. status: 0: the requested operation was successfully completed
03/26/2013 04:20:37 - end writing; write time: 5:51:33
the requested operation was successfully completed (0)
This happens every night, even though the dedupe ratios are near 100%. As a test today, I ran a differntial with accelerator disabled:
03/27/2013 11:16:48 - Info nbjm (pid=1008) starting backup job (jobid=934639) for client sasserver4, policy sasserver4, schedule differential-inc
03/27/2013 11:16:48 - Info nbjm (pid=1008) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=934639, request id:{BA1FDAA4-96F9-11E2-894C-00144FBB31A9})
03/27/2013 11:16:48 - requesting resource sasserver4_disk
03/27/2013 11:16:48 - requesting resource netbackup1.NBU_CLIENT.MAXJOBS.sasserver4
03/27/2013 11:16:48 - requesting resource netbackup1.NBU_POLICY.MAXJOBS.sasserver4
03/27/2013 11:16:49 - granted resource netbackup1.NBU_CLIENT.MAXJOBS.sasserver4
03/27/2013 11:16:49 - granted resource netbackup1.NBU_POLICY.MAXJOBS.sasserver4
03/27/2013 11:16:49 - granted resource MediaID=@aaaad;DiskVolume=PureDiskVolume;DiskPool=nbapp1-dedupe-pool-1;Path=PureDiskVolume;StorageServer=nbapp1;MediaServer=nbsrv1
03/27/2013 11:16:49 - granted resource sasserver4_disk
03/27/2013 11:16:49 - estimated 33934 kbytes needed
03/27/2013 11:16:49 - Info nbjm (pid=1008) started backup (backupid=sasserver4_1364401009) job for client sasserver4, policy sasserver4, schedule differential-inc on storage unit sasserver4_disk
03/27/2013 11:16:50 - started process bpbrm (pid=23506)
03/27/2013 11:16:52 - Info bpbrm (pid=23506) sasserver4 is the host to backup data from
03/27/2013 11:16:52 - Info bpbrm (pid=23506) reading file list from client
03/27/2013 11:16:52 - Info bpbrm (pid=23506) starting bpbkar on client
03/27/2013 11:16:52 - Info bpbkar (pid=11155) Backup started
03/27/2013 11:16:52 - Info bpbrm (pid=23506) bptm pid: 23516
03/27/2013 11:16:52 - connecting
03/27/2013 11:16:52 - connected; connect time: 0:00:00
03/27/2013 11:16:53 - Info bptm (pid=23516) start
03/27/2013 11:16:53 - Info bptm (pid=23516) using 262144 data buffer size
03/27/2013 11:16:53 - Info bptm (pid=23516) using 30 data buffers
03/27/2013 11:16:53 - Info nbsrv1 (pid=23516) Using OpenStorage client direct to backup from client sasserver4 to nbapp1
03/27/2013 11:16:55 - begin writing
03/27/2013 11:17:00 - Info bpbkar (pid=11155) 118 entries sent to bpdbm
03/27/2013 11:17:04 - Info bpbkar (pid=11155) 291 entries sent to bpdbm
03/27/2013 11:17:06 - Info bpbkar (pid=11155) 481 entries sent to bpdbm
03/27/2013 11:17:09 - Info bpbkar (pid=11155) 577 entries sent to bpdbm
03/27/2013 11:17:12 - Info bpbkar (pid=11155) 629 entries sent to bpdbm
03/27/2013 11:17:15 - Info bpbkar (pid=11155) 795 entries sent to bpdbm
03/27/2013 11:17:18 - Info bpbkar (pid=11155) 863 entries sent to bpdbm
03/27/2013 11:17:20 - Info bpbkar (pid=11155) 1105 entries sent to bpdbm
03/27/2013 11:17:23 - Info bpbkar (pid=11155) 1234 entries sent to bpdbm
03/27/2013 11:17:26 - Info bpbkar (pid=11155) 1419 entries sent to bpdbm
03/27/2013 11:17:34 - Info nbsrv1 (pid=23516) StorageServer=PureDisk:nbapp1; Report=PDDO Stats for (nbapp1): scanned: 1458945 KB, CR sent: 161353 KB, CR sent over FC: 0 KB, dedup: 88.9%
03/27/2013 11:17:34 - Info nbsrv1 (pid=23516) Using the media server to write NBU data for backup sasserver4_1364401009 to nbapp1
03/27/2013 11:17:35 - Info bpbkar (pid=11155) 1659 entries sent to bpdbm
03/27/2013 11:17:35 - Info bpbkar (pid=11155) bpbkar waited 3696 times for empty buffer, delayed 3710 times
03/27/2013 11:17:38 - Info bptm (pid=23516) EXITING with status 0 <----------
03/27/2013 11:17:39 - Info nbsrv1 (pid=23516) StorageServer=PureDisk:nbapp1; Report=PDDO Stats for (nbapp1): scanned: 7197 KB, CR sent: 1123 KB, CR sent over FC: 0 KB, dedup: 84.4%
03/27/2013 11:17:39 - Info bpbrm (pid=23506) validating image for client sasserver4
03/27/2013 11:17:39 - Info bpbkar (pid=11155) done. status: 0: the requested operation was successfully completed
03/27/2013 11:17:39 - end writing; write time: 0:00:44
the requested operation was successfully completed (0)
As you can see, this backup completes in minutes, not hours. This client has loads of RAM and CPU, is mostly idle at scheduled backup time, is running 7.5.0.4 Enterprise client, and is backing up to a 5020 appliance disk pool. It even has it's own policy and disk storage unit configured. I have another client similarly configured that I am having this issue on as well, but I also have many other clients that are not experiencing this issue.
I've searched the web for details on how accelerator works but have not come up with anything that would make sense of this. Can anyone suggest where I should be looking to troubleshoot this issue?
Thanks!