netbackup appliance 5220 version 2.5.1, netbackup version 7.5.0.4. Linux master server backing up a windows 2008 r2 VM machine.
My VM admin moved serveral servers from 1 datastore to another. He called it v-motioned. only selected servers fail with status 156. A reboot of the server clears up the problem and goes away but we have some servers that run 24/7 and rebooting takes a act of congress. I seen where re-installing vmtools would help, changing VSS providers, but evrything will cause a reboot. So i dont know really what fixed it, the change or reboot. All VSS is up & running, I can backup without quiescing on selected servers but this is not recommended and not possible on some servers. i have tried stopping netbackup and clearing
/usr/openv/netbackup/online_util/fi_cntl/ folder and restarting.
is any idea how to fix quiescing without rebooting the client?
detail log :
05/30/2013 04:33:50 - Info nbjm (pid=11843) starting backup job (jobid=137332) for client stsymantec, policy vmware_highland, schedule Full
05/30/2013 04:33:50 - Info nbjm (pid=11843) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=137332, request id:{A724CD60-C903-11E2-B1D8-D2F6FB7B2377})
05/30/2013 04:33:50 - requesting resource stu_disk_primaster
05/30/2013 04:33:50 - requesting resource primaster.NBU_CLIENT.MAXJOBS.stsymantec
05/30/2013 04:33:50 - requesting resource primaster.NBU_POLICY.MAXJOBS.vmware_highland
05/30/2013 04:33:50 - requesting resource primaster.VMware.Datastore.vmwhl-hixiv-san3
05/30/2013 04:33:50 - granted resource primaster.NBU_CLIENT.MAXJOBS.stsymantec
05/30/2013 04:33:50 - granted resource primaster.NBU_POLICY.MAXJOBS.vmware_highland
05/30/2013 04:33:50 - granted resource primaster.VMware.Datastore.vmwhl-hixiv-san3
05/30/2013 04:33:50 - granted resource MediaID=@aaaac;DiskVolume=PureDiskVolume;DiskPool=dp_disk_primaster;Path=PureDiskVolume;StorageServer=primaster;MediaServer=primaster
05/30/2013 04:33:50 - granted resource stu_disk_primaster
05/30/2013 04:33:51 - Info bpbrm (pid=25075) stsymantec is the host to backup data from
05/30/2013 04:33:51 - Info bpbrm (pid=25075) reading file list from client
05/30/2013 04:33:51 - Info bpbrm (pid=25075) start bpfis on client
05/30/2013 04:33:51 - Info bpbrm (pid=25075) Starting create snapshot processing
05/30/2013 04:33:51 - estimated 0 kbytes needed
05/30/2013 04:33:51 - begin Parent Job
05/30/2013 04:33:51 - begin Application Snapshot: Step By Condition
Operation Status: 0
05/30/2013 04:33:51 - end Application Snapshot: Step By Condition; elapsed time 0:00:00
05/30/2013 04:33:51 - begin Application Snapshot: Read File List
Operation Status: 0
05/30/2013 04:33:51 - end Application Snapshot: Read File List; elapsed time 0:00:00
05/30/2013 04:33:51 - begin Application Snapshot: Create Snapshot
05/30/2013 04:33:51 - started process bpbrm (pid=25075)
05/30/2013 04:33:52 - Info bpfis (pid=25083) Backup started
05/30/2013 04:33:52 - snapshot backup of client stsymantec using method VMware_v2
05/30/2013 04:38:33 - Critical bpbrm (pid=25075) from client stsymantec: FTL - VMware snapshot failed: SYM_VMC_TASK_REACHED_ERROR_STATE
05/30/2013 04:38:33 - Critical bpbrm (pid=25075) from client stsymantec: FTL - VMware error received: An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.
05/30/2013 04:38:33 - Critical bpbrm (pid=25075) from client stsymantec: FTL - snapshot creation failed, status 156
05/30/2013 04:38:33 - Warning bpbrm (pid=25075) from client stsymantec: WRN - ALL_LOCAL_DRIVES is not frozen
05/30/2013 04:38:33 - Info bpfis (pid=25083) done. status: 156
05/30/2013 04:38:33 - end Application Snapshot: Create Snapshot; elapsed time 0:04:42
05/30/2013 04:38:33 - Info bpfis (pid=0) done. status: 156: snapshot error encountered
05/30/2013 04:38:33 - Info bpbrm (pid=25515) Starting delete snapshot processing
05/30/2013 04:38:33 - Info bpfis (pid=0) Snapshot will not be deleted
05/30/2013 04:38:33 - end writing
Operation Status: 156
05/30/2013 04:38:33 - begin Application Snapshot: Stop On Error
Operation Status: 0
05/30/2013 04:38:33 - end Application Snapshot: Stop On Error; elapsed time 0:00:00
05/30/2013 04:38:33 - begin Application Snapshot: Cleanup Resources
Operation Status: 4294957297
05/30/2013 04:38:33 - end Application Snapshot: Cleanup Resources; elapsed time 0:00:00
05/30/2013 04:38:33 - begin Application Snapshot: Delete Snapshot
05/30/2013 04:38:33 - started process bpbrm (pid=25515)
05/30/2013 04:38:34 - Info bpfis (pid=25523) Backup started
05/30/2013 04:38:34 - Critical bpbrm (pid=25515) from client stsymantec: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.stsymantec_1369902831.1.0
05/30/2013 04:38:34 - Info bpfis (pid=25523) done. status: 1542
05/30/2013 04:38:34 - end Parent Job; elapsed time 0:04:43
05/30/2013 04:38:34 - Info bpfis (pid=0) done. status: 1542: An existing snapshot is no longer valid and cannot be mounted for subsequent operations
05/30/2013 04:38:34 - end writing
Operation Status: 1542
05/30/2013 04:38:34 - end Application Snapshot: Delete Snapshot; elapsed time 0:00:01
Operation Status: 156
snapshot error encountered (156)