Rebuild Report Database Isilon

Introduction

If you ever get a problem with networking between 2 distant Isilon clusters then you might get a problem with SyncIQ reports filling up with nonsense. So if you have a lot of jobs you might end up with thousands of reports. This in turn means the Isilon will not return results from either the CLI or GUI when trying to see reports for synciq policies but will just timeout.

Solution

Here is the ultimate solution for the problem, check how many reports exist:

find /ifs/.ifsvar/modules/tsm/sched/reports/ -name "report-[0-9]*.gc" | wc -l

isi sync policies list

Copy the list so you can enable the policies later and then disable all policies:

isi sync policies disable --all

Change your policies for a maximum age and count to more sensible levels:

isi sync policies modify --policy=<policy-name> --report-max-age=1W --report-max-count=100

Disable the migrate service:

isi services -a isi_migrate disable

Then move the reports to the support directory:

mv /ifs/.ifsvar/modules/tsm/sched/reports /ifs/data/Isilon_Support/reports.bak

Then create a new reports directory:

mkdir /ifs/.ifsvar/modules/tsm/sched/reports

Find all reports over 7 days old and delete:

find /ifs/data/Isilon_Support/reports.bak/ -type f -mtime +7 -exec rm {} \;

Copy all the remaining reports back to the original location:

cp /ifs/data/Isilon_Support/reports.bak/<policy-UID>/* /ifs/.ifsvar/modules/tsm/sched/reports/<policy-UID>/

Rebuild the database:

isi_classic sync reports rebuild

Enable your policies:

isi sync policies enable --policy=<policies>

Finally enable the migrate service:

isi services -a isi_migrate enable

This will do the trick. I would advise you complete for the first time through EMC support however.