AFS ops meeting 2014-11-03

<h1>AFS ops meeting 2014-11-03</h1>

<p><u>present:</u> Dan, Kuba, Massimo, Jan</p>

<h2>Issues:</h2>

<ul>
    <li>BOINC server hangs: was OK over weekend, just started hanging again several time&nbsp;(and is now auto-SEGV-ed by Dan&#39;s cron job).&nbsp;Suspicion is that the CERN rescheduling patch (now?) has some concurrency issue, possibly linked to &quot;fs setacl&quot; (which BOINC/sixtrack did&nbsp;repeatedly, removed from one script but many otehrs had copied that).&nbsp;Dan will try out a minimally-modified upstream 1.6.10</li>
    <li>Backup/ABS:
    <ul>
        <li>will better handle corrupted/0-size CASTOR files, in particular not ever consider these to be useful for incremental backups (and eventually also auto-clean these)</li>
        <li>Massimo: TSM usage decreasing as expected? Should go from initial 2.5PB to 500TB (last fully copy of all AFS, will stay until manually removed - once restores from TSM have been disabled). Kuba will check.</li>
    </ul>
    </li>
    <li>Indico meeting: no &quot;spare&quot; server on critical power? (only afs260-267&nbsp;are in barn, all assigned to volsets &quot;users&nbsp;idle_projects&quot;. afs262 has less users (why?) ).
    <ul>
        <li>could try to get ex-afs240,afs241 back (but already renamed..)</li>
        <li>could wait for Kuba to liberate afs230,afs231 (cernbox) - ETA 2 weeks (OK)</li>
        <li>side discussion - both &quot;critical power&quot; and &quot;UPS&quot; ought to be easily queriable (aka become a puppet &quot;fact&quot;)</li>
    </ul>
    </li>
    <li>PTS IP/DNS cleanup - done, 27k entries gone (have list); will run once/day
    <ul>
        <li>should also remove the &quot;parked&quot; IPs that can be recycled at short notice from OpenStack, have some fairly random name.</li>
    </ul>
    </li>
    <li>afsmisc - move &quot;arcserver&quot; functionality to VM tomorrow morning; can use &quot;afs_admin -s afsarc1.cern.ch&quot; to test</li>
    <li>ASIS removal - why touch this?? some volumes broken; hardcoded; &quot;odd&quot; use case (e.g external IPs). Removal is great way to make friends.. Should doublecheck that AFS is not using anything (perl?) from there..</li>
</ul>

<h2>Discussion:</h2>

<p>Is it worth to run AFS servers without AFS client access? Would need to package the existing code. (several benefits: would allow to attack the mess of dependencies under /p/, can do staged roll-out of code, can use in test cell)? Unclear:</p>

<ul>
    <li>some AFS client access might be required for some of the functionality (mount restore volumes?)</li>
    <li>/p/cernafs/ was an attempt to migrate existing code into some new self-contained structure, seems not to be progressing much (coud use GIT checkouts to get code onto machines, instead of RPMs)</li>
    <li>Alternative is to review existing functionality, rewrite along real existing use (possibly in a different context, aka &quot;test cell&quot; - can use non-admin people for that). Once feature parity is reached, no longer need to look at original code. Of course, new bugs will come in as well.</li>
</ul>

<p>Discussion moved on to whether &quot;arc&quot; can be replaced - use SSH or REST for remote calls. But major mess is in the procedure on the server side, not the actual ARC server. Agreement that &quot;chained&quot; ARC procedure (hop from one user to other, via ARC calls in some perl module) should die.</p>

<p>[no real conclusion; can stay as low-prio target; simplification is obviously good]</p>