I just found the actual answer to this. /etc/periodic/security/ has two periodic scripts that by default run daily: 100.chksetuid and 110.neggrpperm
by default (/etc/defaults/periodic.conf) these are enabled and configured to run daily. these scripts scan your system for files that have insecure setuid and negative group permissions, using `find`.
the problem is that this gets run *per jail* and if the jails mount large datasets it eats a ton of CPU time for several hours at a time.
Comments
Displaying 0 of 2 comments
Graham Sutherland / Polynomial
In response to this postthe biggest problem is if one of these operations ever takes over 24h you'll end up with multiple scans overlapping and sharing CPU/IO load, slowing them down, spiralling into resource exhaustion.
these can be turned off system-wide by setting the security_status_chksetuid_enable and security_status_neggrpperm_enable rc.conf vars to NO in the Tunables tab, or you can manually add those overrides to /etc/periodic.conf on a per-jail basis if you just want to turn this off for specific jails.
the middlewared issue I mentioned earlier in the thread is just a symptom of these scans eating up resources. when the scans aren't running the middlewared process uses a lot less CPU time, so I can only guess that there's some spinlock contention or something causing problems (which would make sense because the python stuff is using multiprocessing)
by Graham Sutherland / Polynomial ;
@gsuberland Hmm, would that actually happen if the system has a reasonable amount of RAM? It should be dirt cheap to keep the dents in ARC (neither of those scripts look at file contents do they?) and an ARC hit should be pretty much free.
I suppose it would matter how much disk activity the system and/or ARC is seeing in that 24h period? 🤔
by fraggLe! ;
@gsuberland lol, “once per day” scripts that take more than one day to run but don’t gate themselves from multiple copies of itself running
by Joel Michael ;
Likes: 0
Replies: 3
Boosts: 0
Daniel O'Connor
In response to this post@gsuberland There are numerous periodic jobs like that, IMO the defaults need updating in this modern world of jails and large ZFS datasets.. My favourite solution would be proper OS/FS indexing so you could just query some index for the answer trivially but that is a lot of work
Mentions: @gsuberland@chaos.social
Likes: 0
Replies: 0
Boosts: 0