Yes, but him, Daala, the warlords are all part of basically Sith Empire (GE is basically culmination of Sith's plan afterall), so they still, however minimum, associated with Sith.
That's not how that works considering we have literal Sith Homeworlds and other hotspots that are far more damaging than anything being from The Empire...
Just like non force user who serves Revan, Malak, Vitiate and so on
That really feels assbackwards there, no joke.