The PCI-DSS requirements define scope in terms of the CDE and systems connected to it. The requirements that specifically deal with physical security (9.x) are phrased in terms of facilities that house CDE systems (and can reasonably be interpreted to extend to facilities that house systems connected by network to the CDE).
However, requirement 11.1, which requires scanning for wireless access points is phrased in absolute terms with no reference to the CDE:
Implement processes to test for the presence of wireless access points (802.11), and detect and identify all authorized and unauthorized wireless access points on a quarterly basis.
Is it reasonable to assume that such WiFi scans only need to be done at company facilities that have some connection to the CDE? That seems to match the intent of the requirement, though interpreted broadly it could apply to any facility.
I agree with Jonah B but I would make sure that there were no devices within the facility that either have the ability to administer any devices in the CDE (authentiation servers) or have access to those devices (jump boxes).
If the provisions for satisfying Req. 11.1 elsewhere are not going to apply to a facility, then all of the devices within that facility have to be out of scope.
We're looking for rogue APs. When is that ever not a good idea? If I have no data flows there that affect storage, processing, or trasmission, you're good - the devices are out of scope. I'm curious why rogue APs would be tolerated, but you're good.