I countered an issue where you can't demolish buildings that are no longer in your influence area (the green grid). It happened when I ordered multiple buildings (firepits) to be demolished on the edge of my village's influence area, and the nearest ones got demolished first. This left the furthest buildings outside the village influence area, before they were demolished.
The villagers seemed to ignore them completely, which was unfortunate since you can't cancel the demolish order and the buildings still consumed a building slot. When I built a new firepit that extended the influence range back to the to-be demolished buildings, the villagers started the job immediately.