The Milky Way contains up to a billion black holes, but they aren't cosmic vacuums. If the Sun were replaced by one of equal mass, Earth's orbit wouldn't shift a single millimeter.
A black hole does not possess extra gravity just because it is a black hole. Its gravitational pull behaves exactly like the gravity of any other object in the universe with the same mass, provided the object remains outside the event horizon.
If a black hole took the Sun's place, the planet would freeze, but it would continue on its exact current path. Black holes only consume matter that crosses the event horizon. They do not pull distant objects out of stable orbits.
Having millions of black holes wandering the galaxy sounds like a minefield until you account for cosmic scale. The Milky Way is about 100,000 light-years across and contains between 100 billion and 400 billion stars. Even at the high estimate of one billion black holes, they account for less than 1% of the massive objects in the galaxy.
The solar system is not sitting still. The Sun, the Earth, and all those millions of black holes are moving together, orbiting the center of the galaxy in a relatively stable pattern. Because everything swirls in the same general direction, and because space is mostly empty, objects rarely cross paths.
The closest known black hole to Earth is Gaia BH1, which sits about 1,560 light-years away. If the entire solar system out to Pluto were scaled down to the size of a US quarter, Gaia BH1 would still be hundreds of miles away.