Why did steam locomotives not have condensers to recycle the spent steam? The thermal efficiency would be a lot higher.

 There have been condenser locomotives, and they indeed do have double the thermal efficiency than the ordinary double acting steam locomotives. They have one major drawback: the condenser required is HUGE. The condenser really doesn't matter on ships or stationary power plants, but trains have these things called "payload" and "load gauge."

Hauling the condenser is about the same as hauling one extra fully laden freight wagon, and the condenser needs to be designed to fit within the load gauge. Moreover, on ordinary steam engines, the spent steam is fed into the exhaust to improve firebox draught, but condenser locomotives need an extra turbine to do so. Condensers could be heavy as well, which reduced the payload able to be hauled.

Condensing the steam meant the added complexity, and maintenance, of the fan needed to feed the blast pipe that pulled air through the firebox. Unless there was a compelling reason, it was easier to take on water at a station stop rather than add the complexity of condensers. Diesels replaced steam primarily because they required far less maintenance and they offered the flexibility of allowing multiple locomotives to be controlled by the one driving crew. The diesel electric locomotive had about 10 times the efficiency of the regular steam engine, and the condensing system couldn't compare even if it doubled the steam engine's efficiency.

Nevertheless, condenser locomotives have been used, and they have proven worth the extra investment in regions where distances are long, where water is sparse, and where fuel is expensive. Another extra benefit is that they do not leave the telltale vapour clouds behind them. The South Africans built 90 of these condensing locomotives from 1953 to work arid regions like the Great Karoo. Steam would be recycled up to eight times, giving the locos a 500 mile range. They had effectively sevenfold the range of ordinary steam locomotives, with no need to stop to replenish fuel and water. Deutsche Reichsbahn also used condenser tenders in Russia during the WWII, where distances are long.

Why can't diesel locomotives be switched off?

 In Indian Railways there is a common practice of not switching off diesel locomotives, even when stopped at a station. There are some reasons for it.

1. Many diesel locomotives are old and tend to take an hour or more to get started and thus turning off the locomotive and again turning it on every time is not pragmatic and will result in a lot of delay.

2. Brakes in the trains are fed by a compressor inside the locomotive and the compressors in older trains were mechanical meaning that they rotated due to the rotation of the engine shafts. Therefore, the locomotives were not turned off because the compressors would turn off. Even today, with electrical compressors they power supply must be maintained so the diesel locomotives are not turned off.

3. Locomotives have a lot of electronic controls that consume a lot of power. These controls will need continuous power and the locomotive batteries cannot sustain it for long, therefore needing for the engine to keep running and keep the generator running.

Yet now we have something called as an Auxiliary Power Unit (APU) in diesel locomotives:

This can be allowed to run even when the engine is turned off so that it can supply power for the electronic controls, brake compressor and keep the batteries charged.

An idle diesel locomotive with the engine running will use 20 liters of diesel per hour. Yet, with the APU it will use only 7 liters of diesel per hour.

(Image Credits = Google Search)