I think you will find that most ICE only achive their peak BTE under "limited conditions". Why is the comparison disingenuous? Production auto diesel engines are designed with fuel efficiency front and centre. Motorsport engines have traditionally been thirstier than production engines - 2014 is a reversal with F1 powertrains leapfrogging not only production SI engines but diesels as well.
The BTE of current F1 engines is good for one reason - the rules impose a fuel flow limit. Peak BTE is the only claim made - time spent at WOT is irrelevant. Turbocharging helps but eg Toyota claims 38% for their current Prius - NA. They also claim over 40% NA in their development engines for the next generation of Prius. They are investigating a turbo alternative with even higher BTE. I don't think the mechanical compression ratio is as significant as you might think. Current F1 engines boast a very high expansion ratio courtesy of turbo compounding.
I would disagree with your statement that "Production auto diesel engines are designed with fuel efficiency front and centre". The primary drivers today with either gas or diesel auto engines are cost and emissions, respectively.
I would agree that the BTE of current F1 engines is good, but designing a thermally efficient F1 is made easier for the reasons I noted: no cost restrictions, no emission control requirements, limited service life requirements, limited operating conditions, etc. Operation at WOT/full load conditions is important for BTE with either gas or diesel engines. Obviously diesel engines do not use an intake air throttle to regulate load like gas engines do, diesels regulate load by controlling fuel mass flow.
As for the claim of 38% BTE from a Toyota Prius, once again that is only at limited operating conditions, likely WOT/low road speed/low rpm. I would imagine that a Prius travelling down the highway at 55mph using just the gas engine (ie. at part throttle) would have a BTE of around 20% at best.