Implement a no-list strategy for wal-restore to improve recovery times

[sjuls]

Hi @mikewallace1979,

I have another proposal for improving wal-restore download times. This proposal eliminates the list operation in favour of optimistically trying to download the wal file, going through supported compressions and falling back to non-compressed variant.

Since list object operations are quite expensive (eg it's 10x the price of a Get on AWS) and also subject to more variance in performance, this change should be both cheaper and yield a shorter and more consistent download time for all use-cases.

I've run a small experiment to verify whether no-list has the desired effect, and it seems to yield an overall better and more consistent result in my current configuration and cloud provider.

Number of seconds spent to download wal-files - measurements taken against Azure with snappy compression;

	List source_dir	List wal_path (#876)	Optimistic no-list
min	6.602070531	4.466363753	4.252058976
max	17.18391819	15.12628784	7.055532052
avg	9.149752375	6.924325793	5.208105985
median	7.51357695	5.796609713	5.204032868
stdev	3.429938044	3.265527009	0.6495604517

I don't have a good way to test the changes against all cloud provider, hope you can assist me here in some fashion.

Let me know what you think.

[mikewallace1979]

Hi @sjuls - according to the data this looks ilke a worthwhile optimisation!

I wonder if it might also be worth adding an CLI argument, so for cases where the same compression type has been used throughout the archive's lifetime you can specify that compression in the restore_command and Barman will always try that compression first.

I can test this against the remaining two cloud providers though I'm not sure much time I can commit to it this week. If we were to merge this we'll add some corresponding integration tests - unfortunately the integration tests are not public so that's something that'll have to happen behind the scenes for now.

@gcalacoci what do you think about the general idea here?