need reduced echam and jsbach output for spinup

[chrisdane]

Hi

Is your feature request related to a problem? Please describe.
The default namelist.echam for a PI-CTRL experiment, which is (almost always?) also used for spinups, creates a lot of data on a high temporal interval (< month) due to the namelist blocks

# cd to esm_tools
git checkout develop
git fetch
# namelists/echam/6.3.04p1/PI-CTRL/namelist.echam:
&mvstreamctl
    interval = 1, 'days', 'last', 0
    target = 'glday'
    source = 'gl'
    variables = 'q:mean'
/
&mvstreamctl
    target = 'g3bday'
    interval = 1, 'days', 'last', 0
    source = 'g3b'
    meannam = 'tslm1', 'tsi'
/
&mvstreamctl
    interval = 1, 'days', 'last', 0
    target = 'g3bid'
    source = 'g3b'
    variables = 'u10:mean', 'v10:mean', 'temp2:mean', 'relhum:mean', 'albedo:mean',
                'dew2:mean', 'ws:mean', 'sn:mean', 'wimax:max', 't2min:min', 't2max:max'
/
&mvstreamctl
    interval = 1, 'days', 'last', 0
    target = 'jsbid'
    source = 'jsbach'
    variables = 'layer_moisture:mean'
/
&mvstreamctl
    interval = 6, 'hours', 'last', 0
    target = 'sp6h'
    source = 'sp'
    variables = 'st:mean', 'svo:mean', 'lsp:mean', 'sd:mean'
/
&mvstreamctl
    interval = 1, 'hours', 'last', 0
    target = 'g3b1hi'
    source = 'g3b'
    variables = 'u10:inst', 'v10:inst', 'wimax:max'
/

For a spinup, this is unnecessary and bad practice since nobody will ever need this data but the disks are full with it.

Describe the solution you'd like
As far as I understand the esm tools, I would like to have a esm_tools/namelists/echam/<version>/PI-CTRL-SPINUP/namelist.echam which is the same as esm_tools/namelists/echam/<version>/PI-CTRL/namelist.echam but with only a few important variables on monthly output frequency, or similar.

I tried to achieve this. I ran a default echam-only PI-CTRL experiment on ollie:

esm_tools branch: develop
runscript: /home/ollie/cdanek/esm/runscripts/echam-ollie-initial-monthly.yaml 
work: /work/ollie/cdanek/out/echam-6.3.04p1/pictrl-grb

The resulting echam output after 1 month is

no	name	interval	time	lon	lat	lev	nsp	nc2	file
1	u10      	hr 	744	192	96	  	    	  	pictrl-grb_200001.01_g3b1hi
2	v10      	hr 	744	192	96	  	    	  	pictrl-grb_200001.01_g3b1hi
3	wimax    	hr 	744	192	96	  	    	  	pictrl-grb_200001.01_g3b1hi
4	lsp      	6hr	124	   	  	  	2080	2	pictrl-grb_200001.01_sp6h  
5	sd       	6hr	124	   	  	47	2080	2	pictrl-grb_200001.01_sp6h  
6	svo      	6hr	124	   	  	47	2080	2	pictrl-grb_200001.01_sp6h  
7	t        	6hr	124	   	  	47	2080	2	pictrl-grb_200001.01_sp6h  
8	tsi      	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bday
9	tslm1    	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bday
10	albedo   	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
11	dew2     	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
12	rhumidity	day	31	192	96	47	    	  	pictrl-grb_200001.01_g3bid 
13	sn       	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
14	t2max    	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
15	t2min    	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
16	temp2    	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
17	u10      	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
18	v10      	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
19	wimax    	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
20	ws       	day	31	192	96	  	    	  	pictrl-grb_200001.01_g3bid 
21	q        	day	31	192	96	47	    	  	pictrl-grb_200001.01_glday 
22	apmegl   	mon	1	192	96	  	    	  	pictrl-grb_200001.01_accw  
23	drain    	mon	1	192	96	  	    	  	pictrl-grb_200001.01_accw  
24	grndflux 	mon	1	192	96	  	    	  	pictrl-grb_200001.01_accw  
25	rogl     	mon	1	192	96	  	    	  	pictrl-grb_200001.01_accw  
26	runoff   	mon	1	192	96	  	    	  	pictrl-grb_200001.01_accw  
27	snacl    	mon	1	192	96	  	    	  	pictrl-grb_200001.01_accw  
28	snmel    	mon	1	192	96	  	    	  	pictrl-grb_200001.01_accw  
29	sodif    	mon	1	192	96	  	    	  	pictrl-grb_200001.01_accw  
30	aclc     	mon	1	192	96	47	    	  	pictrl-grb_200001.01_aclcim
31	t2max    	mon	1	192	96	  	    	  	pictrl-grb_200001.01_g3bim 
32	t2min    	mon	1	192	96	  	    	  	pictrl-grb_200001.01_g3bim 
33	topmax   	mon	1	192	96	  	    	  	pictrl-grb_200001.01_g3bim 
34	xi       	mon	1	192	96	47	    	  	pictrl-grb_200001.01_glim  
35	xl       	mon	1	192	96	47	    	  	pictrl-grb_200001.01_glim  
36	lsp      	mon	1	   	  	  	2080	2	pictrl-grb_200001.01_spim  
37	sd       	mon	1	   	  	47	2080	2	pictrl-grb_200001.01_spim  
38	svo      	mon	1	   	  	47	2080	2	pictrl-grb_200001.01_spim  
39	t        	mon	1	   	  	47	2080	2	pictrl-grb_200001.01_spim  

and for jsbach

no	name	interval	time	lon	lat	depth	lev	file
1	layer_moisture	day	31	192	96	5	5	pictrl-grb_200001.01_jsbid

To test for reduced echam output, I ran the experiment

esm_tools branch: echam_scenario_PI-CTRL-SPINUP
runscript: /home/ollie/cdanek/esm/runscripts/echam-ollie-initial-monthly-spinup.yaml 
work: /work/ollie/cdanek/out/echam-6.3.04p1/pictrl-spinup-grb

The resulting echam output after 1 month is

no	name	interval	time	lon	lat	lev	nsp	nc2	file
1	aclcov	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
2	ahfl  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
3	ahfs  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
4	albedo	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
5	aprc  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
6	aprl  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
7	aprs  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
8	aps   	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
9	dew2  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
10	evap  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
11	friac 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
12	geosp 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
13	qvi   	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
14	srad0d	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
15	srad0u	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
16	srads 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
17	sradsu	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
18	sraf0 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
19	t2max 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
20	t2min 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
21	temp2 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
22	topmax	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
23	trad0 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
24	trads 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
25	tradsu	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
26	traf0 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
27	trafs 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
28	tsurf 	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
29	u10   	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
30	ustr  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
31	v10   	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
32	vstr  	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
33	wind10	mon	1	192	96	  	    	  	pictrl-spinup-grb_200001.01_g3bmon
34	q     	mon	1	192	96	47	    	  	pictrl-spinup-grb_200001.01_glmon 
35	lsp   	mon	1	   	  	  	2080	2	pictrl-spinup-grb_200001.01_spmon 
36	sd    	mon	1	   	  	47	2080	2	pictrl-spinup-grb_200001.01_spmon 
37	svo   	mon	1	   	  	47	2080	2	pictrl-spinup-grb_200001.01_spmon 
38	t     	mon	1	   	  	47	2080	2	pictrl-spinup-grb_200001.01_spmon 

and for jsbach

no	name	interval	time	lon	lat	depth	lev	file
1	cover_fract       	mon	1	192	96	  	11	pictrl-spinup-grb_200001.01_jsbachmon
2	veg_ratio_max_mean	mon	1	192	96	  	1	pictrl-spinup-grb_200001.01_jsbachmon
3	layer_moisture    	mon	1	192	96	5	5	pictrl-spinup-grb_200001.01_jsbachmon
4	lai               	mon	1	192	96	  	1	pictrl-spinup-grb_200001.01_landmon  
5	snow_fract        	mon	1	192	96	  	1	pictrl-spinup-grb_200001.01_landmon  
6	soil_temperature  	mon	1	192	96	5	5	pictrl-spinup-grb_200001.01_landmon  

The total size of outdata of 1 month reduces from

cd /work/ollie/cdanek/out/echam-6.3.04p1/pictrl-grb/outdata
find -name "*200001*" | sort | xargs du -hcs
135K    ./echam/pictrl-grb_200001.01_accw
1.5K    ./echam/pictrl-grb_200001.01_accw.codes
1.1M    ./echam/pictrl-grb_200001.01_aclcim
512     ./echam/pictrl-grb_200001.01_aclcim.codes
79M     ./echam/pictrl-grb_200001.01_g3b1hi
512     ./echam/pictrl-grb_200001.01_g3b1hi.codes
2.2M    ./echam/pictrl-grb_200001.01_g3bday
512     ./echam/pictrl-grb_200001.01_g3bday.codes
63M     ./echam/pictrl-grb_200001.01_g3bid
1.5K    ./echam/pictrl-grb_200001.01_g3bid.codes
109K    ./echam/pictrl-grb_200001.01_g3bim
512     ./echam/pictrl-grb_200001.01_g3bim.codes
52M     ./echam/pictrl-grb_200001.01_glday
512     ./echam/pictrl-grb_200001.01_glday.codes
2.9M    ./echam/pictrl-grb_200001.01_glim
512     ./echam/pictrl-grb_200001.01_glim.codes
164M    ./echam/pictrl-grb_200001.01_sp6h
512     ./echam/pictrl-grb_200001.01_sp6h.codes
1.4M    ./echam/pictrl-grb_200001.01_spim
512     ./echam/pictrl-grb_200001.01_spim.codes
2.2M    ./jsbach/pictrl-grb_200001.01_jsbid
512     ./jsbach/pictrl-grb_200001.01_jsbid.codes
367M    total

to

cd /work/ollie/cdanek/out/echam-6.3.04p1/pictrl-spinup-grb/outdata
find -name "*200001*" | sort | xargs du -hcs
1.3M    ./echam/pictrl-spinup-grb_200001.01_g3bmon
4.0K    ./echam/pictrl-spinup-grb_200001.01_g3bmon.codes
1.7M    ./echam/pictrl-spinup-grb_200001.01_glmon
512     ./echam/pictrl-spinup-grb_200001.01_glmon.codes
1.4M    ./echam/pictrl-spinup-grb_200001.01_spmon
512     ./echam/pictrl-spinup-grb_200001.01_spmon.codes
247K    ./jsbach/pictrl-spinup-grb_200001.01_jsbachmon
512     ./jsbach/pictrl-spinup-grb_200001.01_jsbachmon.codes
102K    ./jsbach/pictrl-spinup-grb_200001.01_landmon
512     ./jsbach/pictrl-spinup-grb_200001.01_landmon.codes
4.6M    total

This reduction is roughly 100 - 4.6 MB / 367 MB * 100 ~ 99% although many of the most important variables are included. Of course, one could argue if e.g. the 3d variable q (specific humidity) needs to be included or if other variables should be included and so on ...

I would be very happy if you could implement this in some way as a standard for echam since I am really sick of this huge unnecessary spinup output everywhere.

Thanks a lot for consideration,
Chris

ps: I couldnt get the jsbach stream veg to work (vegmon in the new namelist.echam in the echam_scenario_PI-CTRL-SPINUP branch).

[pgierz]

Can someone PLEASE PLEASE put this in? It's been on my list for months.

[pgierz]

How would this be best solved, Christopher?

echam:
    scenario: PI-CTRL-SPINUP

Or rather:

echam:
    namelist_variant: spinup # Or reduced, minimal, or something

I think using a new "scenario" may be confusing to some people

[chrisdane]

I dont know. Someone of the esm tools heads need to decide that.

Another point I cannot decide is how to distinguish different streams-definitions in echam.yaml and jsbach.yaml in a clean and esm tools-way.

[chrisdane]

There is one problem to define scenario-dependent output streams of echam/jsbach. The echam.yaml-entries

streams:
        - accw
        - co2
        - echam
        - g3bid
        - g3bim
        - g3bday
        - g3b1hi
        - glday
        - aclcim
        - sp6h
        - spim
streamsnc:
        - aclcim
        - g3b1hi
        - g3bday
        - g3bid
        - g3bim
        - glday
        - glim
        - jsbid
        - sp6h
        - spim

define which output files will be moved from work to outdata/<model>/. This way, some files are ignored if I change the stream definitions in the namelist, or, the other way around, the esm tools complain that there are missing files if the namelist does not define some of these streams.

In my view the solution to this would be to consider all work/<expname>_<current_date>.01_* files and drop the streams and streamsnc lists completely (<current_date> may be YYYYMM or something similar, I dont know all cases). Would this be in accordance with how the esm tools work?

@mandresm @dbarbi could you please comment on that? Thanks a lot!

[pgierz]

Distinguishing that cleanly isn't really so easy. The Namelist for echam also influences jsbach output, but then we have two different yaml files and also two different namelists even just controlling model behaviour. Confusion lurks around every single corner there.

Ultimately, we need to find some way of getting a single source of truth for the streams, and define that only once and have it propagate everywhere.

Chris, did you make a branch? Given that we have space issues, I'd like to put that in and update the handbook

[mandresm]

Thanks @chrisdane for this suggestion, I think it's a very good point. About the streams, I would need to have a look again at the echam files. However, even if scenario is an extra variable, I think it is a very intuitive one and could help us in the setup of the strings and on the selection of the namelist_dir. @pgierz , why do you say scenario might be a confusing variable for some people?

[pgierz]

"Scenario" controls a lot more than just output, also for example which model features should be active. PI shouldn't have human land use change, a future run should. I would therefore recommend separating output control from the scenario.

[chrisdane]

Thanks Miguel. I agree that having another scenario is not confusing.

@pgierz1: Having those streams in the echam.yaml and jsbach.yaml is in my view the source of the confusion. Thats why I suggest to drop them completely. I made a suggestion of an alternative and my question was if this is possible.

@pgierz2: Yes I made a esm_tools branch and wrote that several times in my initial post: echam_scenario_PI-CTRL-SPINUP.

@pgierz3: Separating output control and scenario is a good idea. But for echam/jsbach, the output control is defined in the namelist.echam. This namelist, in turn, enters the esm tools via scenario. You would have to change this workflow if you want to separate output control and scenario. It would certainly be possible but Im afraid this would mean much more work than just adding another scenario.

[pgierz]

Chris is correct, that would need a bit of a stronger rewrite.

We could, for example, add one more layer of folders? Or maybe better, merge in the output part of the name list, separate from whatever defines physical behaviour? Do we have a merge feature for namelists yet? They are just dictionaries on the python side.

I would again strongly suggest against making the output dependent on the scenarios. That (gut feeling) seems too easy to mess up accidentally. We will fix one, but forget the other.

[pgierz]

Chris, sorry I didn't see your branch in the initial post. Info is obviously there, I just need to learn how to read...

[chrisdane]

Paul, the output definitions for echam/jsbach are not as simple as for fesom. Separating the stream definitions from everything else in the namelist.echam would yield more confusion in my view. For example, &runctl:default_output, i.e. a namelist parameter from another chapter within the namelist, affects the output as well.

I dont see a problem in making output dependent on the scenario. In fact, my initial motivation is to have it.

[mandresm]

I think this discussion is very interesting and valuable, but I also think we need to include the main advance users of ECHAM into it if we are going to make a major change that affect the streams. What do you think?

[chrisdane]

Sure please do so =)

[dbarbi]

The original idea was to use the streams and streamsnc arrays to automatically GENERATE the output stream part in the namelist.echam, but we never got there.

[chrisdane]

Ok. I dont think this is feasible. How would you specify that e.g. 1) only certain variables from a specfic stream should be saved 2) in a specific temporal interval? If the yaml lists streams and streamsnc will be extended to support those and more features, I feel you end up having similar entries as in the actual namelist.echam. Why reinventing the wheel?