Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remap-pp-components doesn't confirm that pp components were successfully remapped before exit #13

Open
uwagura opened this issue Aug 21, 2024 · 0 comments

Comments

@uwagura
Copy link

uwagura commented Aug 21, 2024

There have been several instances where cylc has said that the `remap-pp-components task has executed successfully even though it didn't. Two instances where this occurred for me are:

  1. When the pp_chunk_a variable in my pp yaml did not match the chunksize that fre inferred from my history files, causing the remap script to simply do nothing at this point in the loop:

    for c in chunk:
    if c != current_chunk:
    continue

  2. If the link command fails, the remap script also doesn't do anything when it gets a non-zero return value

    if ret != 0:
    if ens_mem is None:
    copy = [f"{copy_tool}",
    f"{input_dir}/{g}/{s}/{f}/{c}/{file}",
    f"{output_dir}/{dirs}/{newfile2}" ]
    subprocess.run( copy, check = False )
    else:
    copy = [f"{copy_tool}",
    f"{input_dir}/{g}/{ens_mem}/{s}/{f}/{c}/{file}",
    f"{output_dir}/{dirs}/{newfile2}" ]
    subprocess.run( copy, check = False )
    print("Component remapping complete")

These problems were quite difficult to debug becausecylc, job.out, and job.err all seemed to indicate that the remap job had completed successfully when they had just failed silently. There could also be several other points where this loop could fail / skip an iteration and the function would still return 0 and print "Component reamapping complete" without actually executing the copy command.

I think it would be helpful to at least add a check before printing that the remapping completed successfully, so that if the files aren't copied to output_dir the user knows to check this script for issues instead of having cylc fail at some future workflow step that may not be related.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants