Remote execution builds are failed on grpc status code UNAVAILABLE

[abderrahim]

Trying to use remote execution to build, I often get these errors (often to the point I can't get a full build done, especially that there seems to be a bug in caching elements)

        [--:--:--] START   freedesktop-sdk.bst:components/pixman.bst: Uploading input root
        [00:00:09] SUCCESS freedesktop-sdk.bst:components/pixman.bst: Uploading input root
        [--:--:--] START   freedesktop-sdk.bst:components/pixman.bst: Waiting for the remote build to complete
        [00:01:00] FAILURE freedesktop-sdk.bst:components/pixman.bst: Waiting for the remote build to complete
        [00:01:09] FAILURE freedesktop-sdk.bst:components/pixman.bst: Running commands
        [00:01:12] FAILURE [87a4afe9] freedesktop-sdk.bst:components/pixman.bst: Failed contacting remote execution server at http://ab84137533a384b2ea288af9529636c2-1891086637.us-east-1.elb.amazonaws.com:8980.UNAVAILABLE/: Socket closed

Looking at the grpc documentation, the UNAVAILABLE status code should be retried.

[nanonyme]

Apparently this https://github.com/apache/buildstream/blob/master/src/buildstream/sandbox/_sandboxremote.py#L102

[nanonyme]

@abderrahim are there any automated tests for RE?

[abderrahim]

There are some tests indeed. See

• https://github.com/apache/buildstream/blob/master/.github/workflows/ci.yml#L61
• https://github.com/apache/buildstream/blob/master/.github/run-ci.sh#L74
• https://github.com/apache/buildstream/tree/master/.github/compose

btw, I had a patch to fix this (or at least work around it) but I don't remember if I ever opened a merge request.

[nanonyme]

I couldn't see PR at least of brief look

[abderrahim]

Yeah, it was just a side branch: 6205b41

[nanonyme]

I don't have test environment. I assumed it was correct to remove UNAVAILABLE from the set or statuses to raise for. Then if resume is not supported, lower condition raises as expected.

[nanonyme]

@abderrahim in other words like #1901