Improve the reliability of the pod pending test case

[johscheuer]

Description

Improve the reliability of the pod pending test case. We sometimes see that the test is timing out and it seems like the test is failing because the reconciliation is requeue:

2024/11/26 01:41:00 Failed (unscheduled) Pod: operator-test-fdnhokqf-stateless-42898 , Pod to replace: operator-test-fdnhokqf-stateless-93723
...
{"level":"debug","ts":"2024-11-26T01:47:47Z","logger":"events","msg":"cannot: exclude processes, clusters last recovery was 0.00 seconds ago, wait until the last recovery was 1 seconds ago" ....}
# Roughly 4 minute break
{"level":"debug","ts":"2024-11-26T01:51:08Z","logger":"controller","msg":"Deleting pod","namespace":"nightly-2002-operator-test-irgo4uxk","cluster":"operator-test-fdnhokqf","traceID":"9ba8d86f-3d1d-4b79-ba60-09fef6771a49","reconciler":"controllers.removeProcessGroups","name":"operator-test-fdnhokqf-stateless-93723"}

Type of change

Please select one of the options below.

• Bug fix (non-breaking change which fixes an issue)

Discussion

Testing

CI will run the e2e test.

Documentation

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c732be2
• Duration 3:08:02
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[johscheuer]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c732be2
• Duration 3:08:02
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

Summarizing 1 Failure:
  [FAIL] Operator HA Upgrades when no remote storage processes are restarted [It] Upgrade from 7.1.63 to 7.3.43 [e2e, pr]
  /codebuild/output/src990390057/src/github.com/FoundationDB/fdb-kubernetes-operator/e2e/fixtures/ha_fdb_cluster.go:314

unrelated failure.

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c732be2
• Duration 2:47:48
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)