nil pointer error in `/api/v1/workflow-templates/{namespace}/lint` #13190

ahaessly · 2024-06-14T21:11:13Z

Pre-requisites

I have double-checked my configuration
I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
I have searched existing issues and could not find a match for this bug
I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

I was able to run argo lint on a file that successfully validated. When I tried the swagger endpoint /api/v1/workflow-templates/{namespace}/lint using the same json, it returned an error:

{
  "code": 13,
  "message": "runtime error: invalid memory address or nil pointer dereference"
}

See json and stacktrace below

Version

v3.5.7

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

WorkflowTemplate JSON

{
  "apiVersion": "argoproj.io/v1alpha1",
  "kind": "WorkflowTemplate",
  "metadata": {
    "name": "poc-dag-template2"
  },
  "spec": {
    "entrypoint": "two-tasks",
    "templates": [
      {
        "name": "rand-num",
        "script": {
          "image": "python:alpine3.6",
          "command": ["python"],
          "source": "import random\ni = random.randint(1, 100)\nprint(i)\n"
        }
      },
      {
        "name": "print-add-one-hundred",
        "inputs": {
          "parameters": [
            {
              "name": "message"
            }
          ]
        },
        "container": {
          "image": "alpine:3.7",
          "command": [
            "echo",
            "{{inputs.parameters.message}} plus 100"
          ]
        }
      },
      {
        "name": "two-tasks",
        "dag": {
          "tasks": [
            {
              "name": "rand-num",
              "template": "rand-num"
            },
            {
              "name": "add-one-hundred",
              "dependencies": [
                "rand-num"
              ],
              "template": "print-add-one-hundred",
              "arguments": {
                "parameters": [
                  {
                    "name": "message",
                    "value": "{{tasks.rand-num.outputs.result}}"
                  }
                ]
              }
            }
          ]
        }
      }
    ]
  }
}

Logs from the workflow controller server

logs from the argo server kubectl logs -n argo deploy/argo-server
attached in the file serverlog.txt

stacktrace from logs:

time="2024-06-14T19:42:20.015Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 32878 [running]:
runtime/debug.Stack()
    /usr/local/go/src/runtime/debug/stack.go:24 +0x64
github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.PanicLoggerUnaryServerInterceptor.func4.1()
    /go/src/github.com/argoproj/argo-workflows/util/grpc/interceptor.go:22 +0x48
panic({0x22c85a0?, 0x4e58e90?})
    /usr/local/go/src/runtime/panic.go:914 +0x218

#  ------ split for readability ------  #

github.com/argoproj/argo-workflows/v3/pkg/apis/workflow/v1alpha1.(*WorkflowTemplate).GetLabels(0x4000bd5428?)
    <autogenerated>:1
github.com/argoproj/argo-workflows/v3/util/labels.UnLabel(...)
    /go/src/github.com/argoproj/argo-workflows/util/labels/labeler.go:24
github.com/argoproj/argo-workflows/v3/util/instanceid.(*service).Label(0x4000e612f0?, {0x37018c8?, 0x0?})
    /go/src/github.com/argoproj/argo-workflows/util/instanceid/service.go:35 +0x74
github.com/argoproj/argo-workflows/v3/server/workflowtemplate.(*WorkflowTemplateServer).LintWorkflowTemplate(0x4000533350, {0x36dfe78, 0x4000e612f0}, 0x4000998a80)
    /go/src/github.com/argoproj/argo-workflows/server/workflowtemplate/workflow_template_server.go:172 +0x8c
github.com/argoproj/argo-workflows/v3/pkg/apiclient/workflowtemplate._WorkflowTemplateService_LintWorkflowTemplate_Handler.func1({0x36dfe78, 0x4000e612f0}, {0x26246e0?, 0x4000998a80})
    /go/src/github.com/argoproj/argo-workflows/pkg/apiclient/workflowtemplate/workflow-template.pb.go:739 +0x74

#  ------ split for readability ------  #

github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.RatelimitUnaryServerInterceptor.func5({0x36dfe78, 0x4000e612f0}, {0x26246e0, 0x4000998a80}, 0x40002f0a20, 0x4000a521f8)
    /go/src/github.com/argoproj/argo-workflows/util/grpc/interceptor.go:65 +0x11c
github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1({0x36dfe78?, 0x4000e612f0?}, {0x26246e0?, 0x4000998a80?})
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x40
github.com/argoproj/argo-workflows/v3/server/auth.(*gatekeeper).UnaryServerInterceptor.func1({0x36dfe78?, 0x4000e61020?}, {0x26246e0, 0x4000998a80}, 0x4000789678?, 0x40002f0a60)
    /go/src/github.com/argoproj/argo-workflows/server/auth/gatekeeper.go:98 +0x74

#  ------ split for readability ------  #

github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1({0x36dfe78?, 0x4000e61020?}, {0x26246e0?, 0x4000998a80?})
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x40
github.com/argoproj/argo-workflows/v3/util/grpc.glob..func1({0x36dfe78?, 0x4000e61020?}, {0x26246e0?, 0x4000998a80?}, 0xb?, 0x2758b65?)
    /go/src/github.com/argoproj/argo-workflows/util/grpc/interceptor.go:45 +0x34
github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1({0x36dfe78?, 0x4000e61020?}, {0x26246e0?, 0x4000998a80?})
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x40
github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.PanicLoggerUnaryServerInterceptor.func4({0x36dfe78?, 0x4000e61020?}, {0x26246e0?, 0x4000998a80?}, 0x0?, 0x0?)
    /go/src/github.com/argoproj/argo-workflows/util/grpc/interceptor.go:26 +0x78
github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1({0x36dfe78?, 0x4000e61020?}, {0x26246e0?, 0x4000998a80?})
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x40
github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus.UnaryServerInterceptor.func1({0x36dfe78, 0x4000e60cf0}, {0x26246e0, 0x4000998a80}, 0x40002f0a20, 0x40002f0ae0)
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/logging/logrus/server_interceptors.go:31 +0xb8
github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1({0x36dfe78?, 0x4000e60cf0?}, {0x26246e0?, 0x4000998a80?})
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x40
github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3({0x36dfe78, 0x4000e60cf0}, {0x26246e0, 0x4000998a80}, 0x4ed67a0?, 0x40002f0ba0)
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:107 +0x74
github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1({0x36dfe78?, 0x4000e60cf0?}, {0x26246e0?, 0x4000998a80?})
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x40
github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6({0x36dfe78, 0x4000e60cf0}, {0x26246e0, 0x4000998a80}, 0x4000afc9e8?, 0xb22a88?)
    /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xb8
github.com/argoproj/argo-workflows/v3/pkg/apiclient/workflowtemplate._WorkflowTemplateService_LintWorkflowTemplate_Handler({0x24bc180?, 0x4000533350}, {0x36dfe78, 0x4000e60cf0}, 0x40010c4400, 0x400082dce0)
    /go/src/github.com/argoproj/argo-workflows/pkg/apiclient/workflowtemplate/workflow-template.pb.go:741 +0x12c

#  ------ split for readability ------  #

google.golang.org/grpc.(*Server).processUnaryRPC(0x40006e4000, {0x36dfe78, 0x4000e60c60}, {0x36ed0e0, 0x4000f0e000}, 0x40003a0240, 0x4000858180, 0x4e75498, 0x0)
    /go/pkg/mod/google.golang.org/[email protected]/server.go:1343 +0xb8c
google.golang.org/grpc.(*Server).handleStream(0x40006e4000, {0x36ed0e0, 0x4000f0e000}, 0x40003a0240)
    /go/pkg/mod/google.golang.org/[email protected]/server.go:1737 +0x990
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    /go/pkg/mod/google.golang.org/[email protected]/server.go:986 +0x88
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 32816
    /go/pkg/mod/google.golang.org/[email protected]/server.go:997 +0x160
"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

N/A

The text was updated successfully, but these errors were encountered:

agilgur5 · 2024-06-15T02:19:35Z

Follow-up from this Slack thread.

I also edited your content to expand the attachments and convert newlines etc so that they're readable at a glance

agilgur5 · 2024-06-15T02:33:08Z

github.com/argoproj/argo-workflows/v3/util/labels.UnLabel(...)
    /go/src/github.com/argoproj/argo-workflows/util/labels/labeler.go:24

corresponds to this line:

argo-workflows/util/labels/labeler.go

Line 24 in dc6a18d

labels := obj.GetLabels()

so obj is nil or an invalid memory reference

backtrace:

argo-workflows/util/instanceid/service.go

Line 35 in dc6a18d

labels.UnLabel(obj, common.LabelKeyControllerInstanceID)

argo-workflows/server/workflowtemplate/workflow_template_server.go

Line 172 in dc6a18d

wts.instanceIDService.Label(req.Template)

meaning req.Template is nil or invalid 🤔 🤔

then generated code and some handlers

agilgur5 · 2024-06-15T02:40:36Z

Ah I see the problem, the API schema wraps around the WorkflowTemplate JSON. Per the schema, it would be {template: <wftjson>}.

So that is an invalid request.

But it shouldn't have a nil pointer, it should return a 400 error

agilgur5 · 2024-06-15T03:10:10Z

We might just be able to add [(google.api.field_behavior) = REQUIRED] to the protobuf and the code generation might handle validation automatically. If that works, there's several places we should add that.

ahaessly added the type/bug label Jun 14, 2024

agilgur5 added the area/api Argo Server API label Jun 15, 2024

agilgur5 changed the title ~~Rest API to lint a workflow template fails for a valid document~~ nil pointer error in /api/v1/workflow-templates/{namespace}/lint Jun 15, 2024

agilgur5 added solution/suggested A solution to the bug has been suggested. Someone needs to implement it. P3 Low priority labels Jun 15, 2024

tooptoop4 linked a pull request Nov 6, 2024 that will close this issue

fix: nilpointer in lint workflowtemplate. Fixes: #13190 #13868

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nil pointer error in `/api/v1/workflow-templates/{namespace}/lint` #13190

nil pointer error in `/api/v1/workflow-templates/{namespace}/lint` #13190

ahaessly commented Jun 14, 2024 •

edited by agilgur5

Loading

agilgur5 commented Jun 15, 2024

agilgur5 commented Jun 15, 2024 •

edited

Loading

agilgur5 commented Jun 15, 2024 •

edited

Loading

agilgur5 commented Jun 15, 2024

nil pointer error in /api/v1/workflow-templates/{namespace}/lint #13190

nil pointer error in /api/v1/workflow-templates/{namespace}/lint #13190

Comments

ahaessly commented Jun 14, 2024 • edited by agilgur5 Loading

Pre-requisites

What happened/what did you expect to happen?

Version

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Logs from the workflow controller server

Logs from in your workflow's wait container

agilgur5 commented Jun 15, 2024

agilgur5 commented Jun 15, 2024 • edited Loading

agilgur5 commented Jun 15, 2024 • edited Loading

agilgur5 commented Jun 15, 2024

nil pointer error in `/api/v1/workflow-templates/{namespace}/lint` #13190

nil pointer error in `/api/v1/workflow-templates/{namespace}/lint` #13190

ahaessly commented Jun 14, 2024 •

edited by agilgur5

Loading

agilgur5 commented Jun 15, 2024 •

edited

Loading

agilgur5 commented Jun 15, 2024 •

edited

Loading