Language dependent testing features

[jrstrunk]

Hey all, I wanted to start a discussion about two relatively simple things I'd love to see in the way we test Gleam code that would require new language features (and could not be implemented by a cool new testing framework):

1. Some compiler-enforced way to show the tests a function is tested by the alongside function definition (so at a glance I can see how a function is being tested, for understanding the function or determining if it needs more test cases)
2. The ability to import private functions / constants from modules into test functions (library code has the @internal function, but application code is stuck with making everything public).

There are lots of ways to implement these two things and there are pros and cons of each, maybe we can all discuss this in the comments :)

[jrstrunk]

A simple thing that would satisfy both is to add a @test annotation that can be added to functions, which would specify a test function to receive the current function as part of it's scope. It could look something like:

// In pair.gleam
// we'd have to first allow importing from the test/ dir, if not just allowed to be called in the @test annotation
import pair_test 

@test(with: pair_test.get_first_string_test)
fn first(pair: #(a, b)) -> a {
  pair.0
}

// In pair_test.gleam

pub fn get_first_string_test() {
  first(#("Hi", "mom"))
  |> should.equal("Hi")
}

Because cyclical imports are not allowed, this would enforce not importing the modules (and their functions) to be tested into the test module, but instead lead authors to add the @test annotation to each function definition in order to test it, achieving item 2.

In tests that use multiple functions:

// In pair.gleam
import pair_test

@test(with: pair_test.map_second_test)
@test(with: pair_test.get_second_test)
fn second(pair: #(a, b)) -> b {
  pair.0
}

@test(with: pair_test.map_second_test)
fn map_second(pair: #(a, b), to c) -> #(a, c) {
  #(pair.0, to)
}

// In pair_test.gleam

pub fn map_second_test() {
  #("Hi", "Lucy")
  |> map_second("friend")
  |> second
  |> should.equal("friend")
}

This Gleam code would compile to Erlang or JS code that has the module imports reversed (so the test module imports the main one and not vice versa like in the Gleam code), and all the functions with the @test annotation are made public and added as unqualified imports to the test module. Compile time checks would need to be run for collisions between function names and emit an error if so.

This @test annotation could then later be expanded to do cool things that require more language features and consideration, similar to #3017:

// In pair.gleam

@test(input: #("Hi", "Lucy"), output: "Hi")
fn first(pair: #(a, b)) -> a {
  pair.0
}

A real world example of library code using this might look like this (using the actual tests and functions in the tempo package):

import datetime_test as test

/// Parses a datetime string in the format `YYYY-MM-DDThh:mm:ss.sTZD`,
/// `YYYYMMDDThhmmss.sTZD`, `YYYY-MM-DD hh:mm:ss.sTZD`,
/// `YYYYMMDD hhmmss.sTZD`, `YYYY-MM-DD`, `YYYY-M-D`, `YYYY/MM/DD`, 
/// `YYYY/M/D`, `YYYY.MM.DD`, `YYYY.M.D`, `YYYY_MM_DD`, `YYYY_M_D`, 
/// `YYYY MM DD`, `YYYY M D`, or `YYYYMMDD`.
/// 
/// ## Examples
/// 
/// ```gleam
/// datetime.from_string("20240613T230400.009+00:00")
/// // -> datetime.literal("2024-06-13T23:04:00.009Z")
/// ```
@test(with: test.negative_offset_from_string_test)
@test(with: test.positive_offset_from_string_test)
@test(with: test.z_offset_from_string_test)
@test(with: test.condensed_negative_offset_from_string_test)
@test(with: test.condensed_offset_from_string_test)
@test(with: test.char_offset_from_string_test)
@test(with: test.space_delim_from_string_test)
@test(with: test.naive_from_string_test)
@test(with: test.date_out_of_bounds_from_string_test)
@test(with: test.time_out_of_bounds_from_string_test)
pub fn from_string(
  datetime: String,
) -> Result(tempo.DateTime, tempo.DateTimeParseError) {
  let split_dt = case string.contains(datetime, "T") {
    True -> string.split(datetime, "T")
    False -> string.split(datetime, " ")
  }

  case split_dt {
    [date, time] -> {
      use date: tempo.Date <- result.try(
        date.from_string(date)
        |> result.map_error(fn(e) { tempo.DateTimeDateParseError(e) }),
      )

      use #(time, offset): #(String, String) <- result.try(
        split_time_and_offset(time),
      )

      use time: tempo.Time <- result.try(
        time.from_string(time)
        |> result.map_error(fn(e) { tempo.DateTimeTimeParseError(e) }),
      )
      use offset: tempo.Offset <- result.map(
        offset.from_string(offset)
        |> result.map_error(fn(e) { tempo.DateTimeOffsetParseError(e) }),
      )

      new(date, time, offset)
    }

    [date] ->
      date.from_string(date)
      |> result.map(new(_, tempo.time(0, 0, 0, 0, None, None), tempo.utc))
      |> result.map_error(fn(e) { tempo.DateTimeDateParseError(e) })

    _ -> Error(tempo.DateTimeInvalidFormat)
  }
}

Another cool thing this could enable is an in-code reminder to add tests later (maybe you thought of a niche test while in the middle of an implementation and want to write it down before you forget), emitting a warning but not actually crashing:

@test(with: pair_test.map_second_test)
@test(with: todo as "get second with custom type test")
fn second(pair: #(a, b)) -> b {
  pair.0
}

[jrstrunk]

Alternatively, @tested(by: pair_test.map_second_test) on the main function and @test(for: pair.map_second) on the test function would be more explicit but requires more back-and-forth setup which is annoying. Plus I am not sure how you'd handle the Gleam import statements since this implies a cyclical import.

[jrstrunk]

A solution that would generally solve item 2 would be to add some way to have friend modules, and then make the test module a friend to the main module so it can import its private values.

[lpil]

Private functions will never be importable, that's the definition of private. If you wish to import them then you would use @internal.

but application code is stuck with making everything public

Applications can use internal modules and definitions. There's no difference there.

[jrstrunk]

I don't think friend modules are a great idea, I just wanted to include it as a secondary comment in case it resonated somehow haha.

But in regards to the internal definitions: in application code usually there are custom, but totally separate modules that live in the same project -- for example even in a simple application I might have a database.gleam module, an api.gleam module, and a user_interface.gleam. If I don't want any other module directly calling the database.get_connections function, but I add @internal to it so I can test it, then all my other application code now gets auto complete suggestions for calling it when I start typing database.get_friends (just for example). This not only could lead to a bug, but also clutters the auto complete prompt. The larger the application gets and the more people collaborate on it, the larger this issue becomes.

Since library code does not live in a project alongside completely separate logic, it does not have this issue and the @internal annotation / internal module is fantastic.

[lpil]

Sorry, I don't know what you mean. There is no difference with @internal whether or not you have defined a function called main or not.

[jrstrunk]

@internal works the same, but is less effective at separating application code because it's scope is the entire project. Applications tend to have code with different concerns living in the same project, whereas Libraries tend to only have code that has the same overall concern in a project.

[lpil]

You can split an application into as many packages as you like. It's common for web applications to have at least 3 packages.

[giacomocavalieri]

I don't love the idea of exposing private functions to a test module, or testing the private bits of a module in general.
In my experience testing the private stuff makes the tests harder to maintain and ties those to some implementation details that become harder to change; I like how gleam forces me to only test the public api of a module.

I'm sure there's plenty of exceptions to this and there could be a use for test functions inside a module to test its private bits but I haven't really felt that need so far :)

[lpil]

By making it @internal, indicating that it is private.

[jrstrunk]

Right, and you made a good point in the other thread: "You can split an application into as many packages as you like".

This is an answer, but can start to feel clunky with big projects. I have a project that has 57 modules that are all related to the same application. Some are small modules, but there are at least 13 that need significant testing. I would have to create 13 different packages inside one repo, then if I made a change in one of those packages that had implications in others, I would have to either go into each project and run the tests manually myself (which would be very error prone), or setup my own test script.

Not horrible, not great. So far Gleam's DX has set a standard for being great.

I don't want a way to share private definitions precisely between modules, but I think it would be nice for testing. If we make testing hard on authors, then it won't be done properly, we'll have broken packages in the ecosystem, and people won't feel good about the things they build with Gleam.

Since Gleam is wonderfully just types and functions, it has a good foundation to be extremely, easily testable -- it's one of the things I like most. I don't want to add frivolous new features to the language, but I think if we support the DX for testing then the entire ecosystem will benefit.

[lpil]

In that situation you would use @internal as it lets you test the code while clearly marking the code as unusable elsewhere, and removing it from all documentation and editor tooling.

[jrstrunk]

When I mark a function as @internal I still see it show up as an auto-complete suggestion in VS code (with no indication that it is internal). Is that what you meant by hiding it from editor tooling?

[lpil]

Apologies, I was thinking of from other packages.

[lpil]

What happens if you forget to add a @test annotation to the other function? It seems like it would be easy to forget to make the pointer to some test modules, resulting in the test never getting run.

What do you do for tests that are not testing any specific function? There's no clear place to add the annotation such that the test function is turned into a test.

[lpil]

The best way to think about language design (or any system design really) is to think about problems and requirements. Once they are fleshed out you can start to work towards some solutions and evaluate them against those goals.

Starting with a solution puts you in a place where you don't have enough context to evaluate it, so it's hard to argue for it.

No particular plans today. Problems I am interested in exploring are test discovery without resorting to brittle naming conventions and runtime reflection, type safe setup and tear down, test parallelism, improving test assertion output, documentation example verification.

[jrstrunk]

That's a good way of framing. I guess to put it that way: a problem is that in a medium project I have lost track of my test coverage, and I am unsure of what test cases may be missing.

For a concrete example: gtempo v5 has 197 public functions and over 400 tests. The details of which functions have which coverage is very difficult to figure out manually. I tried my best to be diligent and write tests immediately after writing a function so I had its full context in my working memory, but that is error prone. If I got distracted while writing a function, I may have lost track of a test I wanted to add. For example, in gtempo v4.3.0 I just straight missed parsing the Z offset in the datetime.parse_any function. I might have caught it before publishing that version if I had an easy to review list of tests that applied to each function. If I saw parse_any_positive_offset_test, parse_any_negative_offset_test, and parse_any_no_offset_test listed together near the datetime.parse_any function, then when I was doing a review before publishing I probably would've realized the Z case was missing. As it is though, it is very un-user-friendly to manually review test coverage when you have a decent amount of them.

It seems the answer right now it to use some external tool (don't think it exists for Gleam yet) or organization process. Maybe that is good enough -- but gtempo is bigger than I planned on it being, and I didn't start off thinking I'd need detailed test organization: meaning I accidentally walked into tech debt that could've been avoided if the compiler was guiding me from the beginning.

I'm planning on spending my next few months of free time studying testing for critical software. If I learn something useful I'd like to think of how it might apply to Gleam. Gleam has so many wonderful things about it; I'd love to see the testing DX be one of those things :)

[lpil]

Wouldn't you want a traditional coverage tool in this case? Having just the annotation on the function wouldn't be sufficient as it wouldn't show you if you hit all the paths though the function so you'd still be lacking the information required to know you missed a case such as Z offset parsing.

Erlang's xref and cover could be a good place to start.

[jrstrunk]

In the past I've never used a coverage tool that was smart enough to provide clear, useful feedback. Maybe I just haven't used the good ones. I'll give xref and cover a shot and see how they are!

[jrstrunk]

Also just in my opinion I like having test cases presented in the forefront instead of hidden behind a separate command, because then developers are lead into thinking about them more often :)

[lpil]

It is sometimes useful to have setup and teardown around all the tests, or around groups of tests. Did you have any thoughts for how this might work?

[jrstrunk]

The implementation that I suggested first would not control the execution of the test, just what is in scope of the test. Whatever test runner you were using would determine how to do test setup and teardown.

Since these annotations really only have the semantics of testing (and nothing that really strictly makes them apply to tests), they could also be applied to any setup/teardown functions like so:

@test(with: pair_test.setup_helper)
@test(with: pair_test.complex_test_1)
@test(with: pair_test.complex_test_2)
fn first(pair: #(a, b)) -> a {
  pair.0
}

[lpil]

The programmer would still need to define the setup and teardown, so a design would be needed for this.

What would that proposed syntax do?

My concern with that one is that it looks extremely similar to the syntax for saying that pair_test.setup_helper is the test for the first function. It would also mean that adding a label changes the meaning of the expression, which is not how Gleam works today.

[jrstrunk]

We might as well hop in a call hahaha, the annotation I suggested does not execute any code itself. The setup and teardown design comes from whatever test runner the user is using, like gleeunit. This is not a replacement for a separate test running library, it is just a way to manage the scope of functions a test function in any arbitrary test framework has access to.

[lpil]

If the language is going to have syntax for defining setup and tear-down for tests it will need to be fully specified and type safe, including the interface that test runners would interface with in order to use it.

[inoas]

I'd love to have doctests:

• lsp to run them on hover or at least validate the example code to be actual gleam code
• something that the testing framework can hook into to run them

And I'd like to have some coverage info from gleam tooling if a larger testing feature is added from to gleam tooling

[jrstrunk]

Me too! Though that is different and much harder to implement than what I outlined above. Though if we were to expand what I suggested, what do you think of using the @test(input: _, output: _) annotations (or similar) for a doctests like experience instead of having the tests literally in the doc comment? I personally find it strange to have code in a comment be executed haha.

[inoas]

doctests make certain that the examples are not out of date
they also offer basic/minimum coverage

[jrstrunk]

Yeah and that's great! We could also generate docs from the @test annotation, so I think they bring the same value. I was just thinking about syntax and semantics mostly.

But also, in functions that have a lot of tests, we may not want to display the tests as part of the main documentation snippet. It would be nice to have a way to separate the most value-bringing examples and the repetitive tests. I think allowing for formatting in the doc comment (like we currently have), plus having a second syntax for a test annotation would bring the best feature set.

[lpil]

Tests are not useful primary documentation, they communicate information very poorly and are too numerous and too verbose. Collecting all the tests to present them as the documentation is not an option.

[jrstrunk]

Totally agree, not primary documentation. They are useful to present in some niche view though: I can't count how many times I've learned how a library works from reading over the tests in lieu of incomplete documentation.