Translate marshmallow schema to jsonschema

[hancush]

Define a script using the marshmallow-jsonschema library to create a jsonschema from the marshmallow schema.

It would be pretty cool for this step to happen automatically so the jsonschema is always in sync with the marshmallow schema. Consider programmatically running the script as a build step in GitHub Actions.

Some potential candidate Actions for this behavior:

• git-auto-commit
• add-commit

[hancush]

I set up a build job that compiles a JavaScript bundle for the docs site here. This provides a good starting pattern for generating the JSON schema. I think we'll want a separate job to create the JSON file and commit it to master, and then update the build job to move a copy of the schema to the docs/js/ directory so it's available to the demo UI deployed from the gh-actions branch.

[hancush]

@smcalilly This is the next issue on deck.

I talk a lot about CI, but the first step is actually to translate the marshmallow schema to jsonschema. As stated in the description, the marshmallow-jsonschema library looks like a promising tool for that. Please evaluate the library by writing a Python script to perform the conversion. Pay special attention to how, if at all, it handles the conditional logic you implemented in the schema-level validation methods you wrote.

I'd like to review a draft once you've drafted the Python script a written a brief inventory (a bulleted list is fine) of any validation that isn't automatically translated from marshmallow to jsonschema.

Any questions up front?

[smcalilly]

I've added a basic script to the branch called schematize. You can see the output here.

It didn't convert any of the validation decorators. There is custom validation for these fields:

• county_names
• other_jurisdiction
• sample_location
• sample_location_specify
• pretreatment
• pretreatment_specify
• rec_eff_percent
• rec_eff_target_name
• rec_eff_spike_matrix
• rec_eff_spike_conc
• hum_frac_mic_conc
• hum_frac_mic_unit
• hum_frac_target_mic
• hum_frac_target_mic_ref
• hum_frac_chem_conc
• hum_frac_chem_unit
• hum_frac_target_chem
• hum_frac_target_chem_ref
• other_norm_conc
• other_norm_name
• other_norm_unit
• other_norm_ref
• inhibition_detect
• inhibition_adjust
• inhibition_method
• time_zone (field-level)
• sample_collect_date (field-level)
• test_result_date
• sample_matrix
• flow_rate
• sars_cov2_std_error
• sars_cov2_cl_95_lo
• sars_cov2_cl_95_up

It appears that the marshmallow-jsonschema library doesn't add the schema-level validation, but somebody has opened an issue to add it. I'm wondering if this is even compatible with the JSON schema spec? Reading this right now.

Some solutions/questions/brainstorming:

• I also don't see the OneOf in the output, but maybe this is because we're using a custom CaseInsensitiveOneOf class?
• Somehow use the dependencies?
• There are like three different *_conc fields that are similar to each other? Perhaps this could be a reusable custom type, that we then we could specify the type with the schema conversion?
• Would it help if some of the dependent fields were be refactored into a nested field?

[hancush]

A couple of notes ahead of our chat:

• Well observed about the OneOf subclass. Looks like you can set _jsonschema_base_validator_class as an attribute of your subclass to determine how it is interpreted. Relevant code. One catch is that there isn't a great way to do case insensitive matching in JSON Schema...
• Yes to dependencies. I used them when I wrote a JSON Schema for another project and they worked quite well.

Look forward to hearing more from you about the other bullet points in your comment, which I'm not familiar with.

[hancush]

I think this documentation has good examples of requiring fields based on the value of another field: https://react-jsonschema-form.readthedocs.io/en/latest/usage/dependencies/

[hancush]

Another good thread for conditional requirements: https://stackoverflow.com/questions/38717933/jsonschema-attribute-conditionally-required

[hancush]

@smcalilly The next step here is to replace the schema.json file in docs/js/ with the actual JSON schema. You can use your script for this!

Then, you should do some test uploads locally to confirm it's working as expected and, if not, address any problems that shake out. Any immediate questions?

[smcalilly]

No questions. I'm working on this. The script setup is useful! Finding/working through some problems with the schema validation.

[hancush]

The first draft of this is done, as of #15! We have issues open to address outstanding bugs, so I'm going to close this.