Module development: can an analysis overwrite or replace existing columns?

[scro1461]

Hi,

I'm developing a jamovi module to help my students clean raw data: standardising inconsistent category labels, trimming whitespace, converting codes like -99 to missing, and tidying variable names. The aim is for them to do this in jamovi rather than Excel.

As I understand it, a module can only add new columns through `Output` options, and those stay linked to the analysis. Is there any way for a module to overwrite the original columns, or to convert an output variable into an ordinary data variable so the raw columns and the analysis can be deleted afterwards?

If neither is possible at the moment, would you consider adding a "convert to data variable" option for output columns? At present the only route I can find is copying and pasting each column by hand, which is error-prone for students with a lot of variables.

Thanks,
Joshua

[jonathon]

hi,

yeah, an analysis modifying existing columns is totally verbotten.

but in general terms, jamovi allows users to do these things already:

- standardising inconsistent category labels
- trimming whitespace
- converting codes like -99 to missing
- tidying variable names

in fact, trimming whitespace shouldn't be a thing ... jamovi *should* do that automagically, and typically does, so if you've found a gap, we'd appreciate a bug report.

but we'd be interested in hearing your use-case (user story) in more detail, and how jamovi currently falls short.

jonathon

[scro1461]

Hi Jonathon,

Thanks, and sorry, I hadn't realised Setup lets you edit several variables at once. That covers most of what I thought was missing on the fixing side.

Having thought about it more, the gap for my students is detection rather than fixing. They're undergraduates in criminology and the social sciences working with raw survey or administrative data, often exported from Qualtrics or supplied by agencies. They open a file and don't notice that a variable has "Male", "male" and "M" as separate levels, that 999 is sitting in age, or that a numeric variable came in as text, and they run analyses on it anyway.

So I'm planning a read-only audit module. Students select variables and get:
- a summary table per variable: type, inferred type, missingness and issue count
- an issues table flagging case-inconsistent levels, likely missing-value codes not yet declared, out-of-range values, numbers stored as text and similar problems
- a missingness chart showing which variables tend to be missing together
- a brief guide alongside each issue explaining what it means and how to fix it in jamovi, e.g. "set -99 as a missing value in Setup", with the multi-variable editing pointed out where it applies

Nothing gets written back to the data, so no column access is needed. The fixing stays in jamovi's own tools, and students re-run the audit to check. I'm looking at building the checks on the dataReporter R package rather than writing them from scratch.

Does that sound sensible from your side, and is there anything similar planned that I'd be duplicating? Any pointers on presenting guidance text in the results (an Html element, or notes on tables) would be welcome too.

Thanks,

Joshua

[jonathon]

Yup, this is sounding useful ... a lot of overlap with descriptives of course ... but i can see real merit in it. It's maybe something we should support out of the box! let's start with your implementation though.

For guidance text, use the Text options rather than Html

https://dev.jamovi.org/api/text

We're trying to move people away from Html as much as possible.

jonathon

[sbalci]

In ClinicoPathDescriptives module there are functions checking data quality. Benford, single and multivariable quality checks, and outlier detection. You may check it and use the codes here:
https://github.com/sbalci/ClinicoPathDe ... e/master/R

[scro1461]

Hi Jonathon,

Thanks for steering me towards an audit module rather than one that edits columns. Data Check is now finished and I’ve submitted it to the library: [GitHub link].

It reports problems per variable and lets students decide in the options panel what each value becomes, e.g. F → Female or blank → Not Asian for a follow-up question. It then creates cleaned variables or opens a cleaned dataset in a new window. Nothing touches the original columns. It also handles the quirks of Qualtrics, Microsoft Forms and JISC exports.

Building it turned up three things:

+ Whitespace on CSV import. In jamovi 28.8, leading and trailing spaces are kept, so “ Female” and “female “ become separate levels. The test file in the repo reproduces it (data/messy_test_data.csv, rows 11, 14, 23, 24 and 27).
+ The export action. The docs list action: export from 28.7, but jmvtools 28.4 rejects it: “is not one of enum values: open, openExternal”. I’ve used openExternal for now. Is a newer jmvtools due?
+ Read-only cells in a ListBox. TextBoxes with enable: false don’t appear disabled. Is there a control that shows a string from the option value without letting it be edited?

Any feedback on the module itself would be welcome.

Joshua

[jonathon]

hi,

good catches. I've fixed the stripping of spaces consistent across all the importers now. fix will go out in the next version. i've also pushed an updated jmvtools.

wrt UI stuff, let me pass you over to damo. i'll attend him to your question.

jonathon