Filtering large datasets with the same filter

General help and assistance with jamovi. Bug reports can be made at our issues page: https://github.com/jamovi/jamovi/issues . (If you're unsure feel free to discuss it here)
Post Reply
Fabian
Posts: 1
Joined: Wed Sep 25, 2019 10:58 am

Filtering large datasets with the same filter

Post by Fabian »

Hey boys and girls,

does anyone know a good workaround for filtering many variables (> 50) with the same filter? For example, I want to exclude all values that are outside 3 standard deviations from further analysis. So far I have used the following as a filter: -3 <= Z(Variable_A) <= 3. However, it is quite impractical to create a separate filter for each variable or Jamovi becomes quite slow and often crashes if I have set more than six filter layers.

Thanks for your help!

Best regards,

Fabian
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Filtering large datasets with the same filter

Post by jonathon »

hi,

if you can provide us a data set which runs very slowly and crashes, we'd be keen to take a look.

here's a few options for formulas (you can use `or`):

-3 < Z(Variable_A) < 3 or -3 < Z(Variable_B) < 3 or -3 < Z(Variable_C) < 3

ABS(Z(Variable_A)) < 3 or ABS(Z(Variable_B)) < 3 or ABS(Z(Variable_C)) < 3

MAX(ABS(Z(Variable_A)), ABS(Z(Variable_B)), ABS(Z(Variable_C))) < 3

it seems like we should add a ABSZ() function, maybe i'll do that for the next release. then you could go:

MAX(ABSZ(Variable_A), ABSZ(Variable_B), ABSZ(Variable_C)) < 3

and perhaps a MAXABSZ formula isn't a terrible idea either:

MAXABSZ(Variable_A, Variable_B, Variable_C) < 3

cheers

jonathon
Post Reply