Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to specify random effects? #556

Open
Jeffrothschild opened this issue Jun 25, 2023 · 1 comment
Open

Option to specify random effects? #556

Jeffrothschild opened this issue Jun 25, 2023 · 1 comment
Labels
question ❔ Further information is requested R 🐳 Related to R

Comments

@Jeffrothschild
Copy link

Hi, I'm wondering if it would be possible (or even make sense) to have the option to specify random effects in the model explainer?

I thought about this because when looking at feature importance, the full model RMSE is quite different to one that accounts for random effects. For example...

library(tidyverse)
library(tidymodels)
library(lme4)
library(DALEXtra)

df <- nlme::Oxboys 
df

# model using lmer

lmr_mod <- lme4::lmer(height ~ age + Occasion + (1|Subject), df)
sjstats::rmse(lmr_mod)
# RMSE is 1.2

# model with tidymodels
mixed_model_spec <- linear_reg() %>% set_engine("lmer")

mixed_model_wf <- workflow() %>%
  add_model(mixed_model_spec, formula = height ~ age + Occasion + (1|Subject)) %>%
  add_variables(outcomes = height, predictors = c(age, Occasion, Subject))

fit <- fit(mixed_model_wf, df)

explainer <- 
  explain_tidymodels(
    fit, 
    data = dplyr::select(df, c(age, Occasion, Subject)),
    y = df$height,
    label = "lmm",
    verbose = T)


var_imp <- 
  feature_importance(explainer)

# full model RMSE is 8.0
@mayer79
Copy link

mayer79 commented Jun 25, 2023

ML workflows with clustered data are a delicate thing. Using a clean train/test split (grouped split on subject) and then evaluating the model on the test data is often a good choice. Then you wont have this problem.

@hbaniecki hbaniecki added the question ❔ Further information is requested label Jun 26, 2023
@hbaniecki hbaniecki added the R 🐳 Related to R label Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question ❔ Further information is requested R 🐳 Related to R
Projects
None yet
Development

No branches or pull requests

3 participants