.SD in grouped operations causes update() to refit on wrong data subset

[PMildenb]

I often fit models to subsets within data.tables, similar to Section 3.3 in the .SD vignette. However, is the following behavior intended?

A model fit m1 on the subset Grp == "A" is identical to the respective grouped model fit m2, as expected. Yet, a trivial update() call yields vastly different results, which did surprise me a little.

library(data.table)
library(lme4)

set.seed(2026)

dat <- CJ(Grp = c("A", "B"),
           ID = factor(1:8),
           Time = factor(c("T1", "T2", "T3", "T4")))
dat[, y := 3 * (Time == "T2") + rnorm(20, 3)[as.integer(ID)] + rnorm(.N, sd = 3)]

# Fit models using .SD --- ungrouped subset vs. grouped
m1 <- dat[Grp == "A", .(model = .(lmer(y ~ Time + (1 | ID))))]$model[[1]]
m2 <- dat[, .(model = .(lmer(y ~ Time + (1 | ID)))), by = Grp]$model[[1]]

all.equal(m1, m2) # equal as expected
all.equal(update(m1), update(m2)) # different results

As far as I understand, this issue stems from how environments are evaluated:

sd_m1 <- get(".SD", environment(m1@call$formula))
sd_m2 <- get(".SD", environment(m2@call$formula))

all.equal(sd_m1[, y], dat[Grp == "A", y]) # TRUE
all.equal(sd_m2[, y], dat[Grp == "B", y]) # TRUE, but should be Grp == "A"!

I would greatly appreciate it if there were (or someone could point me to) an elegant way to ensure that .SD retains the correct group-specific subset when used with a grouping variable :)

> sessioninfo::session_info()
─ Session info ────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.6.0 (2026-04-24 ucrt)
 os       Windows 11 x64 (build 26100)
 system   x86_64, mingw32
 ui       Positron
 language (EN)
 collate  German_Germany.utf8
 ctype    German_Germany.utf8
 tz       Europe/Berlin
 date     2026-07-23
 pandoc   NA

─ Packages ────────────────────────────────────────────────────────────────────────────────
 package     * version   date (UTC) lib source
 boot          1.3-32    2025-08-29 [2] CRAN (R 4.6.0)
 cli           3.6.6     2026-04-09 [1] CRAN (R 4.6.0)
 data.table  * 1.18.4    2026-05-06 [1] CRAN (R 4.6.1)
 lattice       0.22-9    2026-02-09 [2] CRAN (R 4.6.0)
 lme4        * 2.0-1     2026-03-05 [1] CRAN (R 4.6.1)
 MASS          7.3-65    2025-02-28 [2] CRAN (R 4.6.0)
 Matrix      * 1.7-5     2026-03-21 [2] CRAN (R 4.6.0)
 minqa         1.2.8     2024-08-17 [1] CRAN (R 4.6.1)
 nlme          3.1-169   2026-03-27 [2] CRAN (R 4.6.0)
 nloptr        2.2.1     2025-03-17 [1] CRAN (R 4.6.1)
 rbibutils     2.4.1     2026-01-21 [1] CRAN (R 4.6.1)
 Rcpp          1.1.1-1.1 2026-04-24 [1] CRAN (R 4.6.0)
 Rdpack        2.6.6     2026-02-08 [1] CRAN (R 4.6.1)
 reformulas    0.4.4     2026-02-02 [1] CRAN (R 4.6.1)
 rlang         1.2.0     2026-04-06 [1] CRAN (R 4.6.0)
 sessioninfo   1.2.4     2026-06-04 [1] CRAN (R 4.6.0)

[aitap]

Will try to give a better answer later, but when you're struggling with data mutation during by-group operations, use copy().