Don't access rows of empty table

NEWS.md

@@ -50,6 +50,8 @@
 
 8. `test()` now reports multiple expected warnings more clearly when `warning=` has length greater than 1L, instead of printing a collapsed or repeated mismatch summary after messages like `Test 1 produced 1 warnings but expected 2`, [#7092](https://github.com/Rdatatable/data.table/issues/7092). Expected and observed warnings are now printed on separate aligned lines, making small differences easier to spot. Thanks @MichaelChirico for the report, @ben-schwen for assistance, and @lucaslarson25, @tjdavis51, @D3VTHSTVR, and @car723 for the fix.
 
+9. Grouping operations on empty (0-row, 0-column) data.tables work as intended, [#7749](https://github.com/Rdatatable/data.table/issues/7749). Thanks @rickhelmus for the report and @MichaelChirico for the fix.
+
 ### Notes
 
 1. {data.table} now depends on R 3.5.0 (2018).

inst/tests/tests.Rraw

@@ -21613,3 +21613,9 @@ test(2371.1, test(0, {warning("a"); 2L}, 2L, warning=c("a", "b")), FALSE,
              output="Test 0 produced 1 warnings but expected 2\nExpected: a\n          b\nObserved: a")
 test(2372.2, test(0, {warning("a"); warning("b"); 2L}, 2L, warning="a"), FALSE,
              output="Test 0 produced 2 warnings but expected 1\nExpected: a\nObserved: a\n          b")
+
+# group-by on empty table works
+empty_dt = data.table()
+test(2373.1, empty_dt[, 1, by = numeric()], data.table(numeric=numeric(), V1=numeric()))
+test(2373.2, empty_dt[, 1L, by = numeric()], data.table(numeric=numeric(), V1=integer()))
+test(2373.3, empty_dt[, .(x=1), by = .(b=numeric())], data.table(b=numeric(), x=numeric()))

src/dogroups.c

@@ -190,7 +190,7 @@ SEXP dogroups(SEXP dt, SEXP dtcols, SEXP groups, SEXP grpcols, SEXP jiscols, SEX
   // We just want to set anyNA for later. We do it only once for the whole operation
   // because it is a rare edge case for it to be true. See #4892.
   bool anyNA=false, orderedSubset=false;
-  check_idx(order, length(VECTOR_ELT(dt, 0)), &anyNA, &orderedSubset);
+  check_idx(order, length(dt) ? length(VECTOR_ELT(dt, 0)) : 0, &anyNA, &orderedSubset);
   for(int i=0; i<ngrp; ++i) {   // even for an empty i table, ngroup is length 1 (starts is value 0), for consistency of empty cases
 
     if (istarts[i]==0 && (i<ngrp-1 || estn>-1)) continue;