A pretrained transformer that never trains on your data, set against XGBoost, logistic regression, neural nets, kNN and Naive Bayes on three targets: response, high-value response, and gift amount. Scored with standard metrics and with the metric that pays: net revenue from the mailing.
This study began with a LinkedIn post about TabICL, a "foundation model" for tabular data that claims to match tuned gradient boosting without being trained on your data at all. Claims like that deserve a harder test than a benchmark table. So I ran TabICL on a data set I know well: the KDD Cup 1998 donor data. It is messy and imbalanced, the dollar amounts are heavy-tailed, and the predictors are weak. It is also best judged by a metric that isn't statistical at all: the net revenue a mailing earns. TabICL knows nothing about that metric. It optimizes what it was pretrained to optimize. The question is whether that is good enough when the money is what counts.
A gradient-boosted model learns from your data: it fits trees to your rows, and the fitted trees are the model. TabICL moves the learning somewhere else. Its authors pretrained a transformer on millions of generated tables, so it learned a general procedure: given labelled rows, predict the label of a new row. When you call fit(), nothing is learned. The rows are only stored. At predict(), your labelled rows and the rows to score go through the network together, and probabilities come out of one forward pass.
The labelled rows you hand it are called the context. The rows being scored are the test rows. Those two colors carry the same meaning in every diagram below.
Inside that forward pass, the network works in three stages. Each one shrinks the table into something the next stage can handle.
Each column is handled on its own. A set transformer reads the column's context values, together with their labels, through 128 learned "inducing points", which act as a compact summary of that column's distribution in this table. Every cell then gets a 128-number embedding from that summary. So a LASTGIFT of $25 is encoded by where $25 falls in this table's LASTGIFT values and how those values relate to the target here. Only context rows feed the summary, so test labels never leak in.
Each row now holds one embedding per feature (38 for this KDD Cup prep). A second transformer lets the features within the row attend to each other, which is where interactions such as "frequent donor and recent gift" can form. Four summary tokens (CLS) collect the result. Joined together, they give a 512-number vector for the row, the same size whether the table has 10 columns or 500. Rotary position encoding tells features apart. The ensemble described below shuffles feature order between members, so no particular order matters.
Now each row is one vector, and context vectors also carry their label. Twelve transformer blocks let every context row attend to every other context row, building up the dataset's pattern. Each test row attends to the context rows and asks which labelled rows look like it and how those turned out. Context rows never attend to test rows, and test rows never attend to each other. That is why the 47,706 test rows can be scored in batches of 2,000–3,000 with identical results, and why cost grows with context size times test size.
Every predict() actually runs eight versions of the table: different feature orders, the class labels relabelled, and different normalizations (power and quantile transforms). It clips values beyond 4 standard deviations and averages the eight outputs, with a temperature of 0.9 on the final softmax. "No tuning" means these settings were tuned by the authors on public benchmarks and then frozen.
TabICL has never seen a real dataset during pretraining. Its training tables came from a random generator. The package's generator mixes roughly 70% random causal graphs built from small neural networks with 30% tree-based functions built with XGBoost. About a fifth of the features are categorical, and columns and labels are shuffled. So it carries no memory of other datasets that it has to set aside for yours. What it carries is a learned sense of which input–output relationships are common: smooth effects, thresholds, a few interacting features, many irrelevant ones, label noise.
That learned sense is a prior, in the Bayesian meaning. Pretraining teaches the network to approximate the posterior prediction for a new row, averaged over every table the generator could produce. That gives it a strong edge when data are scarce, the same way an informative prior helps a small sample. It also carries the usual price of a prior:
When real data don't match the prior, nothing in the method tells you so.
A boosted tree on unfamiliar data at least shows you its training curve and its holdout error. TabICL returns probabilities with the same confidence either way. The only check is the one you would run on any model: a holdout set and a baseline it has to beat.
The KDD Cup 1998 donor data are a demanding test: 95,412 lapsed donors, a 5.1% response rate, heavy-tailed dollar amounts, coded recency/frequency/amount (RFA) fields and many weak predictors. The split is 50/50 (seed 22): 47,706 rows to learn from and 47,706 to score. Every model uses the same 38 inputs. TARGET_D, record IDs and a row-order artifact are removed, because TARGET_D leaks the answer into a response model.
One hypothesis I have is that the more imbalanced the data set is, the more TabICL adjusts the data, and the greater the risk that it will miss patterns in the data. I tested this by creating a narrower response variable, HV15, which labels a donation as a "good" donation only if the gift was $15 or more. That changes the response rate from 5.1% to 2.4%. It also leaves a 5,000-row TabICL context with about 125 positive cases instead of about 250.
| Target | Definition | Test positives | Rate | What a model learns from |
|---|---|---|---|---|
| TARGET_B | responded to the mailing | 2,422 | 5.1% | train half (TabICL: a 3k or 5k context) |
| HV15 | responded with a gift ≥ $15 | 1,136 | 2.4% | train half (TabICL: 5k context, ~125 positives) |
| TARGET_D | gift amount, given a response | 2,422 gifts | mean $15.63 | the 2,421 train responders (all of them, for every model) |
Six model families: TabICL with default settings; XGBoost with 300 trees, depth 4 and no tuning, trained on all rows and on the same 5k sample TabICL saw; logistic regression (ridge regression for TARGET_D); a neural net (MLP 64-32 with early stopping); kNN (k = 200, or 25 for gifts); and Gaussian Naive Bayes. The sklearn models get median imputation, standardization and one-hot categoricals. TabICL and XGBoost read the raw columns. Every model scores the full 47,706-row test half.
Two families of metrics. Standard: AUC, log loss and Brier score for classifiers, and MAE, RMSE and correlation for gifts. Custom: lift in the top 10% and 20%, and cumulative net revenue. That is the KDD Cup's own scoring: mail the list in score order, and each mailed record adds its actual gift minus the $0.68 mailing cost. Mailing everyone nets $5,405. A perfect list, mailing only the 2,422 responders, would net $36,198.
Hits = responders (TARGET_B = 1) inside the mailed group; HV = responders who gave $15 or more. Peak = the most profitable depth, chosen with hindsight on the test half, so it is an upper bound and not an operating rule.
This is the finding AUC hides. TabICL's top 10% loses money. Its 4,771 best-scored donors include 433 responders, close to XGBoost's 448, but only 15 of them gave $15 or more, against 78 for XGBoost. TabICL puts frequent, small-gift donors at the top of its list. The two models have nearly the same AUC and the same hit counts but very different dollars. It isn't a fluke of one run. TabICL's top 10% lost money in all three seeds (−$249 to −$329, with only 14–15 donors of $15 or more), while XGBoost's made $1,349 to $1,731. We haven't established why. One hypothesis is that TabICL's built-in power and quantile normalizations flatten the long dollar tails, so big-gift history carries less weight. That hypothesis is testable.
Every TARGET_B curve peaks at 91–100% depth. A response model alone can't tell you whom not to mail profitably, because the gift size drives the money. That is the case for the next two targets.
HV15 tests the imbalance hypothesis from the setup. Responders who gave $15 or more are 2.4% of the test half (1,136 people), and each TabICL context of 5,000 rows holds 125 of them. Everything else is unchanged: the same inputs, split, models and settings as for TARGET_B.
This is TabICL's most natural fit. Every model learns from the same 2,421 train responders, and all of them fit inside a single TabICL context, so nothing is subsampled away. The TabICL regressor is a separate checkpoint, pretrained the same way on synthetic regression tables. Accuracy is measured on the 2,422 test responders. Every model also predicts a gift for all 47,706 test rows, which the expected-value section needs.
A gift model on its own is a surprisingly good mailing list. Its top 10% nets $3.3–3.6k, more than any TARGET_B model's top 10% (−$0.2k to $1.5k), because people who gave large amounts before tend to give large amounts again. It doesn't account for how likely each donor is to respond. Combining the two is the next section.
A response model ignores the gift amount, and a gift model can only learn from the donors who gave. Expected value combines the two while keeping every record in play. For each donor, multiply the probability of responding by the predicted gift, and mail if that expected value is greater than the $0.68 cost. The thinking is a bit like the Heckman correction for sample-selection bias, but it works on the predictions rather than on the model coefficients. I first saw the idea in Zadrozny and Elkan (2001)1.
It is also where calibration starts to matter. The probability enters the decision as a number, not just a rank, so a probability that runs low mails too few people. Unlike the profit peak, the rule uses no test labels, so it is a real operating decision. The grid pairs every response model with every gift model. The "constant gift" column gives every donor the mean train gift ($15.62), to show what the gift model adds.
All TabICL timings are from an Apple M1 with 16 GB, running on the CPU with 4 threads, because a GPU (MPS) run crashed the machine. Every other model fit and scored in under 6 seconds.
Batch scoring is slow for a structural reason. Each test row attends to every context row, through 12 transformer layers, in each of the 8 ensemble members. In our runs, without a cache, the whole 5,000-row context was also re-encoded for every batch of 1,500 test rows. TabICL v2 has a kv_cache option that encodes the context once at fit() and reuses it. We didn't use it in the experiments above. The benchmark below measures what it buys.
TabICL is an interesting technique when you don't have much data and can rely on "common knowledge". That common knowledge is the prior learned from millions of synthetic tables, and it is a far better alternative to creating synthetic data yourself. On KDD Cup 98 it proved that point twice. It was the best model at equal sample size, and it was the best gift model when all the responders fit in one context.
It also showed where the free lunch ends. With the full training set, well-built classical models matched or beat its response model. Its top-ranked donors were the wrong donors for the money. Its probabilities needed fixing before any expected-value decision. And it is slow: building and scoring at this volume took minutes, where every other model took a fraction of a second. For anything operational, speed is a real hurdle. It is not quite a wall, as the speed section shows, but it changes the cost of every prediction.
The practical use is as a component, not a replacement. Use it where the data are small (a gift model, a new segment, a new campaign with little history). Use it as a no-tuning benchmark that a production model has to beat. And always test it on your own holdout with the metric the business pays for.
The next test is the DTNP data from the error-metrics talk. The same harness applies (one model per process, guarded memory, standard and custom metrics side by side), so the question becomes whether the AUC-versus-dollars gap shows up there too.
Method: KDD Cup 1998 learning file (prep_6c8725699af8), 50/50 stratified split (split_86b7d7947fdc, seed 22), 38 inputs. TabICL v2.2.0 with default settings (8-member ensemble), checkpoints tabicl-classifier-v2-20260212 and tabicl-regressor-v2-20260212, on CPU. TabICL contexts are stratified samples of the train half; seeds change the sample and TabICL's internal ensemble. Curves and depth tables use seed 0 of each model. "Prior run" models are from the earlier kddcup1998 pipeline on the same split. This is one dataset and one split, so treat AUC differences under about 0.005 and profit differences under about $300 (the seed-to-seed spread) as noise. Code: Analytics Projects/tabicl_study. Claude Opus 5.5 ran the analyses, built the charts and tables, and suggested reporting insights. Dean Abbott edited all final content and approved it.