Introducing AdVal

“You know, we use Grok to launch X ad campaigns but honestly it’s not very good.”

That statement (which we heard from a friendly marketing team) sent us thinking on how to assess and improve agents on this kind of tasks. Launching live ad campaigns is costly and slow. Maybe we can simulate them?

Meet AdVal, evaluation of frontier models’ capabilities in ad targeting. The task is to launch and manage an ad campaign on (simulated) X given a client brief which includes product description, some priors about the audience, suggested posts and the goal of the campaign (e.g. sign-ups). The agent is given the official X Ads MCP (which we simulate) and nothing else.

Results Each agent’s geometric mean over AdVal’s 45 held-out briefs, as a multiple of par, a scripted baseline that runs one ad group on the brief’s rules; faint ticks are single briefs. Every agent ran each brief once in its own harness. Yvligo is DeepSeek V4.1 Flash after training on separate briefs; it never saw AdVal’s. Post-training, below, explains how.

This MCP is the interface to the simulation, which roughly splits into 3 components:

X users (aka audience)
X advertisers (aka competition)
your X ads account

1. X users

That’s probably the hardest part to simulate. You need to predict the reaction of users on posts and landing pages while taking into account differences in region, language, age, gender, interests and so on.

Not only that but you also want it to be cheap and fast. Naively making millions of LLM-backed personas is not feasible. So we had to come up with something.

The gist of our current audience model is:

the world is divided into cells by demographic axes (country, age band, gender, platform)
each cell hosts a number of atoms each representing a set of users with a certain profile (interests, click propensity etc)
the reaction of the user on posts and landing pages is modeled both by mechanical laws (e.g. the relation between the times the user saw a post and the probability of engagement) and by embeddings similarity

We chose to use embeddings instead of LLM-as-a-judge for the simulation to be deterministic and fast. We started with a small embedder but then switched to EmbeddingGemma 2 which turned out to be pretty good for our use case.

Agent sees: 0link clicks
Graded on: 0.0expected installs

Click that can convert
Click that can’t
Conversion

Every setup ran three times in the simulator, each with the brief’s budget, one ad group and X’s automatic bidding. The map replays the flight from those runs: each dot is a share of the brief’s audience on X, drawn in its country.

Try it Two briefs from our task set. Pick the targeting and the map replays the flight in the simulation: each dot is a share of the brief’s audience on X. The agent sees clicks; it is graded on what the client counts.

Keyword targeting happened to be particularly tricky to simulate: X supports multiple types of them, for each you have to properly size the reachable audience and words popularity changes each year (LLMs and agents were not a thing just recently). That took quite some iterations to get right.

That said, we didn’t just come up with a model and called it good. We had to make sure the model fits the data of 1000s of campaigns that we had on our hands. So it was a cycle: propose a model => fit on data [aka train] => assess discrepancies [aka test] => repeat.

Besides using private and public data to train and test on, we also had to run live campaigns to close some modeling gaps.

2. X advertisers

X ads follow an auction model, where advertisers make bids on placements of their posts. There are multiple ways to place bids, each with its pros and cons. Unlike the X algorithm, this part is not open-source. Moreover X uses their own ML model under the hood to guide who sees your ads, and you have to take its learning rate and capabilities into account.

Again, we had to run live experiments to make sure that our model corresponds to what actually happens on X.

MAX against AUTO X’s automatic bidding (AUTO) against a capped bid (MAX), with the cap set as a multiple of the ad group’s own AUTO cost per click. Simulator: 15 ad groups from our live tests replayed for 72 hours, 5 seeds each; the band is the range over groups. Live: our AUTO-vs-MAX test on X.

3. Your X ads account

This is relatively easy part where we had to model some state (balance, campaigns, ad groups, promoted posts etc) and to conform to the official X Ads MCP with all its quirks. Nothing fancy here.

The MCP The 74 tools of X’s Ads MCP, which the simulation serves word for word, and the four we added (outlined): a platform search, a policy lookup, and a clock the agent can read and wait on.

Tasks

It’s not enough to make a realistic simulation. You also have to make tasks for it to evaluate and train agents on, and those tasks should have certain properties. We had to make sure tasks test different angles of the problem: consumer products vs B2B, age-restricted verticals which vary by country, presence or absence of X pixel on the landing page etc.

The held-out tasks AdVal’s 45 held-out briefs. Filter by angle, open one to read it as the agent does, and see how every agent scored on it, as a multiple of a scripted baseline (par).

We assessed tasks both ourselves and with an automated QA pipeline, looking for inconsistencies, running scripted baselines and reviewing agent traces.

Post-training

Okay, now we have the simulation (which fits our campaigns data and live experiments) and 100s of selected tasks for it. Does training a model on the subset of these tasks transfer to the rest or is it all just noise?

We split our tasks into 3 parts: train, validation and held-out. Then we selected an open-weights model (DeepSeek 4.1 Flash 1017) which did well on our eval and gave it a best-practice targeting guidance (the one you would give to a human junior targeting specialist).

We ran DeepSeek on the train set with this guidance multiple times and selected the best runs from there. Then we used Tinker to SFT the base DeepSeek model using the selected traces (the technical term is “context distillation by rejection-sampled SFT”). We used validation set for checkpoint assessment during training. The student learned faster than we expected and started overfitting so we discarded the latest checkpoint and used the one before that.

Checkpoints Validation loss on held-back teacher runs, and the student’s campaigns on 8 validation briefs as a share of the teacher’s, after each training block.

Once the model was trained we ran it on held-out tasks and turned out the performance held! It was on par with the teacher and well above other models. But that was just the beginning.

Brief by brief Each dot is one held-out brief: one agent across, another up, both as multiples of par. Above the diagonal, the agent up did better. The teacher is DeepSeek V4.1 Flash with the guidance in its prompt.

More live experiments

After we trained the model we started to conduct A/B tests against other models on real X. That was a sobering experience: turns out real life is quite nuanced (who would have thought). We used the learnings from these experiments to improve our simulation and are still continuing to do so.

Simulator against live X Every gap we measured between the simulator and our live runs on X: what each said, and what we did about it. Filter by status, open a row for the details.

As we improve the simulation we expect the leaderboard to change quite a bit and for more trained models to come. Stay tuned and let us know what you think.