Nine Homepages and a Dollar Sign
  • My Role
  • Lead Product Designer, Lead of Product Optimization Team
  •  
  • Timeline
  • January 2013 – May 2014
  • Competencies
  • Consumer Conversion
  • Experimentation & Testing
  • Interaction Design
  • Cross-Functional Leadership
  •  
B
y 2013 Grubhub had the thing every startup wants and nobody knows what to do with: enough traffic that a bad decision was expensive and a good one compounded. We were past the stage where you improve the product by having better taste than the other guy. The remaining wins were smaller, less obvious, and only findable by looking.

So we built a team to look. Officially it was the Product Optimization Team. We called it Spoon. A couple of years later the industry decided to call this growth hacking, which made it sound like a trick.

Two designers — me and a senior product designer I managed — two engineers, a data analyst, a QA engineer, and a psuedo-product manager we borrowed from another stream team. I built it and led it for 18 months.
Go Forth, and Conquer
Challenge Description
The team's real asset was permission. We could change anything on the site — no sign-off from a feature pod, no business case written in advance, no design review that ended in a compromise nobody tested. That sounds like a small structural detail. It's the entire thing.

Product pods owned features, and features were both the unit of work and the unit of celebration. That works, and it has a blind spot: nobody's job was the space between the features. The sign-up field nobody questioned. The confirmation screen everyone passed through and no one owned. The moment in checkout where people quietly left.

Those aren't features. They're seams. At scale, seams are where the money is.

Design note. The argument I made for the team wasn't "we should test things." Everyone agrees with that and nothing happens. It was that the seams needed an owner with the authority to change them without asking, because a problem that belongs to everyone belongs to no one.
How We Picked What to Run
Challenge Description
Four metrics: conversion rate, repeat order rate, cart size, bounce rate.

Every test moved through the same ten stages. Identify the opportunity with product, finance, and marketing. Investigate — what do we need to know, quantitative and qualitative both, questions written down. Write a hypothesis. Brainstorm, prioritize, assign. Mockups. A test overview session where everyone had to understand what was being measured before anyone built anything. Tickets, implementation, MixPanel events. Then a reflection period, a written analysis, and a decision: ship the winner or iterate.

The requirement I'd defend hardest is the hypothesis. Every test started as one sentence that could turn out to be false. Minimums and fees are psychological barriers rather than actual barriers to ordering. One text field instead of two will reduce homepage abandons. Cuisines are inferred from the restaurant name and logo and don't add value in search results.

Write the sentence before you make the thing. A test without a falsifiable hypothesis isn't a test — it's an opinion with a mockup attached, and it will find a way to be right.

The analyses went into a company wiki — the Conversion Lab — readable by product leadership across the org, not just by us. A losing test is only worth what it cost you if the next person can find out you already ran it.

Monetate first, Optimizely later. Tests ran a week or two, and the analyst called significance. Not me. A designer who can call his own test the moment it looks good will do exactly that, every time, without noticing.
The Dollar Sign
Challenge Description
The problem. Cart size was the metric we had the least leverage on. Conversion you improve by removing friction. Cart size means convincing someone to order more food than they planned to, which is not a thing you do with a better button. The dollar sign wasn't a one-off. It was part of a line of inquiry we ran across the whole funnel: how much of what looks like a price objection is actually a presentation problem. Further up, we tested pulling delivery minimums and fees out of the search results on the hypothesis that they were psychological barriers rather than real ones — the number that makes you hesitate isn't always the number that makes you stop. That one won too.

The call. We took the dollar sign off the item prices in the menu list. Not the price. Just the symbol. It still appeared on the item detail modal. Worth being precise about the line, because "we made the price less salient and people spent more" is one sentence away from something I wouldn't ship. The price didn't move, shrink, gray out, or hide behind an interaction — same size, same place, full price with the symbol on the detail view before anything enters a cart. If we'd needed to obscure the number to get the lift, I'd have killed it. Refunds didn't move, which was the thing I actually watched.

Cart size went up 7–10%. Restaurant menu designers had known that for years — we didn't discover anything. What we did was refuse to assume it transferred. A printed menu is a fixed object you read once at a table with people watching you order. A delivery cart is editable, private, and running a live total on screen. "It works on menus" is a hypothesis, not a reason. That's most of what the team did, honestly: take a thing that's supposed to be true and find out whether it's true here.



Design note. $12.00 and 12.00 contain identical information and do not read identically. The symbol is what tells your eye it's looking at money, and money is the thing you're being careful about. Remove it and the number becomes a number.
Nine Homepages
Challenge Description
The problem. The homepage was busy. Restaurants, logos, promos, cuisine filters, a search field competing with all of it.

The call. We didn't pick a side. We wrote down both.

One hypothesis: stripping the page down creates a strong call to action and lowers abandonment. Headline, address field, nothing else. It looked like the kind of page you'd put in a portfolio.

The other: images of food, social data, and diner quotes build validity and trust for new diners, and that's what gets them to search.

Those are opposite claims about the same page, and they went into the same deck in the same week. Nine variations, tested against each other.

Design note. Trust won. The clutter was the product — a restaurant grid isn't decoration on a food delivery site, it's the proof. Three hundred places near you deliver, and here are their names. The stripped page asked a first-time visitor to type their address on faith. The crowded one showed them why they should.

I was on the losing side of that. Every instinct from twenty years of graphic design said the quiet page would win — hierarchy, one idea per screen, negative space as a tool. Those principles are right about composition and wrong about a page whose only job is to establish that inventory exists.

What I'd point to isn't that I was wrong, though. It's that being wrong cost the team nothing. Nobody had to lose an argument, and I didn't have to be talked out of my own taste by a colleague, which is a conversation that rarely ends with anyone changing their mind. The disagreement was a matter of record before either version was built, and the answer came back in two weeks.

The nine variants moved the numbers less than removing one character from a menu did. That pattern held for three years: the size of the result tracks the size of the change to the user, not the size of the change to the design. Most of those homepages were a big change to the page and no change at all to the person looking at it — different layout, same job, type your address. That's restyling, and restyling moves nothing. The stripped one took away the proof, which is a real change, and it swung hard in the wrong direction. That's the other half of the rule: a real change gets you a real result, and a real result is just as likely to be bad.

So the useful question isn't "is this a big project." It's "does this change anything for the person on the other side." Those two come apart constantly, and a design team's default assumption is that they don't.
Waiting On Statistical Significance
Challenge Description
Conversion rose 23% against the pre-team baseline, from the team's founding to my move off it. That's the number, and it's the one the 20% was actually describing. Repeat order rate rose around 5%. Cart size moved — the dollar sign alone was 7–10% — but I don't have a full-period figure I'd stand behind, so I'm not going to give you one.

Three metrics did not move 20%. One did.

None of it is only the team's doing. The company was growing fast, the brand relaunched in the same window, and mobile ordering went from novelty to default. What the team can claim is the direction and the discipline: every change shipped because a version of it beat another version, called by someone with no stake in the outcome.
What I'd Do Differently
Challenge Description
We never made a real swing at the order flow.

Enter your address, see restaurants, add items, review the order, check the details, buy. Six steps. We'd just come off a redesign of it and conversion through it was good, so we left it alone and worked everywhere else. Defensible at the time. I still think it was wrong.

By the rule the team spent three years proving, that flow was the highest-value target on the site — six steps a user has to complete is the biggest change-to-the-user available anywhere in the product. Everything else we touched was a page they looked at. This was the thing they had to do. And we had the safest possible conditions: traffic, a holdout, an analyst calling significance, and nobody's permission to ask for. A strong conversion rate isn't a reason to skip the experiment. It's the cushion that lets you run it.

Optimization teams drift toward local maxima because local maxima are where the wins are legible. Somebody has to keep asking whether it's the right hill. That was my job, and on this one I didn't ask.