Beyond the demo

You lead a team of eight in the claims department of an insurance broker. Over the weekend, Hugo, one of your team, used an AI agent to build a working app. Monday's demo is impressive. "It's 90% done," he says. "We could launch next week."

A demo shows the happy path: the right user, the right input, a good connection, on the builder's laptop. A real product has to handle everything else, and that's where most of the work is:

• Odd cases and errors: two people booking the same thing, a file that's too big, a connection that drops.
• Security and access rules: who can see what, and how the tool knows who's who.
• Personal data: what's kept, for how long, and who can reach it.
• Tests on real devices and with real users, not only on Hugo's laptop.
• Running it: hosting, backups, support, updates, and someone who is responsible for it.

Programmers have long repeated a joke attributed to Tom Cargill, of Bell Labs: the first 90% of the work takes 90% of the time, and the last 10% takes the other 90%. A weekend demo is often that first 90%. And "the agent can do the rest in another weekend" is rarely true, because the rest is decisions, checks and tests with real people.

How much is left depends on who uses the tool and what's at stake. So size it for the case.

• An app where the 40 people in your office book meeting rooms: internal and low-stakes, since the worst case is a double-booked room. What's left is logging in with company accounts, a week of tests with a few colleagues, and someone who looks after it. That's days, not months. Agree to a pilot next week, then open it to everyone a week or two later.
• A portal where clients log in, upload the documents for their claims and pay their fees: here the demo is a small part. It needs secure logins, rules so that each client only ever sees their own claims, safe payments, a security review, GDPR checks and support. That's weeks to months, with professionals. Involve whoever handles security and the GDPR before any client data goes in.
• A mobile app where clients follow their claims: two platforms, developer accounts with Apple and Google, their review of every version, a real connection to the claims system, and updates whenever iOS or Android changes. That's weeks to months, then a yearly cost. First ask whether a page that works well on phones would give clients the same thing.

Don't simply say no, and don't throw the demo away. Thank Hugo, and keep his prototype: a working prototype is the clearest brief you can give a developer, and it saves weeks of explaining. Then name who looks after the tool once it's live, because "Hugo's weekend project" can't be a company service.

The same applies to your portfolio. The agent may build its first version in an afternoon. What's left is small, because the stakes are low: your real texts and photos, a check on a phone, a form that really reaches your inbox, and a domain name. That's days, not months.

You're a freelance project manager for a small travel agency. The marketing manager wants to announce a date in next month's newsletter, so you ask the developer how long the next piece of work will take.

An estimate is a guess. How good a guess depends on how familiar the work is. For work the developer has done many times, it's a good guess. For anything new, unclear, or dependent on another company's system, the real time can be several times the estimate. The parts that nobody understands yet are the unknowns.

• "Two days to add a contact form. I've built dozens just like it." This estimate is reliable. Give a date about a week away, which leaves room for testing and the usual small surprises.
• "About two weeks to connect the site to our partner's booking system, if their documentation is good. I haven't seen it yet." It could take two weeks or two months. Don't announce a date yet. Ask the developer to spend two or three days reading the documentation and trying one connection. This short investigation clears the biggest unknown. Then ask for a range.
• "An afternoon. The AI agent can translate the whole site into English." The translation itself is quick. What isn't quick: a fluent speaker who checks it, the legal pages, the emails and error messages, text inside images, and every future change, now made in two languages. Give a date a week or two away, and tell the marketing manager that the English site is also a small ongoing cost.

Don't give the developer's estimate to marketing as the date. The developer gave you a guess for the work going well. A date in a newsletter is a promise to customers, so it has to allow for the work not going well.

Ask two questions. What's the range: the best case, the likely case and the worst case? And what does it depend on? Then match the date you announce to the risk. For familiar work, add a small margin. For work with a big unknown, clear the unknown first. Or announce without a precise date, such as "this spring" rather than "on 3 April". If marketing can't wait, give the pessimistic end of the range.

Once a date is public, protect it by cutting what's included, not by moving it. For example, launch the booking connection for one destination first, and add the others later. The writer Douglas Hofstadter described the problem in a law that carries his name: it always takes longer than you expect, even when you take Hofstadter's law into account.

You've just been hired as a digital manager. The hosting company calls with an offer. For €800 a month, it offers monitoring day and night, an engineer on call at night, and a promise of 99.99% availability instead of the current 99.9%. "Most of our serious clients take it," says the salesperson.

Decide by comparing two figures: what an hour of failure would cost you, and what the protection costs. €800 a month is €9,600 a year. Translate the promise too. 99.9% availability still allows about 9 hours offline a year, and 99.99% about one hour. So the offer buys, at most, about eight fewer hours of downtime a year. Ask what the provider pays if it breaks its promise: usually a small credit, not your lost sales.

• A small history museum, where tickets are sold at the door: an hour offline, or even a day, costs almost nothing, because visitors find the opening hours on a map app or by phone. Decline.
• An online shop selling €3 million a year, a third of it in the six weeks before Christmas: €1 million in six weeks is about €1,000 an hour, and more in the evening. One hour down in December can cost more than a month of the offer. Take it, at least for the season. Have the site tested under heavy traffic before November, and stop all changes in December.
• A festival that sells 30,000 tickets on one morning: the risk is in the first hour of the sale, when traffic is many times its usual level. An engineer on call at night won't stop the site from collapsing at 10 a.m. Sell through a ticketing platform built for rushes, with a virtual queue, or test the site under that load beforehand.

The salesperson sizes the offer to their own revenue, not to your risk. Start with the cheap measures every site needs: an alert when the site goes down, sent to a shared address, and a recent backup.

Ask these questions when someone shows you a demo, gives you an estimate, or sells you reliability.

After a demo, ask what it leaves out: odd cases and errors, access rules, personal data, payments, tests on real phones with real users, hosting, backups and support. Ask who uses the tool, because that sets what's at stake, and who will look after it once it's live. Then decide on a realistic timeframe: a pilot soon, or an estimate in weeks or months.

For an estimate, ask for the range and what it depends on. Ask whether the developer has done this before, and what nobody has looked at yet. Clear the biggest unknown before you announce anything.

For reliability, ask what an hour offline would cost you, and when: every day, in a busy season, or on one morning. Ask how many hours offline the provider's promise allows, and what it pays when it fails. Then buy the protection that matches the risk.