A friend brought me this scenario.
The Situation
A software company is testing an AI model that conducts job interviews in a paid pilot with an enterprise client. Candidates can start whenever they're ready, and a researcher should watch every session that tests a new combination of industry, role, and seniority to judge the model's performance. At the same time, a developer can adjust the model mid-session. The sponsor, who funds the pilot, sometimes joins from the team room to observe. It’s not uncommon for six candidates to be ready to start the AI interview at pretty much the same time in the same afternoon, and the research co-ordinator has to work out who watches what as sessions go live.
The Constraints
The team is trying to stick to the rules they've set for themselves.
- Only new combinations need live observation, as every session is recorded and repeats can be viewed later.
- Each individual must be prepared to attend one live session at a time - no session switching.
- Researchers and developers prefer to work in pairs, as one person can spot the problem, and the other can fix it.
| Constraint | Type | Source | Cost of violation |
|---|---|---|---|
| Each person observes only one live session at a time | Hard | Physics | A session goes unwatched, or an observer leaves one session for another |
| Only new combinations need live observation; repeats run unwatched and are reviewed from the recording | Hard | Policy | Observer time goes to scenarios already seen while new ones are missed |
| A session starts when the candidate says ready | Hard | Policy | Candidates wait, and the pilot breaks its promise of on-demand interviews |
| Researchers and developers observe in pairs | Soft | Policy (team preference) | A session is evaluated with no one to correct the model, or corrected with no one to evaluate it |
| The sponsor is most interested in executive-level sessions, and they get priority whenever the sponsor watches | Hidden | Culture | The sponsor loses confidence in the pilot, and the client relationship suffers |
| Candidate notice, consent to recording, disclosure of observers, and rules on automated hiring decisions | Hard | Law and regulation (context) | Legal exposure and candidate complaints |
The rule that actually dictates how this afternoon runs is one that nobody has written down: the sponsor cares most about the executive interviews, so the team will always prioritize those if the sponsor is available to watch. Alongside those logistics are additional rules about candidate consent, covering the AI tools, the recording notice, and disclosing who is watching from the other side.
Where It Collides
When the six candidates say “ready”, the coordinator looks at the schedules of the seven people on call, which at first glance looks like enough. Two candidates are being interviewed for combinations the team has already seen the AI model handle, so those can run as standalone sessions and be reviewed later from the recording. The other four are new (two at the exec level and two at the mid level), and the researchers and devs work better together in pairs. With three pairs available for four new sessions, at least one session will start with someone missing.
Of course, the team can keep the pairs together and leave one of the new mid-level scenarios unwatched, or they can split Researcher 3 and Developer C so every new scenario has someone on it, with two sessions running on half the support. The real rub, though, is that the sponsor will only see one of the two executive sessions.
| Session | Scenario | Needs observation | Sponsor priority | Option A: pairs stay together | Option B: one pair splits |
|---|---|---|---|---|---|
| 1 | New executive scenario | Yes | Highest | Researcher 1 and Developer A, with the sponsor | Researcher 1 and Developer A, with the sponsor |
| 2 | New executive scenario | Yes | Highest | Researcher 2 and Developer B | Researcher 2 and Developer B |
| 3 | New mid-level scenario | Yes | Normal | Researcher 3 and Developer C | Researcher 3 alone, with no developer to adjust the model |
| 4 | New mid-level scenario | Yes | Normal | No one | Developer C alone, with no researcher to judge the model |
| 5 | Repeat scenario | No | None | No one, reviewed from the recording | No one, reviewed from the recording |
| 6 | Repeat scenario | No | None | No one, reviewed from the recording | No one, reviewed from the recording |
How a Solver Thinks About It
Scheduling software handles what that coordinator spent those first few minutes scrambling to do, long before anyone needs to rush. The moment a fourth new candidate confirms readiness, the system spots the missing Developer and Research pairs. It flags the shortage immediately, giving the coordinator time to reassign staff before a session begins unobserved.
To handle this decision, the team first needs to agree on which sessions have the highest priority. It’s similar to a restaurant host with four tables and three servers: do you give three tables full service and leave one to wait, or stretch one server between two tables so everyone is served, but less attentively? The host decides based on which tables matter most that night - and the coordinator faces the same trade-off. Once priorities are clear, the Software can lay out the trade-offs: showing the cost of keeping pairs together versus splitting them up, so the coordinator can make an informed choice.
What the Software Cannot Decide
Suppose the coordinator keeps the pairs together, meaning the second mid-level session goes unwatched. Is that a problem? It depends on how often candidates with the same industry, role, and level show up. If those combinations are common, another similar session will come along soon, and not observing this one doesn’t matter much - the team can catch the next. But if those combinations are rare, skipping this session could mean missing a valuable opportunity, and splitting the pair today would have made more sense. The Software can’t make that call: it doesn’t know how often these scenarios come up or how patient the client is willing to be.
The sponsor's interest in the executive sessions raises a question: should we write that down at all? It currently lives in the coordinator's head, and the coordinator respects it when the sponsor is present. If we put it into the scheduling rules, it would be consistent, and it would put it in writing for the researchers, the developers, and the client to see that mid-level sessions get less of a go whenever the sponsor is watching. The team will have to decide whether they are comfortable with that.
Then there's the candidate in the session that nobody watched at all. For that candidate, it was a genuine interview, and the team needs to be confident the AI model treated that candidate the same way it would have if a researcher and a developer had been watching.
The Question for the Reader
Most teams that do live work will have at least one priority that exists purely in someone's head (the sponsor's interest in executive roles here being a prime example). When you have more sessions coming up than your team can cover, who decides which ones get the pass and which don't, and is that prioritization documented somewhere your coordinator can see?
The Constraint Ledger arrives weekly with one real scheduling collision, worked through to the decisions that stay with people. Subscribe on Substack.
The Constraint Ledger is a product of the team at Calgebra (calgebra.com), which builds scheduling optimization solutions.