Shadow systems: the process that actually runs

Every client has two processes. One is written down: the SOP, the flowchart, the field list inside the system. The other is running: a spreadsheet on someone's desktop, a group chat used as a task queue, a paper ledger kept because the system cannot hold the exception. The formal system records what should happen. The shadow system records what does.

This is not fraud and it is not laziness. It is accumulated repair. Each time an exception appeared, someone patched it in the cheapest available place, and the cheapest place was never the system whose change process is measured in months. After a few years the repairs are the process, and the document describes a company that no longer exists.

Description drift

The gap between the documented process and the executing one is one of the inputs that cannot be delivered into an interface, and it is the one that hides best. Nobody lies about it. The people describing the process genuinely believe they are describing it, because they are describing the version they were taught.

There are two ways to see the gap. The first is to watch a real person through a real day, sitting beside them rather than interviewing them, and to watch the workarounds rather than the flow — the places where they sigh, open a second window, or suddenly message someone. The second is to refuse second-hand accounts. A pain point that has passed through IT, procurement or a consultant has been translated at least once, and each translation loses the specific.

Three states of data, and the deceptive middle

Structured data is regular: tables, messages, fields, protocols, readable by someone outside the company. Unstructured data is handwritten notes, private spreadsheets, dialect, muscle memory — readable only by the handful of people involved, who do not consider it knowledge at all.

Pseudo-structured data is the middle case and the expensive one. It has the form of structured data — tables, fields, even a flowchart — but it describes the SOP version, while the executing version is something else. The form passes every check you would run on the content.

That is the answer to the question people ask about every large organisation: why does a company with a system still run on Excel? Because the system records how things should be and the spreadsheet records how they are. The company is not failing to adopt software. It is running two databases, and only one of them is maintained.

Why the middle case fails silently

Unstructured data announces itself. Nobody mistakes a paper ledger for an API, so the work of making it explicit gets planned, and it gets priced.

Pseudo-structured data does not announce itself. The columns are complete, the schema is clean, the numbers are plausible, and any pipeline built on it will run without error and produce fluent output about a process that does not exist. That is the failure worth fearing in this work: not a crash, but a confident report on the wrong workflow.

The signature is easy to test for and easy to miss. A column that is always filled but never read. A field that always carries the same value. A status that is only ever updated by one person, by hand. Each of those marks a table that exists to satisfy the form rather than to run the work.

Extracting the executing process

Four methods do most of the work, and they can run in parallel across two weeks.

Record and transcribe a post, then extract the decisions rather than the tasks — what each decision depended on, and what the person looked at before making it. Trace one exception event backwards: take something that went wrong last month and reconstruct who touched it, in what order, through which tools. Walk the process against its own document and mark every place the walk diverges. Read the handover log, if one exists, because a departing person's notes are often the only written record of the executing version.

Two questions from the seven questions feed the same inventory. Asking which table is most unpopular, who fills it in and who reads it produces the candidate list; asking where the knowledge lives produces the answer to whether the shadow system is a spreadsheet, a chat thread, or a person.

Then compare versions. Ask for the last three versions of the same table and look at which columns are always blank and which are always written by hand. The blank columns are the parts of the form nobody uses; the handwritten ones are the parts the system cannot hold. Both are the executing process showing through.

This is also what makes the who-loses map countable. The hours that matter are the hours being spent inside these tables, not the ones written into a job description.

What to do once you have it

The wrong move is to make the shadow system the project. Asking a client to migrate to the official process before anything works is asking them to lose their repairs before they have replacements.

The principle that holds is asymmetric compatibility. Be deeply compatible with the old system at the data layer, because data will not move to accommodate an architecture. Do not become a parasite of the old environment at the architecture layer, because during validation more than half of the design will be overturned, and the old change process is measured in months. And at the process layer, obey people's habits rather than the system's habits.

In practice that means reading from the spreadsheet instead of demanding its retirement. Instrument where the work already is, and let the migration happen because the new path is easier, not because the old one was switched off.

The test

Any number maintained in two places is a shadow system. Find the copy that gets corrected first — that is the executing one, and every metric definition has to point at it. A definition that points at the other copy produces a project spent defending a number nobody on site recognises.