Scheduled Data Delivery, Part 3: A Destination That Holds Secrets
Part three of four. Adding a destination where the platform stores a customer's credentials and connects outward to a server it does not control: write-only encrypted secrets with a mask sentinel, trust-on-first-use host key pinning, stage-typed connection failures, and a server-enforced rule that a profile cannot be enabled until a test has passed against the exact target that is stored.
The destinations up to this point were comfortable. Writing to our own storage fails in ways we control, and posting to a customer's HTTP endpoint fails in ways that are well-understood and stateless. Delivering onto a server the customer operates is a different proposition: the platform now stores their credentials, initiates outbound connections to an address they supply, and depends on the state of a machine nobody on the team can log into.
Three problems fall out of that, and each of them shaped a design decision. The platform holds a secret it must never leak. It has to decide whether a remote host is the one it expects. And when a connection does not work, it has to tell somebody something useful about a machine it cannot inspect.
Write-Only Credentials
Passwords are encrypted at rest with a key held outside the database, which is the unremarkable part. The decision worth explaining is that there is no way to read one back. Not a restricted endpoint, not an audited reveal flow for administrators: the capability does not exist in the API surface at all. Reads always return a fixed mask, and the only supported operation is overwriting the value.
That makes the mask do real work. When the client submits a form containing the mask, the server interprets it as an explicit instruction to keep the stored secret unchanged, which is what lets somebody edit a remote path without retyping a password nobody has written down. The sentinel is rejected in the one context where it would be meaningless: an ad-hoc connection test for a profile that does not exist yet has nothing stored to keep, so submitting the mask there is an error rather than a silent no-op.
Trusting the Host, and Letting It Change
Accepting whatever host key a server presents means a machine that answers on the right address can receive customer data. Requiring the key up front means every onboarding starts with an email asking somebody to read out a fingerprint. The compromise is trust-on-first-use: the key is pinned during the first successful test connection and verified on every connection afterwards.
The refinement that makes it usable in practice is that the pin is scoped to the target it was recorded against. If the host or port is edited, the stored pin no longer applies and the next connection pins afresh. A customer migrating to a new server is doing something legitimate, and the system should not require an engineer to clear state by hand before they can complete it. A changed key on the same address, which is the case that should worry you, still fails loudly.
Failure Needs a Stage
The single highest-value decision in this slice was refusing to let connection failures collapse into a boolean. A connection attempt passes through four distinct stages, and which one it died in tells the customer's operations team exactly whose problem it is: reaching the host at all is a firewall or address question, the key exchange is a fingerprint question, authentication is a credentials question, and touching the remote directory is a permissions question.
The result surfaces the stage alongside a sanitised message, and that turns a support thread that used to begin with it does not work into one that begins with we are being refused at authentication. Almost every ticket this feature could generate is answered by knowing which of those four things happened.
The Enable Gate
The failure mode nobody wants is a profile that has been scheduled, looks active in the console, and fails silently every night because the password was mistyped at three in the afternoon six weeks ago. The rule that prevents it is that a profile with this destination cannot be enabled until a connection test has passed against the target currently stored on it.
Two details make that rule hold up. First, it is enforced by the server, not the interface: the enable operation itself checks for the stamp and refuses without it, so there is no path through any client that can produce an enabled but untested profile. Second, the stamp is bound to the target it was earned against. Editing the host, port, username or remote path clears it, because a test that passed against the old address proves nothing about the new one.
An ad-hoc test from the creation wizard is therefore advisory. It genuinely connects and gives immediate feedback, but it cannot stamp anything, because there is no row yet to stamp. That leads to a small piece of flow design that took a couple of attempts to get right: the obvious move is to block the save button until a test passes, and it deadlocks, because you cannot obtain a stampable test without first saving to get an identifier. The flow that works is save, land disabled with a status reason explaining exactly what is missing, test, then enable. The state the user cannot leave is the one that tells them how to leave it.
The Toggle That Was Never There
The bug I enjoyed most was not in the new code. Enabling and disabling a profile had exactly one entry point in the entire interface, a switch on the profile card, and that switch was wrapped in a condition that rendered it only for the other destination type. Profiles of this family had therefore never been enableable from the console.
Nobody had noticed because nobody could. Profiles of that family had always been created in an active state, so the switch was never needed. This feature introduced the first one that is deliberately created disabled, which meant it would have been permanently stuck: disabled, unenableable, and blocked from the manual run and replay paths that also require an enabled profile. A dormant defect became a blocking one the moment a new state became reachable.
The status checklist for the feature had this capability marked as complete, with the card switch cited as the evidence. It was written by someone who had reasonably assumed that a control which exists is a control that renders. The habit I took from it is to treat the completed rows of a status document as claims to verify rather than as facts, particularly the ones that were true for the configuration that existed when they were written.
One Quirk Worth Knowing
Many managed file-transfer products present a chrooted, tightly restricted environment, and reject the attribute-setting call that client libraries routinely make after a transfer completes. The upload has fully succeeded at that point, so the default behaviour is to report a perfectly good transfer as failed. Disabling that call is a one-line configuration change that matters much more than it sounds like it should, because the servers that need it are exactly the commercial products enterprise customers are most likely to be running.
Key Technical Decisions
No read path for secrets, at all. Not a restricted one. A capability that does not exist cannot be misconfigured, audited incorrectly, or abused, and the cost of losing it is one retyped password.
The mask is a meaningful value. Returned on read and interpreted on write as keep what is stored, so unrelated edits do not require re-entering a credential. It is rejected only where there is nothing to keep.
Trust on first use, scoped to the target. Pin the host key on the first successful test and verify it thereafter, but let a changed address re-pin. Legitimate server migrations should not need an engineer.
Failures carry a stage. Reachability, host key, authentication and path are four different people's problems. Naming which one occurred is the difference between a two-minute ticket and a two-day one.
The server owns the state transition. Enablement is refused unless a passing test is stamped against the stored target, and editing that target clears the stamp. The interface reflects the rule rather than implementing it.
Never gate an action on something it must exist to obtain. Blocking save until a test passes deadlocks a create flow whose test needs a saved identifier. Land in an explicit blocked state that names its own exit instead.
