Article URL: https://earendil.com/posts/session-portability/ Comments URL: https://news.ycombinator.com/item?id=49118781 Points: 57 # Comments: 5

The original promise of an inference API was wonderfully simple: send some input, receive some output. If you kept both, you had the conversation. You could inspect it, archive it, replay it, or give it to a different model. That abstraction was never completely true. Prompt caches live on somebody else's GPUs, tokenization differs between models, and sampling is not reproducible (and quite intentionally so). But the semantic record of a session in the form of a transcript could still belong to the user. A transcript should contain the instructions, messages, tool calls and tool results. Another sufficiently capable model might not continue identically, but it could understand what happened and take over. Inference APIs are frustratingly moving away from that property, at least somewhat. They increasingly return a mixture of text and provider-bound state that is very intentionally non-portable. Each feature comes with a basic justification that's trivial for a provider to come up with, along with good arguments for why this is good for the user. Together all of these things change the ownership reality of an AI session: the transcript on your machine is no longer your session but a partial view of a session whose operational state belongs to an inference provider and not you. We are not fans of this direction, and we want to talk a bit about what it means to you, as a user, and what it means to us, as people developing tools in this space. By a portable session we do not mean that switching from one model to another must produce the same next token. That's a given because models have different capabilities, trained personalities, context windows, and ways of working with tools. And well, it's all quite nondeterministic anyway. Portability means something more modest: The archive should contain enough intelligible information for another model to continue the work. It should not require the old provider to dereference an ID, decrypt a blob, remember a search result, or reconstruct a summary. A response ID is not a transcript, a ciphertext the user cannot decrypt is not user-controlled state, a list of citations is not the evidence that was placed in the model's context by a search result.