Article URL: https://eardatasci.github.io/c/ambiance/index.html Comments URL: https://news.ycombinator.com/item?id=48921077 Points: 57 # Comments: 38

I've been thinking about how to free LLMs from the chat pane for a few years now. Watching what people have tried and seeing what worked (and didn't), I've formed my own opinions on the right way to approach it. Here are my thoughts, and what I've been working on recently. With the advancing intelligence of LLMs, harnesses will eventually be reliable. The real matter at hand is reducing how much cognitive load (measured in tokens) you are putting on your bot. A good harness MUST make use of the a priori coding knowledge of the LLM. Coding and systems administration are heavily overrepresented in the LLM's training data, so give it an environment it is already comfortable in; wrangling it through a novel one ultimately wastes tokens. Similarly, precious context should not be wasted on things like file discovery, traversal, etc. -- good harnesses make delegation easy and efficient. At the same time, a harness should feel light to the LLM, but actually do a lot of things in the background. These include logging, sanity checks, failsafes, sanitizations, etc. Everything is vulnerable, and all agents eventually fail. There are two types of Agent failures: Failures at the LLM level cannot be directly patched, but the risk of such a failure can be mitigated by the harness. Harness-level failures can be recovered from, and should be fixable at runtime due to the turn-based nature of the LLM. In order to fix a defect, the Agent needs two things: great logging, and a clear error message. Most of these requirements are age-old questions; only the verbiage has shifted from Users to Agents. So it's worth asking: what can we learn from the before-fore times, when people used to actually write code? I am, by no means, an expert on Unix or any of its descendants, but I have spent the last decade learning about its history, design choices, and usage. A lot of it maps beautifully onto our predicament; plenty of it doesn't. Think of U/L as a motivating analogy rather than a direct comparison.