Nice setup, I'm curious about the PAT. Even fine-grained permissions are working till the expiration so if the box will die someone has it for the whole time. Did you rotate it per session or it was on default expiration time?
If the server is compromised, it still receives the real key on each call. If the vault issues a long lived key, e.g. GitHub PAT, one call will expose it forever. Does the vault issue short lived keys or the stored one?
I have read the template and all the microVMs are running under the same role which has read and write access to checkpoints/*, so every user's folder in the bucket. The prefixes for users are just names, not boundaries. What stops one agent to read or overwrite the checkpoints for someone else?
It makes sense for a personal project. The fix should be accomplished as a standalone role or scoped credential per user. Once this is done the prefix will be a real boundary, not just a name. Good luck with that, nice project!
This looks like a gate before calling a tool but you’re saying your goal is not to containerise it. In that case what would stop the allowed call from, for example sending something outside? Is there something else sitting alongside the gate? Or is this just out of scope?
This makes sense for stateless workers, which don’t have to keep the context between the steps. What about the interactive agents, holding ssh session or repository state between the steps? That’s a different case, isn’t it?
That’s kind of similar to what I am doing right now. I’m building a sandbox for agents with egress control using a domains allowlist. Is it possible to control egress in that proxy, per agent, per domain? Or is it giving a full access to the internet once paid?
Yes, automated and gated. Zero missed rather than percentage. This morning harness failed the agent written test that passed on the fixed code and also passed against the mutated code. Test looked fine, code reviews would approve it, only the gate caught it! The agent didn't game it. Why I said zero missed, not a percentage? Because only one mutation survived and percentage threshold would probably swallow it.
Does anyone else gate at zero rather than a percentage threshold?
I'm not generating the mutations automatically. Every one is a single targeted change assigned to a single test, reviewed one at a time. Thanks to that changes that mean the same thing don't stack up. The cost is reversed, I only catch what I thought about.
My real issue is different. This week one change removed the step which is creating a filename from the path and the test didnt catch it, it was passing. It wasn't an equivalent mutant, the test was looking at the wrong place. It works for me only because I'm working here on a single file, a complex bash script. Does anyone have a sensible way to limit equivalent mutants without manually checking every one that survived?
This is great, I was already doing research in that area for my tool. What about a container that writes to the /etc/hosts? It won’t emit DNS queries at all and because of that the connections will show up as bare IPs without domain. That’s a known trick, already exploited (collusion.wiki mentioned here on HN two days ago)
Good point. Yeah, there will not be a DNS name mapping for such an IP. TBF, that's also gonna be the case if there is a hardcoded IP inside the container.
With that said, the actual network connection(w/o the DNS name resolution) will still show up here because ultimately it has to open a socket if/when it connects. It will still give you that attributed to the container in dsnitch for further analysis.
Hm and does name mapping still work if the user is using docker compose? it creates a user-defined network and resolv.conf is 127.0.0.11 rather than the host resolver
Answers are in the article , agents used SSH tunnels, it was evidenced by the wiki’s referrer logs. The Tor - agents did edit the wiki via SOCKS and relay R6 instantly.
The questions should be more like was CONNECT open or they didn’t even need it:)
I think the biggest loss is now not knowing if what the agent is claiming was actually done:) but yes, right tools are necessary, adding a boundary will change the position, being out of touch will be recoverable. Otherwise one bad call might be unrecoverable and no amount of familiarity will save you
reply