So each prebuilt environment is basically a container that is exclusively accessed by your test runner. We set up the environment in the container and code and tests are run there. So there is no sharing of the build environment between customers.
We are pretty paranoid about protecting people's code (as you might expect). We do all the "normal things", encrypt connections, etc. One novelish idea we use is that different customers' code never shares a binary. So, what that means is that the CLI connects through a load balancer directly to it's dedicated server (a binary running in it's own container) which connects to it's dedicated workers (more binaries in their own containers). Once we are finished with a server or worker we destroy the entire container. We have multiple layers of security, and for example once a customer has connected to a container, only a customer with the correct key can connect again (so it should be the same customer). We also offer on-prem for people who need more security wise.
I feel we are a long way from getting the pricing right. At the moment most people are on the free tier just trying things out and getting started. I think as we grow we'll probably fill the space between the two tiers as you suggest. But to be honest, the details of exactly what this will cost us at scale is still very much open. Usage rates and test suite size are very variable and until we have a good idea on that we are just trying to make the best guesses we can and try not to shoot ourselves in the foot too much :)
Just saw the question about testing splitting - yes there is automation magic. You give us a command to list your tests and we split them as optimally as we can.
These are really good answers. I couldn't find any of this info in the docs on your site though!
And BTW, it's been a while since I last checked the numbers, plus we've moved to GitHub Actions recently. Our current setup time is almost 5 minutes for each parallel runner. Saving that time by using prebuilt runner images starts to sound like a great idea.
I'm curious, do you keep an image for each customer? And you must have some heuristic for determining when their setup environment changes and requires rebuilding?
Yes, I'll need to update the site. I'm always torn between putting too much info there or not enough, but I guess I need more.
We don't keep an image per customer. But we do have general images, so something like ruby-2.8-node-14 that contain a lot of the setup. Then building is just running the customers build commands, "yarn install" etc.
The rebuilding heuristic is a list of relevant files. We hash the contents and if the hash doesn't match we know we need to rebuild.
I sympathize with the tension. These are the big questions I still had after reading your site docs. Thanks for taking the time to answer them!
We use RSpec, Capybara, Selenium, Chrome Driver. We parallelize our tests semi-manually, which is a minor inconvenience.
Recently moved to GitHub Actions, which makes it easy to cache dependencies (mostly gems), although it's not quite a pre-built image. Startup cost with fully warmed cache is about 55 seconds, almost all of which appears to be spinning up the container. Not sure what the underlying tech is, but that seems slow.
So each prebuilt environment is basically a container that is exclusively accessed by your test runner. We set up the environment in the container and code and tests are run there. So there is no sharing of the build environment between customers.
We are pretty paranoid about protecting people's code (as you might expect). We do all the "normal things", encrypt connections, etc. One novelish idea we use is that different customers' code never shares a binary. So, what that means is that the CLI connects through a load balancer directly to it's dedicated server (a binary running in it's own container) which connects to it's dedicated workers (more binaries in their own containers). Once we are finished with a server or worker we destroy the entire container. We have multiple layers of security, and for example once a customer has connected to a container, only a customer with the correct key can connect again (so it should be the same customer). We also offer on-prem for people who need more security wise.
I feel we are a long way from getting the pricing right. At the moment most people are on the free tier just trying things out and getting started. I think as we grow we'll probably fill the space between the two tiers as you suggest. But to be honest, the details of exactly what this will cost us at scale is still very much open. Usage rates and test suite size are very variable and until we have a good idea on that we are just trying to make the best guesses we can and try not to shoot ourselves in the foot too much :)