Farm Goes Open Source: Preview Environments Without the Pain

You don’t just want every pull request to pass the tests — you want to show it live: to the designer, the manager, the neighboring team, so all of them can poke around the UI — and be delighted, or find bugs — and be upset. Sounds easy enough: spin up an environment and share a link. But when pull requests come in by the dozens every day and there are lots of teams, “just spinning up an environment” turns into a full-blown infrastructure project.

Hi! My name is Mikhail Golbakh, and I’m a Senior Frontend Developer at Yandex Cloud. For several years now, my team and I have been building Farm — a service that spins up a preview environment for every pull request. In this article, I’ll walk you through how Farm grew from a simple self-hosted script on a single virtual machine into an orchestrator running on Docker and Kubernetes — and why we ended up open-sourcing it.

How We Arrived at the First Version of Farm

Yandex has many frontend teams that create hundreds of pull requests a day. Each pull request triggers a multitude of tests and other CI checks. Unit tests, type checks, and linters run just fine on isolated virtual machines.

But what do you do when you need to bring up a full application from a branch? This question leads to a task frontend developers face all the time: spinning up a “beta” for every pull request. The first thing that comes to mind is to set up a dedicated dev environment mirroring production and share it between developers. That’s what small teams often do early on, but they quickly run into a scaling problem. A single environment can no longer be shared, and you can’t really run e2e tests on it either — their stability would clearly raise questions.

Traditionally, every team sets up its own process for deploying dev environments, spending a fair amount of effort on it and resources on maintenance. We were no exception and came up with a solution of our own. However, we designed it from the start to be a shared technology that could scale to dozens of teams despite possible differences in infrastructure.

Back in 2018, the Yandex Cloud frontend team built the first version of Farm — essentially a simple self-hosted service that downloaded code from a repository branch, ran the build commands for the target application, and started its process on a VM, routing traffic from a dedicated domain. Even the command set itself was hardcoded: Farm was aimed primarily at our own projects, which are built on shared components for Node.js web services. What we got was a kind of simplified deploy system with a Yandex flavor.

Over the years, the Farm service evolved: it learned to work with separate build and start configurations for each project, gained a full-fledged UI, a database, and a lot of extra code simplifying work with Yandex’s infrastructure. But the essence of the service didn’t change — it still solved the following tasks:

• downloading code and building the project from a pull request;

• routing traffic to the running application;

• unifying the solution in a single infrastructure component that’s easy to distribute and embed into Yandex’s CI processes.

How It Works

Farm is a small Node.js application, and its simplified workflow looks like this:

1. Farm receives a generation request. It can come from a CI task or directly from a user via a form in the UI.

2. A new application instance lands in the database with the queued status, joining the build queue.

3. Farm works through the queue, downloads the code from the target repository and branch, reads the configuration, and starts the build.

4. The application process starts: it’s expected to open a socket at <instance_dir>/dist/server.sock.

5. Farm — or rather nginx with a special config at this point — starts routing traffic to a unique domain, for example: d008838956420e285cee652262dd5d18b2ed2a2a.farm.example.com.

As you can see from the diagram, we didn’t make Farm responsible for load balancing — we used nginx. This approach is here to stay: maintaining traffic proxying is a non-trivial task we’d rather not solve ourselves. There was also support for Git and Arc — Yandex’s own internal version control system.

The build and start themselves were a straightforward matter of Farm running commands. Everything was as blunt and direct as it gets:

# Build
npm ci
npm run build

# Start
npm run start

For the full picture, let’s also look at an example farm.json config:

{
  "preview-generator": {
    "build": ["npm ci", "npm run build"],
    "start": {
      "command": "npm",
      "args": "run start"
    },
    "env": {
      "APP_BUILDER_CDN": "false",
      "IS_FARM": "true"
    },
    "instances": [
      {
        "name": "preprod",
        "urlTemplate": "https://{hash}.farm.example.com",
        "env": {
          "APP_INSTALLATION": "russia",
          "APP_ENV": "preprod"
        }
      },
      {
        "name": "prod",
        "urlTemplate": "https://{hash}.farm.example.com",
        "env": {
          "APP_INSTALLATION": "russia",
          "APP_ENV": "prod"
        }
      }
    ]
  }
}

What we ended up with was a real workhorse. All a target team had to do was prepare a virtual machine, describe the configuration of Farm itself and its projects, bring it up, and set up a CI process for pull requests. Sounds easy, right? But…

Growth and New Challenges

Farm lived in this state for quite a long time and did a decent job of solving its tasks. At first we even had a communal setup — one shared Farm for several frontend teams, so as not to waste effort maintaining multiple installations. But the teams grew in size, their number increased, and Farm could only work within a single VM. Over time, even the most powerful configurations stopped being enough.

Running several builds in parallel consumed all of the machine’s resources, and an instance could sit in the queue for quite a while. So an organic process began of splitting the communal Farm into smaller ones, sometimes even one per project.

Obviously, this splitting didn’t solve the problem — it merely postponed it. Compute resources kept running out anyway, and new teams had to spend extra effort supporting and maintaining Farm. For example, the VM’s disk would fill up quite regularly, forcing a manual cleanup of unneeded files. So the first problem is limited resources with no management or scaling. Let’s keep that in mind.

Another problem was that under the hood Farm used SQLite as its database without any abstractions like Knex. This created vendor lock and also forced us to wipe all data on every schema change, since there was no migration mechanism. Given that Farm was split into many per-team installations, it also meant that hardly anyone wanted to update it: who knows what might break — and you’d most likely have to wipe your data, too. As the saying goes, “if it works, don’t touch it.”

On top of that, Farm could only run Node.js applications, and all processes ran directly on the host without any isolation. And since there was no isolation, applications couldn’t simply use their usual port to receive traffic, because Farm didn’t do port management. That’s why unix sockets were used for communication, forcing applications to comply with this slightly odd contract.

So, we needed to solve the following problems:

1. Limited resources with no management or scaling.

2. No database migrations and a hard dependency on SQLite.

3. No instance isolation, plus the constraint of being a Node.js application publishing a unix socket.

4. The burden of support and operations — after all, we originally wanted to make teams' lives easier, not add headaches (this is an extra problem we’ll set aside for now, but we’ll come back to it closer to the end).

How We Tackled It

An experienced engineer looking at these tasks would most likely suggest moving Farm onto the rails of a k8s cluster. And that’s the idea we gradually arrived at: move application builds and deployment onto the cluster’s compute, into separate pods, and use its capabilities for automatic scaling and resource management. The applications themselves would be delivered as the familiar docker container — a very clear and simple contract for our industry. On top of that, we needed to keep the ability to run applications the old way, as processes, so that small teams wouldn’t have to go through a laborious migration to the k8s solution.

Simply dropping Farm and switching to deploying applications directly from CI to a k8s cluster was impossible, for the following reasons:

1. Farm had grown deep roots in all our existing processes and CI. Forcing teams to migrate to an entirely new solution made no sense.

2. If you dig into the details, Farm doesn’t just build and start the application — it also takes responsibility for routing (albeit via nginx), stops and removes unused instances, and does a lot more.

3. And the convenient UI matters a lot, too: it lets not only frontend but also backend engineers spin up an instance of an application to test various features.

In practice, adding k8s support turned out to be harder than we initially thought. From the earliest versions, the logic for working with application instances was scattered across the codebase, and over the years plenty of details and elegant crutches had accumulated inside it — squeezing another implementation in alongside was tricky.

A Major Refactoring

As a first step, we undertook a major refactoring of Farm’s code. First we needed to encapsulate the instance-handling logic in a dedicated provider class — a sort of backend for our Farm that would define the strategy for building and deploying an application. This would let us support the legacy scheme and k8s side by side for the time being, and add new implementations later through a single interface.

After taking the code apart, we managed to extract all the provider-specific methods and prepare an abstract class describing them. Simplifying a bit, it all came down to this:

class BaseFarmProvider {
    startup(): Promise<void>;

    buildInstance(
        generateData: GenerateInstanceData,
        observer: SubscriptionObserver<InstanceObservableEmitValue>,
    ): Promise<void>;

    stopBuilder(hash: string): Promise<void>;

    startInstance(instance: Instance): Promise<void>;

    stopInstance(hash: string): Promise<void>;

    restartInstance(instance: Instance): Promise<void>;

    deleteInstance(hash: string): Promise<void>;

    getInstanceStatus(instance: Instance): Promise<InstanceProviderStatus>;

    getInstances(): Promise<Array<InstanceProviderInfo>>;

    getInstanceLogs(params: {
        hash: string;
        stdout?: LogParams;
        stderr?: LogParams;
    }): Promise<{stdout?: string; stderr?: string}>;
}

Around the same time, we abstracted away all database access and moved it to Knex, so that in the future we could switch to a different, more powerful and persistent DBMS. As a bonus, we got a cheap migration mechanism that makes changing the database schema easier.

The New Kubernetes Setup

In the new setup, Farm is deployed in the cluster, works as a controller, connects to the k8s API, and starts managing cluster resources: launching pods for builds and for the application, creating and deleting other resources.

To understand it, it’s enough to cover the two main processes: building and deploying an application instance.

The build consists of several stages:

1. A user or CI starts the build (the buildInstance method).

2. The Farm configuration for the target application is fetched from the branch (the farm.json file in the repository root).

3. A builder pod starts: it downloads the code from the branch, builds a docker image (Dockerfile.farm by default), and pushes it to a predefined registry.

4. All the builder pod’s logs are streamed in real time and shown in the UI for debugging.

5. Once the build is finished, Farm deletes the builder pod.

The build’s output is an artifact — an application image pushed to the registry, ready to be deployed and started. As a bonus, we got build caching via Docker, which is very useful for speeding up CI.

Deployment starts right after the build, in the same buildInstance method, so let’s continue from the moment the image is pushed:

1. A Deployment is created in the cluster, using the image produced in the previous stage.

2. A Service is created with the port we specified in the Farm configuration, along with an Ingress with a domain based on the instance ID.

3. Then a new player enters the game — the Ingress NGINX Controller. It handles all the routing and makes the application reachable from the outside. Here Farm stays true to itself, delegating traffic proxying to another component of the system.

Technically, any other Ingress controller could be used for routing — up to some cloud ALB if needed. We picked the Ingress NGINX Controller for its simplicity and the speed of config updates.

That’s roughly how we get a working application instance, ready to be opened and tested. On top of that, Farm makes sure resources aren’t duplicated and all operations are idempotent — which is crucial for stable operation.

A Simpler Setup with Plain Docker

Once the idea of k8s support in Farm appeared, a rather obvious proposal followed: why not support running instances simply on a virtual machine in Docker? That would solve the instance isolation problem, lift the Node.js-application and port restrictions, and give Farm one more mode of operation — this time for small teams without a k8s cluster. Down the road, we could drop the old way of running plain Node.js processes entirely, which would greatly simplify maintenance by letting us delete a pile of legacy code.

How does it work in the end? Farm connects to the Docker Engine via a socket and acts as the builder itself — there are no separate build pods here. And here’s the first important difference from the k8s setup: the instance image is built locally, on the same daemon, and stays there. There’s no need to push it to a registry — the registry only comes in handy for pulling base images during the build. It’s noticeably simpler, and that’s exactly what small teams without a cluster need.

We designed two modes of operation:

• docker_container — Farm itself runs as a container with the host’s socket mounted and conducts its “neighbor” containers on the same daemon;

• vm — Farm lives right on the VM with its own Docker Engine.

Both Farm and all the instances join a shared Docker network (farm by default), and every application instance gets a container with the self-explanatory name farm-docker-<hash>. That’s the name we later use to find it when it’s time to route traffic.

As in the k8s setup, build and start live in the same buildInstance method, just without separate pods:

1. A user or CI starts the build (the buildInstance method).

2. The code is pulled from the branch and the Farm configuration is read from farm.json — that’s where the Dockerfile path comes from, along with the build and runtime variables.

3. Farm builds the docker image locally and tags it farm-docker-<hash>, streaming all the build logs to the UI for debugging along the way.

4. From the resulting image, a container is created and started in the farm network — with the environment variables passed through and, if desired, an overridden start command.

Traffic Routing

Here Farm once again delegates proxying to nginx. The instance container doesn’t publish any ports on the host — it can be reached by the name farm-docker-<hash> inside the farm Docker network, where Docker’s built-in DNS resolves that name.

The only question is where that nginx capable of resolving the name lives — and that depends on the mode:

• In docker_container mode, a request from the host goes into the Farm container, and its internal nginx, sitting on the farm network, delivers the traffic straight to the instance.

• In vm mode, Farm’s nginx runs directly on the host and can’t see container names, so it proxies the request to a separate Docker Proxy container (the same nginx, just running inside the farm network), which then hands the traffic over to the right instance.

And that’s how we get a working instance on an ordinary virtual machine with Docker, no cluster involved: the code is built into an image, the image becomes a container, and nginx delivers traffic to it.

Along the way we also solved the isolation problem together with that odd unix-socket contract — the application now ships as a regular docker image and simply listens on its port, knowing nothing about Farm’s internal kitchen.

Farm’s Evolution and Going Open Source

After all the improvements — the provider abstractions, support for different build and run schemes, and the move to Knex — Farm already looked like a mature, full-fledged product where Yandex specifics no longer played a big role. Farm had gradually become an orchestrator with no hard coupling to the technologies it ran underneath.

It now resembled a simplified deploy system along the lines of Heroku, only as a self-hosted service. So more ambitious plans emerged — to publish Farm’s code as part of our open-source project Gravity UI, so that it could be useful not only inside the company but to the whole community.

But this beautiful idea of opening the code had one catch. Even though Farm had become more abstract and clean, plenty of Yandex specifics still remained: internal authentication, Arc support, posting statuses to our issue tracker, telemetry, and other bits tied to the internal infrastructure. Publishing all of that isn’t a good idea — and the community has no use for it anyway. But simply cutting it out wasn’t an option either: our own installations, used by dozens of teams every day, run on those very specifics.

So the task boiled down to this: take everything Yandex-specific out of the picture and extract a configurable core from Farm, into which the specifics could be plugged back from the outside without touching the core’s code. Essentially, we needed to draw one more boundary between what we’re ready to open to the world and what stays internal.

In the end, we physically split Farm into two codebases.

• The open core: all the orchestration, the Docker and k8s providers, Git support, the Knex-based database, the API, the UI, the build queue, healthcheck — in short, everything that makes Farm a product. The core knows nothing about Yandex and can safely be published as is.

• A thin extension layer that adds everything proprietary on top.

The glue between them is a plugin registry: the core boots itself up and provides a single entry point through which the extension layer registers its implementations against well-defined interfaces. As a result, the entire Yandex part now lives in one small module and is plugged in declaratively:

initCoreExtension(async () => {
    // a custom way of running instances
    coreRegistry.farmProviders.plugIn('process', {
        constructor: ({internalApi, config}) => new ProcessFarmProvider(internalApi, config),
    });
    // a custom version control system
    coreRegistry.vcs.plugIn('arc', {constructor: () => new ArcVcs()});
    // a custom webhook action — for example, posting a status to the tracker
    coreRegistry.webhookActions.plugIn('tracker', new TrackerWebhookAction());
    // custom authentication
    coreRegistry.authProviders.plugIn('internal', {constructor: () => new InternalAuthProvider()});
    // ...and also telemetry, CSP domains, UI menu items
});

The registry is organized as a set of separate extension points, each with its own interface — everything that may vary is exposed through them:

• providers (farmProviders) — how and where to deploy;

• version control systems (vcs) — public git, internal arc;

• authentication (authProviders) — how to let users into the UI and API;

• webhook actions (webhookActions) — what to do in response to a CI event, such as posting a status back to the tracker;

• the farm.json schema (farmJsonConfig) — custom fields in the application configuration for a specific provider’s needs;

• UI and security (uiConfiguration, cspDirectives) — menu items and the list of trusted domains in the CSP.

The key point is that to the core, all these implementations are equal: arc is no different from git, and process is no different from docker or k8s. So the core can be opened up without looking back at the internal kitchen, while our installation keeps working as before — simply by building the core together with the extension layer.

Rolling Out Farm on k8s

Having come a long way, we finally got to moving our Farm installations to k8s. The strategy we chose: migrate the largest installation onto Kubernetes rails, and then, if everything works well, revive the communal setup by moving the remaining projects from local Farms to one shared installation. This approach is exactly what solves the maintenance complexity problem we flagged at the beginning. Teams once again don’t have to think about running their own installation and watching its resources, because the shared cluster scales automatically under load.

We’re free to use our own services and cloud offerings to build the infrastructure. So we created a cluster with Yandex Managed Service for Kubernetes® and described absolutely all resources with Terraform. We also set up separate secret management for each project with atomic access permissions on every resource. The result is a reusable Terraform module that makes it quite easy to deploy a k8s installation of Farm.

In a single day, we successfully moved all current projects to the k8s installation of Farm. It wasn’t too hard, since the original contract and configs stayed the same. The new Farm performed quite well, and other projects started migrating to it too. Soon enough a new problem surfaced: although the cluster scales by RAM and CPU, disk space still kept filling up, because plenty of images were built every day, and they took up storage.

Here we solved the problem head-on by adding a cleaner mechanism:

• for Docker it works like this: you set a specific time as a cron expression for when unused images should be deleted;

• for k8s the mechanism is built on the native CronJob resource, which launches pods on every node that likewise delete old images.

We declared the Farm-on-k8s experiment a success based on the teams' response: the feedback, the number of problems, and the number of requests to the maintainers.

What Farm Can Do Today

Over this journey, Farm has grown from a single-stack workhorse into a mature orchestrator of preview environments. Here’s a quick summary of what it can do now:

1. Build and run from a branch. Triggered by a webhook or from the UI, Farm fetches the code, builds the application, and brings up an isolated instance with its own URL.

2. Multiple providers. k8s for large installations with horizontal scaling, docker for a single virtual machine without a cluster. The application contract is the same across all providers.

3. A configurable core with a plugin registry. Providers, version control systems, authentication, and webhook actions plug in through a single registry, letting you extend Farm without changing its core.

4. A scheduler and a build queue. Generation requests are queued, and the scheduler works through the queue respecting limits on concurrent builds and running instances.

5. Lifecycle management. Automatic instance start from the UI, plus stopping and deleting idle instances on a timeout.

6. Healthcheck. Farm monitors instance health and reflects the current status in the UI and API: with its own availability checks in Docker and process mode, and via native liveness/readiness probes in Kubernetes.

7. Environment variable passing. Flexible environment setup: build-time and runtime variables, protected variables that can’t be overridden at generation time, and for Docker — inheriting variables from the host or container as well.

8. A persistent database with migrations. Knex on top of SQLite, PostgreSQL, or another DBMS.

9. UI and API. A web interface for launching and debugging instances and an HTTP API for CI integration.

10. Disk space cleaners. Regular cleanup of unused images in Docker and k8s.

What’s Next

Now, about our plans. There are still many things we’d like to improve in Farm. Here are the main directions at the moment:

1. Moving to the Gateway API as a replacement for Ingress in the k8s provider.

2. Alias support, so an instance can be reached by a human-readable name instead of a hash.

3. Full-fledged authorization — a system of users, roles, and access permissions.

4. Separate quotas for different projects.

As with any other open-source project, we’ll keep improving the documentation and examples to make Farm easier and more convenient to use — and to help the number of external users grow quickly. We’d love Farm to eventually become one of the go-to tools for preview environments in the industry — but we’ll see how it goes.

How to Try It

Come visit our repository, check out the README and the relevant documentation. To give it a try, you can freely spin up Farm locally with Docker Engine installed. We’ll be grateful for any feedback, so file issues, send PRs, ask questions, and bring feature requests if you need some extra functionality!

Thanks for reading! If you like our project, we’d be happy to get your stars! And if you’re interested in the latest news from the Yandex Cloud team, join our channel.