RPC Code Generation

Explicit code and no magic - this is one of the core principles of Monstera. The goal was to achieve type-safety and utilize compile-time checks without reflection, but eliminate human mistakes from vast boilerplate code at the same time. That is why all boilerplate RPC code is generated by the framework.

monstera.yaml defines all application cores and their operations. The following command runs codegen (from the same directory where the YAML file is located):

go tool github.com/evrblk/monstera/cmd/monstera code generate

Codegen produces four files into the same directory:

  • api.go with interfaces for application cores and stubs, plus the request/response type aliases (see below).
  • adapters.go with adapters to application cores, that turn binary blobs into method calls on a core.
  • stubs.go with service stubs, that turn requests into binary blobs and route them to the correct application core.
  • validation.go with a Validate-checking wrapper for each application core (see Request validation below).

The adapters record per-method Prometheus metrics, but the metrics are not auto-registered: call the generated RegisterMetrics(prometheus.DefaultRegisterer) once at startup (alongside monstera.RegisterMetrics(...) for the framework’s own metrics) in each process that hosts a node.

Let’s take a look at the example monstera.yaml from Grackle, a service that provides distributed synchronization primitives. It defines five application cores (GrackleLocks, GrackleSemaphores, GrackleNamespaces, GrackleWaitGroups, GrackleBarriers) grouped under a single Grackle stub. Only the GrackleLocks core is shown in full below; the rest follow the same shape.

go_code:
  output_package: github.com/evrblk/grackle/pkg/coreapis
  core_types_package: github.com/evrblk/grackle/pkg/corepb
cores:
  - name: GrackleLocks
    read_methods:
      - name: GetLock
        method_number: 1
        sharded: true
      - name: ListLocks
        method_number: 2
        sharded: true
      - name: ListLocksByLeaseId
        method_number: 3
        sharded: true
      - name: ListLockLeases
        method_number: 4
        sharded: true
      - name: ListLockLeasesByProcessId
        method_number: 5
        sharded: true
      - name: GetLockLease
        method_number: 6
        sharded: true
    update_methods:
      - name: AcquireLock
        method_number: 1
        sharded: true
      - name: ReleaseLock
        method_number: 2
        sharded: true
      - name: DeleteLock
        method_number: 3
        sharded: true
      - name: RunLocksGarbageCollection
        method_number: 4
        sharded: false
      - name: LocksDeleteNamespace
        method_number: 5
        sharded: true
      - name: CreateLockLease
        method_number: 6
        sharded: true
      - name: RefreshLockLease
        method_number: 7
        sharded: true
      - name: RevokeLockLease
        method_number: 8
        sharded: true
  # GrackleSemaphores, GrackleNamespaces, GrackleWaitGroups, and GrackleBarriers omitted for brevity
stubs:
  - name: Grackle
    cores:
      - GrackleLocks
      - GrackleSemaphores
      - GrackleNamespaces
      - GrackleWaitGroups
      - GrackleBarriers

Each method has a method_number. Like protobuf field numbers, method numbers identify a method on the wire and must never change. Numbers must be explicit (>= 1) and unique among methods of the same kind (read/update) within a core; method names must be unique across all cores of the manifest, because each becomes a package-level *Request/*Response type alias. Both rules (and the overall manifest shape, including unknown keys from typos) are validated by monstera code generate before any file is written. sharded: true means the request is routed to a single shard by its shard key (see below); sharded: false is for cluster-wide or fan-out operations that target a shard by id explicitly. Read methods can also set allow_read_from_followers: true to allow serving them from follower replicas when slightly stale data is acceptable.

This config will generate:

  • GrackleClientApi - interface for the stub Grackle which consists of methods from the GrackleLocks, GrackleSemaphores, GrackleNamespaces, GrackleWaitGroups, and GrackleBarriers application cores. There are two generated implementations of this interface:
    • GrackleMonsteraStub - the stub that wraps the Monstera Client. All the routing is done there. Constructed with NewGrackleMonsteraStub(monsteraClient).
    • GrackleNonclusteredStub - nonclustered stub. Constructed with NewGrackleNonclusteredStub together with the generated GrackleNonclusteredApplicationCoresFactory. See Single Node Mode for more details.
  • GrackleLocksCoreApi, GrackleSemaphoresCoreApi, etc. - interfaces for application cores (this is what you implement).
  • GrackleLocksCoreAdapter, GrackleSemaphoresCoreAdapter, etc. - adapters for the corresponding application cores, constructed with NewGrackleLocksCoreAdapter, NewGrackleSemaphoresCoreAdapter, etc.
  • GrackleLocksValidatingCore, GrackleSemaphoresValidatingCore, etc. - a Validate-checking wrapper around each *CoreApi, constructed with NewGrackleLocksValidatingCore, etc. See Request validation below.
  • Request/response type aliases for every method (e.g. GetLockRequest, GetLockResponse) in the output_package. These wrap your proto payloads (see below) and only improve code readability.

A stub can route requests to one or several application cores. Since cores tend to be small, it makes sense to group multiple of them in a single stub. The stub interface (GrackleClientApi) is essentially the union of the methods of its application cores, plus a ListShards(applicationName string) helper. Unlike the core methods, stub methods take a context.Context and the raw request/response types directly (sharded methods derive the shard key by calling ShardKey() on the request; unsharded methods take an extra shardId string argument).

Currently, Monstera codegen relies on several conventions in order to make it work with your code:

  • Methods of an application core must have corresponding *Request and *Response structs in the go_code.core_types_package package. For example, GetLock of the GrackleLocks core must have GetLockRequest and GetLockResponse structs messages in github.com/evrblk/grackle/pkg/corepb.
  • Those *Request and *Response structs must implement encoding.BinaryMarshaler and encoding.BinaryUnmarshaler (MarshalBinary() / UnmarshalBinary()). The framework moves them around as opaque bytes inside a generic envelope, so it does not need to know their concrete types. (Grackle generates these one-line wrappers over the proto’s MarshalVT / UnmarshalVT with a small helper tool of its own, but you can choose whatever you want).
  • Every *Request of a sharded method must implement ShardKey() cluster.ShardKey. There are no annotations or reflection — you specify explicitly how to extract a shard key from each request, usually with one line of Go code (utils.GetShardKey derives one from arbitrary bytes). A cluster.ShardKey is a plain uint32: every value is a valid key, and the framework routes it to the shard whose range contains it.
  • Every *Request, sharded or not, must implement Validate() error. See Request validation below.

Packages output_package and core_types_package must be different, otherwise the generated *Request/*Response type aliases would collide with the payload types (validated by monstera code generate).

You define one *Request/*Response pair per method. This is conceptually similar to a gRPC service declaration. The envelope exchanged over the Monstera client is the framework’s own rpc.Request (carrying a method_number, the marshaled payload, and a request timestamp) and rpc.Response (carrying the marshaled payload and an application rpc.Error).

Here is an example proto file with all definitions necessary for the YAML config from above:

syntax = "proto3";

option go_package = "github.com/evrblk/grackle/pkg/corepb";
package com.evrblk.grackle.corepb;

message GetLockRequest {/*...*/}
message GetLockResponse {/*...*/}
message ListLocksRequest {/*...*/}
message ListLocksResponse {/*...*/}
message AcquireLockRequest {/*...*/}
message AcquireLockResponse {/*...*/}
message RunLocksGarbageCollectionRequest {/*...*/}
message RunLocksGarbageCollectionResponse {/*...*/}
// ... one *Request/*Response pair per method, for every core

That is all the proto you need: just the per-method request and response messages. No high-level containers, no shared Error field — those are provided by the framework’s rpc package.

Core API interfaces

The generated *CoreApi interface is what you implement. Its methods receive the request payload wrapped in one of the framework’s generic types and return the response payload wrapped accordingly. Codegen emits a type alias per method in the output_package so the signatures stay short (the generated code imports github.com/evrblk/monstera/rpc aliased as mrpc). For the config above:

// sharded read
type GetLockRequest = mrpc.ReadRequest[*corepb.GetLockRequest]
type GetLockResponse = mrpc.ReadResponse[*corepb.GetLockResponse]

// sharded update
type AcquireLockRequest = mrpc.UpdateRequest[*corepb.AcquireLockRequest]
type AcquireLockResponse = mrpc.UpdateResponse[*corepb.AcquireLockResponse]

// an unsharded method uses the *Unsharded* wrapper for the request
type RunLocksGarbageCollectionRequest = mrpc.UpdateUnshardedRequest[*corepb.RunLocksGarbageCollectionRequest]
type RunLocksGarbageCollectionResponse = mrpc.UpdateResponse[*corepb.RunLocksGarbageCollectionResponse]

The request wrapper exposes the proto payload as Payload and the request timestamp as Now (Unix nanoseconds) — read Now instead of calling time.Now() so updates apply deterministically on every replica. The response wrapper carries the proto Payload and an optional ApplicationError *mrpc.Error that becomes the deterministic domain error returned to the caller. So the generated GrackleLocksCoreApi looks like:

type GrackleLocksCoreApi interface {
    Snapshot() monstera.ApplicationCoreSnapshot
    Restore(readers ...io.ReadCloser) error
    Close()

    GetLock(req *GetLockRequest) (*GetLockResponse, error)
    ListLocks(req *ListLocksRequest) (*ListLocksResponse, error)
    AcquireLock(req *AcquireLockRequest) (*AcquireLockResponse, error)
    RunLocksGarbageCollection(req *RunLocksGarbageCollectionRequest) (*RunLocksGarbageCollectionResponse, error)
    // ... one method per read_method and update_method
}

The second returned value error is an internal error that killed the application core (not a domain error). See Core Principles for more details.

What is left for you to implement:

  • The application cores themselves (GrackleLocksCoreApi, GrackleSemaphoresCoreApi, etc. interfaces).
  • ShardKey() cluster.ShardKey on every sharded *Request payload type.
  • Validate() error on every *Request payload type, sharded or not. See Request validation below.
  • MarshalBinary / UnmarshalBinary on all request and response payload types.

Request validation

A request that reaches an application core’s Update/Read method through the Raft-applied path gets applied to every replica identically — so the adapter constructors (NewGrackleLocksCoreAdapter, etc.) deliberately do not wrap the core you pass them in a validating layer. Rejecting a request is itself a decision that has to be applied identically everywhere: if Validate()’s logic changed between the old and new binary during a rolling deploy, one replica could reject a request (never calling the core) while another accepts it (calling the core) for the exact same Raft log entry, diverging the FSM. Keeping validation out of the adapter means cores must be written to cope with whatever reaches Update/Read — the framework gives no guarantee that Validate() ran first.

This still makes Validate() error a required method on every *Request payload type, the same way ShardKey() is required on sharded ones — codegen’s request/unshardedRequest constraints require it, so the package fails to compile without it. Unlike ShardKey(), its implementation has no framework-provided helper; write whatever checks matter for that request (required fields, id shapes, bounds on values the core dereferences or feeds into arithmetic), or return nil if the payload genuinely has nothing to check.

In practice, Validate() is exercised by the generated stub (GrackleMonsteraStub), which calls it on the request before ever marshaling or sending it. This is a client-side check only: it saves a wasted network/Raft round trip on an obviously-bad request, and it is the main place Validate() runs against real traffic. A caller that reaches the core adapter through some other path — a raw RPC, single-node mode, a unit test — skips it unless it opts in some other way.

Codegen also generates, for each core, a *ValidatingCore (NewGrackleLocksValidatingCore, etc.) that implements the same *CoreApi interface by wrapping another instance of it: every method calls Validate() on the request payload first and, on error, returns an ApplicationError (mrpc.NewError(mrpc.InvalidRequest, err.Error())) without ever delegating to the wrapped core. Nothing wires this in automatically — reach for it explicitly wherever you want Validate() enforced ahead of a core call. The main use is core-level unit tests: wrap the core under test in it (NewGrackleLocksValidatingCore(core)) instead of constructing it directly, so tests exercise requests through the same rejection path the client-side stub does.