REST to gRPC using Twirp

Photo by Caspar Camille Rubin on Unsplash

In 2000, Roy Fielding presented REST (Representational State Transfer) software architectural style in his doctoral dissertation, which was a turning point in the web development world. It provided a set of architectural guidelines which helped with scalability, deployability, reduced latency, and encapsulation of legacy systems.

REST relies on a stateless, client-server protocol, where the client and server are completely separated. REST’s uniform interface simplifies and decouples the architecture, which enables each part of the service to evolve independently.

For these reasons, REST became popular within a microservices architecture.

A structure is stable if cohesion is high, and coupling is low.

— Constantine’s Law

Enter Microservices

As the software industry started maturing, many aspects were brought to the surface like scalability, deployability, rapid test processes, ease of maintenance, etc., which monolithic applications couldn’t solve. That’s when microservices architecture came to the limelight. Although the concept of microservices was not new. Google, Amazon, and Facebook have been running microservices for a long time.

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

— Conway’s Law

Microservices architectural style allows you to develop a single application as a suite of small services, that is loosely coupled following Single Responsibility Principal (SRS), and can be deployed independently. Most organizations have moved to, or are in the process of moving to microservices architecture.

When to use REST

RESTful services have been a good choice for microservices architecture when:

• you want request-response to be human-readable.
• your architecture needs to support different types of clients like mobile, browsers, etc.
• you need a standard interface and message format such as JSON.

But it may not be ideal when:

• your microservices architecture has internal service-to-service communication.
• performance is of concern. Most RESTful services use JSON as the data format but it has several limitations. JSON serialization and deserialization take a lot of memory and CPU, which makes inter-service communication slow. JSON objects are far more bloated than binary format. Binary serialization is way faster than JSON serialization.
• you want to create more services without the maintenance overhead. While designing our APIs, there are various aspects we should be thinking about: how are services going to talk to other services? what should be the response for each endpoint? How should we handle errors? How do we handle versioning? What query parameter should we pass? As we start adding more features these concerns will also start growing, especially in a microservices architecture.

A developer is now spending all their time and energy reading the API docs, and what HTTP verb and payload to use. Instead of writing the business logic, the developer is now focused on how to call the services. Wouldn’t it be cool if we have standard URL schemes, request and response payload types, and error messages?

Meet Twirp by Twitch

Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API definition files and lets you focus on your application’s logic instead of thinking about folderol like HTTP methods and paths and JSON.

Twirp is similar to gRPC but without the custom HTTP server and transport implementations. It can run on both HTTP 1.1 and HTTP/2, and supports both JSON and Protobuf serialization.

gRPC is a modern open-source high-performance Remote Procedure Call (RPC) framework that can run in any environment. It was created by Google in 2015 as an open-source project that is now under the umbrella of the Cloud Native Computing Foundation (CNCF).

Twirp addressed some of the pain points of gRPC —

• gRPC only uses protobuf over HTTP/2, which means browsers/mobile can’t easily talk directly to gRPC-based services.
• Twirp works over Protobuf and JSON, over HTTP 1.1 and HTTP/2, so any client(browser/mobile) can easily communicate.
• gRPC is a full framework with many features like streaming, channels, etc., but Twirp is a lightweight framework with few features and is easily manageable.

Twirp solves service-to-service communication problems through code generation from the protocol buffer (protobuf) file.

Protobuf

Protobuf is a data serializing mechanism written by Google which is language-neutral and platform-neutral. It’s like JSON, except it’s smaller and faster, and it generates native language bindings.

Protobuf performs six times faster than JSON

Some of the advantages of using protocol buffers include:

• Compact data storage
• Fast parsing
• Availability in many programming languages like Java, Python, Kotlin, GO, etc.
• Optimized functionality through automatically-generated classes

Sample CRUD API using Twirp

Let’s first define our service in proto file

Sample proto file for sample gRPC TWIRP API

In our proto file, we have 5 CRUD functions: GetAlbums , PostAlbums , GetAlbumByID , DeleteAlbumByID , and UpdateAlbumByID with their request and response message types.

The next step is to use the protocol buffer compiler protoc command to generate go code from the proto file above:

protoc --twirp_out=. --go_out=rpc/twirpAPI/ rpc/twirpAPI/twirp.proto

The above command will generate two new files twirp.pb.go and twirp.twirp.go which have an interface, a client, and some server utils to start an HTTP listener.

protoc command will generate an interface that we will need to implement which maps cleanly to the service definition.

Twirp interface generated via protoc

A sample implementation of the interface connecting to PostgreSQL

Interface implementation

In the above gist, I used sqlc to generate type-safe code from SQL.

Run the server!

go run ./cmd/server/main.go

You can now use the auto-generated Client to make remote calls to your new service by running

go run ./cmd/client/main.go

Protobuf client

Notice in the above gist, that you have an option to create both ProtobufClient and JsonClient .

You can also call your Twirp service with cURL, using either Protobuf or JSON.

For Protobuf call:

Use the header Content-Type: application/protobuf to signal that the request and response are Protobuf.

cURL using Protobuf

For JSON call:

Use the header Content-Type: application/json to signal that the request and response are JSON

cURL using JSON

Notice how every request is a POST , we don’t have to think about which HTTP method or URL to call. A developer can now focus on writing the business logic and not get bogged down with details like routing, headers, HTTP methods, etc.

To Sum it up

I recently worked on Twirp and was impressed with how lean and developer-friendly it is. Thank You, Twitch Engineers!

I was able to focus on writing the business logic and not worry about other details like routing, serialization/deserialization, or reading API docs. And you can run on either HTTP/1.1 or HTTP/2. If you have a microservices architecture, I’d try adding your next service using Twirp or perhaps migrate your existing REST API to Twirp. Although, if your use case is more around bidirectional or streaming communication, then gRPC may be your solution, not Twirp. If Go is not your preferred programming language, there’s a list of some third-party implementations in major programming languages.

Check out the full code for the sample CRUD Twirp API in GO.

Let me know in the comments about your experience with Twirp.