Context, context, context

At Yoyo wallet, we are always handling a lot of RESTful API requests to our platform.

We built that platform from the ground up based around a microservice architecture. All communication between layers and services is routed via message queues. This pattern gives us separation of responsibility and the ability to scale where it is needed.

The API by design is is stateless, however an API request will have additional metadata that will need to be passed through the relevant microservices. This metadata holds information like the identity and roles of the user making the request. We wanted to do this in a way that didn't transform the payload and was consistent across all microservices.

To satisfy this requirement we hijacked the pattern used here in Google's golang context package. This package is designed for sharing context across multiple parallel processes in a single application. We have taken the concept and expanded it for multiple parallel microservices.

This allows us to easily implement features in our microservices such as:

  • authentication (who are they?)
  • authorisation (are they allowed to do it?)
  • auditing (remember they did it)
  • monitoring (how many times have they done it)

Service Context

The simplest form of context we care about is when one microservice calls another microservice.
Service context In this scenario the only context information we pass are details of Microservice A as it calls Microservice B.

ServiceContext information includes:

  • service name (which service?)
  • service version (which code?)
  • service instance (which running instance?)

Service version and service instance information is essential when things go awry.

Service to Service Authorisation

This allows us to implement Service to Service Authorisation. We may wish to restrict which services have permission to call a particular service.
Service to service authorisation As you can see in the diagram Microservice 1 can call Microservice 3, however Microservice 2 is denied.

User Context

Passing details of a user that initiates an API call is handled in the same way as a Service context. (Actually behind the scenes we also pass ServiceContext details too)

UserContext information includes:

  • userID (who they are?)
  • sessionID (which session?)
  • roles (what are they allowed to do?)

The UserContext is derived in our edge-api when the user is authenticated. The UserContext then travels with the request from service to service.
User context

Tick tock

With all these service calls going on it is often hard to decide how long we should wait for requests to complete. "Deadlines" are another feature hijacked from the Google context package.

The concept is simple, when you initiate a request you specify a hard deadline. This is a fixed time that the initial request will wait until. If the time is exceeded the service will return a timeout error.

This deadline is passed from service to service as a hard deadline for the overall request to complete. This prevents downstream services from continuing to run when there is no-one waiting to receive their response.

The diagram below shows an example of a successful deadline where all the associated service calls complete within the time limit.

Deadline Success Deadline success

The diagram below shows when a deadline has been exceeded. In this example Microservice 3 was taking an excessive amount of time, therefore the request was cancelled and returned a timeout error.

Deadline Exceeded Deadline exceeded

Deadlines naturally rely on accurate clock synchronisation between machines, or deadlines which are “fairly” relaxed.

Making life easy

Enough about context, how about some code? Being developers we like an easy life, so we have developed a client package that we use to interact with all our microservices.

There are two ways to call our microservices. With a context and without.

type Client interface {  
    Call(endpoint string, req proto.Message, rsp proto.Message) errors.Error
    CallWithContext(ctx context.Context, endpoint string, req proto.Message, rsp proto.Message) errors.Error
}

Intra-service calls are being made using messages on a bus. We are using RabbitMQ as our chosen messaging layer. The client pacakge handles the marshalling and unmarshalling of context information as message headers.

Calling with default context

Here is an example from some of our test code:

    // create client
    sc, err := NewServiceClient(service.Name)
    ...
    // call service
    err = sc.Call(endpointName, req, rsp)

When calling a service endpoint a ServiceContext is implicitly added.

The client package serialises the context information into a series of AMQP headers. The service package deserialises the context information in the receiving microservice to reconstruct the context.

These headers allow the context of the request to be propagated between multiple microservices without effort.

Calling with user context

    // create client
    sc, err := NewServiceClient(service.Name)
    ...

    // init vars
    sessionToken := "session1234"
    userID := "user1234"

    // create context
    ctx := service.NewServiceContext()
    // add extra context info
    ctx = actor.WithSessionToken(ctx, sessionToken)
    ctx = actor.WithUserID(ctx, userID)
    ctx = actor.WithAccountID(ctx, accountID)

    // call service
    err = sc.CallWithContext(ctx,endpointName, req, rsp)
    ...

Calling with a deadline

    // create client
    sc, err := NewServiceClient(service.Name)
    ...

    // create context
    ctx := service.NewServiceContext()
    // deadline 1 second in the future
    deadline := time.Now().UTC().Add(time.Duration(2 * time.Second))
    deadlineCtx, _ := context.WithDeadline(ctx, deadline)

    // call service
    err = sc.CallWithContext(deadlineCtx,endpointName, req, rsp)
    ...

Handling deadlines

This is an example of code from the service package that calls an endpoint handler and uses channels to wait for timeouts:

// get context from request message
ctx := req.GetContext()

// call endpoint handling code
go func() { rspChan <- handlerWrapper() }()

select {

// wait for context cancel/timeout
case <-ctx.Done():  
    logger.Info("Server request has been cancelled")

    // timeout error
    msErr := errors.Timeout("com.yoyo.service.service-lib", fmt.Sprintf("Request cancelled: %s", ctx.Err()))
    returnMQError(d, msErr)

    // wait for service response
case msRsp := <-rspChan:  
    if msRsp.err != nil {
        // endpoint handler returns error
        returnMQError(d, msRsp.err)
    } else {
        // endpoint handler success
        returnMQRsp(d, msRsp.rsp)
    }
}

Endpoint handlers

All our microservice endpoints implement the same interface.

// Handler interface
type Handler func(ctx context.Context, req *Request) (proto.Message, errors.Error)  

Each one can receive a context and therefore pass it into any child services it invokes.

AMQP Gotcha

We implemented this code and 90% of our requests were working fine but we would get occasional timeout failures on a handful of requests. Upon investigation we realised that we were using an AMQP Timestamp as a header for the deadline. Unfortunately the precision of timestamps is in seconds. Therefore if a request was made at 12:59:59.999999 the timestamp would default to 12:59:59. This would leave us only 1 nanosecond to complete the request, when we expected a full second.

The code has been changed now to use an int64 header and persist the deadline as a value in nanoseconds.

Benefits

We have found this to be a very helpful pattern to implement thoughout our stack as you are always going to want context information at some point in your code. By passing in explicitly, all the time in the same way it is always there when you need it.

It can be used for role based authorisation, database update auditing, business logic rules, API rating limiting etc.

Your experience

We're always interested in hearing about other people's experiences with developing microservices. Have any of you used similar tools or had similar experiences around the propogation of context between services?

Let us know what you did, what challenges you faced etc. using the comments below, or tweet us @yoyoengineering