Marshalling monadic types

In the previous post we’ve seen that spray directives are monads and how we can use composition to make new building blocks for effective construction of complex route structures. Now let’s look at another trick with monadic types and what Spray can offer us for this time.

Part 1. Simple use case with monads

At first we need to take some common and simple case involving monadic types, so the easiest i could come up with is authentication REST-like service. We have a basic route expecting a POST request with credentials (name, password) in json format, some logic to be executed on the provided data and a couple of rejections to denote that something is wrong with the data or the request itself. Let’s start with rejections part:

1
2
3
4
5
6
7
trait Rejections {
  type MJR = MalformedJsonRejection.type
  type VR = ValidationRejection

  case object MalformedJsonRejection extends Rejection
  case class ValidationRejection(msg: String) extends Rejection
}

In this example we need only two types of rejections. MalformedJsonRejection works in cases when the json isn’t what we expected (required field is missing) and ValidationRejection(msg: String) if the data itself is invalid. Type aliases are declared just to make our code shorter. Now to the main function:

1
2
3
4
5
6
7
8
9
10
11
12
def auth(json: JsObject): Future[MJR \/ ValidationNel[VR, JsObject]] = {
  future {
    json.getFields("name", "password") match {
      case Seq(JsString(name), JsString(pwd)) =>
        val auth = (validName(name) |@| validPwd(pwd)) { (n, p) =>
          JsObject("result" -> JsString("Logged In!"))
        }
        auth.right
      case _ => MalformedJsonRejection.left
    }
  }
}

Auth takes an extracted data, with pattern matching extracts required fields and then proceeds to the validation step or rejects the request. For validation we are using ValidationNel, from Scalaz library, which allows us to aggregate validation errors, if there are some, or return the result, in this example it returns a login message in json. Also for demonstration purposes let’s wrap this into a future. So we have a Future with either (\/) a MalformedJsonRejection or a NonEmptyList with ValidationRejections, in case there were some errors validation errors, and a response json message if everything is good. Validation functions are also simple:

1
2
3
4
5
6
7
8
9
def validName(name: String): ValidationNel[VR, String] = {
  if (name == "Alex") name.successNel
  else ValidationNelRejection("No such user").failureNel
}

def validPwd(pwd: String): ValidationNel[VR, String] = {
  if (pwd.isEmpty) ValidationNelRejection("Password is empty").failureNel
  else pwd.successNel
}

Now we need define some route to test all this stuff:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def entityAs[T: Unmarshaller] = entity(as[T])

val mainRoute = {
  (path("test") & post & entityAs[JsObject]) { json =>
    onSuccess(auth(json)) {
      case -\/(rej) => reject(rej)
      case \/-(valid) =>
        valid match {
          case Failure(rej) => reject(rej.list: _*)
          case Success(response) => complete(response)
        }
    }
  }
}

When the auth function successfully completes (onSuccess directive), the request would be either rejected (with MalformedJsonRejection or ValidationRejection) or completed with a json response. But we still need to add handlers for our rejections:

1
2
3
4
5
6
7
8
9
10
11
trait RejectionsHandler extends Rejections {
  implicit def handlerRejections = RejectionHandler {
    case MalformedJsonRejection :: _ =>
      val response = JsObject("reason" -> JsString("Bad json!"))
      complete(BadRequest, response)
    case nels =>
      val res = nels collect { case rej: VR => JsString(rej.msg) }
      val response = JsObject("reason" -> JsArray(res))
      complete(BadRequest, response)
  }
}

Case for the malformed rejection is easy, in case of validation rejections we need to collect all rejections of type ValidationRejection into a List[JsString] and just convert it to JsArray. Now everything is fine, we can launch the service:

1
2
3
4
5
object Launcher extends App {
  implicit val system = ActorSystem("TestSystem")
  val handler = system.actorOf(Props[Service], "service")
  IO(Http) ! Http.Bind(handler, "0.0.0.0", 9000)
}

A couple of test request:

1
2
3
4
5
6
7
8
9
10
11
request: { name": "Alex", "password": "asd" }
response: { "result" : "Logged In!" }

request: { "name": "Alex", "password": "" }
response: { "reason" : ["Password is empty"] }

request: { "name": "hello", "password": "" }
response: { "reason" : ["No such user", "Password is empty"] }

request: { "username": "hello", "password": "" }
response: { "reason": "Bad json!" }

Looks like everything is fine, now to the best part… refactoring.

Part 2. Refactoring with Marshallers

Let’s change the approach for this time to the bottom-up and then “derefactor” our code. Here is how the final route looks like:

1
val authentication = authRequest { auth(_) |> completeAs(OK) }

All the routing logic just in one line! It takes a request, and completes the request with a response or rejects it. And it looks concise and readable, at least to me. Now to the internals.

Let’s break this route into three parts: authRequest, auth and completeAs. The first part, authRequest directive, it is a simple example of directives composition (in semigroup style) as described in official documentation:

1
val authRequest = path("test") & post & entityAs[JsObject]

It specifies our route to accept a post request with a json payload on path "test". auth function implementation hasn’t changed at all, we still have the same result type, so were all the rejection handling? And here comes the power of Spray marshallers.

For our example we need to add two types of marshallers: for either (\/) and for ValidationNel. Both marshallers have a very similar implementation – handle rejection(s) or delegate object marshalling further.

1
2
3
4
5
6
7
8
9
type EitherZ[T] = Rejection \/ T
implicit def eitherZMarshaller[T: Marshaller] = {
  Marshaller[EitherZ[T]] {
    (either, ctx) => either match {
      case \/-(res) => implicitly[Marshaller[T]].apply(res, ctx)
      case -\/(rej) => ctx.handleError(RejectionError(rej))
    }
  }
}

And for ValidationNel looks similar, with the exception that in case of failure we have a NonEmptyList:

1
2
3
4
5
6
7
8
9
10
11
type Valid[T] = ValidationNel[Rejection, T]
implicit def validationMarshaller[T: Marshaller] = {
  Marshaller[Valid[T]] {
    (valid, ctx) => valid match {
      case Success(res) => implicitly[Marshaller[T]].apply(res, ctx)
      case Failure(nel) => nel foreach { // we have a list with rejections
        rej => ctx.handleError(RejectionError(rej))
      }
    }
  }
}

With this marshallers in scope, we can delegate all the handling job to the Spray. As for the Future and onSuccess directive, there is already a futureMarshaller implemented in marshalling package.

Final part, completeAs directive, is defined in a similar way to entityAs and allows us to handle response entity implicitly:

1
def completeAs[T: Marshaller](sc: StatusCode)(response: T) = complete(sc, response)

One more note on |> operator (or Thrush combinator), it is defined is scalaz IdOps, allows us to apply self to the given function.

So we have a value of type Future[MJR \/ ValidationNel[VR, JsObject]], returned by auth function, and a function of type T => Unit (actually it has a bit different type, complete directive returns a StandartRoute with an apply method which returns nothing (Unit)). To complete the request we need to have a marshaller to convert our response JsObject to HttpEntity. To do this Scala goes through the chain of implicit marshallers until it reaches the sprayJsonMarshaller, which delegates stringified version of json to the StringMarshaller, handling all rejections on its way.

If you don’t have any specific logic for rejections handling or transforming values, then, it would be better and cleaner to delegate this job to the combintation of directives and marshallers.

Part 3. Abstracting Marshallers

But that’s not all. Let’s go further and think about modularity, cakes and abstractions. If you might have read the Precog’s article The Abstract Future and you like things like modules, cakes and abstraction, then you might ask, what if we want to abstract over some types? How can we abstract marshaller itself?

Let’s take the Precog’s article way and abstract over the Future, we can do this for other monadic types as well, but i think Future is the most interesting one, because you might easily want to switch to Scalaz version of Future or use an Id monad (from Scalaz as well) for easier testing. To do this we have to abstract our module to something similar:

1
2
3
4
trait AuthModule[M[+_]] {
  implicit val M: Monad[M]
  def auth(json: JsObject): M[MJR \/ ValidationNel[VR, JsObject]] = ???
}

Where M is a Monad, so we can switch between Future and Id. Id is included in the Scalaz library and for future we should provide our own implementation:

1
2
3
4
val futureMonad = new Monad[Future] {
  def point[A](a: => A): Future[A] = future(a)
  def bind[A, B](fa: Future[A])(f: (A) => Future[B]): Future[B] = fa.flatMap(f)
}

Now a little change to our auth function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def auth(json: JsObject): M[MJR \/ ValidationNel[VR, JsObject]] = {
  val res = json.getFields("id", "name", "password") match {
    case Seq(JsString(id), JsString(name), JsString(pwd)) =>
      val auth = (validName(name) |@| validPwd(pwd)) { (n, p) =>
        JsObject(
          "id" -> JsString(id),
          "result" -> JsString("Logged In!")
        )
      }
      auth.right
    case _ => MalformedJsonRejection.left
  }
  res.point[M] // method, defined in a Monad trait
}

But this won’t help us much, cause there is no an implicit for M in scope:

1
[error] couldn't find implicit value for evidence parameter of type Marshaller[M[MJR \/ ValidationNel[VR,JsObject]]]

Natural way to do this would be to add another abstract member similar to M:

1
implicit val MM: Marshaller[M]

But this also won’t help us either, cause Marshaller is not a higher-order type and expects a first-order type, whereas M is a second-order. So we need a define higher-order Marshaller. Let’s make one:

1
2
3
trait MonadMarshaller[M[_]] {
  def apply[A: Marshaller](monad: M[A], ctx: MarshallingContext): Unit
}

But Spray is working only with standard Marshallers, so we need to provide a marshaller instance for some type A. Now a couple of instances, for Future and for Id:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
object MonadMarshaller {
  implicit val futureMM = new MonadMarshaller[Future] {
    def apply[A: Marshaller](value: Future[A], ctx: MarshallingContext) = {
      value onComplete {
        case Success(res) => implicitly[Marshaller[A]].apply(res, ctx)
        case Failure(ex) => ctx.handleError(ex)
      }
    }
  }
  implicit val idMM = new MonadMarshaller[Id.Id] {
    def apply[A: Marshaller](value: Id.Id[A], ctx: MarshallingContext) =
      implicitly[Marshaller[A]].apply(value, ctx)
  }
}

This allows us to provide an implicit marshaller for monadic types into the scope, so we can abstract our module:

1
2
3
4
5
trait AuthModule[M[+_]] {
  implicit val M: Monad[M]
  implicit val MM: MonadMarshaller[M]
  .........
}

At the last step we need a new version of a complete directive to complete requests with monadic response:

1
2
3
4
5
6
def completeAsM[N[_]: MonadMarshaller, T: Marshaller](sc: StatusCode)(body: N[T]) = {
  implicit val m = Marshaller[N[T]] { (value, ctx) =>
    implicitly[MonadMarshaller[N]].apply(body, ctx)
  }
  complete(sc, body)
}

Now we can specify our production to Future:

1
2
3
4
5
class Service extends HttpServiceActor with ServiceLogic[Future] {
  implicit val M: Monad[Future] = futureMonad
  implicit val MM: MonadMarshaller[Future] = futureMM
  def receive = runRoute(mainRoute)
}

And in your test code you can specify it to Id monad respectively:

1
2
3
4
5
class ServiceTest extends HttpServiceActor with ServiceLogic[Id.Id] {
  implicit val M: Monad[Id.Id] = idMonad
  implicit val MM: MonadMarshaller[Id.Id] = idMonadMarshaller
  ....
}

Important note.

As of this PR, marshallers for monadic types were added into Spray as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
trait MarshallerM[M[_]] {
  def marshaller[T: Marshaller]: Marshaller[M[T]]
}

// Example for \/ monad

object MarshallerM {
  implicit val eitherZ = new MarshallerM[({ type λ[β] = Rejection \/ β})#λ] {
    def marshaller[T: Marshaller] = Marshaller[λ[T]] {
      (either, ctx) => either match {
        case \/-(res) => implicitly[Marshaller[T]].apply(res, ctx)
        case -\/(rej) => ctx.handleError(RejectionError(rej))
      }
    }
  }
}

Implementation didn’t change much, but now there is an implicit convertion to a simple marshaller, so no need for a custom complete for monadic marshaller.

I guess that’s all for today. Thanks


Binding Directives in Spray

One of the fundamental concepts in spray-routing is the idea of Directives – building blocks for creating complex and flexible route structure. With directives you can:

  • Transform the incomming request
  • Reject the request according to some logic
  • Extract some value from the request
  • Complete this request

Not much, but simple and very powerfull. So lets take a look at some example of working with directives and for the test case lets write some little improvised registration services. At first we need some data type with user data for registration purposes:

1
case class UserData(login: String, pwd: String)

In our example we are expecting this data to be send as a POST request on http://localhost/register. There are a couple of way how we can send the data, using JSON or as a simple payload and how are going to extract it from the request. To extract our data from the request a simple directive is what we need. If the data was send from the browser then formField is our choice, but to make this example more explicit chooice spray (un)marshalling module. Lets go with the second way and again Spray lets us choose a couple of ways how to extract it from the request context (what the directives actually do), with formField directive or through (un)marshalling. All this marshalling stuff is a bit harder, so lets define an unmarshaller for UserData. We need an implicit object in scope, for this cases a companion object is a perfect choice:

1
2
3
4
5
6
7
8
object UserData {
  implicit val userDataUM = Unmarshaller(`application/x-www-form-urlencoded`) {
    case HttpEntity.NonEmpty(contentType, buffer) =>
      val str: String = buffer.asString
      val Array(login, pwd) = str.split("&").map(_.dropWhile('=' != _).tail)
      UserData(login, pwd)
  }
}

With such unmarshaller defined we can extract entity with another directive entity. Lets write the full route (i think that other function are easy to implement, so i will place only their signatures) :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
def validate(data: UserData): Option[UserData] = ???
case class Account(data: UserData)
case object MalformedDataRejection extends Rejection
type Id = String
def storeAccountInDB(acc: Account): Future[Id] = future("123123")

val registerRoute = {
  path("register") {
    post {
      entity(as[UserData]) { entity =>
        validate(entity) match {
          case None => reject(MalformedDataRejection)
          case Some(data) =>
            val account = Account(data)
            val res = storeAccountInDB(account)
            onSuccess(res) { id =>
              complete(s"Your new id is $id!")
            }
        }
      }
    }
  }
}

Not that hard, looks clear and readable, but Spray can help us and make this even more clearer with directives concatenation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
val extract = post("register") & post & entity(as[UserData])

val registerRoute = {
  extract { entity =>
    validate(entity) match {
      case None => reject(MalformedDataRejection)
      case Some(data) =>
        val account = Account(data)
        val res = storeAccountInDB(account)
        onSuccess(res) { id =>
          complete(s"Your new id is $id!")
        }
    }
  }
}

That’s better and, maybe, for the most cases this would be enough, but, still, i really think that this pattern matching construct,couple of vals, explicit rejection and onSuccess parts breaks all the real beauty of this code. Can we do better? Yes, we do. How about the idea to convert all this stuff into a bunch of directives?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
def validateDir(user: UserData): Directive0 = {
  if (user.pwd.nonEmpty) pass
  else reject(MalformedDataRejection)
}

def createAccount(data) = provide(Account(data))

val registerRoute = {
  extract { entity =>
    validate(entity) {
      createAccount(entity) { acc =>
        onSuccess(storeAccountInDB(acc)) { id =>
          complete(s"Your new id is $id!")
        }
      }
    }
  }
}

Just what i wanted! No explicit rejections, useless vals and pattern matching. Now the code is more readable and it’s clear how the request goes through this branch: extract user data from request ~> validate the data ~> create account ~> write it to some store ~> return some id to the user. The execution flow is sequestial and clear, when the request hits our route, we are extracting the data, transforming it with directives and completing the request. This gently leads us to the hail concept of Monads, which is perfectly described in this book. So can Monads help us and make our life better? Lets try.

In Scalaz library, Monads is defined in terms of its two primitive operations: point and bind (flatMap). If we take a closer look at the Directive implementation we can see that it’s actually a Monad (as well as a Monoid, but i didn’t find any good use case for this, yet). That’s define one:

1
2
3
4
implicit val directive1Monad = new Monad[Directive1] {
  def point[A](a:  A): Directive1[A] = provide(a)
  def bind[A, B](fa: Directive1[A])(f: (A)  Directive1[B]): Directive1[B] = fa.flatMap(f)
}

The only problem for which i couldn’t find the solution connected with the internal implementation of Directives. To be able to extract data from the context and be typesafe, they are build on top of Shapeless HLists, but it looks like it’s not possible to define a general Monad for HList as so as for Directives. But in 90% i’m using Directive1 type and in rest 10% Directive0, so that’s not hard to define seconds version. Still, if needed, we can “lift” directives with standard implicit convertion:

1
implicit def dir02dir1(dir0: Directive0) = provide(HNil)

Such a small piece of code, but it opens us the hole power of monads. So lets flatmap out last snippet:

1
2
3
4
5
6
7
8
import scalaz.syntax.bind._

def completeRegistration(id: String) = complete(Created, s"Your new id is $id!")

val registerRoute =
  (extractData >>= validate >>= createAccount >>= store >>= andOnSuccess) {
    copmleteRegistration
  }

With the power of monads composition we can compose different directives, i.e:

1
2
3
4
5
6
7
8
9
10
11
val register = extractData >>= validate
def newAccount(data: UserData) = createAccount(data) >>= store
val registrationRequest = register >>= newAccount
def whenDone(f: Future[String]) = onSuccess(f)
def completeAs(sc: StatusCode)(id: String) = complete(sc, s"Your new id is $id!")

val registraterRoute =
  (registrationRequest >>= whenDone) {
    completeAs(Created)
  }
}

And that’s the very tip of the iceberg what can Spray and Scalaz do together.