Hierarchical Data Exchange: Protobuf services made easier

We’ve all been there. You have nicely defined your LogiQL schema, you’ve powered through writing your protobuf services to get the data in and out of the workspace, and you’re slapping stuff on a web page, solving problems. Eventually a requirement changes or the data changes or you just realize you didn’t perfectly model your application on the first attempt. Shocking, I know. Instead of going about solving the actual problem, you get to embark on the mind-numbing journey of rewriting parts of your model, which means rewriting your protobuf specification which means rewriting your import AND your export services (and debugging them) and then, finally, you get back to doing real work.

Simply put, protobuf services are a real pain to write and even harder to write correctly. Enter Hierarchical Data Exchange, or HDX. With HDX, you declare your message format and then bind it to your data model. Once you learn the LogiQL message schema, changing a message isn’t much harder than updating a protobuf specification. But don’t take my word for it, let’s do a comparison! About year ago I started making a Football Pick’em LB application while on Xmas vacation. The point was mostly to learn how well AngularJS & ServiceBlox would work together. The good news is that I learned quite a bit about both. The bad news is that at least 50% of my time on the application was writing / tweaking protobuf services, which taught me nothing except how to complete boring and repetitive tasks. In this exercise we’re going to rewrite the services for exporting the NFL football schedule & results. The previous application had the following services that we’ll want to duplicate in our HDX example:

  • export all seasons (including weeks, games, etc.)
  • export the current season (including weeks, games, etc.)
  • export the current week (including games & teams)
  • export all teams

Model

Here is the model we used in the original application. The only thing that changed for the HDX version was to add the _trigger predicates.

//lang:logiql
Season(s), Season_year(s:year) -> string(year).
Season_weeks(s, w) -> Season(s), Week(w).
Season_trigger(s) -> Season(s).

Week(w), Week_id(w:s) -> string(s).
Week_games(w,g) -> Week(w), Game(g).
Week_trigger(w) -> Week(w).

currentSeason[] = s -> Season(s).
currentWeek[] = w -> Week(w).

Game(g), Game_id(g:s) -> string(s).
Game_winner[g] = t -> Game(g), Team(t).
Game_week[g] = w -> Game(g), Week(w).
Game_homeTeam[g] = t -> Game(g), Team(t).
Game_awayTeam[g] = t -> Game(g), Team(t).
Game_dateStr[g] = s -> Game(g), string(s).
Game_trigger(g) -> Game(g).

Team(t), Team_id(t:s) -> string(s).
Team_abbr[t] = abbr -> string(abbr), Team(t).
Team_trigger(t) -> Team(t).

This simple model is actually surprisingly annoying for writing protobuf services. You have four entities that form a hierarchy which means you write a lot of rules that map the two entities to the two protobuf messages that represent those entities. Some levels of this hierarchy have multiple references to other entities (namely Game to Team). Additionally the resulting JSON message if you just export the entity graph would result in the same Team being exported at least once per Week (twice if they won).

Message Specification

The first thing we want to do is set up our message specification which describes both what our messages will look like as well as binding the attributes of the message to predicates.

//lang:logiql
abbr:binding_by_name["Team"] = m,
abbr:binding_trigger[m] = "model:football:Team_trigger",
abbr:binding(m) {
    abbr:binding_entity[] = "model:football:Team",
    abbr:key_by_name["team"] = abbr:key(_) {
        abbr:key_index[] = 0
    },
    abbr:attribute_by_name["abbr"] = abbr:attribute(_) {
        abbr:attribute_predicate[] = "model:football:Team_abbr",
        abbr:attribute_key_index["team"] = 0,
        abbr:attribute_is_default()
    }
}.

abbr:binding_by_name["Game"] = m,
abbr:binding_trigger[m] = "model:football:Game_trigger",
abbr:binding(m) {
    abbr:binding_entity[] = "model:football:Game",
    abbr:key_by_name["game"] = abbr:key(_) {
        abbr:key_index[] = 0
    },
    abbr:attribute_by_name["winner"] = abbr:attribute(_) {
        abbr:attribute_predicate[] = "model:football:Game_winner",
        abbr:attribute_key_index["game"] = 0,
        abbr:attribute_is_default(),
        abbr:relation_binding[1] = "Team",
        abbr:relation_is_eager(1),
        abbr:relation_creation[1]="none"
    },
    abbr:attribute_by_name["home_team"] = abbr:attribute(_) {
        abbr:attribute_predicate[] = "model:football:Game_homeTeam",
        abbr:attribute_key_index["game"] = 0,
        abbr:attribute_is_default(),
        abbr:relation_binding[1] = "Team",
        abbr:relation_is_eager(1),
        abbr:relation_creation[1]="none"
    },
    abbr:attribute_by_name["away_team"] = abbr:attribute(_) {
        abbr:attribute_predicate[] = "model:football:Game_awayTeam",
        abbr:attribute_key_index["game"] = 0,
        abbr:attribute_is_default(),
        abbr:relation_binding[1] = "Team",
        abbr:relation_is_eager(1),
        abbr:relation_creation[1]="none"
    },
    abbr:attribute_by_name["date"] = abbr:attribute(_) {
        abbr:attribute_predicate[] = "model:football:Game_dateStr",
        abbr:attribute_key_index["game"] = 0,
        abbr:attribute_is_default()
    }
}.

abbr:binding_by_name["Week"] = m,
abbr:binding_trigger[m] = "model:football:Week_trigger",
abbr:binding(m) {
    abbr:binding_entity[] = "model:football:Week",
    abbr:key_by_name["week"] = abbr:key(_) {
        abbr:key_index[] = 0
    },
    abbr:attribute_by_name["games"] = abbr:attribute(_) {
        abbr:attribute_predicate[] = "model:football:Week_games",
        abbr:attribute_key_index["week"] = 0,
        abbr:attribute_is_default(),
        abbr:relation_binding[1] = "Game",
        abbr:relation_is_eager(1),
        abbr:relation_creation[1]="none"
    }
}.

abbr:binding_by_name["Season"] = m,
abbr:binding_trigger[m] = "model:football:Season_trigger",
abbr:binding(m) {
    abbr:binding_entity[] = "model:football:Season",
    abbr:key_by_name["season"] = abbr:key(_) {
        abbr:key_index[] = 0
    },
    abbr:attribute_by_name["weeks"] = abbr:attribute(_) {
        abbr:attribute_predicate[] = "model:football:Season_weeks",
        abbr:attribute_key_index["season"] = 0,
        abbr:attribute_is_default(),
        abbr:relation_binding[1] = "Week",
        abbr:relation_is_eager(1),
        abbr:relation_creation[1]="none"
    }
}.

One thing to note is that while this may seem somewhat verbose, it’s only about a 2x increase in line count from the actual protobuf specification itself.

Queries

We want to be able to execute the same queries we did in the previous version. In order to do that, we need to write just a tiny bit of LogiQL code to decide which entities to export based on the query parameters.

//lang:logiql
// NOTE: These queries are written for 3.10.x, for HDX on 4.0.x you can omit the '@prev'.

lang:pulse(`params).
params(k,v) -> string(k), string(v).

+params(k,v) <-
    +messages:Query(q),
    +messages:Query_param(q, p),
    +messages:QueryParam_key[p] = k,
    +messages:QueryParam_value(p, v).

// return all seasons
+fb:Season_trigger(s) <-
    +params("message", "Season"),
    !+params("current", _),
    fb:Season@prev(s).

// return current season
+fb:Season_trigger(s) <-
    +params("message", "Season"),
    +params("current", "true"),
    fb:currentSeason@prev[] = s.

// return current week
+fb:Week_trigger(w) <-
    +params("message", "Week"),
    +params("current", "true"),
    fb:currentWeek@prev[] = w.

// return all teams
+fb:Team_trigger(id) <-
    +params("message", "Team"),
    fb:Team@prev(id).

In each case we execute a query by posting a message that lists a bunch of key / value pairs which can be thought of as equivalent HTTP query string parameters. Once we ‘parse’ the query params, we look up the entities want to export and inform HDX that we want to export them by pulsing the appropriate trigger predicate.

Conclusion

And that’s all. We’ve replicated and actually improved on the functionality of the original pick’em application and it took roughly 122 lines with HDX vs 364 lines for hand-written protobuf. And keep in mind the hand written protobuf version didn’t have the ability to import or delete! A side benefit is that due to the way the HDX messages are constructed, the resulting JSON data is considerably smaller, although at the cost of a bit of Javascript on the client to look up message references. I hope you’ve enjoyed this preview of HDX. Documentation on this feature is in progress and we hope to make it available to you soon!

4 Comments
  1. Jeff Vaughan 7 years ago

    Trevor, Great post!

    This post brings up a high-level question for me. To me, the football data seems more naturally “relational” as opposed to “hierarchical.” When building an app how do you decide whether it makes more sense to export via tabular data exchange or via hierarchical data exchange? Is the choice mostly dictated by tooling, by schema, or something else?

    Also, many of the attributes have similar properties, e.g. abbr:attribute_is_default(). Have you thought about techniques to even further reduce the verbosity of HDX specs?

  2. Author
    Trevor Paddock 7 years ago

    I suppose the relational vs hierarchical nature of this particular data set depends on how you are using it. In the context of a web application, there are a few reasons why you’d want a hierarchical data format over a relational version.

    First off, JSON is the default data exchange format for Javascript MVC frameworks such as AngularJS, etc. Second, when declaring your data this way, it’s very easy to derive RESTful services from this format. For example, it’s easy to see how /weeks/1 would return a list of games which in turn returns the teams. These would be the same team messages you would get from the /teams URL. With the relational (i.e. delimited) version, there is no shared format between the teams data from /weeks & /teams since each URL is a completely different file specification.

    As far as the verbosity of the message specification, keep in mind it is just logiql code. You could write a rule such as the following to make all attributes on a particular message default:

    attribute_is_default(a) <-
        binding_by_name["Week"] = b,
        attribute_by_name[b,_] = a.

    That being said, we do plan on improving the schema to make things as brief as possible. Key indexes are an example of one area we'd like to improve sooner rather than later. For the majority of cases, we should be able to determine key order for the user, except in the few cases where the keyspace of a predicate has multiple keys of the same type.

  3. Arnaud Sahuguet 7 years ago

    The BitBucket repo says “Access denied”.

  4. Shan Shan Huang 7 years ago

    Hi Arnaud, sorry about that. We are working on opening up example repos to the public, but it’ll probably take some time. Please stay tuned!

Leave a reply

© Copyright 2020. Infor. All rights reserved.

Log in with your credentials

Forgot your details?