One of the challenges of working at Procurios, is that we have many similar-but-not-same customers. Case in point are the three Dutch political parties that we host at the time of writing.
We have a rule of thumb here: 1, 2, many. It's ok to make features for specific customers, rather than for the product. It's ok to make a similar feature twice. But when we have several instances of similar feature requests, then it's time to make a proper feature for the product. While onboarding our third political party, we ran into a "many".
TL;DR
Event sourcing makes significant model changes harder, so understand your domain when you use it.
Membership
Political parties in the Netherlands share a similar structure when it comes to membership. If you are a member of a party then you may already be aware of this. When a person becomes a member of a national party, they automatically enroll in a hierarchy of ever more local departments. National, provincial, municipal, local, neighborhood... You name it, there is a party out there that does it. One party bases this hierarchy on postcodes, another on towns, a third on both.
So imagine you live in lovely Wijk bij Duurstede, Utrecht, Netherlands, and you wish to join one of these parties. You will likely become a member of the Utrecht division too, and of the local Wijk or Heuvelrug area department as well.
Many things depend on this membership concept. Payouts of contribution (retribution), voting privileges for party congress, data privileges for party employees and volunteers, personalization of mailings and other forms of contact, etc. Having access to this data is vital for the core business of any party bureau.
Why we chose event sourcing
There were several good reasons to pick event sourcing for this problem:
-
Time and order of events plays a major factor, something you get "for free" when event sourcing
-
Memberships are time dependent, as people subscribe, cancel or move
-
Organization subdivision is time dependent: the Netherlands merges a couple of municipalities every year, and then does a major reshuffle every decade or so
-
History is important, not just the current state of affairs, for both reporting and accounting purposes (subsidies and retribution are calculated and paid out months after they are set)
-
Business intelligence options on the history of memberships, especially related to marketing strategies and events, could become a major defining product feature
There were also less good reasons to pick event sourcing for this problem:
-
We just finished v1 of our event sourcing infrastructure and didn't have a good case for it yet.
So let us point out, before moving forward, that we have not discovered anything wrong with event sourcing, just with our choice for it.
What we created
The module that we created to tackle this problem has three major features:
-
Managing the structure and rules of an organization
We chose to create a hierarchical representation of the organization with a rule-engine powered, inheriting set of rules. The local department would inherit the rules of the greater organs and determine its members and the memberships then bubble up. -
Broadcasting that structure to other modules
We already have a huge product that offers everything that "The secretary of the local department of Wijk bij Duurstede" could need. It just needs to be configured automatically, rather than by one of our consultants. -
Tracking organization and sub-organization (organ) membership
Now that we know a) what it means to be a member and b) what should happen with members in other modules of the product, go track some members!
There are all kinds of events in the product that lead to a potential membership. All we now have to do is listen to those events and fire up the rule-engine as needed.
What we learned
The first thing we learned, is that we were quite wrong about time in this domain. You may have already come to the same conclusion. We aren't sure why we didn't see it up front. But there is a big difference between time being an important factor, and time and order of recording being important. In other words, the recording of events into the system differs from the actual occurrence of interesting events happening in the domain.
The nature of our solution is highly abstract. Using the rule-engine to determine membership, membership events rarely if ever follow directly from other events. And even if it follows directly from other events, it is not dependent on the recorded time of those events, but the interesting time those events speak of.
For example, say we configure the rule engine to place members in Wijk bij Duurstede when they both live in Wijk bij Duurstede and have a paid subscription to our party. The moment that we know someone wishes to subscribe is far less interesting for the purposes of determining membership, than the start date of said subscription, which is often January first, April first, July first or October first. For the purposes of business intelligence it may be very interesting, but that's not in scope right now.
The second thing we learned is that not all rules are hierarchical. We have discovered that in the cases of our clients, we can distinguish between two sets of rules:
-
Rules that determine how the organization is subdivided
-
Rules that determine conditions under which a person is eligible for a type of membership
We discovered this when we tried to cover a new feature, partitioning newsletter subscriptions, with this module. Someone who wishes to receive national and local mail, but not provincial or municipal mail, is not part of a hierarchical structure. The "membership" does not bubble up and the rules don't inherit. The subdivision rules do, but the eligibility rules don’t.
The rules also change for different reasons. Eligibility rules change when the business changes. For example, when a new kind of subscription is offered. Subdivision rules often change under outside influences, such as the Dutch government merging municipalities. We had not looked for these events sufficiently.
What does this have to do with event sourcing, you ask? Well, we now have a client with 2 membership types and ~500 organs, giving us about a thousand rule sets. We would like to introduce our new understanding of the problem and solution, for which we need organ types, and 2 different sets of rules. 1 For the organization subdivision, for 500 rules, and 1 for the memberships per organ type, which for 5 organ types and 2 membership types would be 10. This would make an odd 500 rule configuration events superfluous, and 10 events missing.
It may very well be possible to fix this and still use event sourcing. There are promises of fortune in the future, with the recording times of events providing us meaningful insights in how the business operates, not just performs. But right now it costs us a lot of time to fix our poor understanding. Migrating a state to a different state is much easier than migrating an event stream to a different event stream (which could mean amending it with correcting events, or even rewriting history).
We were not ready to settle on a solution model. We needed more time to discover if we understood the problem. We needed more feedback before committing to the current solution. What we need right now and in the upcoming time from this module, is the ability to try a few things, and see how they pan out.
So we're dropping the event store for this module. We're currently merging several of the projections and will come out with a module like most others we have.
Will we use event sourcing in the future
Most certainly - but we will be more hesitant, and require a greater promise of fortune within the time frame the module should pay off.
Thank you for reading! If you have any experiences with event sourcing you want to share then please do so in the comments. We're very curious what you think.