"I NEED data from another service!"... Do you really?
How do you share data between services? This is one of the most common questions when you have a system composed of many different services, and each service owns a data set. For example, you may need data from another service for validation purposes, or you may need to send data to another service as a part of a workflow. I'm going to explain a few different situations where you could share data between services, but more importantly, when you shouldn't.
🔗 EventStoreDB
eventsto.re/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to a private Discord Server and any source code in my videos.
🔥 Join via Patreon
/ codeopinion
✔️ Join via KZhead
/ @codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
0:42 Workflow
4:13 Reference Data
8:00 Example
#softwarearchitecture #softwaredesign #microservices
Several years ago, a senior software designer told me about Pull-Push mechanism, up to now that term still sticks in my head.
One thing I like to think about with caches is to think of an "ideal" refresh rate and an "acceptable" one. We try to refresh at the ideal rate, but things don't actually start to break down until we go past the acceptable one. For example, when we had an ecommerce site that had to deal with exchange rates, we would update daily, but it was considered acceptable to use values up to 7 days old.
Thanks for the comment and good way to wording it. Which I would of said it like that!
So relevant. Last two organizations I’ve been working with the last two years have had this issue. Both of them had workflows with this issue. Tying availability to several network separated components, increasing load on these components and creating difficulties when considering retries.
Great content with explicit explanations. I love it. Thanks
This video contains a lot of good information. It's so easy for me to recommend your channel to others. Please keep up the great work.
Much appreciated!
You're in Canada? I never would have figured that oot. ;) Great video my man, seriously.
Great video and I like the new studio setup and lighting :D
Thanks!
How do you handle new services that need reference data, but missed out on the events? Would be nice to see a video on that.
I would have an endpoint on the service that provides a way to request the data. On service start up it can ask the other service for the reference data and then start listening to events. This can be an internal endpoint only not something that can be hit outside of the private network.
What I do is create these events artificially and point them to this new service. Obviously, there are some caveats to that but works without much thinking 90% of the time.
What we usually do in sophisticated event driven systems is that we tend to use event/message brokers with acknowledgment systems. This improves the consistency and stability of the system in 2 ways. The first is that events which get acknowledged are locked for the service or removed from the queue (depends on the setup of the system) so that they don’t get processed multiple times in the same way (for example by different instances of the same service if the service should be scaled). Second is that if events don’t get acknowledged within a certain timeframe (maybe because a service has an outage) they can be pushed to some sort of dead letter queue. If the service comes back to life, it will check the dead letter queue of the message broker first, process all events in there and then continue with the regular subscription.
I replay them from the data lake
@@maximilianbeck2301 Sounds like a nice feature, any AWS alternatives that you can think of? Or any recommended brokers in general?
Can you extend the video where ahipping options depends on the weight or size of the order. Then after that, also define rules where the price of certain shipping methods depends on the total amount of the order. Finally define a reasonable api for a UI to display to customers to nudge e.g. large order to get free shipping. I know this is a mouthful but it'd be cool to go a level deeper in complexity
Thanks for the video, again, this channel makes programming better! How do you deal with a situation, when you need to get data from a service, but depending on some conditions, owned by the other service? e.g. I need all the orders, shipped in a specific date range? This might require require large amount of data to be transferred between the services.
It goes back to the question of why do you need this data. If its for some type of query composition would be my assumption, check out this video kzhead.info/sun/fLCbmq-KaHiOops/bejne.html
Nice information. Having one scenario in my project like promotion, tax computation during invoice. In this I’m having dependency with product, price, customer info, promotion definition, inventory,tax info etc. Keeping all in cache is best if some reference data huge ? Your view please
Hard to give an answer because there's a lot context needed, one of which is where is the initial request generated from. Check out: kzhead.info/sun/f7GEht6frJp8qmw/bejne.html
Interesting video. How do you ensure that when the event from "ordering" reaches "shipping" all the "shipping" information is already in the "shipping" database? Could you have a race conditioner there, or the "client" is orchestrating the first steps(generating ID, saving ordering data, saving shipping data, and finally placing order) in a synchronous way?
The event would be deadlettered by the shipping service because it doesn’t know the id. The queue can be configured to automatically resend the event later (until a preset number of failures is reached, at which point it is dropped into a separate queue for manual processing).
Great video!! So if we are having user-service which manages login/sessions and user info and there are other bunch services that are dependent on user-service for user info, does that qualifies the user info as a reference data for other services?
Problem is what's "user info". Is it their name? Payment information? etc. Auth can handle login creds, but payment info.. not so much.
How would you handle situations where some servers are down when calling multiple endpoints from the client in your data ownership exanple? For instance, while the shipping service is down, I would expect the ordering service to continue working. However, the approach you described would result in a failure of the client's ordering process and may require compensacting transaction.
Not following. Why would there be a compensating transaction needed? If the shipping service is down and not processing the OrderPlaced event, it will happen when it comes back online. If you have some process timing requirements around that, then you'd likely model it differently using timeouts and/or async request/reply or some type of orchestration.
Leaving this flow to the customer won't expose things it shouldn't? The client could not send the information to the shipping or send any other id there, wouldn't it break everything?
Considering your last example but from the point of view of the Credit service. Imagine that this service is responsible for calculating the score based on Sales service. Should it store the data of sales ahead of time or directly call the Sales service in time of calculating the score?
Depends on your concerns around availability and coupling. If credit service only needs all sales data once per month, then it's not concerned about availability. Loosely coupling between the two via events or even an async reqest-reply is still an option though, it doesn't need to be a sync request-response even if it's just a limited number of call per month.
@@CodeOpinion Thanks for the thoughts for consideration
At 04:20 - 04:30 you're talk from another video. Do you have the link to that video?
Here you go: kzhead.info/sun/fLCbmq-KaHiOops/bejne.html
Thanks for the video! I'm thinking about a provider agnostic payment micro-service. Let's say I want one place where all payments flow through and the service returns either payment success or payment failed result. Different providers require different information, for example: -Stripe requires card token information -PayPal requires order item information In your opinion, would it make sense to setup separate services such as a Stripe integration gateway, PayPal integration gateway, provider x integration gateway etc. Then pass something like the order reference through the payment microservice, and allow the specific integration gateway to use that reference to gather specific data required from other services before making the call to the external payment provider.
Ultimately yes, you want specific integrations for each individual service. Those services could define commands/messages (API) that is gathered higher upstream
What about having one or several services or DBs involved in, let's say, customer information? Might that be a scenario when you should not should not share or replicate data across consuming services? I would think that some data is better fetched on demand if the costs is low.
It's always about use-case for me and how that data is used and what type of data it is. If you want to fetch data "on-demand", what are the implications if you can't fetch that data is the question. Also, how are you versioning the APIs for fetching this data on demand.
@@CodeOpinion Thank you. Exactly. What the implications are is important. Not the least, how it impacts the development of the system as a whole.
We have this exact scenario at my current company. A customer service with numerous internal clients depending on it. Initially we were making synchronous calls between those clients and the customer service, but suffered system wide outages whenever there was any issue with the customer service. We have since moved to the approach described in this video, and have those dependent services caching the customer data that they need. Relying on eventing to update when things change. It does remove the single point of failure that we had before, but does introduce a little more complexity and the possibility for data to get out of sync.
CodeOpinion design architecture for real chad
What if the data is volatile and we really need to get the data from another service? Only rpc and request response?
Then that's what you need to do. Especially if it's a 3rd party that only supports that. Use async messaging where appropriate, use RPC where appropriate. Understand the trade-offs that come with each.
Let me see if I got it, imagine that there is one microservice (A) that consumes data from other 10 microservices (B, C, D...) . In this case there would be 9 different caches in the context of microservice A? Is that?
They don't need to be separate/different. How you persist them is going to be dependent on you're read/write/expiry needs.
But shouldn't a service be stateless? Doesn't a cache make it have a state?
That statement is referencing that the applications process/memory itself doesn't have state. That's typically for scaling out purposes.
No Code example ?
what is wrong with just reading from the database
Great points but I strongly believe the cache for exchange rates should be maintained by exchange service team and not order service team. Order service team really should not determine the data quality of exchange rates from another service
Curious, if you needed to store the exchange rate with an order, the exchange rate that was used. You think that's different? Meaning you'd query the exchange rate service for backdated orders too?
@CodeOpinion not for backdated orders as the oder rate is already determined. What I meant was for the new orders. Take scenario when business determine to consider exchange rate from real-time forex platform and not from a bank tomorrow? Will the order service or exchange service team know about the business change?
@@hrtummala The exchange rate service was 3rd party service in the example. So they would not know anything about what rates your business wants to use. But if your business wanted to use real time rates from 3rd party service then the business has to just accept the losses from lost orders when the rate was not available. Or update the rate every hour or every 30 mins, those are still highly cacheable update rates. Also even if you do sync call every time, the data is stale as soon as you get it, so what is need for such a fast updating real time that cannot be cached?
@@hrtummala Btw forex real time is only updated every 15 mins, so instead of caching it every 1 day you cache it every 15 mins
A good way too accomplish this separation of concerns is to let the exchange rate service publish rate change update events that subscribing services keep a local copy or cache of. The result will be that the client does not have to worry about cache invalidation as it becomes the concern of the publishing service.
I would like video on a pattern I loathe - Central Configuration Service. The stated reasons are often: - I need one place to change everything - Avoid duplication of "configuration data" - Usually a clickity-click GUI (git/svn is too difficult) The real reasons and issues are: - Afraid of making changes - commit, build, test and deploy - so we need a way to bypass all that "structure and rigor". - Overly complicated processes, bad culture, CCB, CAB, ST, RM, etc... - Not knowing where to place some set of "configuration data" (ownership) and lack of business domain modelling. - Not understanding the difference between static data, environment, deployment and runtime configuration. Conflating it all to "configuration/metadata", just like the common/utils anti-patterns. - The services need to integrate towards this CCS API that becomes a single-point of failure at runtime. - Then we end up having to implement audit, access-control, versioning, undo, and structures and in the end re-invents a graph database in MongoDB, that we now also need to have backups of. - And finally we loose control over which services has access to which "configuration", in what format/structure/schema because it's just a centrally shared database, but hiding behind an API, and no-one owns "schema". My take: 1. If we have some set of "configuration data" that is static, smallish and changes a few times per month - just put it the same repo/binary/package; commit, build and deploy as needed. 2. Environment configuration can go into a central system, but I think it should be the deployment/provisioning systems job to provide that to the applications at deployment/provisioning time, even if it has to generate yaml-files from it. And the source-of-truth shall be version-control. 3. Some applications may benefit from being able to fetch "configuration" at runtime - but is it really configuration then and not a missing abstraction/service in the architecture? 4. Feature flags and stuff like that should be pushed to the deployment/provisioning system that then provides it to the applications. Whether that is via API, config-files, env-vars, hot-reload, or restarts is context-dependent. 5. Application system-configuration (think tomcat server.xml) shall have a optimized default built-in production-safe in the package and any values that could be changed shall be provided to the application by any of the means mentioned above. I would love to hear what your thoughts is around this?
Ya, good suggestion for a topic! A lot of it I agree wit what you wrote with some caveats.
What about a scenario where you have a business rule that must be enforced in 2 different boundaries/services? For example, we have Orders and Subscriptions boundaries. Whenever you buy something, what you are buying is a Price. If the Price has a recurring frequency, then, after Ordering, you create a Subscription of that Price. Well, theres a business rule that says that a subscription can have more than one Price, but those prices must be of the same frequency (for instance, two Daily Prices, or two Monthly Prices, but not a Daily Price and a Monthly Price). Orders, whenever you create an Order, you have the exact same requirement. So, it seems to me that we need to have Price in both boundaries and the Frequency. In this case I cannot have a cache... I have business rules to enforce, and the business rule is the same. Whats the better way to approach this? I was thinking about making the Price aggregates in both boundaries but only allowing one boundary to change the Frequency, but it still doesn't seem right. Particularly because i want everything to be as async as possible through messaging. But if the frequency changes, eventual consistency could be a problem because the business rule could not be correctly validated in one of the services because of stale data.
The rules for ordering the subscriptions (products) exists within that boundary.. If you want to change your subscription, cancel the subscription and place a new order. That doesn't need to be manual by the customer, but a "change in subscription" can really be that workflow of cancelling the subscription and creating a new order.
Why use cache instead of db table?