The Pattern You MUST Learn in .NET
Use code MODULAR and get 20% off the brand new "Getting Started with Modular Monoliths" course on Dometrain: dometrain.com/course/getting-...
Get the source code: mailchi.mp/dometrain/032sfebfijs
Become a Patreon and get special perks: / nickchapsas
Hello, everybody, I'm Nick, and in this video, I will introduce you to the Outbox or Transactional Outbox Pattern in .NET. I will explain what the outbox pattern is and how it works, and we will see how we can use the MassTransit library to implement it.
This video is sponsored by AWS
Workshops: bit.ly/nickworkshops
Don't forget to comment, like and subscribe :)
Social Media:
Follow me on GitHub: github.com/Elfocrash
Follow me on Twitter: / nickchapsas
Connect on LinkedIn: / nick-chapsas
Keep coding merch: keepcoding.shop
#csharp #dotnet
I want to know what is your other approach to doing this!
+1
The "listen to yourself pattern" is more scalable in my opinion
Another way, back in the day you could have another program you run and register a row change listener in SQL, you can receive an event any time a row in a table is added/change/delete. This registered call back could place the "outbox message" into the outbox table for example, and then i've see signalR used for pushing events to subscribers.
+1 I normally use cdc direct from DB transport events to message bus or queue, also good when you re happen to change the data directly on the DB you want miss the messages
@@ahupond Listen to yourself works, provided it does not cause timing issues with a consumer knowing about the message before the producer. If the Db supports it another trick is to listen to its transaction log, and write to your outbox from that. The issue there is that your code that knows how to create the outgoing message has to be able to interpret the outgoing message from the Db transaction. The reason why Outbox is so common is it doesn't bump into these issues.
"Hello everybody I'm naked" once you hear it you can't unhear it 😂
what have you done to me
Hellovrybody
Clowns 😂
I have heard about this pattern twice for past 2 weekends on interviews, great topic
Hey Nick, great video as always! One thing was a bit unclear to me, which was the total message flow, so I did a little research. From the video it seems that it is only one pattern you are using and that both services still need access to the same database. But in fact, you are using two patterns: the outbox and inbox pattern. The message is first saved to the outbox in the database. Then behind the scenes, MassTransit picks it up and sends it on the message queue (and removes it from the outbox table). Finally, the message is received by your worker which will save it to the inbox table. Then the worker reads from the inbox and processes the messages. So the worker and API can be running as separate processes (on different nodes) and they can work with a different database. The worker only touches the inbox table and the API touches the outbox tables. Correct me if I'm wrong, cause I only did a quick 5 minute research to clarify for myself :)
Hi Nick, thank you for sharing. Please give us some more insight on your experience dealing with the alternative method for the outbox pattern.
Excellent video as always. I'm interested in hearing about your other approach that you mentioned at the end of the video. Cheers
Love the abstraction over message queuing that MT offers. And this feature you've demonstrated is definitely something I'm keeping on the radar. Last project that had something similar was using MSSQL ServiceBroker, but that was ages ago and that technology has become a long forgotten relic. Very cool stuff, Nick.
Awesome video on Information Systems. I think this was a great introduction to the development side of Information Systems. I think you touched on it, but did not say anything specifically, but what about Event Sourcing. And I vote yes on a video detailing your alternative method of solving this problem.
Love videos like this - explain useful pattern and show a good way to implement it using OSS 🙌
Thanks once more, Nick! Very interesting pattern! I want to know about the other approach using native cloud products!
I'd love to see the "Nick" approach to this, you tease 😊
This is nice. I solve things like this just with a Task / Thread Process in the background and a messagequeue as a command/message store to process dependend tasks like this. It is simple and easy and only as many features in this construct that i need for my problem that i have to solve. But i really love this content, because it validates my way of doing things, if big companies walk same ways in a bigger radius of problem solving.
Interesting to hear about pettern for reverse situation. When you read from queue, and something happens with database, or the next queue sending
Love you're "very scientific tool" as you mentionned :D
Yes, please also show your approach to this problem.
This is a very informative video. Kudos!
Glad im not the only one using paint to explian things! Brother id love to see a video on the other approach you mentioned to solve for atomic operations over a database and queue!
A common issue with the polling worker is that if you scale your app you would then have multiple workers processing and publishing the same messages. That's why people would usually process the outbox before acknowledging the original message. If the publish fail then that original message would retry, you would see that an outbox already exists and then only process the outbox instead of the entire message.
You can deal with it by adding a field like "ProcessingStartedAt" which will prevent messages from being processed by other workers until the one who picked them will complete processing and delete the messages. Having a date instead of bool will allow you to re-process messages which became stuck because of a crushed worker.
One question regarding the worker. Considering the behavior we're trying to solve is usually part of a microservice architecture, or some fully separated systems, doesn't the fact that the worker needs access to the outbox kind of defeat the purpose? What I mean is that if I have access to the outbox, I have access to the DB. If I have access to the DB I can create a trigger for example in the DB, or a scheduled task that gets the info and handles it. I don't see a purpose of the queue in these cases, because I can just create an artificial one locally.
I think the idea is that you create another service that has access to both DB and MessageBroker. It reads data from the outbox and passes it into the queue. Other services consuming from the queue don't need connection to DB since they can use message acknowledgement to avoid duplicate processing of the message.
I think the example in the video was a bit confusing but that's not how it works. MassTransit automatically configures a Background Service in the API that polls the Outbox and sends the messages. The consumer application does not change, it still reads from the queue.
If you're using the database as the message queue/broker, you should be aware of SQL Hints, particularly UPDLOCK, HOLDLOCK, and READPAST. UPDLOCK will make it take the necessary exclusive locks to modify the row, HOLDLOCK will make it hold the locks until the end of the transaction (e.g. shared locks tend to get released otherwise), READPAST makes it skip rows that are already locked. Combine this with a SELECT TOP 1 read, and you've got yourself an easy and fairly robust (though not idempotent) queue. Can run as many competing readers on it as you like (to diminishing returns), and they won't step on each other's toes. That all said, I'm sure MassTransit has stuff built in to deal with any idempotency concerns.
I don't think the worker has access to the DB in the example
consumer reads messages from the queue (not db). It can use inbox table for duplicate detection feature and it can have it's own database.
Hey Nick, Could you please tell us your favorite way of solving this problem, And also tell us about how we can guarantee message is handled only once in the consumer? Also wondering how to preserve correct order of the messages.😊
There is no need to configure the consumer side. Doing so only makes this example confusing (as people are already asking if it uses a database instead of a queue). The consumer can be also configured to use an outbox pattern (which actually combines inbox and outbox), but it's not obligatory.
Another fantastic video, definately interested in the other option.
Thanks for the video!
Awesome! Thank you 😊
it is worth noting the trade-off here. By assuming you are on EF Core, Mass Transit can avoid you needing to inform it about the "unit of work" that the Outbox needs to participate in. For Brighter we take a different view, and support multiple approaches to your DB: Dapper, EF Core, DynamoDb etc. but you need to pass us a unit of work. We will create this for you, via config, so that you can just use DI to inject it into your code. You then use the unit of work to create the transaction for your Db and the publish to the queue. As ever, trade-offs, but it removes the hard dependency on EF Core for us.
We have a legacy system that implements something like the outbox system. Some action will write a message to our custom outbox and some separate background app will check that outbox and send the message. But it doesn't have retries and no idempotency.
Atomic vs Self-healing is always a fun conversation. I've had good success with self-healing designs where atomic was not feasible. Could you do a video on self-healing design patterns?
A good follow-up for this would be a video detailing the Inbox Pattern, for receiving messages.
I'd be interested in know what you have actually implemented and used in practice for you previous requirements.
Hi Nick, very nice video! One question: In api methods that work with related entities(aggregates), when for example I want to update all data in related in main and related entity, is a good idea to pass the related entity id or should get directly with the main entity id all the childs and update then? Sorry, maybe the explanation is not the best... The question is if should I send directly to the api related entity's IDs or should I obtain them from the main aggregate? Thanks in advance.
So far I used Azure Functions but this looks interesting, I also wanted to know what other approach do you suggest, thanks for the great turor
Hey Nick, I would like to congratulate you on another excellent video. I have a question regarding to the need to add EF Core packages to Consumer. The way I was assuming it work is following: - publish push the event to the outbox table - a background process (on the publish side) is responsible for scanning the table, pushing de event to the queue, and the update de status of the event row as processed - the consumer still consumes messages from the queue with no changes needed What am I missing here? 😅
I use outbox all the time. What's your preferred way to do it? We use masstransit, SQS, RMQ & rdb but not the built-in masstransit outbox.
Please create a video for your preference for dealing with this issue, thanks
I've been seeing this pattern a lot lately. Many teams implementing it at work. What is the other approach that you suggest (besides two-phase commit, obviously)?
Yes I’d really like to know your preferred solution to this problem. I’m not a big fan of this approach, it smells a bit “automagic” - i.e, it isn’t obvious when looking at the service code that the queue message isn’t published until the save changes transaction. I can imagine picking up this codebase and scratching my head as to why the message isn’t in the queue when it looks like it should be.
In this approach is there a behind the scenes polling of the outbox? If so, that could be costly from a cloud perspective. I've seen patterns that inert into a db, then add a service bus message with a reference to the record in the db, then ack to the client. You need to have a cleanup process if there is a failure between db and service bus, but there would be much less db polling. Not quite as elegant as the Listen To Yourself pattern, but it works with large messages that might not be suitable for service bus
Yo dawg, I heard you like queues so we added a queue for adding messages to a queue so you can add message to a queue while adding message to a queue!
literally. I am looking and thinking, wth? The whole point of MQ was its simplicity of use, even replacing a MQ engine is not such a big deal that it reauires this massive MassTransit abstraction on top of it.
@@MilYanXo Yeah, the problems you run into using MassTransit for a while are not worth it in my opinion.
Hey Nick, you mention the sponsored AWS course is free from the link in the description, but there is no link? And the course has a price tag on the website? Interested, thanks!
Thanks bruda
12:32 I am confused about how id created and is passed to outbox table which was done before save changes. Does the masstransist handles to update id and send to outbox table?
In EF you can implement interceptor for SaveChanges operation - so this is how MassTransit, most probably, does that
@@ealeykin got it thanks
one of the things I learned is that not everything has to be programmed. There are many automation tools that can be spawned in a docker container to do this kind of background processing. In this case you just set a deleted flag on your record, then your cron job will spin up and do all the extra processing like email or clean up without having to rewrite all that code yourself and bloat it.
I think I like the simplistic of save changes the fire the message. Having a relay poll the database and then enqueue the message seems complicated and oddly familiar to using the database as a queue. Maybe my problems aren’t enterprise-enough to warrant this.
But in enterprise scale of micro services each service has its own database and common queue. So it is not the same as DB will be pulled only by one service just to publish messages to the queue while in your solution all services would need to access single DB and pull it.
Defo want to hear the other solution, currently building this very thing at work and we lol'd becuase your released the vid as soon as our meeting about it was done
I'm really spooked because this literally happened just after a meeting I had which spoke about the outbox pattern
@Nick, is it possible to use with LiteDB?
I like to let you know that i am interested to learn how you solve that problem without a outbox. Is it the build-in transaction-support in some messaging frameworks (i think azure service bus supports that).
I am so excited to read this comment
I too would like to know the alternative approach, especially if it's cloud centric.
How does duplicate message detection work, particularly if the subscriber has many instances and does not have access to the DB?
The subscriber must have access to the database
But what happens if there is no consumer? Is there a timeout and api returns error?
I'm also curious about the other approach
Please let me know how you are dealing with distributed transactions.
Would you send the message to the queue AFTER saving the record to the database? What if you get a DB exception and the message was already propagated?
At 16:37, you use AddAsync() to prepare adding a customer to a database. Why? Normally you would only need to use the Add() method unless you have some reason to use AddAsync()? The documentation says as much (e.g. SequenceHiLo).
I want to know your favorite way of solving the problem!
I've not used the outbox pattern so this is interesting to see in action. My gut reaction is it feels complicated. What would be wrong with publishing the messages regardless and relying on the consumer to validate the message before continuing? You should do this anyway for a future-dated message in case the underlying entities have changed in the meantime. The same can logic applies for an immediate message. I guess the risk you have is occasionally getting messages in the queue which are not going to lead to anything, but hopefully we're talking about edge cases here and not failing transactions as a flow control.
As I understood this, the main idea is to save changes to the db and not loose the message. Without outbox pattern you may have the next problems: 1. If you publish first, db save changes may rise an exception, but the message already sent so undesired behavior. 2. If you save changes first, on the next step message broker may be down or the entire process down, and message lost in this way. Both cases can be critical, that's why some solution is needed. So the thing is not related to consumer, it's only about atomic saving and publishing. Hope got it right)
7:30 When you said "Don't log like this", to me it was slightly unclear what you were referring to. Do you mean we shouldn't have log lines that change the semantic behavior of a function? Or do you mean abusing the message queue to log something that the creator of the message should have logged? It's hard to tell exactly what you meant, and I can think of multiple reasons why it might not be a great idea.
He was referring to the concept called "Structured Logging"
He meant don't use the object .ToString() as a log message template. Why he didn't just do it "right" is beyond me though, would have taken less time.
7:25 what is the right way to do the logging?
Nick, with the pattern you ensure the message is published exactly once, but how can you make sure the message is also consumed exactly once?
It includes a "lock" column which should be enough for queries to ignore it if it's not filled with zeroes. I didn't see it but it must also have a "processed" column or something too.
The message queue should support locking so a message can only be consumed by a single subscriber.
He didn't explain this. From what I read, one solution is to save message id's to be able to identify already handled message. If already handled - then skip.
Too bad that there is no easy way to stop listening for messages from the queue. Or maybe there is? What if my app received some message that starts up some heavy process and I don't want to listen on new messages like this from the queue until I'm ready?
Idk if I miss something about DI injection magics but who guarantees that Publish will work after inserting to database? Isn't the same problem as the beginning?
I believe they're both added to the changetracker which EF will ensure happens atomically (they either both are created or neither is). So it's not one then the other. It's either both or neither.
The Publish method is hijacked behind the scenes and it turns into a database operation that is part of the transaction
Thanks! @@nickchapsas
Could someone clarify why there is a service that reads from the outbox table and pushes the message to a queue to be read and dealt with as opposed to just resding from the outbox table and dealing with the messages at that point in time? IE read and process directly from the outbox table and cut out the message queue.
Other microservices probably have its own DBs. Still if you have a single DB for all then you need to pull it even more often because each service will be pulling it. And you loose built in support for topics etc for queues.
Anyone else not getting the email when registering in the dometrain course?
use function app with cosmos db trigger, cosmos db is your outbox
The only issue I can see consumer should have access to publisher database or outbox configuration should be enough on publisher side only?
It doesn't need to have access. It depends on your setup
Why does the working need to use DBContext? didn't it just listen to sqs and process?
Pub sub is the best. I demo this in paint for our new hires.
How would we configure something like this if we were trying to queue multiple transactions between multiple microservices, each one independent of each other... I wonder... 🤔
The outbox pattern is applicable for some use cases, in your example it is not the case though. You added extra complexity to your design and pooling in the database. A DTC with retries would simpler, atomic and efficient.
It would be great to hear how else you deal with this.
Is there a way to do this in DynamoDB? Does EF support DynamoDB? As I saw there's still no support for dotnet8 in AWS Lambda! :(
Theres something really close to EF for dynamo that I used its called poco dynamo, and you could use cdc + lambdas in dynamo instead of this.
Brighter supports an Outbox for DynamoDb
How does this compare with Sagas? Or is it an implementation of Sagas?
Saga involves taking counter measure if the message has not been delivered is not it?
Hey Nick, I wan't to hear your preferred way of dealing with this problem.
Why does the Worker also need the Outbox pattern and Entity Framework as a dependency? Please watch 15min.
Why use await to publish to the queue?
I wanna know your preferred approach very much
@Nick Chapsas - I would like to know your approach, please.
When you need to write the message to the DB - why not simply pull the DB via an IP or push some notification to a consumer of the "mesage" service that sometging int he DB has changed
Yes, I wan't to hear your preferred way of dealing with this problem.
But does the worker has to connects to the same database?
The worker doesn't need to be aware of the outbox. There are multiple types of setup you can have and one of them is for the worker to be agnostic of the db stuff
thanks for replying@@nickchapsas, I really liked it the concept and started exploring it in a demo project.
13:52 Somebody knows what console it is?
Does somebody use MassTransit with IBM MQ? Doesn't seem like it is supported :(
Why there are three tables, and what is inbox?
What is publishEndpoint? This was not explained. It can't be any custom object.
Its so funny to see the resurrection of MSMQ spinoffs now that cloud providers decided to sell (err, rent) them. I've been there 3000 years ago when MSMQ with DTS was used for all this and we didn't call it outbox pattern :D
whats ur approach?
Did I get it right that it uses db as a synchronization source instead of messaging queue? Can it be done without using db? As I know message queues have their own persistence layer, so the messages can be saved to a file and then be restored in case of failure
What if request modifies data in your database which other parts(microservices) of your system are interested in?
@@dy0mber847 Not only that, but I'd rather write to a database instead of a file... those are locks and queues I don't want to get into.
File scales better than db. In microservices you have to create a new table for every db to make this happen or you get into dtc. And scaling reading from 1 table transactionally is very slow. I was originally doing this via triggers and the inserts were fast but i couldnt get my transactional reader to read/queue/commit fast enough. Instead, when queue write fails i write to disk with a prefix of the app name so that later i can rename it then read it and resend it to the queue. This is much more consistent during networking or rmq outages.
Thanks for covered topic, but for me seems like overkill. Interesting how everyone deal with this problem, from my side I don't see any problems with creating transaction, save changes, publish message , complete transaction. Interesting about other opinions.
What happens if after you publish the message but before you complete the transaction, any of the following happens: a network outage, app pool recycles, hardware failure occurs, or web process is shut down?
@@TheMikernet Got it, thanks,I understand the problem. To implement all this staff you need background worker which read DB (possible bottleneck) and push messages to queue. And background works can have the same problem (network, hardware) so we have to add some logic to prevent problem with background worker and logic to prevent duplication of messages in queue. Maybe it's better avoid using the outbox pattern and just add logic to subscriber of this event where we check that operation completed. Or if it is so critical to use Service Bus transaction. Thanks why I think that this pattern is overkill.
The point of using MQ is to not have to poll a table. If you have polling, why use a MQ at all?
I write to a table have a status column and my service reads from it :/
What about Debbezium or CDC ?
The very idea that there's a line of code that says Publish and yet nothing seems get published after it's execution bothers me so much, there's so much going on that is invisible to me that it seems like I would never use it.
No... I Will not mix messaging with my application data layer.
It's a resiliency layer. The messaging is still handled by a messaging service.
@ I am not concerned about how or what handles the service but the implementation. If you watch the video, you will see that the implementation becomes part of your data access code. Nick briefly makes an excellent and concise explanation using a diagram. I prefer a separate service to handle messaging; it is not part of my application and deploys separately.
@@2SHARP4UIQ150 I see your point but there's probably not a good way to create a transaction that would affect both the data and the messaging service. The goal is to ensure that either both or neither of the writes happen. So yes, messages are temporarily (probably for less than a second) saved in the database along with the other data. It's a trade off but makes sense to me.
TL;DR - Have your services store messages in a database table and then have separate service that sends them from there to your message broker.
The videos where you are introducing us to libraries and all that is great. Outside of that, in my opinion, there is a consistent lack of disconnect between your intent and what you are really trying to accomplish through pattern-type videos. You need explain a lot more technical background via system design and diagrams without even touching the code. Of course, this is not to a suggestion to make hour long videos. At least share system examples you have worked on and why in the hindsight you think you were right or wrong. I do not really care you talking about food delivery service or ride-hailing service. That would be so because you have not worked on them. Any heavily used enterprise system would pose massive technical challenges, so there should be enough to discuss on that itself. If only seeing through the complexities of the system would be easier.
Is there a reason you use insomnia now instead of postman?
want to know your other approach
using a side car? dapr?
i guess he uses Event Sourcing
Registation for the AWS Course doesn't work, mail isn't send.
I can see that some emails are stuck. Will try to fix it
Fyi I do get other mails from dome train. Address is nothing special, just Gmail.
The issue with this approach is that the publishing is completely decoupled from the context
What do you mean? The DbContext?
@@nickchapsas yes. You're using the DbContext in the method, but Publish is using it secretly. What if a context is instantiated in the method?