Getting started with Pub/Sub Messaging in .NET
Get $50 free AWS Credit by adding "AWS CREDIT #NICK" in the "Tell us about your project" area: bit.ly/nickaws
Get the source code: github.com/Elfocrash/aws-videos
Check out my courses: dometrain.com
Become a Patreon and get source code access: / nickchapsas
This video is sponsored by AWS
Hello everybody I'm Nick and in this video I will show you how you can get started with Pub/Sub or Publisher Subscriber messaging in .NET. In this example we will be using AWS SNS and SQS but you can use everything you learn here with other messaging technologies too since the concept is fairly standardized.
Don't forget to comment, like and subscribe :)
Video on queues: • What is Azure Service ...
Social Media:
Follow me on GitHub: bit.ly/ChapsasGitHub
Follow me on Twitter: bit.ly/ChapsasTwitter
Connect on LinkedIn: bit.ly/ChapsasLinkedIn
Keep coding merch: keepcoding.shop
#dotnet #microservices #aws
I'd love to see more in-depth videos on what it's like to work with messaging. For example, now that you're not awaiting synchronous calls, how do you detect errors in the system, how do you make sure not to end up with invalid state in one service when another fails, how do you aggregate the results of messages you've sent to be worked on in parallel, how different is it to design a microservice ecosystem using asynchronous messaging, etc.
YES, please make a video about FIFO and the slution in a microservice worl
Make one video for Channels in .NET (ChannelReader, ChannelWriter, BoundedChannel, etc.)
Second this, an implementation comparison (comparing with MediatR and/or MassTransit, for example) would be very interesting.
Beside channels it would be also great to create some videos for TPL Dataflow and Akka.NET.
@@branislavpetrovic7486 are people still using TPL?
@@ivaniliev93 Ofcourse, TPL is siuted very well for parallel processing of CPU-bound work and I use it regularly on my projects.
I would absolutely love to see more content like this.
@4:20 Great plug Nick!
Thank you for the great content you offer here Nick.
That's more of the Nick Chapsas, keep going forward!
I can only recommend doing it like so! 🙂 Also the new developers I'm working with currently love it, too.
Great video. Can't wait for more.
Thanks Nick, best aws content so far - the only thing missing is I now it's horrible, but if you could add the cdk logic to have all those roles added etc. we would have more time to watch world cup :) apart from joking thanks mate for excellent videos.
Love to see how you solved ordering with out a fifo queue. I'm doing something similar now by building state time checks to determine if the event is out of date. If it is, reject the event. Would love to see how you resolve it without fifo
Thank you a lot! You helped me to become a good developer
Thanks for the video. I think it would be much better if the subscriber could define the filter expression in client side instead of we should create a pre-configured filtered queue in the AWS. Or is this possible?
Hi Nick, great video as usual, 1q, how is this compared to Amazon EventBridge? is EventBridge an older solution, or a new one, is it a simple way to do SNS+SQS? how about performance, is the custom SNS+SQS better for high scalability or EventBridge is just the same? Thanks!
Great Video but the : Basket added in the item so funny :D
Hi Nick, this is a bit off topic. I looked for a video on this topic in your channel and couldn't find it. What is your view on Reactive Extensions (Rx.NET)? Is it something .NET developers should bother learning or can they, for the most part, ignore it and perhaps focus on other technologies (cloud, Pub/Sub, IAsyncEnumerable amongst others) which might accomplish the same thing in a different way? Thanks
Hi Nick, are you not bothered by having to poll SQS for new messages? wouldn't an event driven option be preferable?
I got three questions for you, 1. How does it ensures a published message is consumed by a consumer only once(consumer on same message for scalability reason) 2. How does it ensures a published message is consumed by all consumer only once(consumer on same message for different different actions) 3. How to ensure a distributed transaction among multiple consumers(so if a consumer fails, rollback all consumer's action), is there any design pattern for that?
Yes. We or at least I would like to know how you're not reading the past in the future without FIFO queue
Thanks for the video @Nick Chapsas I really appreciated. But I see a logic condition filter added into the AWS service. Is that a good practice in real life production apps?
It is if you care about throughput. Client side filtering will bottleneck your services. The subscriber is part of your service
@@nickchapsas Thanks for your reply. It is a valid point. But what if I wanted to move into Azure service bus or RabbitMQ implementation. It's not the same way of filtering for each technology.
Love the video, can you some content in dapr as well??
For example we have all databases, SQS, SNS etc environment setup on the dev AWS account and we need to create and setup all of the things in the customer's environment. Does AWS support import and export of these things ? Or you need to re-create all of it manually ?
The tool you're looking for is called 'Cloudformation'. You can create a YAML file (or JSON file) that defines all the infrastructure you want in your environment and then tell AWS to create all your infrastructure from that file. This way you can re-create the same architecture multiple times. You can also parameterize it for variations between environments.
Please do a video with the same scenario you described here, but with Azure PubSub Service, and compare it with the sns solution
Put me down for wanting to hear what you do to handle message ordering when not using a FIFO queue.
18:18 nice
Really powerful tool and as all powerful tools it is a very easy to turn you project in untestable an undebuggable mess :) Every one how had met Flux/Redux/other inheritor applications would feel the pain, I believe. The only way to control this is the strong discipline inside the team, heh, because the "callback hell" can be met in all realisations of Pub/Sub, not only in Events, sadly
420? 69? -- Nick's testing is based.
"Like you all hopefully" xDDDD
interesting choice of the lifeTimeSpent amount 😈
Showing up at the end of Alyx was a gigachad move. Keep it up Gordon
Looking forward to test this soon. AWS CREDIT #NICK
Can you please make a performance comparison between the C Family languages (C#, Java, etc...)? I am curious which is faster, C# or Java. Thanks.
I am not an expert in Java so I am not qualified to do such a thing
Hey Nick, great content. Just curious, is it proper way of filtering events in SNS? the LifeTimeSpent >= 69 seems like BL code, no? also in term of testing this logic seems to be little harder to test and also you need to keep in mind that there is some filtering happening in SNS, when you dont receive some events in consumers. Thanks
It is totally fine. You own the subscription while the publisher owns the topic. Technically you are leaking it from a DDD perspective but if you normally were listening on a high volume topic then this filtering is needed to ensure good performance. Ultimately I think it’s a balancing act. If performance is fine then keep it in the app. If you need even more then move it to the subscription as a filter.
I was unable to get the $50 credit. The first message I got back from AWS said they didn't have enough context, so they asked for some more information. After I mentioned this promotion, they said that I needed to contact the sales team.
I will follow this up with my AWS contact
Do you really need to restart the consumer after changing the rules (sub dfilter) in AWS?
No you don’t. I just had to wait something like 5-7 minutes for the changes to propagate and in the meantime I stopped it
Hi, i need ef core multiple provider/db.. anyone have references?
nice video again, Nick! I am wondering whether it would be more concise to create a topic per message. That way, a subscription would mean you are interested in that message. If you are not, you don't subscribe. It feels more comprehensive to link the handling of a message to the subscription. Not to the subscription + some filters that may or may not apply. How do you feel about this?
One topic per message is a horrible convoluted approach. It is totally acceptable for subscriptions to have filters in pub/sub. It’s part of their design.
How can you do that without cloud services on a single server? Best way i know for now is to install Kafka in single node configuration but i'm still looking for something simpler.
rabbitmq in docker
@@adrian_franczak Kafka is in docker too, still not ideal to deploy on end user devices to communicate between front end winforms app to backend windows services.
Redis has pub/sub, queues, db, etc
I normally enqueue from a database trigger
Don't want to nitpick but your whiteboard is actually a gray board. In which case it's actually called a blackboard.
Let's bring the white board, and literally brings a black board. Great content. Thank you
@Nick Chapsas makes a video about AWS, which means lots of AWS vs Azure comments are incoming.
Isn't that a blackboard?
How about offline development?
You can use localstack for local development
Being new to this area, to understand slowly, I had to reduce playback speed to 0.75 .. you talk so fast :)
I’ve been doing this for a couple of years as well, but via Azure Service Bus. Different interface but same concept. One thing I really do not like in your example is the fact that you put all the attributes as metadata, In effect increasing your message size by double, an also making the raw content of the message obsolete. I know you can not filter on raw content, but this solution is not a good way to go. I would rather do the filtering in the subscriber instead, where the rest of your business logic would probably be located. Just my 2 cents. Thanks for your videos, learning a lot from them. 😊
Subscriber based filtering destroys thoughput in both SQS and Azure Service Bus. You should have this responsibility on the topic level. You don't need to have the property as a message attribute if you don't need it, meaning you can reduce the payload drastically. There is publisher-consumer communication so if they need filtering on a property you can retroactively add it
Create a video for messages queues but this time using azure
I already have that
AWS access policy/access control architecture is unnecessarily complicated...
"convenience is the enemy of security"
Make a good explanation, in your style, of SignarlR and Realtime in .NET
FIRST
Have u ever explored dotnet orleans? That stuff looks neat but rarely spoken about.
Hey Microsoft, take back MVP from that guy unless you did it already🙂
More like, Microsoft start sponsoring some .NET content creators