Pipeline-oriented programming - Scott Wlaschin - NDC Porto 2023
This talk was recorded at NDC Porto in Porto, Portugal. #ndcporto #ndcconferences #functionalprogramming #developer #softwaredeveloper
Attend the next NDC conference near you:
ndcconferences.com
ndcporto.com/
Subscribe to our KZhead channel and learn every day:
/@NDC
Follow our Social Media!
/ ndcconferences
/ ndc_conferences
/ ndc_conferences
Passing data through a pipeline of transformations is an alternative approach to classic OOP. The LINQ methods in .NET are designed around this, but the pipeline approach can be used for so much more than manipulating collections.
In this talk, I'll look at pipeline-oriented programming and how it relates to functional programming, the open-closed principle, unit testing, the onion architecture, and more. I'll finish up by showing how you can build a complete web app using only this approach.
Any Scott Wlaschin's speech is a quality mark. Thanks!
Scott's talks are marvelous. Here's a list of all his best talks so far ordered by the number of views: 1. "The Functional Programmer's Toolkit - Scott Wlaschin" - kzhead.info/sun/gdapj7CSZIp9ppE/bejne.html 2. "The lazy programmer's guide to writing thousands of tests - Scott Wlaschin" - kzhead.info/sun/fL2zdKqAr2iHg5E/bejne.html 3. "Four Languages from Forty Years Ago - Scott Wlaschin" - kzhead.info/sun/Y8qpdNB5fIOfomw/bejne.html 4. "The Power of Composition - Scott Wlaschin" - kzhead.info/sun/qaieXZinb4etn3A/bejne.html 5. "Pipeline-oriented programming - Scott Wlaschin - NDC Porto 2023" - kzhead.info/sun/nNSclbitgZ6tYIU/bejne.html 6. "Reinventing the Transaction Script - Scott Wlaschin" - kzhead.info/sun/iLeMm82cpJOJaa8/bejne.html 7. "Thirteen ways of looking at a Turtle - Scott Wlaschin - NDC London 2021" - kzhead.info/sun/mpRvmcujr5R8anA/bejne.html 8. "Building confidence in concurrent code with a model checker - Scott Wlaschin - NDC Oslo 2020" - kzhead.info/sun/p9Wwk95lkKZvmok/bejne.html btw. I'm building a newsletter called Tech Talks Weekly (techtalksweekly.substack.com/) where once a week I send out all the recently uploaded tech conference talks across +100 engineering conferences including NDC, GOTO, Devoxx and more. Consider subscribing if this sounds useful.
btw. I highly recommend his book called Domain Modeling Made Functional where he presents Functional Programming as a perfect paradigm for doing DDD.
A presentation by Scott Wlaschin - instant click!
I've been using this pattern for some years. Data train goes from station from station, no side effects. Just accumulating operations as data (makes testing very easy is well). On some stations, some choo-choo is performed (fire the side-effects) and then the train continues its journey...
"Pipeline-oriented programming" is more properly termed "concatenative programming". There is an entire family of languages devoted to it, namely Forth, Joy, Factor, and Listack. There is even a concatenative hack of Clojure called Gershwin. Manfred von Thun did a lot of the theoretical work on this programming style 20 years ago. It's really the ultimate extension of function composition. The basic principle is simple - you send data through functions in the order in which they act upon the data. (As opposed to the usual inside-out or backwards function call style.)
Or dataflow programming.
I'd like to stick to pipeline as terminology. Thanks to both of commentators for invaluable input as to what "proper" might be, but pipeline works just great so if you don't mind - I'll use that one. Cheers.
Also Functional Programming
Amazing presentation.
To me, in the narrow realm of software engineering, systems and software design are the two of the most sexiest areas ever. The best part is it will never get old or out of fashion.
A nice approach to software development and coding 👨💻😎, but I find it somewhat similar to the middleware and loop pipe approach that was introduced in .NET Core 3.2, and also bears resemblance to the Aspect-Oriented Programming approach that focuses on the business management layer, so I see it as taking an advantage from AOP. And an advantage of middleware, and the result was this beautiful approach. I do not deny that it contains its own features, but the basic things that represent the essence of this approach are what I mentioned previously. Thank you, teacher, for this nice lecture.
52:15 It’d be nice if it was possible to use React in this way. React being the pure code parts (explicitly calling React.render) and I/O being user input or network requests. Either you could have all fetches in the top level, respecting the purity of this Onion architecture (aka. ‘functional core, imperative shell’). Alternatively, the React.render call could collect all side-effects (like fetches) and then you could batch them and execute them in the I/O step. Like a structured concurrency approach with EffectJS. It would solve the fetch waterfall problem.
maybe you could co-locate fetches in XState machines next to the components, but pass that machine down from the top of the page, so it is always called (with all fetches for each component on the page) on page load?
Hmm. This is what I've been doing for years, but I've been calling it OOP. I probably started building like this around the time I discovered TDD or Clean. What a great talk.
fantastic talk !
Is this any different than Clojure's thread[first|last|as] macro with a sequence?
How is this diffrent form the fluent interface design pattern?
can we overload operator | in C#
I'm a fan of fluent interfaces as well as functional programming styles, but when it comes to pipelining, I'd love to see an example of connecting functions with more complex signatures. that can get tricky..
What exactly can get tricky? "More complex signatures"? What does that mean? F# arguments are always curried. A let binding can hold any data or partially applied function.
This is an OO answer, but you can use the Command pattern to encapsulate all the details of an operation, thus making it possible to build a chain of commands. The output of a command could be used as input to the next command and so on
Didn't Scott originally use the name "Railway-oriented-programming" instead of "Pipeline-oriented-programming"?
Railway is for error handling.
Interesting, but how do you break a pipeline if the remaining functions should not be executed? With a for loop you can break or return early.
I suspect you'll get the answer on the other talks. There were several examples of this in the simple web server: GET fails if the request is not valid and the following functions (route and setStatusCode) are not executed.
Mix it with the Result pattern
basically you prepare each function to short-cut. See more on this in railway-oriented programming: kzhead.info/sun/mb2oY7CGcIickoU/bejne.html
He showed it in the webapp example, it uses maybe monad without saying a word monad and making railway analogy instead.
Just when I thought that I need to learn more about pipelines 😊
PowerShell!
actually functional programming
Aren't these just called streams?
"Grokking Simplicity" (2021, Eric Normand) is a more thorough introduction to Functional Programming (also the book crashes a lot less than this guys laptop, yeesh windows)
Great talk but Windows let him down, switch to Linux and say bye to your blue screens
Wow - just like core idea in UNIX in 1975.... LOL! When world is going to completely reject this madness with OOP?
When Java dies... 😅
@@raptoress6131 which is not happening
That process has started. It won't be a complete rejection, but with 'Out of the Tar Pit', OO-everwhere began its descent.
OOP is good
This is just OOP. The arrows are the class data the boxes are methods. The good thing about OOP is scope. The data in the arrow is private
This appears to be nothing more than pure functions plus method chaining, which is indeed a useful idiom, but it's hardly a "paradigm". I don't see how it's more composable or testable than pure functions in general, nor do I see how it is an alternative to OOP. Actually, in most languages, this is only possible thanks to OO method call syntax.
I think you are wrong. He just misnames it. What he is talking about seems like DDD, but he for some reason calls it pipeline instead.
Was thinking the same. We do this all the time in pure functional code. there is literally no other way. He most likely just tries to find a less intimidating name for functional programming. For whatever his reason is...
I think this talk is really great for people who are unfamiliar with FP. It highlights the benefits of composition without putting FP in the forefront. I still work with "senior" developers that refuse to learn or use LINQ based solely on opinion and dogmatic principles. Giving it a new name might make those people tune in.
Yeah, I actually give Scott a lot of merit for simplifying FP concepts and design for all the tech leads and CTOs out there that wouldn't otherwise sit for even 5mins to listen to an FP talk, unfortunately for them and for all the developers actually interested in finding better ways to write composable code
amazing its the same talk he has been giving the last 5 years.. NOTHING NEW!!! Read his book (years old by now).. same content..
education never stops, there are always more people and new generations.
Компьютер перезагружался потому, что народ охренивал от того, что он снова в очередной раз рассказывает одно и тоже! Сколько можно уже одно и тоже рассказывать-то? Может уже пора на пенсию?!