Why would anyone still work in branches when Continuous Integration exists? In this clip from Jez Humble's appearance on The Engineering Room podcast, Jez & Dave talk about the stubborn nature of software developers and why that might cause people to stick to processes that can't keep up with other modern software engineering techniques.
You can watch Jez's FULL episode, HERE ➡️ • “Industry Changing Boo...
___________________________________________
🙏The Engineering Room series is SPONSORED BY EQUAL EXPERTS
Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ bit.ly/3ASy8n0
___________________________________________
#softwareengineer #softwaredevelopment
I watched this hoping to find why the hell do I still branch, instead I saw two guys agreeing with each other: "yeah, yeah, indeed, I wonder, I remember, I agree".
What the hell is wrong with: 1. Branch from main branch into your own feature one 2. Work on it, finish ticket, test it 3. Create a PR to merge into main Why the hell do I need to merge incomplete code into the main branch??
Exactly! Also, one of main points in continous delivery and trunk based development is that the code/trunk should always be in a releasable state. I would say that code not passing unit tests, static code quality analysis, secrets scanning, commit message standards etc, is most certainly not code in a releasable state! One does not even necessarily need a human review despite using feature branches and pull requests (although, putting notes for future reference in a pull request and keeping conversation about the code out in the open is never a bad thing). All the checks I rambled about earlier can be implemented on the CI pipeline, triggered by pull requests and once all checks are passing: Just merge the branch, delete it and pick up the work on the next ticket. As long as the branches do not exist for more than one or two working days, they are an excellent tool for ensuring that only releasable xode ends up back in trunk.
@@MLA916 Who cares about how long the feature branch is off of the main one? Your "one to two days" sounds like another dogma
@@Whyoakdbi of course that one to two days can be flexible, depending on how frequent changes are in the codebase, dependencies from other developers on the changes in your branch and so on.
A few side effects of letting branches live for too long is that things grow over time. If you pass on one-two days worth of work on a pull request off to a teammate for review it means a lot less work with reviewing. Smaller changes to review enables the reviewer to give more specific feedback instead of going "Screw it, looks good. Ship it." If one practices continous deployment from merges to trunk, one also gets an easier time with monitoring how your changes affect production, and can act accordingly to that feedback in the next piece of work yoy take on. I wouldn't take the one-two day lifetime as dogma, more as a rule of thumb to make things sligthly more easy to manage and review
@@MLA916 I know about the drawbacks of having large PRs and that it becomes worse if they had been off of the main branch for too long. Every experienced developer knows those things, so it kinda goes without saying..
Being dogmatic about continuous integration/delivery is just as dumb as being dogmatically against it. There are plenty of massively successful software projects and teams that don't use CI/CD (particularly CD). I use it, but not every industry or product has the same requirements or level of risk as the software I work on. Projects involving highly-regulated industries, safety-critical systems, legacy integrations, external partner timelines, embedded systems, etc. all have different cost/benefit needs that may rule out CI/CD as an option.
Totally agree with Dave and Jez about long-lived branching being problematic, however the title of the video is also problematic. Short-lived branches are very effective and are a valid CI practice. DORA shows that benefits from TBD and short-lived branching are the same. However when criticisms of branch-based dev are made it is always exclusively about long-lived branches.
Martin Fowler changes the language from "branches" to "independent code lines" to detangle the git software feature from the code management process. With that change it's easy to consider the local cloned repo an "independent code line" as well. So main-line-healthy-branch development where people are only integrating weekly is not better than short lived branches (repo-side independent code lines) that live only a day in terms of technical debt from delays in integration. However, as a developer who prefers main-line-healthy-branch TBD, I do get a bit annoyed with the overhead of managing the short lived branches in the repository software, since it's not part of my discipline. Things like writing my code and then realizing I hadn't switched branches locally, or somehow ending up with a detached head.
Yes, but the problem is that short lived branches more than often turns into long lived branches. Might just as well stick to trunk all the time.
@@pompiuses that is circular reasoning, if your short-lived branches are turning into longer-lived branches then they are not short-lived branches. If someone is waiting many days to merge with main then you would have the same problems as someone working to trunk and not pushing their commits to origin/main for days. This isn't an argument against short-lived branches. Also saying "might as well stick to trunk instead of short-lived branching" is a poor argument, it is as pointless as saying to those on trunk to go the short-lived branch route. If your tooling is set up for one or the other, then stick to it. As long as you're not waiting days/weeks to merge (branching) or to push commits to origin/main (trunk) then you're gaining the benefits of CI. Ideally committing/merging more than once a day will realise these benefits.
@@pompiuses Wtf is a "trunk"?
Have said it many times: if long lived branches are a problem, there is an easy fix. Just merge daily from the originating branch. All the benefits without the risks of committing half working code.
Can't agree with this dogma. If you want to abandon or postpone a feature, it will be a headache. Also, this setup is not good enough for remote work, because you introduce a dependency: having pairs that coordinate on exact timeframes and schedules. To put it into perspective, in today's world, XP can be done less and less. Working on the latest version of code is important, but with large teams, this is not possible, unless you idealize the idea that there will be no code merge conflicts ( good luck with that ).
I am sorry to be blunt but you are simply wrong in your statements. This can and does work at scale for some big complex software, Google practice Trunk Based Development with 25,000 engineers, Amazon, Facebook, Microsoft and Tesla all work in the way that Jez and I describer here. How can it be "dogmatic" to state a fact - if you aren't evaluating your software all together, you aren't evaluating what goes into production. If you aren't evaluating what goes into production, you are guessing about whether or not it will work. That is fact not "dogma"
@@ContinuousDelivery You are wrong. The way Google practices Trunk is that they **do not** commit straight to origin/main. They commit only to local main and local main will stay divorced from origin/main until they have passed code review - and yes they will need to rebase, and yes they need to resolve merge conflicts. They do not sit and pair program and push each commit to origin. They only get to "land" commits to main once their tooling builds local main and they have passed code review. Satisfying code review may even take days. Trunk BD as practiced at Google and short term branching are swings and roundabouts, they resemble each other more closely than the kind of trunk based development you advocate.
Hmm, strange that they do not mention that trunk-based development increases burnout according to the last DORA report, and slows down operational performance. Actually, the opposite should be achieved with trunk-based development. No word about it?! As many mentioned don’t be too dogmatic! In case you have short-lived branches (1 - 2 days max) with a proper CI pipeline, I think you are sorted out. Even the official trunk-based development page calls it “scaled trunk-based”. More important than the branching strategy is, that you automate the majority of your tests and regressions and the code review process is fast enough (if you have any). Deployments are done anytime you want, without big “deployment pain”. On top of it, good observability of your system. "Trunk-based development" would be the least of my problems today.
💯. The things you mentioned are 100% the first things a dev team needs to do to achieve CI. Fix deployment pain, increase observability and testability. CI isn’t just something a team can achieve overnight. There are steps that must be taken to make it happen. Dogmatically saying CI is the only way and branches are bad skips over all the important discussions and learning that a dev team needs to take on board to make CI happen and also skips teaching about all the nuances and tradeoffs that can legitimately be made along the way.
I think Jez's theory is bang on! I used to work in the healthcare industry on safety-critical projects. Dave, what do you recommend for these kinds of situations where automated testing must be supplemented with manual/formal review process for a release? Another thing I think people can do in a CI environment is use feature-branches to do independent experiments and prototypes, but then throw them away afterwards, and I think people think it's all or nothing.
Not sure if this is the correct forum to ask you and or if you are going to respond. Anyway something I struggle with my team is the concept of "pre studies". My team time to time spend plenty of time investigating a subject to get an idea of how to implement something in the entire solution. This is typically done on a feature branch. Most of the time we do not merge to master because it's an "investigation". Is there a time and place for a team to do a pre study? Or is this simply a waterfall process I should try to remove from the team?
That's an exception. Big refactorings, proof-of-concepts, etc should be on a branch (I wouldn't call it a feature branch though). Nevertheless, if you are unsure if the final work will make it to the master, do a branch. Branches are not evil. Feature branches, where one develops and tests without integrating to master, are the problem.
I’m actually really concerned about the messaging in this vid and others like it. It’s a disservice to our community. People shouldn’t be taught fanatically that one thing is good and another thing is bad. They should instead be taught the trade-offs, and taught the skills to be able to make informed choices. Branches are good for the things I mentioned in another comment, but there is a trade-off to be made the longer a branch is left open and the more code that goes into it. By jumping to “branches are bad”, you’re skipping important teaching points like how to break large problems down into smaller problems, and how to introduce fundamentally different code without breaking everything. For example a fundamental breaking change from a one-one relationship to a has-many relationship in your DB when a lot of app logic is implemented as role-based logic, and now needs to shift to resource-based logic. It takes a different kind of skill to do that in a CI way. You’re missing the opportunity to talk about that and teach those skills. Please stop being so fanatical about “branches are bad”, and instead focus on the skills devs need to be able to push changes to production faster.
Is this the first and only video you've watched from the channel??
@@michaelrstover I’m quite well watched. If Dave won’t provide balance, I will. I hope others do too.
I agree, context is crucial - in a vacuum CI is better, but most people are not working in environments conducive to true CI. They are working on legacy systems, are under-resourced and poorly directed, or they're working with people who are putting in minimal effort or care. Branches work in situations where the system does not yet support clean, trustworthy continuous integration - which I think is more than likely a majority of situations.
I totally agree. It is somehow fantasy by assuming that all team members are equally skilled. In reality, they are not. Sometimes you even have people introducing negative value. Branches and pull requests get the last resort against anarchy and chaos.
@@ralftaraschewski6094 the other day I had a conversation with a colleague who was asking "how would we know what was in development and what was in production without separate development and production branches!?" and it's that sort of bad mental model of how git - or just version control in general - works that we're up against when trying to put continuous integration in place. For now, I'm stuck with their ignorance and poor development practices
Where can I buy Dave’s T-shirt??
In the mid 90's I worked on a system that had I suppose around 30 devs working on it and it was a continuous integration setup. We didn't call it that in those days. And the tools weren't very flexible (proprietary in house repository and build systems) but we never had any troubles with the overall approach. Fwiw I've always had trouble with the idea of branches. I spent a lot of time through the years at other companies resolving one nasty merge conflict after another. But here's the thing, I don't see branching going away. Many approaches work (whether they're ideal is a separate issue), so long as the team understands them, follows the process, and does even a half descent job of communicating. Heck, I worked in one place that used a stuffed animal as the version control system. Worked great, once the rules were explained to you. And people only ever broke the rules once - they got to clean up the mess all by themselves.
The thing that hit the nerve with me BIG TIME is the practice of Silo the engineers, not sharing the big picture, and micromanaging workers. They still think it's a "slow coding problem" and not "a management's problem." Ever hear the "failure to plan is planning to fail"? I guess they hadn't... When I give up waiting for them to do the planning, and I sketch out an idea and say we need something like this fleshed out by everyone on the team (usually 1 or 2 other ppl) ... That will predictably set into motion the following chain of events: Management thinks because of my effort to draw up a design that I'm not focusing on my siloed project assignment that I am overstepping my boundary, and/or that I'm telling them how to do their job. I am sat down in a private office, and given a stern talk to... I am told that the planning activities I propose will take away from the work schedule, that we have an ambitious and over aggressive schedule, that "We already know what we need to build and this planning stuff you're doing is only a distraction from the real work that needs to be done." Then the management focuses down on my performance, scolds me that I already don't deliver fast enough for them, blames me for not heads down at my desk slaving away, and says that I should be able to pull this code out of my "brain" faster, use the "Fire, Aim, Ready" approach to coding. Never is the quality of my work in question, if fact it's often the best they've seen... it's just that it "took too long" maybe it was all those comments that I put into the code that took so long LOL? Of course I'll promise to "do it their way" but it's still a slow-motion trainwreck. Predictably, the project timeline slips, one project I worked on was almost a year late... They're blamed the engineers for not fulfilling "their vision" (that they failed to share with us). This was the story at all the previous companies I've worked at, the last one 12 years... now I'm at a new place, and guess what? The common issue is that management never sees itself as having a management problem, and they rarely, if ever, took responsibility for failure to deliver a project on time... because they're too busy blaming their engineers for their lack of leadership, planning, and ignoring industry standards and practices, they don't think that they need to get better at what they're doing, but the engineers need to get better at what they're doing... we're the perceived bottleneck. And even when the company pays for Agile/Scrum training, they've got their minds already set that the training itself is a waste of time... they already know it won't work for our company... so they go back to the "tried and true" silo method and blaming the engineers for not know how to do things and delivering faster code. (Insert the sounds of whips cracking here) Lather, Rinse, Repeat! 30 FLIPPING years of this!!!! Guess what!? It's Groundhog DAY... Again! and again, and again... (If you get the movie reference of the time-loop Bill Murray finds himself in on Groundhog day)
I'm about 25-30 years into it myself started in embedded, then windows and then web... gotta say this has not been my experience so far. Yes some places do suck but I left those places if I couldn't change them. Other places I changed through example. In every place from my first job to now I have come in with ideas, had pushback, demonstrated ideas working myself and then these practices spread through the team and/or org. I'm not perfect, not every idea I have grows like that but the basics like testing, QA, CI, code reviews, and some basic SOLID are things I think you can easily get buy in on any dev team for and at least in my experience are relatively easy to get management buy in once you've built the necessary trust to be given the space to show them these practices working. I'm sorry about your experiences but just wanted to say that I was able to implement good practices in every org I've worked for even starting as a junior developer. I would say focussing on less us vs them and more on how we can both be successful (management, devs) has been successful for me. Recently my org changed sales incentives to make sure the sales team aligned more with our development roadmap. It can be done :)
@ozzyaararon Yep 30 years embedded firmware developer. It's funny that embedded is the last hold out and they think because "we're special, we are a small thing and those rules only apply in big organization, and Java Architectures, they don't apply to us" Then you look at the code and OMG: They write the monolith, tightly couple your code, low cohesion, no sense of reuse, portability, API, and then there's the patch and repatch, and presto you've built a "ball of mud." So I'm back in that conversation again at my new job... Just left the previous company that said the same damn thing! The definition of insanity is repeating the same thing and expecting a different result... right!? You bring up good points about being the example... however... some places that will backfire on you and you'll get in trouble... I've done this exact thing: (try to) leading by example, that is what I meant by doing an architecture design and through my coding practices such as a bit of TDD. I once thought as they did, I realized that after a few go arounds that there has to be a better way... because the hero mode of product development, then the fire fighting afterwards gets tiresome after a few decades. This scores NO points with them, in fact is scores NEGATIVE points because they think it's a huge waste of time, evidence of success is boring, or course it "worked" so what!? They'll never do it nor learn from it, and worse yet, they think this is Jr. engineers that need "training wheels" That is pretty insulting because it is often by people that are half my age and I've been doing development before they were born. I had video games sold on the market by a commercial company in 1982... sometimes I have to remind them that I've improved myself beyond that in the last 40 years... (A) hat have you done after college to improve your skills? and (B) I'm not a kid anymore... I have to bite my tongue so I don't say too much... because I can call them out on their stuff, so what a bunch of ignorant arrogant buffoonery and hubris they have in their skills to think they "know better" and that gives them permission to pull rank and cut me down, they love using HR as a weapon against me... I see them as basically kids that are given authority to do so. I should be clear, not all people or managers are like that, I just have had more of the negative experiences that the positive ones. And I am not ageist either, a young person can teach me a lot too... Age is not the factor, it's experience that is... age just means you've lived it longer than they have... Some old folks aren't any better, they sometimes don't get it either. What I love about folks like Dave Farley is that they question things, they inform us on ideas and advance the conversation. This is "continuing education" and improving not only workflows but also the quality of work, and the quality of life so you're not in the office on weekends and firefighting the problems out in the field.
From my perspective, this happens because the people managing software teams and projects have absolutely no formal or informal training in software engineering. What exacerbates this, is there is an IT department and CTO and/or CIO, but no software department and CSEO to discuss, encourage, and assist with good consistent practices. Only people with software engineering training/experience should manage software engineering/development teams, and there should probably be a corresponding department and representation at board level. Not only this, but software engineers are the only profession that could also extend their training to refactoring company structures to make them more efficient, and solve all the silo issues. (Assuming that company processes and hierarchies are same enough to the patterns seen in software).
@@br3nto The scenario in the places I've worked is that they promote the engineer to management positions that they don't deserve or have training in... they're promoted as a reward, they've lost a good engineer and gained a bad manager... Lacking in leadership, and they blame the people that work because they don't see it or do it like they do. While I'd want to believe that I could handle a management role better because I taught at a University for 10 years and have people skills, plus I have 30+ years working in industry and not only do I understand the reasons for the designs and implementations, I really understand that you don't wait until the last minute and just tell your people to start coding and hope that it will come together. But I am humble enough to know that I don't have training and I'd seek that out if I ever got put in such a position... I've never asked or sought after it. The current situation is that the new manager, engineer person, is drowning and not delegating nor communicating, doing the engineering tasks plus assuming that everyone is doing their part... So I've been dumped on with an impossible task to complete in 3 days, I'm putting in 18 hour days to meet an unrealistic deadline, but I'm shooting for it in good faith, but there's no upside or reward for stretching... just potential burn out. I'm already being punished for not producing fast enough for them, so I'm just sinking in quicksand... The hope I'm holding out for is that we have someone at the top, VP exec, that wants to change the organization, but the engineering department is so big and there's a lot of other things that have priority... I think it's coming, but not anytime soon, and not soon enough in my book. Sigh... Oh well.
From my experience and opinion based on my experience, its nostly to do with comfort zone (make sure things are good before letting other see your code) and also something they are used to do or started with. Change is always difficult and scary. When I was trying to convince my team to TBD, I raised some questions like, if you are working on your own project or alone, would you even create a branch? It makes no sense, unless you want to test something separately or build a separate implementation, but generally, you wouldnt create a branch. If you are working in a team of 2 and both of you knew each other well enough and trust each other well enough, would you create a branch? If yes, why? Is it because its was you have been practicing or know and would you have to wait for each other approval and doing async review? I also did interview one candidate where she and her team has been working together for more than 2 years on the same project, they have been very comfortable with each other and I ask if they're still doing feature branching? It was a yes, and I asked how was the review done and the answer was, usually anyone would just approved the PR when asked to review. So, I asked, why then are they still practicing feature branching? It seems that they didnt know there was other option, and it was just something they do, follow or has been practicing and there was no continuous improvements practices. I came from a time or used to write code witout knowing what is a version control, during that time, we used to write a lot of code in a single java file or jsp file, it can contains thousands of lines a mix of html, css, javascript, server side code. Imagine how does 2 or more developers work on a single file and without version control? Well, in the past, it used to be 1 developer work on a single feature and bear in mind, in the past, there is no such thing as story slicing, every requirement is a huge change that could take a dev weeks to complete and we try to avoid more than 1 dev work on the requirement that touches the same file. But, we do have times when we need to work together on a big feature or change, and imagine how can we integrate without version control? Well, we communicate, and communicate a lot. We discuss how should we write our code, and how often should we merge our code. How do we merge? 1 person send over his/her file and the other person will merge the changes manually; but how do we make this easy? We had to discuss if there is a new function, where should we put it and when one needs to modify existing function, we should inform one another and constantly integrate our code. Hey, arent we doong continuois intgeration? When I started working in version control project, it was a dissaster, we have master, dev, feature and feature by dev. I felt so indifferent and wonders, why with version control and so called professional working environment with "proper" process and tools, it was so inefficient, slow and having a lot of issues like resolving conflicts, we dont communicate, long-lived branch that resulted in PR so big that is difficult to review and continuously need to resolve conflicts. How come we were able to work and integrate better before using version control? I didnt even understand continuos integration until much later after I attended DevOps training that explain very well about DevOps being a culture and introduces CICD. Before that, people always says that by using GIT or having a build pipeline meaning the team is doing CICD, and when I was interviewinf candidate, I always asked this question, what does CI mean, what does CD means, most of the candidates always answer using Version Control or having a build pipeline; most dont even know what it means by Continuos Delivery and Continuous Deployment.
I work as a solo developer and I do branching from the master for requests for simple bug fixes and larger updates to the system functionality - both of these types of requests are added to Jira as tickets and result in branches being created for each ticket. It is completely unnecessary, but it keeps me organized. I find that I need to work on bug fixes while also doing larger updates (so, two tickets at once) and thus I need to put aside the project work, implement/deploy the bug fix, and then resume the project work. Branching and merging keeps that organized and makes it easy to locate code changes associated with a given ticket (branches are named by ticket number). I guess this isn't necessary, but when I've tried deviating, I seem to get stuff mixed up too easily.
@@rangerrecon unless you're separating work and working on multiple tickets, perhaps branching helps to organize the work, but it can be in your local branch, no point pushing to remote and create PR. But most people work on one thing at a time, so the question is, whether there is really a need for branching. But if you use it to organize your work or separate your work locally, that's fine, it's like keeping multiple version of a document. The same problem will be there, is you keep multiple copies of different version, and then you had to fix something, and that fix has to apply to every version of the document and what if some of the version has conflicts; which is the truth? Well, working alone is still ok. You can do that, but do you really need to? It's for you to decide, there's no wrong or right. If you're a developer and you were never expose to branching,would you really think of branching? I don't know. I came from a time before version control, everyone has a copy of the same file in their machine and we had to continuously ensure our local copies are up to date; imagine having already different copies in different people's machine, you wouldn't want to create more copies in your local machine. Even when I was working on my own project in those days, I use to keep backup copies of the project and sometimes a copy of stable version, maybe some other versions; backups are ok since we don't need to update them, but we would need to update if there is different copies, and when this is done manually, you start to question whether what you're doing is efficient..with version control, since it helps us, we may start to abuse this and starts creating many branches, which maybe is not needed but we do it because it can be managed much easier. Anyway, sure, branching is not wrong but it's whether when you need it and whether how many branch you need, whether you work on different problems at a single time. Even so, do you need to push to remote, but even if you push to remote, if it's just for managing your work (you wouldn't create PR to merge and take time for yourself to review the code, would you?), it's just like saving your work in different state or version. You would just need to manage it yourself and you can decide whats best for you.
When my team moved to TBD, most people have the same challenge, they're working on a feature implementation and there was a bug reported and need to be fix asap. There's many ways to handle this, the easiest approach is just to create a branch in their local and fix the bug, but there's no need to push to remote to create a PR, we can just do peer review and push directly to master. The local branch is just used to separate the work when their feature implementation is not ready to be pushed. Some people use git stash, so they don't even need to create separate branch but they need to ensure they don't have local commits not yet pushed. But a better practice is to practice small non-breaking change and push to master. This is a challenge for many people, for example, what's wrong with pushing just a single function? If nobody calls it, it does not cause any issue, so you don't have to wait until your feature is complete. And with this practice, people can start to learn about breaking things smaller and working on smaller piece of change, but also, how can push something that we are confident it is working? We should test what we deliver, even if it's a small function. For a simple feature, in a day, there could be multiple commits+push and finally all the units are integrated and with each unit tested, the integration is smoother, and even if there is issue, it's very easy to detect and tell where and what is the problem. This is what our team practiced over the years, in the beginning when switching to tbd, it was challenging because people are used to large commits, long lived branch, rely on PR comments and having the safety net of branch failing, there was always push that did not pass the build in the branch; it's a bad habit but since it didn't cause problems to anyone, this habits continued... When we switch to TBD, in the early stage, we often see people pushing changes that failed the build pipeline and had to quickly fix them as it blocks others. Eventually, it trains our team dev to be a better developer and also started to do small commits/change and having proper automated test. In my opinion, it's a big impactful good change to the team.
For me, branching is a key part of XP. Working in an ensemble, we: 1. Pull the latest main 2. Refactor, commit, in a tight loop 3. Push and merge to main Duration varies but it's not uncommon for this to take an hour. I don't see any conflict between this and Trunk-Based Development.
It's really the long lived feature branches that cause the problem, and specifically the long lived part, not the branch part.
@@georgehelyar I agree. I usually define "long lived" as "exceeds the current workday".
I think that is TBD, it is just a slightly less efficient version, why not save typing by pulling from master to you local copy of master and merge the changes from there at the end?
@@ContinuousDelivery I am fine with working on main and then doing `git push origin HEAD:foo` and then making a PR of foo. Not sure if that answers your question?
Because, as a contractor typically hired to lead teams comprised of junior/unqualified employees, there is a lot of code I want nowhere near the releasable code. I want that barrier and protections. If I had a team of people I trusted and knew got the job done, I'd still want that barrier. There are also reasons why it is a must. I once managed a team where we owned the main "Portal" (menu) application to about 60 applications. On Monday of week 1 we would get say 6 request for changes. One Monday of week 2 we would get say 5 new request, but 3 approvals for the previous week to move to VAL and say 2 approvals from last weeks VAL to move to PROD. Until we setup separate branches for DEV/VAL/PROD releasable states, it was chaos. Branching, like everything, is not a silver bullet. There are cases when it is very useful, and cases when it is not. I do not have the advantage of picking my team or the process of my client, so I required MASTER (PROD), release (DEV/VAL and current efforts), and feature branches for everything. That has allowed me to control what gets released and when.
4:56 Developers who have worked all their career in an environment that strictly uses GitFlow pattern believe the trunk is sacred. Trunk-based development is akin to walking on a tight-rope across the Grand Canyon blindfolded with your hands tied behind your back. Tech leads and managers told them that the trunk/master had to be kept pristine at all costs. Forget using tags, feature flags or release branches. Too much to learn. Just never merge to trunk... never ever. _Ever_ . Merge to trunk. And y'know what? Don't even merge to the dev branch. You might break that too. Just work in your own branch. Everything's safe there. It's your own personal bubble where you can be "in the zone" and don't have to worry about being hassled for breaking the build. Sure, you're not integrating your changes with everyone else's, but the longer you work on your own branch, the more time you have to make sure the code is extra, extra perfect so that when you do merge, everything will just work. What could go wrong?
Working in the main branch sounds idiotic!
@dave, I would like to see how this would integrate with scrum. Of corse I understand this is the very definition of agile, you keep building, testing ,pushing and deploying, but if you are giving a huge scope and this is on you what you have to change to make this a team owned scope and not fall into each developer has this massive individual scope. I mean for example, I don’t think that in this model you could handle the user authentication to one developer and a checkout/gateway integration to another. Big branches will be created.
The more I think about this I ask myself: Isn't TDD a mixture of specification/requirements and the actual (requirements) testing?
Many interesting points discussed... But what is the alternative to branching?
True Continuous Integration, sometimes called Trunk Based Development.
Working on windows computers and having to deploy to both Windows Server and Solaris is exactly why I chose Java back in 2006!
Love this conversation. The "working with branches" vs "everyone working in one trunk" ( 4:21 ) debate breaks down the same way many impassioned disagreements happen: All the assumptions that go unspoken. _Of course_ working in a single trunk is better... if you also have all the necessary things that come with continuous integration. Whereas _of course_ working in branches is better... if you don't have any integration tests, don't have automated testing, or automated deployments, or etc. which really isn't better at all, but is necessary IFF you believe your current context can't change. So when people reject agile principles or Continuous Integration practices, I have to ask "Can you enumerate the list of things that would go horribly wrong, and then can we address each of them?" to change the context we're working in.
I think it's possible for branch-based development to be leaner and more agile for a small team, regardless of if you have automated testing and deployment systems. Really TBD shines in large organizations where communication becomes harder to manage, and thus the trunk becomes the source of truth. In this situation, an interface might change quickly and often, and having to push/pull daily means you are always aware of those changes without direct communication. But to do TBD you need a lot of abstractions, and systems to be able to gate new features that are incomplete, and team resources to manage multiple configurations as new features are ready for testing. In a small team, where it's easier to talk to the whole engineering department, and thus get on the same page on what is required for an interface; TBD just serves to add more overhead by forcing people to build these safeguards that gets in the way of actually implementing these changes, on an already resource constrained team. In this case it's often easier to just spin a new branch, and directly communicate when something needs to pivot. Also, when you only have a handful of engineers, you can't afford to split your resources in half to pair program all the time, so these new branches serve as a good way to review code before they are merged in. I think it's more about team size and communication structure that impacts the efficacy of TBD rather than it being the best in all situations.
Branches don't mitigate the issues that come from lack of automated tests, they just mask them. With TBD changes are integrated frequently, so breaking changes usually quickly become apparent. If there's no automated tests to ensure key business logic rules work as intended, this might feel as though the main codebase is constantly breaking. Whereas if work is done in branches that are merged infrequently, the codebase does not change as often so it may feel more stable than TBD. The catch is that when code is finally integrated, it's usually much more difficult to resolve issues because the commits are much larger. Pull requests can be used to mitigate somewhat, but PR's generally become less effective at catching bugs the larger the request is.
I really don't get the long lived branches argument, surely your re-basing in from the teams mainline branch at least once per day? If the changes are so significant as to cause merge chaos then surely the same problem is going to occur working directly in main?
No, because CI means "at least daily". You assumed that the same amount of code is committed like you use to see in PRs for long lived branches. But that's not true. With CI and TDB you get small commits that should align with your Code as you design it, because you commit daily as well (so others also adjust to your design).
@@seNick7 in other words, your act of committing unfinished codes makes others waste time adjusting their code to changes that may not survive the following day. That or you never change committed code, which leads to messy code that keeps being patched with layers over layers and never refactored.
@alfbarbolani That's not the case, because every commit is reviewed BEFORE it's commited to repo. Every commit is fully reviewed (live review or by a tool like Linkedin does) and is an atomic change. That's the way professional software engineers do it. Continous Integration is a professional standard, sorry. Go and see how Linkedin does it - there is a talk available on YT about trunk based development and pre-commit code review.
@@seNick7 if there is a daily review of a daily commit of a piece of unfinished code that will be changed and reviewed again tomorrow then that explains the low productivity of LinkedIn developers. Please separate concepts: CI is not related to branching strategy. They are independent.
@seNick7 Large org does {x} therefore its the professional standard does not make for a convincing argument. For every example of {x} you will find countless others successfully using strategy {y} or {z}. As you may well be aware the worlds largest open source project uses the Benevolent dictator workflow, they seem to be doing just fine. Perhaps there is more than one way to manage a code-base successfully depending on the context of the project.
I've had my so called "urgent" jobs blocked by my previous work being stuck out on a feature branch with an outstanding pull request for 5 weeks now. Yes... I did say "5 weeks"!!!!
I'm not sure what they mean. I create a new local branch for every task/ticket I do and do a PR back into the main shared dev branch when each of them is done (after merging in any new changes from the dev branch), and delete the local branches at the end of the sprint. Are you saying there are people who don't do that? We used to have one long-lived dev branch, but now we get a new one every sprint because it helps with keeping track of when changes get moved to prod.
So you are using Version Control, but not Continuous Integration.
You can just merge and delete the branch immediately after a feature or fix is done. Why wait till the end of the sprint?
Yes, we are saying there are people that don't do that, and saying "don't do that" to the people that do, because there are better ways of working.
You can easily do CI and use branches together. Branches are a creativity and productivity tool that can be effectively used to do CI. The thing being glossed over is all the skills that a dev needs to develop in order to do CI. How to break problems down into smaller pieces without breaking the app is not something taught well. It requires a different type of thinking and methodology than simply writing an entire feature or change in a branch and then merging it. It’s a shame that this conversation skips talking about those needed skills to make CI work.
I understand the one branch and how it can be better etc. but I would like to see it in practice, I can't get my head around as to WHEN you would code review person X code before merging it into the branch. I do a lot of code reviews at work on pull requests and I very rarely let one through with no comment on it. So if everyone put it into one branch, when do you code review them?
Apparently never, because you never have juniors working on your project and everything everyone commits is flawless. If there is a bug, well… let’s just have the system crash in production and then try to fix it, but we all know that’s not going to happen because we seniors are perfect and don’t make mistakes. I can see them arguing that you do pair-programming with your juniors to have them learn as you go which means the code is “already reviewed”. Of course, then you would need one senior per junior on the projectand the seniors wouldn’t be able to do jack-all due to XP with the juniors. Sounds like a wonderfully Utopian way of doing things. It might be fun to try it out in some potential future project, that might or might not be imaginary.
The easiest way is to adopt pair-programming, so the code is reviewed as it is developed.
it depends what you mean by "senior" in the pair programming teams that I have been part of, the time for a junior to get good enough to lead a pair with another junior is usually measured in a few months. Pair programming is the fastest way to get juniors, or anyone else, up to speed that I know.
@@SuperPranxYea, we do some pair programming, though the main manger of the whole team hasn't "authorised it", but in our little sub team, I talked my team leader into it and we have great results, I would recommend doing it, but sometimes as a senior it feels I waste my time just watching. so we sometimes let them get on with it, and hence the code review as well, so I can then go over the code in full. Seems to work well, but all on one branch as being suggest by Dave & co, I just can't get my head around it.
I agree about pair programming being a good way to go, not just with juniors. I just can’t see it working with trunk-based development. Maybe I’ve never had the right setup/project. Also, I can’t really see it working on a project(not a product), for a client, in a very specific domain that the dev team is not super-knowledgeable about. Seems like it would be more detrimental than beneficial.
What the alternative to not working on the main branch? I’m on a team with continuous delivery I guess I can’t imagine the alternative way of working.
You're lucky. The alternative is to open a branch, keep it open for a long time, open more branches, keep them open a long time, merge the first branch fine, then realise all your other branches are broken and waste a bunch of time trying to fix them. Then merge the next branch and go back and waste more time fixing all the others. Over time build up a collection of branches that take longer and longer to get merged, and eventually get to the point where some never do, even though they are "done". People like it because they only think as far as the branching which is easy, and don't think about the merging, which is hard.
@@georgehelyar Sounds awful and error prone. We have the main branch. Create a bug/feature branch off of it. Commit code. Open a pull request. Test it. Merge it back into master. Next bug/feature. We do monthly releases so we can always sit on the bug/feature if we run out of time to get the bug/feature into the next release. I’ve never known alternative to this and I never knew people do it differently.
Mate at least the things they’re still doing wrong we can still have hope of changing. I’m now dealing with them returning to old bad practices and then complaining about the old problems recurring in turn and a slavering set of Medium writers writing articles on how to [fail to] fix them using the old failed solutions! They just reinvented solos with all new names for crying out loud! And literally 5min later there’s articles about “Are your developers too dependent on you New Silo X?” 🤦♂️
There is an unfortunate myth that branches and pull requests serve to protect the quality of the code. Of course, this is false. For one thing, PR code reviews are no substitute for pair programming. My client requires branching, so the best I've been able to do is convert all of the projects I touch away from gitflow to github flow. The worst part is even at project inception, when I'm the only person working on a code base, I still have to abide by these idiotic rules. My bypass is to create a temporary branch where I can work unimpeded by bureaucracy for a few weeks and then I PR this branch an initial push to the main branch. After that I'm stuck with requesting PRs from my teammates who are all 15 to 20 years my junior. I push probably 10 or 15 times a day, so it's a lot of waiting on people to hit approve. At least our deployment is fully automated, so once code hits main branch it's basically live.
Most leaders want to parallelize everything and think of developers as independent units, i.e. 2 developers do a task 2x faster, or, we can tackle these 2 tasks faster if we use 1 developer for each. I cannot agree more with Jez & Dave about avoiding branching, in all my experiences branching and PRs have created bottlenecks and annoyed developers. I think a team doing peer-programming works much efficiently than all the traditional PR-based methods.
I think that one of the reasons for this disconnect is the polarization of the arguments. Some people really love branches, and some people really hate branches, so it makes sense to NOT have a connection there. I, myself, discovered these ideas last week, after almost 10 years of work, and hearing "branches are stupid, not using CI is stupid, is a waste of time, how are people NOT using CI", etc. isn't helping me a lot, it's putting me down and making me feel bad about myself, because I am yet to understand those arguments. I always said this: I don't like when people DON'T know helpful things, but I equally don't like when people act like they don't know how it's like to NOT know helpful things. I think it's pretty safe to say that we all went through NOT knowing about CI. And I think that the best way to make the world a better place would be for people that love CI, to think about WHAT made it click for them, what caused them to switch their thinking, even if it's a wrong idea, instead of ONLY focusing on the end goal, which, again, can't be seen by most of us.
I'm moved, totally listening to this. That's my experience since years, IMPOSSIBLE is the word I heard quite the much over everything else, and IMPOSSIBLE to continuous integration and pushing to main branch multiple times a day is probably the most heard. I'm embarrassed about that, mostly when dealing with wonderful software engineers that still doesn't accept this possibility. And it's true, it seems the feel attacked personally! That's incredible. Thanks again for your contents
"Working in a silo" This is why I advice all starting programmers to spend at least 4 years in the secondment / consulting industry. People who spend their formative years stuck at one company stop learning far too soon.
to be honest, I do not really like this out of context format cut from the longer calls
What if you're rebasing frequently from the main line into your long live branch?
That is what most sane people (including me) do, and it avoids the risks of his CICD no branch strategy. Because if you rebase (or merge) daily, your branch never is older that one day. All the benefits without the fragmented commit history of half finished work spreading across your repository. Something Mr. Farley et al seem to surprisingly ignore. In my experience the “no branch” strategy, given enough time, leads to layers upon layers of patched code in small commits. If you are lucky and the team is senior enough they recognize the mess and go back to a saner process. If you are unlucky and believe consultants rather than your own experience, you end up in a code base where the usual reaction from outside people when they see it is “this needs a total rewrite “
It makes no difference if you pull/rebase daily. If there are 3 teams and 5 long lived branches, you could have merged every hour and that won't save you from the merge hell (and conflict resolution errors) if all of them start to merge to master during single day. Frequency doesn't matter if other teams also have gigantic branches/PRs
@@seNick7You just need to push your code in before they do
@@seNick7 please learn what means to rebase before trying to contribute to the discussion. In a nutshell, rebasing means putting your changes on top of the latest changes from the remote. If you do it daily, it is like you started your branch each day and you won’t have any more merge conflicts than the ones done after your daily rebase. Hence, frequency does matter. To understand the topic you are talking about before commenting is also very important.
@@alfbarbolani Is it easier to win an argument by offending others? You should know that rebasing still yields conflicts. If someone pushed a longed lived branch today and you have a longed lived branch, what does it change if you had been rebasing it every hour before that big branch came in? You might still get tons of conflicts if you worked on same files (and the previous branch for example made 20 file splits)?. I know. I've been there.
For me personally there is a lot of advantages in using branches and I'm not convinced by the author of this channel to change my approach :)
Would be nice if you name those advantages so people could discuss it more, maybe you will change your conviction then :)
@@Timelog88 For last several years I"m mostly working in large organizations and with big codebases. The approach is the following: 1. When I need to develop smth, I create a branch 2. When I want to test how my changes fit into the bigger picture - I can deploy a new env from this branch 3. Once I"m done with changes - I create MR or PR(depending on gitlab or github), it"s reviewed bz my collegues. 4. Once MR is approaved - I merge it to master and it"s getting deployed to prod env. For me it seems much more structured and convenient then just committing everything I do to the master branch. Maybe in a very small codebase with the team of 1-3 people it can make sense, but when it's like 10 people in your team and you are working on a shared codebase with other teams - sorry, it's just insane to avoid branching :)
i dunno, why doesn't this channel have a tutorial or walkthrough of this actually happening? because I struggle to imagine it
I've also asked this before, and I feel like a complete idiot every time I watch videos on this channel as there's just so much ideological knowledge and not even a simple practical example of how to do thing X
@@mirceastaicu4131 Dave believes in the scientific process. I believe in thought through action, not through debate. Some things on this channel sound good when he sells it. But you go back to your team, and you think for a second and say, " Actually, no, let's branch. That works better."
Real Example of a Deployment Pipeline in the Fintech Industry. Covers this exact topic.
@@toonsoffun5733 kzhead.info/sun/layEeMioa3pja68/bejne.html
@@toonsoffun5733 This is still all talk. It's just "they run tests." There's nothing about this that makes feature branching a bad idea. There is no proof of the advantages of CI here.
I guess the only way this can work is that each and every feature is properly supported by tests. The high test coverage gives you the confidence you aren't breaking anything when you push to main all the time. Edit: reading more comments, it looks like I might be wrong. I have no idea how you can expect good results by committing code to master that is unfinished and not passing tests.
Code in master is not necessarily code that runs in production... There usually are (automatic or manual) gates that decide when code is pushed to production, forming the deployment pipeline. By checking-in unfinished code to master, you communicate (loudly) to the team that you need help getting this code ready. Anyone can chip in after it has been checked in. Test-driven development does complement this nicely, as it will yield high test coverage.
It’s literally impossible to not do branch based development. When you check out a branch from a remote repo, a new branch is created. It’s actually awesome that we can do that. We should create branches. How often are you in the middle of making a change when a new more urgent requirement comes up? Either stash or commit your change in a branch, then create a new branch for the new work. You can’t do that if you don’t use branches. They let you be creative. They let you try new things without effecting the main branch. It’s perfect for the creative process. It is the worst advice to not use branches. Don’t listen to it. Stale and long-lived branches on the other hand are the devil. If you have a branch, you need to frequently rebase it, but the more code in that branch it’ll start holding up other dev work that is planning to rely on that work. This forces work to be done in serial instead of parallel, and that really slows a team down.
Why do branching at all then? Can't everyone just push to the main branch? This is considering there's no need for deploying all commits and some can be skipped. Also in these settings there are always three environments i.e. test, staging, and production, and deploying to test is for testing purposes in the actual system context! So this way any new feature can be tested safely and very easily.
What I liked the most was the cultural explanation at the end. The Idea of a solitary guy in headphones in a dark place with a glowing screen in front of him (using gender on purpose here) is the ultimate myth / barrier for opening software development to many more people.
We have 2 female grads on the team currently and they are disheartened with the ways of working they just want to get on with stuff and software engineering in the enterprise seems to hate them, doesn't trust them, makes them doubt themselves and makes everything painful. I always try and bring it back to, if you were doing this for yourself how would you approach it, autonomy and ownership on each piece of work but honestly I don't know if they will survive the system.
I was l listening to this with headphones on and I feel pertained attacked
In my case it's because i have large features that often are interdepentent of making large changes to the codebase and i don't want it in production on my upcoming scheduled deployment where other team members are working simultaneously on other things that are supposed to go to PROD. You could manage it with feature flags but honestly it's more of a pain in the ass to set that up than to just use a separate feature branch as you're making lots of changes throughout the codebase which are temporarily in nature and have to be reverted once your feature goes to PROD. Or, you'll leave them in and then end up with a lot of nonsensical IF statements throughout your code and dumb useless configurations left behind from your feature flag adventures, it's just better to feature branch tbh and keep things tidy
If you don't want it in production, are you sure it's worth doing in the first place? Does it belong to the same project?
@@pashadia Yes it's just an upcoming feature, that requires all parts to be completed in order to be releasable, otherwise it would appear broken to users on the production website
I think one underlying assumption not mentioned is that your code base is well tested. If that's the case then you can ahead of integration issues between your work and the trunk much earlier when your silo periods are much shorter. Especially if you're making changes everywhere. Especially if there are feature flags, conditionals, etc then having tests and integrating early are even more important. Also as with any practice you can choose a default and then *sometimes* their may be good reason to break your default.
Abstractions help considerably. Not only during the initial feature implementation phase (when you want to control it by feature-flags) but also as clear intent communicators for the long-term. Often, these eliminate the need of a feature flag. All you end up with is a 'desirable' config. E.g. If you abstract the distribution policy of a load-balancer functionality to be a strategy type, adding a new strategy becomes very low cost. Say, you have Round-Robin or Weighted distribution policies. An existing config specifies which one to use. Initialisation ensure this is used for the entire program execution. Later, supporting a new one for your new big feature is by itself a very low-cost/small change in config and initialisation. So, adding this new Sender-Based distribution policy does not impact any existing deployments until the newest policy is configured explicitly. If you do not want to control this via config, make it a compile-time selection or a hard-coded decision. As you see, the abstraction made the cost of adding the new functionality negligible. Additionally, it becomes an independent functionality and enables the very desirable separation of concerns, allows for independent testability and explicit control of it's use. This is called 'Branch by Abstraction'. See trunkbaseddevelopment.com/branch-by-abstraction/. The more parts of the new big feature you can support this way, the less your code is littered with feature-flag conditions. And, the functional parts you really must control by explicit feature-flags, hopefully, become very small and manageable.
@@ankursatle Abstractions definitely make cutting this up easier, however, now you are suggesting that I implement new abstractions to help safeguard the old code from the new code. While a strategy pattern itself is pretty simple, implementing that also carries its own risks depending how that system was integrated into the project previously, and requires more effort/complexity to an already large/complex task. Additionally, adding a new variable to a config file, now means there are more combinations of configuration that need to be tested overall. Undoubtedly these new abstractions and configurations will add bloat to the project overtime unless more tasks are added to remove them once the old code is deprecated and removed, this again is more overhead and costly to a small team. I think TBD makes sense in large teams that intrinsically have problems communicating how an interface might change over a short amount of time, but this adds complexity to ensure that communication is done through the code. I think of it similar to how microservices scale really well for large projects, but the overhead of creating and setting up microservices just doesn't make sense for a small project that won't ever scale.
Sorry, I'm off-topic… but did you hear about the DevTernity fake speakers mess? Horrible situation for you and the other speakers and attendees who signed up in good faith. You're bigger than DevTernity and I look forward to hearing you speak at many future conferences not run by people who believe female software developers are unicorns…!
There are only a few scenarios where branches are great (a) People develop on different parts on the evening as a side project and do not want to ruin the work on the main branch, as their contribution is not finished yet. Usually, they merge the main branch onto their part every now and then. (b) Library versions which you actually do not want to improve much, but you need to do maintenance on them. (c) You work in science and do experimental stuff you usually want to see in the main branch ever. (d) Your ticket system loves it, but honestly, you if merge back with the main branch multiple times a day, this is just a specifically named local main branch which you also store in the project history. Any additional suggestions?
Switching priorities causes you to need to temporarily work on another feature. You want to run an experiment or two or three to compare different approaches to the same problem. Branching can be a great creativity and productivity tool. I don’t agree with b or c. I think b is problematic because there’s a good chance a patch will need to be applied to both versions, but now you need to apply the same code twice (or more, depending on how many versions you’re managing). It would be better if both versions were supported in the main branch. I think c is probably because data scientists and others have less formal software engineering training. There is less incentive and/or knowledge to write maintainable and shareable software. Even throwaway code can be useful in the future, so merging it into the main branch and making it shareable and maintainable would be more beneficial in the long run. That just an assumption, but backed by some first hand experience.
What? Talking to other people - no way. I'll quit !!!2111121!!
Once again, too much judging and a lot less viable and practical examples of how the "right" way he is proposing should be done ... funny thing, is that KZhead is showing me the "Rockstar developers are the WORST" video as the next to watch ...
CI works with open source non proprietary software... most of corporate america uses proprietary software where most of the core functionality is locked in a black box. The scope of the regression testing for this software can take up to a month... CI isn't applicable to this sort of development in my opinion.
Dave is unfortunately skipping over all the steps required to make a codebase Continuously Integratable. Any codebase could be transformed to use CI. Some are just written so poorly that a lot of work is required to make it happen. That time and effort would probably be worth if it takes up to a month to validate a change. How can a company keep their competitive advantage when their codebases move that slow?
I know of several proprietary software solutions developed by major corporations that you probably interact with every week that were developed using CI processes. I think what you are speaking to is the fact that many software applications developed for enterprises are so poorly architected and difficult to test that it makes the sort of smaller incremental development iterations promoted by CI very challenging to adopt. As an aside, I've worked on highly healthcare software applications which directly impacted the lives of people where teams were leveraging CI. In this scenario, it's hard to imagine not using CI practices because the risks were so high.
Why would anyone still work when beaches, champagne, and caviar exist? You guys have a skewed conception of reality. Not everyone is positioned to work the way they would like, and not everyone would necessarily like to work that way even if they could. You can say they are doing it wrong, but if they are delivering value for their clients and users then who are you to talk? You aren't the ones who get to make that evaluation. Jez's attitude in dumping on everyone who doesn't do it his way certainly isn't going to help align people with his views. Enjoy your virtue signaling, the rest of us have actual practical problems to address, including doing the hard work to actually make our processes better rather than just telling people they should be better.
It is not "virtue signalling" what Jez says is based on research, Jez is not dumping on people he is dumping on processes, these are not the same thing.