Making decisions - When should you listen to the tech community in adopting a new technology?
There's an entire ecosystem dedicated to churning out bad advice out there. Be judicious.
The tech community has mastered the art of giving terrible advice.
It is everywhere. Not a day goes by where I don’t run into a blog article espousing a new framework or library, encounter some massively upvoted Hacker News post on yet another data management technique for the current flavor-of-the-month javascript framework, or a meet-up talk about a brand new technique implemented at some company that saved them tons of time or money.
It seems as if the entire tech ecosystem is yelling in unified consensus about the latest and shiniest new thing.
Sometimes, it makes me feel like an idiot because I’m just not using it.
What do junior developers do?
When a junior developer encounters that kind of community consensus, what are they most likely to do? They’ll adopt whatever it is the community has “achieved consensus” on, often without regard to whether it is appropriate for the context.
They’ll start adding the new technology to their company’s projects or implementing the new technique in the codebase.
As they do, they’ll gain experience and expertise in using it, and then start convincing others on their team to do the same. They’ll link to a multitude of very convincing articles and references that simply can’t be argued against — an appeal to the ambiguous authority of the community.
What’s the problem?
The problem is that not all of these ideas are good ideas for all situations, and junior developers haven’t built up the experience and knowledge to determine appropriateness.
By definition, they don’t have the knowledge or experience to filter the good from the bad. Without guidance and appropriate direction from senior engineers, they are likely to cause insidious long-term damage for a small boost in short-term gains, or struggle needlessly to adopt something that just won’t work in their situation.
The reality of it — there’s way more junior developers than senior developers in our industry. It may not be the case in your company, but realize what this means:
There are products out there being built entirely by teams solely composed of junior developers, without any guidance from senior engineers, and with only the community to guide them. This is the case in every single industry, including defense, finance, medical, etc. To achieve success even with the odds stacked against them is impressive, but there’s always a cost.
These products and technologies might work well on the surface, but have hidden issues lurking beneath the functioning veneer. Do you want data leaks in your financial systems? How about dosage miscalculations in your medical dispensing?
It’s a scary thought.
I’ve made these mistakes, too.
I’ve been bitten by the very behavior I’m writing about here. I used to be that developer, blindly listening to the community.
Back in the days of Google Web Toolkit, a pattern called Model-View-Presenter was all the rage. It called for abstractions upon abstractions in order to decouple data, the views, and the wiring in-between.
As a new developer, I didn’t have a senior developer to guide me. I did a lot of research. I judiciously studied and followed the advice of the community. I understood the theoretical reasons for why it was being done and how to do it, but since I was a junior developer, I didn’t have the experience or knowledge to decide whether I should do it or whether the issues it was addressing were even present.
As a result, I applied it everywhere. That’s what the community told me to do. I added it even on pages where it would never have a variant or take advantage of the benefits of the pattern. I perfectly followed the pattern in all situations, as instructed. I convinced others to do the same with links to articles and podcasts and videos and lectures.
I used to be that developer, blindly listening to the community.
After a while, the system became an unmaintainable mess. I would have been better off hard-coding everything. The layers of unnecessary abstraction it created would later make even changing text a significant burden to whatever unfortunate soul had to maintain the system.
I was lucky because that developer who had to later make changes years down the line was still me. Having to struggle through fixing my earlier mistakes helped me learn a lot of lessons early on. While the MVP pattern was useful in many situations, the one I applied it in was not that situation. As a result, it caused significant friction in implementing later changes.
It taught me what might have been my most valuable lesson of my career so far: context is king.
Context is king
Of the good advice out there, much of it is written for a specific context or situation. However, many developers encountering this advice forget that what works for one company might not work for another. What makes one team highly effective might bring another team to a dead halt. A technology that works for one situation might not be a good fit for another.
Look at what happened when the NoSQL hype train arrived. Articles confidently declared that document storage was going to be the wave of the future and that structured data was dead. Technologies like MongoDB made it easy to set up, easy to run, and easy to build on. Coding bootcamps around the world taught it to hordes of new developers, without teaching them when to use it.
Who needed schemas anymore? As far as the tech community was concerned, nobody.
Well, it turned out that people who needed schemas needed schemas, and that ended up being most software that did anything even remotely interesting.
Now, these trendy schema-less, free-form data stores that everyone implemented were liabilities that caused security, performance, and integrity issues in many software systems.
By then, the community had moved on to the shiny new resume-builder.
Trying to lift a technology, strategy, or technique wholesale just because some internet articles tell you to do so is like buying the world’s top rated stroller when you don’t have a baby. It might be a fantastic stroller, but it’s a waste of resources for you specifically.
What makes one team highly effective might bring another team to a dead halt. A technology that works for one situation might not be a good fit for another.
Blind leading the blind
If the “tech community” has a significant number of junior developers consuming and creating material, it means two things:
A lot of the articles and talks will be from junior developers who are unqualified to be giving advice
The articles and posts from experienced developers will get misinterpreted, misapplied, and written about as a success story, which will lack the nuances present in the original messaging.
Talks with titles like “Monoliths are bad” or articles calling GraphQL the “REST killer” pop up, promising a new silver bullet. These talks are then attended by many other junior developers who happily and eagerly consume the information and go off and do it themselves.
Now, don’t get me wrong —it takes courage to publish a piece of content and open yourself up to criticism and judgement with the intention to better their community and help others. I applaud that tremendously.
Nor am I knocking the drive to learn and desire for improvement — these are things I and many others look for in developers. A developer that doesn’t learn is stagnating, especially in an industry that is constantly changing.
However, many developers who read or consume these pieces of content don’t look past the advice itself. If they did, they’d realize that a lot of advice comes from people who haven’t built very much, if anything at all.
Being a part of a large or well-known company doesn’t automatically make a developer qualified to tell you how to structure your teams. Nor does releasing a library make someone an expert in how to architect your software. It might be an indicator, but it’s not assured and certainly shouldn’t be assumed.
a lot of the advice comes from people who haven’t built very much, if anything at all.
Most developers are not experts
The uncomfortable truth is that most developers are, at best, mediocre.
It makes sense — most businesses don’t need expert developers to succeed. A vast majority of problems encountered by businesses that technology solves are not complicated. How many developers out there are working on the world’s umpteenth CMS, or building yet another bar chart? Probably a lot, and maybe most of them.
This might make some founders and engineers bristle. Every company likes to think they are unique and game-changing — special snowflakes. However, at the end of the day, a lot of the technology out there is just a rehash of stuff that has existed before, slapped with a new label, facade, and re-organized in a way that provides distinct value. CRUD apps are CRUD apps.
Even in organizations with more complex challenges, I’d wager many have a handful of senior developers making the complicated decisions, which are then executed on by more junior developers.
It’s how companies work: few have teams that consist solely of experts.
In reality, the skill distribution of developers is not a bell curve. It’s a cliff with a very long tail representing the truly skilled, following a power law.
Most of the developers in the industry have less than 5 years of experience. Of the ones that have more, there’s many that have simply coasted: we’ve all run into senior engineers that didn’t know how to build anything, or haven’t advanced their skillsets.
There’s only a small percentage of software engineers who are experts, and consistently do it well.
Consequently, most of the articles out in the community are not written or read by experts. While senior developers can discern and apply the advice appropriately, most junior developers cannot tell apart what is baloney from what is not.
These junior developers are incentivized to parrot the information and follow the advice, to destructive results. When a developer’s only argument for doing something boils down to “this company does it” or “the community does it this way”, it’s a signal that it is being applied without proper consideration.
Systematic issues
There is a perverse incentive system within the tech community that greatly amplifies the effects and reach of terrible advice.
Open source contribution is associated with notoriety and is a marker for success
Fierce competition amongst junior (and senior) developers causes self-promotion to be a viable ways to compete
Developers self-promote and produce content to help them stand out and get a job
Other developers consume and implement this content in their companies to ensure that they are doing “the right thing” and that they don’t fall behind
These inappropriately implemented tools and techniques end up crippling long-term viability of systems, damaging organizations and increasing costs
Open source = skilled developer?
Our industry generally regards open source contributions as a bragging right.
There’s an entire incentive system out there that rewards developing something that is widely used. Creating a successful, widely used library is (correctly) a noteworthy resume-maker. Having a repository with thousands of Github stars is used as a mark of success. These are credentials that says “hire me”. Who wouldn’t want to hire the creator of Redux, a maintainer of Ruby on Rails?
The fame is a huge allure for many developers of all skill levels. It incentivizes them to create libraries, promote them, and get them used by as many people as possible, regardless of whether it will actually benefit them or not in the long run. Developers hop on to this bandwagon by creating content for the new technologies, writing articles and promoting it.
The problem is that there’s a larger reward in creating new things over creating valuable things.
The lowest hanging fruit to build are libraries that do the same thing as other popular libraries, but only better. Look at how many ways there are to upload an image on Rails — Paperclip, Shrine, Refile, Carrierwave. How about the number of ways you can do state management— Redux, Mobx, Recoil, context, etc. The options are numerous, and ever-increasing.
When everyone will praise you for creating something new, people are incentivized to reinvent the wheel and talk about how awesome their new wheel is, which drives usage.
Within this ecosystem exist junior developers
Junior developers have to operate in this ecosystem we have to succeed, so that’s what they do.
It’s easy to forget, but the competition at the junior level is fierce. The junior developers most likely to be hired are the ones that stand out. They’re told by everyone from their community to mentors to teachers that the best way to stand out from the hordes of other junior developers is to show passion and knowledge.
They show knowledge by working on side projects, studying data structures and algorithms, and demonstrating they can use these new hot, in-demand technologies. They demonstrate their passion by writing blog articles, giving talks, and engaging with the community as a whole. It’s what helps give them an edge in their job hunt.
In their studying, they consume a plethora of content that talks about things on a surface level with an air of authority. Much of this content is essentially marketing spiel that heaps high praise on a specific technique or technology and recommends it without limit.
Just look at articles on Redux, GraphQL, or MongoDB. These tools are absolutely useful and appropriate in the right contexts, but it’s rare to see a nuanced recommendation or one that adequately explains the context behind the decisions.
When everyone will praise you for creating something new, people are incentivized to reinvent the wheel and talk about how awesome their new wheel is, which drives usage.
Lowering the bar
A lot of the new content is designed to lower the barrier to entry so that it is easy to adopt. Code samples show you the exact code you need to copy-paste to get up and running. FAQs show basic errors you run in to. They often invest significantly in the ease of setup.
Unfortunately, much of the guidance after that stops there. Rarely do I see examples that show you what not to do, or how doing something as shown will create an integration challenge in the future.
That’s to be expected — integration is not the library’s job.
However, this lowered bar and lack of context does cause people who don’t know better to follow these patterns blindly. Eventually, through experience they realize that they need to make changes, but by then it is often too late — the damage has been done.
It’s easy to understand how this happens: most articles rave about how you should use whatever they are talking about starting today because it is just so easy to use and set up, and light years above the current status quo. It deceptively leads people who don’t know better down the wrong path. They then repeat this misinformation in their own presentations, articles, and conversations until it becomes solidified as truth in the community.
People then chase shiny things despite marginally better value. Systems end up having to get rewritten, or start incorporating a dozen different ways to do the same thing, costing companies significant time and money.
How do you avoid it?
How are you, as a developer who is rightfully consuming as much literature as possible, going to distinguish between content that is written from a tried-and-true perspective or content written as a puff piece with no experience beyond one or two sample projects?
Get background information
A right solution applied to the wrong problem can be like throwing water onto an oil fire.
The best way to avoid incorrectly applying the wrong solution to the problem is to understand the principles and context under which that solution is being implemented
If you’re lucky, the creator of the piece of content has explained their context to you. Perhaps they work in a massive company on a R&D focused team. Maybe they work on the sole infrastructure team responsible for keeping systems running. Perhaps they are the only developer in their company and need to crank out features as fast as possible. Maybe they are in a product organization that has to balance the needs of dozens of stakeholders. If there’s a significant mismatch between their problem space and your’s, that should raise a yellow flag.
If you’re not lucky, you’ll have to suss out clues. Don’t blindly trust advice from a faceless developer — dive into their background and experience. Look up the author’s employment history. See what kinds of companies or systems they have worked on or are currently worked on. For example, a developer with significant agency experience may be good at starting projects, but I would look further for evidence of an ability to maintain them. Likewise, a developer who spent a decade at a Fortune 500 company may give advice that is inappropriate for a startup. Weight advice accordingly (but don’t dismiss good advice just because a junior developer gave it).
Whoever it is, make sure you understand in what situation the solution is being applied. A right solution applied to the wrong problem can be like throwing water onto an oil fire. Be diligent and judicious.
Adapt the solution
Solutions need adapting and molding to be implemented in your company, even if it is the right solution for the right problem.
Maybe MongoDB is the right solution, but your department’s budget doesn’t allow for another vendor license. While writing everything as a micro-service is truly the answer for an organization with a dozen teams, but your team of 4 developers doesn’t quite need it right now, and a service-oriented approach will be more valuable instead.
Whatever it is, check to see how you can adjust the solution to fit your context.
Avoid inconsistencies
I’ve written about the challenges of adopting new shiny things. From consistency issues to wasted effort, adopting a new technology causes business and technology issues that can last the lifetime of the software you are incorporating it into.
Here’s the deal — if you’re a professional software developer, you have a responsibility to not do things that will screw up your business’ future. This means doing what you can to minimize risk. For most developers, that means experiment on your own time and in your own space. Don’t bet the business’ future on something unproven unless it is your job to do so. If you have to experiment, do it on a side project or in an isolated, unimportant area.
This advice generally applies to organizations that have limited resources. If you’re in a larger company where moving fast and breaking things is par for the course, you can afford to do things differently because innovation risks are supported by the rest of the business. However, if you’re in a small startup with only a year of runway, it is likely that major mistakes you make can ultimately sink the company’s chances of success.
Hopefully, there will be effective senior leadership keeping an eye out for you. But, as we all know, many companies have no such leadership. In these situations, it’s your responsibility to weigh things carefully and with due consideration.
you have a responsibility to not do things that will screw up your business’ future
To writers
Finally, if you are writing an article — provide context and disclaimers. The solution you are proposing or writing about is likely not applicable in every situation. Even mentioning that explicitly is a gentle reminder to use thought before applying it.
Even better, go in depth as to why this solution worked in this particular case, and in what situations it would fail — this would help others build context and determine whether it is appropriate for their environment or not.
I know I’ve forgotten to do this many times, and who knows how many I’ve led astray.
Now that you’ve reached the end of this post/rant, what’s the first thing you should do? That’s right — make sure I’m not just spoon-feeding you a bunch of crap, because I totally could be. Odds are, I’m one of the mediocre developers, so take my advice with a grain of salt, and weigh it accordingly.