I started using Herdr and I really like it. Since the main features I wanted are already there, I have no issue with the commercialization. I still wonder how this can be turned into a profitable business. It'll be really interesting to follow Herdr's future. All the best to the founder!
They said the same when Japanese car makers were trashing European and US manufacturers in the 1980s. Yet they adapted. Then they said the same when the South Koreans trashed them, yet they adapted. Now they say the same about the Chinese.
Sure, past performance is no indicator of future performance, but all the doom-saying and hysterics are just that, fear of the unknown.
They adopted in the sense of surviving. Considering their head start and massive government support for the industry, their market share compared to Japanese and Korean cars is a joke, with the Chinese it will get even worst because this time they will be pushed out of their home market too.
This comes from a die hard bmw fan. Facts over feelings.
I'm not a car fan of any kind or shape. I'm talking about things that happened. I say let's wait and see. You say you know the future. You tell me "facts over feelings".
I accept your negative outlook and respect your right to hold that opinion. But, and I'm saying this in the most respectful way possible, let's have a little intellectual honesty here, please. Your statement is more speculation than mine.
I'd like better understand the German engineering culture. Why was it special in the past? What about today?
As a kid in the 80s, I had some things from Germany and their quality was astounding. Fast forward to present time, our Volkswagen car made in Germany is the worst car I've ever had in terms of reliability.
There is no incentive to build a reliable product that can‘t compete in price. People are trained to consume and throw away. People are getting used to throw away cloths and household appliances after 1-3 years.
Miele once built washing machines that outlast 30 years. They offered spare parts and regional service hubs. But the market decided to just buy the 50% cheaper ones every 5 years.
In the last 25 years, German companies got into "reliability engineering". Which is the art of adjusting your drawings in such a way that every component lasts exactly as long as it has to, and not a second longer.
Which is why back in the day, there were some car/washing machine/tool brands and models that seemed to last forever. Whereas now the current models last exactly their warranty/leasing period.
Edit: Other manufacturers do this, too. But for them, it's often coming from the other side of the reliability curve, so they are improving, the ones that were better back then are getting worse.
What I'm trying to understand is why Toyotas can still make great cars at prices comparable, if not inferior, to Germany's (at least here in the US). There's a quality gap there. I don't understand why German cars used to be great looking AND reliable but no longer.
Toyota builds cars that a private person might want to buy. Their market is real people who want to keep the car for longer, pay for their own maintenance and maybe even do their own oil changes. And who maybe want to put some kids in the back seat and stuff the trunk full of their shopping.
Many German manufacturers (esp. the "premium" ones) don't build for people. They build mostly cars that get leased as company cars. So price isn't that much of an issue, longevity beyond the 4 year leasing term isn't a problem, excessive maintenance isn't a problem since it'll be included in the lease. Oh, and the trunk is just for a laptop bag and the overnight clothes for that one passenger anyways, so nobody cares it's impractically small.
Also, from what I know, design cycles for German cars are relatively long. A model won't just be changed, at least not once per year (as USians seem to expect to the dismay of German manufacturers). While this is great for cost-optimizing the manufacturing process and might be nice because replacement parts are easier to find, this has a downside: Japanese car makers practice continuous improvement, meaning that a change can go in at any time, as long as it seems advantageous. So maybe they learn more quickly from their past mistakes and overall build a more solid engineering experience through this continuous fiddling...
They still do. But as you said,most people go for the 300€ product and not Miles 900€ one. I had the chance to tour their electronics production lines in Gütersloh, Germany a few years ago which were top notch, almost automotive standard, and they told me, that they plan, develop and manufacture for 18 yrs lifetime. Not sure how long the cheaper products last, any experience?
The problem is that even if you want to pay a premium, you can't be sure you're getting a better, more reliable product. It's safer to gamble on the economy option and replace as needed.
If you inflation adjust prices of mid-range major appliances from the 80s they'd be massively expensive today. The price differential is the quality that has been engineered out of all products to be competitive.
Yeah, even if you look up reviews of a particular model, chances are the manufacturer doesn't make that one anymore (it's over a year old!) and just has some seemingly-identical one with different failure modes.
I'd love to buy quality products (and do when I can!) but it's really hard to get good information as a buyer.
> I'd like better understand the German engineering culture. Why was it special in the past? What about today?
German engineering culture is to think things through. To take everything into account. To do it properly, and not mess around. To rather think a bit more and make sure you get everything right.
Basically very similar to German rules culture.
Problem is, both of these cultures have their downsides as well. Rules as well as plans get overengineered. The belief that "one more rule can't hurt". The belief that "taking a step back yet again and discussing it a bit more is always worth it". At some point, you get stuck in a maze of rules, contradictory plans, meetings over meetings with no action. Engineers are scared to invent and engineer, because there surely will be some rule that could be violated...
Since dieselgate I no longer believe in German engineering. That, and that they just can’t make their trains run on time. Never have I taken an ICE train that was not delayed. How could that happen in an actual engineering culture?
The difference is the planning, I'm building my tool around a collaborative use case. I'm often doing multi day bikepacking trips with my close friends. The planner supports real time collaboration, so multiple people can plan on the same route together.
Also for me it's an experiment in AI driven development (which I hope won't alienate people to much).
After my experience in consulting museums, I found that they spend a lot of money on software, which could potentially be replaced with open source software. Therefore I started an awesome list with FOSS software for museums: https://github.com/smartcompanion-app/awesome-open-source-mu....
i am not sure acctually of the math is acctually that complicated/important. the math around neural networks is calculus/chain rule etc and for model comparison/validation one needs statistics. the required math for e.g. understand transformers is quite accessible.
When reading your comment, it just reminds me on how feature flags can be misused as application configuration/customization. An antipattern i could observe at various organzations already.
For me feature flags go along with trunk based development to enable features in QA settings, but not on PROD yet, for PO/PM testing. Trunk based development allows for fast/easy devops, without complicated branching strategies.
Application configuration is, for me, part of the application and has the business context for customizing the application accordingly. Not sure if there are specific frameworks/tools out there. But one should clearly distinguish these two.
> it just reminds me on how feature flags can be misused as application configuration/customization. An antipattern i could observe at various organzations already.
feature flags are perfect for configuration and customization, why using them for this purpose is 'misuse' is beyond me and I've heard this claim from multiple people. they're literally configuration. feature with a flag to turn it on, off or give the flag a value. where's the misuse? is it a problem I'm not running experiments when switching over redis to valkey or whatever?
Feature flags need to be treated as short-lived and experimental otherwise they end up getting abused for everything and make it very difficult to reason about your application.
If it's config/customization, it should be in code. If it's experimental it can be a flag until it solidifies, and then it needs to get moved to code.
When I was at Shopify a couple of years ago they mandated that feature flags had to be short-lived (Like 2-4w lifetime tops, some had exceptions) because they would end up getting left in code and never cleaned up, or for extended periods of time like months. Hard to tell if it's genuinely a "feature flag" or actually just a normal part of the system at that point.
Feature flags being flipped in prod was also a major source of incidents, in part because people didn't treat them as experimental and with the associated risk profile of something experimental.
The only exception where having long-lived flags was useful and required was for operational killswitches (E.g. disable Apple Pay because it's having issues), but that is explicitly not application config.
This is the kind of design wisdom that’s both true and difficult to win an argument over.
It reminds me of arguments related to over-engineering and complexity. The principles are super important to having a codebase that scales and continues to be efficient to work in as the team grows, but they are hard to objectively measure.
Locally or in isolation something may sound like a great idea. Being able to step back and see the greater ripple effects require some experience and intuition that can’t always be used to convince people otherwise.
I disagree with just about everything you said being a problem except the process of cleaning up is absolutely required.
Notably feature flags triggering incidents is expected and desired vs the alternative of shipping the code and having to roll a release back because there is no other way to remove the feature from prod.
In a company the size of Shopify people flipping their feature flags would very often impact *other teams*, and like I said feature flags got abused with even seemingly innocuous changes being put behind them or being left long periods of time before being fully used.
When someone else flips a flag that impacts your team and they have no idea they even caused a problem, it becomes very difficult to resolve the issue. Usually you can check for recent deploys, instead you have to go and guess at which feature flag which was recently flipped could possibly be affecting your code. I experienced this several times.
Also, it was actually more desirable for most of these things to go straight to production. Test it properly before shipping, then when you ship it soaks on a 5% traffic canary at which point you can monitor and cancel the deploy if you see errors. That is generally safer than a feature flag rollout unless you are doing something very high impact/risk, in large part because it gives any other team affected by your rollout the ability to respond and be able to easily find the source of errors.
In my org it was a fairly common failure mode to ship something and accidentally cause an issue for another team. Usually it was other teams/orgs shipping things that impacted us.
Runtime evaluated feature flags can always be used for control plane levers and emergency handbrakes.
You just have to label them as such and prevent other teams from fiddling with them.
This is not an antipattern, it's just semantic hand-wringing.
My team managed critical systems in the online flow of billions of dollars of daily payment volume. We also wrote the feature flag system that the rest of the company used. Not only were we completely fine with feature flags as long-lived control plane levers, we heavily used the system that way ourselves.
You just have to clearly distinguish between ephemeral rollout flags (and clean them up or expire them) and the permanent control plane levers.
It's the exact same functionality for both sets of tools. Just different practices around the two usages.
I completely agree with your distinction and that is exactly what they mandated :)
I don't think that is what most people colloquially mean by "feature flags" though. Even most teams in Shopify abused "ephemeral" flags for long periods of time.
When they rolled out the mandate it was very annoying for my team because we had a lot of operational flags like you're describing that we needed to get exemptions for.
One well known issue is that when you have a lot of separate feature flags that can interact, you explode the number of test cases you have to cover. For example if you have three feature flags that can interact in a module that has 100 test cases, you actually have 900 test cases if you are going to test with each possible combination of flags. Many teams don't test them all because they "already know" that doesn't apply here, and find out in production which combination of feature flags is unworkable.
But you have this same issue if you store those 3 bits of state as "app config" instead of as "feature flags".
I think it's useful to distinguish between kinds of feature flags -- "traditional" feature flags for safe rollout of new features (these should be removed on a strict timeline, to preserve maintainability) vs. "config"-type flags that are designed to remain indefinitely and optionally be configurable by certain end users. But I don't yet see a reason why the actual mechanism for these two things (namely, a function with a descriptive name that can be called to quickly return a small datum that is periodically refreshed from somewhere in the background) can't be the same.
Yes people are speaking past each other to some extent here; but if we're going to talk about "Feature Flags" we aren't talking about using a flag service for "normal" configuration - whatever that means. But what, exactly - does that mean? I normally consider things like the following configuration:
API Host Names, client Ids, secrets,
Database connection strings
Other runtime settings like pool sizes, replica counts etc
Most of those things are secrets, or they are specific scalar values closely tied to the application runtime, that often need to be known at startup. Are people putting those in a "feature flag" service? If not, what is a good example?
My comment only applies to "configuration" that alters behavior, usually in binary off/on manner. Which is what I'd call a "Feature Flag".
Yes, feature flags are conflated with remote configs (or its more useful variety: "dynamic configs"). The difference is subtle, hence why people are talking past each other.
Feature flags are gates for whether a piece of code runs; basically, an if-condition. Remote configs are a mechanism for changing runtime values without redeploying[1].
For example:
# Feature flag — variant gate for rollout
flag = sdk.check_gate(user, "checkout_flow")
if flag == 'open':
render_new_checkout()
elif flag == 'warning':
render_warning_checkout()
else:
render_old_checkout()
# Raw remote config pulled — structured values for tuning behavior
config = sdk.get_config(user, "checkout_settings") # if the config changes based on user or context, this "remote" config is considered "dynamic"
timeout_ms = config.get("timeout_ms", 5000)
max_items = config.get("max_items", 50)
allowed_tlds = config.get("allowed_tlds", [".com", ".org"])
In practice, feature flags are implemented on top of dynamic configs[2] to manage the temporary lifecycle of a feature — aka, ship a new block of code, ramp its execution up to 100%, then delete the flag. Whereas dynamic configs are a deeper primitive meant for semi-permanent/safer operations like tuning rate limits or changing text copy on a marketing website.
As I've seen it: the forcing function that separates the concepts are experimentation platforms: when human-control of feature flags is shared (via dynamic configs) with automated & randomized assignments. That's how Statsig built their system and, in part, why they could sell for a billion. Whereas companies that ignored the difference, like LaunchDarkly, struggled outside of feature flags.
I think feature flags, remote configs, and experiments are all the same thing. Semantically they differ in how you're applying the config and interpreting the outcomes.
That's not a feature flagging service then (config as a service! not a thing really...)
I've done both client and server side implementations of the launch darkly sdk and that's how it's done to know client context.
If you're initialising the entire SDK only to load 1 set of configuration items, I'd argue you can host the config as a json file on a CDN and be done with it - feature flagging is overkill.
All good - it is something that has been over-complicated for marketing/product reasons.
If you know what A/B testing is, feature flagging can really allow you to nail down how you should deliver an experience to an end user.
We track end-to-end engagements on our websites based on how the content is displayed on a page with more performant layouts/content winning the test to drive users through the funnel.
I don't love it though because it's a lot of waste that gets left over and not cleaned up leaving billing to grow exponentially as flags are continually called for no reason.
reply