This is part two of the “From Demo to Done: Engineering Hardware That Actually Ships” series discussing some key high-level challenges of the hardware product development process and how to tackle those problems so that you can mature your design to be ready for manufacturing.
Today we’ll discuss an attribute that everyone wants their product to possess: reliability.
Reliability Is Not Free
Before joining Simplexity, I was a Senior Systems Engineer for a startup whose product was a 10kW Wind Turbine. The CEO of the company did not have a product development background. Typical of those without experience in the field, his expectation was that if the engineers did their job right, we’d CAD up the design, order the parts, assemble and the product would work perfectly. Of course, that’s not how things work in the real world.
Any product design is the result of many thousands of decisions made by the design team. Beyond that, thousands of additional decisions are made implicitly by the designers, suppliers, purchasing, management, etc. There simply isn’t time to think through every minute detail of the design unless you have limitless schedule and budget. I’ve never seen a project without those constraints, so it’s unreasonable to assume that the work will be perfect.
Once a product is released into the wild, all those decisions, explicit and implicit, will be tested. Mother Nature is brutally unforgiving and will eventually check every decision you make in the design of your product. If any single decision isn’t quite right, your product can experience a failure of some sort.
What does this mean for a business that wants to develop a product? The critical takeaway is that you need to treat reliability, specifically the level of reliability, as a requirement, and you need to understand the cost and the business value tradeoffs associated with different levels of reliability. High reliability is expensive to develop and of considerable business value. Low reliability is cheap to develop but often costly in the marketplace due to product returns, unhappy customers and loss of reputation.
At Simplexity, I’ve dealt with a lot of different companies doing product development over the years. There are three errors I’ve seen repeatedly from clients when it comes to thinking about how to deal with reliability in a product:
- Thinking that they are in some way special: “We’ll just do it better” but not having a better approach. Basically, they just ignore the issue and hope it won’t become a problem.
- They “trust” us to do a great job because “you’re the experts”, but then they don’t trust us when we estimate the budget and schedule required to deliver on their quality goals. They cut the budget and schedule for doing the risk reduction, testing and iterative design work required for a reliable product. In this case, they essentially violate the “iron triangle” of project management. That unforgiving law states that out the three high level constraints: features (high reliability), schedule and resources (budget), only two can be fixed. The other must remain a degree of freedom for the project to be managed successfully. These program teams try to specify all three, over-constraining the problem, then they end up dealing with the unfortunate consequences down the road.
- Accepting risk up front, but not willing to accept the consequences of risk: “We’re willing to take risk to go fast”. This is a valid strategy if you do it with eyes open: if you place risky bets, you’re going to lose some of those bets. Play out the scenarios around each risky decision to see how much risk you can really afford to take. Some things simply won’t go your way.
When you start thinking about how reliable your product needs to be, you need to be brutally realistic. Understand both the cost of developing a reliable product and the costs associated with different levels of failure rates. Then make the tradeoff that is right for your business.
Reliability and Development Cost
So, how does one go about assessing the cost of developing a reliable product? It is, of course, highly dependent on the complexity of your product. Developing a reliable remote control for a TV is going to be much less costly than a reliable electric car. However, there’s some math that is not too complex that can help guide you, and that math will lead us to a rule of thumb for getting an idea the cost.
Product reliability is commonly described of in terms of “nines”, where each nine is one digit in the overall reliability with respect to time in operation (ex: time between reboots for a computer) or number of uses (ex: number of cups of coffee produced for an automatic coffee maker). For the coffee maker example, reliability of one nine (fraction of 0.9 or 90%), means that you would have one failure out of every ten cups you attempted to brew. Two nines (0.99) is one failure out of a hundred, and so on. High quality consumer products tend to be about four nines (99.99%) reliable. Beyond that, the development costs tend to be too high, even at premium price points. Products at five or more nines of reliability are required in commercial applications like telecommunications infrastructure, aircraft, etc.
The math of reliability follows the power law of the Pareto curve. Typically, the equations are defined to follow an “80/20” distribution, where the majority of the failures (80%) come from a small fraction (20%) of the causes. When using this relationship to analyze reliability, we rank each failure mode by how likely it is to cause a failure. At the Kinetic Conference, Michael Hammersley showed what the curve looks like:
“Chasing 9s: failure analysis & hardware reliability”, slide 23, courtesy of Michael Hammersley
The first failure mode causes most of the failures. Finding and fixing that is obviously critical. By the time we get to the third failure mode, we’re at > 90% reliability. We’re solving problems and feeling great! However, the graph reveals a harsh reality of the Pareto distribution: by the time we resolve the third failure mode, we’ve already hit the knee of the curve, and we’re quickly experiencing diminishing returns. Michael’s next slide shows exactly why reliability is so difficult, and thus so expensive:
“Chasing 9s: failure analysis & hardware reliability”, slide 23, courtesy of Michael Hammersley
The axes of this chart are logarithmic, not linear. Each major tick mark is 10x the previous tick mark. To increase reliability to the next nine, we follow the probability curve down and to the right. We see that increasing reliability an additional nine means that we have to find and address many more failure modes, and the number keeps increasing! Resolving the first four defects nets us our first nine. The next one means resolving 32 defects, then 236 and so on.
As demonstrated in the graph above, our first rule of thumb for understanding the cost of reliability is:
Each nine of reliability means finding and fixing approximately another order of magnitude of defects.
There’s another implication here: each incremental defect has a slightly lower probability of occurrence than the last one. That means it occurs less frequently and is thus harder to find and reproduce. At the Kinetic Conference, Robert Luan gave us our second rule of thumb for the cost of reliability:
Every “9” of reliability you add is the same amount of additional effort as the previous one.
Think on that statement a bit. Generating your first design and building it is a major undertaking. Getting that product to a high level of reliability in production is going to require several multiples of that level of effort.
As a practical example: when I worked at HP designing consumer grade InkJet printers, we would typically do about 4 full prototype builds per product development. Those were fully tooled builds, not prototype production processes. Builds of 500 to 1000 units, which were then tested for weeks to months, and we’d discover and fix new failure modes the whole time. That was the level of cost and effort required to keep the cost of field failures to a reasonable level for a product that would ship millions of units per year.
Minimizing the Cost of Reliability
We covered why reliability is not free and gained some insight into how much it might cost to develop a reliable product. Is there anything that we can do to reduce the cost of reliability? The whole point of the product development process is to mitigate and contain the financial risk of product development; to get your product to the market at the target reliability levels as cheaply as possible. There are some thoroughly vetted, industry standard practices that should be followed when developing your product.
Test early and test often – The earlier you find and fix a defect, the cheaper it is.
- If you find an issue when you’re generating your product requirements (say conflicting requirements), it only costs you the change of a requirement.
- If you find it once you’ve cut steel for a molded plastic part, you’re in the tens of thousands of dollars of scrap, not to mention the schedule hit.
- Follow a defined product development process and go through the phases carefully to root out as many defects as possible at each phase.
Design for testability.
- Testing is expensive. During your various prototype builds, design in features specifically to help you run tests and to collect as much data as possible. If these features have meaningful cost, they can be removed before manufacturing launch.
- Any “smart” product has a microcontroller built right in. That means it can become a data acquisition system for testing. Invest in the firmware to get that data.
Don’t integrate subsystems until you’ve tested and proven them to be working in isolation – Integration is really hard. When you integrate subsystems, you’ll discover a whole new set of defects that didn’t arise in subsystem test. Often resolving these defects requires the expertise of designers from all of the interacting subsystems. Make sure they’re not fighting problems that could have been resolved before integration. Developing reliable subsystem before integration always has a net positive return on investment. The proper strategy for subsystem test depends on whether your product is high volume or low volume.
- High volume products – These products tend to be highly integrated to minimize material use and thus overall product cost. Often a subsystem relies on other subsystems for critical infrastructure like mechanical mount points, electrical power, or computational resources. For high volume products, invest in dedicated testbeds. Development of these testbeds are often small projects in and of themselves that can be great for outsourcing.
- Low volume products – Typically these products are much more modular. Each subsystem is independent enough to test mostly stand-alone. Make sure that you develop your integration plan so that time and resources are provided to the subsystem teams to verify their subsystem ahead of integration.
I’ve seen these practices skipped many times in a misguided attempt to reduce development cost or to pull in schedule. Unfortunately, I can’t think of a single time where doing so actually worked. In every case I can think of, these shortcuts ended up increasing both the cost and the schedule relative to what they would have been if they just would have followed the correct process in the first place.
Product reliability is not something you get for free by hiring smart engineers and hoping they make all the right decisions. It is a business requirement that must be chosen deliberately, funded appropriately, and developed through disciplined engineering work. If you want a product that delights customers in spite of everything that those customers and Mother Nature will throw at it, you need to make reliability a priority. Doing the work up front may feel expensive, but in my experience, it is always cheaper than the alternative.
Software: Design Complete
Hardware: Pre-production units for design verification
MFG. Readiness: CM schedule and budget, Unit build tracking