Pushed by the necessity to objectively measure the progress of their verification efforts and the contributions of various verification methods, IC designers have adopted protection as a metric. Nonetheless, what precisely is being measured is totally different relying on the underlying verification know-how in use.
In Half 1 we outlined the textbox definitions of simulation and formal protection, after which we briefly launched the dangers inherent in arbitrarily merging these metrics collectively. On this article, we’ll use progressively extra complicated RTL gadget beneath check (DUT) code examples for instance the difficulty that may happen when evaluating the outcomes from simulation and formal side-by-side.
Instance 1: Primary illustration of simulation vs. formal code protection
Recall that code protection is solely the proportion of RTL code measuring the variety of statements in a physique of code which have been executed by means of a check run. Whereas it’s vital that the testbench can train all the RTL code—there isn’t any lifeless code, implying a bug within the DUT—the design can nonetheless be lacking vital performance, and/or the paths to key capabilities may be in violation of the specification.
The next easy design can be used for instance the totally different code protection outcomes that simulation and formal analyses ship when run on the identical DUT. This design has a single clock and reset. There’s a 2-bit enter: “sel”. There are three single bit outputs: “A”, “B”, and “C”; and every output may be lively for just one clock cycle. The “sel” sign specifies which outputs to drive per the next fact desk:
Desk 1 That is how the “sel” sign specifies the outputs. Supply: Siemens EDA
To simplify and focus the dialogue of outcomes, we’ll confine consideration to output “B”. Clearly, in an actual verification move, all necessities and outputs can be examined.
Think about there’s a property for verifying the requirement that output “B” may be lively for just one clock cycle. A simulation check is run to stimulate the “B” output, and formal is run on this property. Code protection outcomes for this circuit, written in Verilog, are proven under for simulation (left-hand-side of Determine 1 under) and formal (right-hand-side). Be aware that VHDL designs would exhibit related protection metrics on this instance, in addition to the next examples on this article.
Determine 1 In code protection outcomes from simulation (left) and formal (proper), traces with inexperienced checkmarks and inexperienced shading are coated. Traces with purple Xs and purple shading will not be coated. Supply: Siemens EDA
Referring to the simulation outcomes on the left of Determine 1, you possibly can see that many of the design is roofed, regardless that solely one of many outputs was pushed. The one factor not coated was associated to the “sel” enter not being pushed with “01” or “11”. This reveals the beneficiant nature of simulation code protection in that logic unrelated to what’s being examined may be coated too. For instance, all statements of a block are coated. Why is that this? In brief, the simulation stimulus—enter vectors—is pushed into the inputs of the Verilog mannequin by the simulation instrument, with the evaluation progressing from the inputs to the outputs of the design being verified.
Clearly, the formal code protection outcomes are fairly totally different, reflecting the character of formal evaluation, which successfully works backwards from the given DUT output being verified utilizing a System Verilog Assertion (SVA) property that specifies the conduct of the node(s) designated therein. Particularly, the way in which that formal proof algorithms work is they begin by a small quantity of logic that’s within the rapid fan-in cone of the output sign specified by the SVA property.
The property could also be confirmed proper at that time, and the evaluation can be full. But when the property isn’t confirmed on this first step, extra logic is “pulled-in” simply forward of this space. If that is sufficient logic to acquire a proof, you’re finished. If not, the method continues till major inputs and, probably, “assumptions”—constraints—are used within the evaluation. Once more, you possibly can consider the formal code protection evaluation as continuing from the node of curiosity—output “B” on this instance—towards the inputs of the design. Consequently, the one traces of code declared “coated” by the formal evaluation are people who have been wanted to acquire the proof of the property; on this case people who immediately management output “B”.
Given these very totally different approaches—and the next very totally different outcomes—it’s crucial to maintain the simulation and formal outcomes “parallel and separate” from one another when exporting this knowledge to a typical verification outcomes database. Solely on this method can customers precisely assess progress and make use of downstream verification administration methods for monitoring and check plan completion.
Instance 2: Simulation vs. formal code protection of a finite state machine
The easy finite state machine (FSM) proven in Determine 2 can be used for one more illustration of the totally different code protection outcomes that simulation and formal analyses ship when run on the identical DUT.
Determine 2 State diagram of the instance FSM illustrates totally different code protection outcomes. Supply: Siemens EDA
The next property was run in simulation and formal towards a Verilog mannequin of this state machine:
a_mout_mutex: assert property (@(posedge clk) $onehot0(mout) );
Simulation was run with one worth of the “sel” sign used, which exercised the output. The property handed in simulation and was confirmed in formal. The protection for each is proven under in Determine 3.
Determine 3 In FSM code protection outcomes from simulation (left) and formal (proper), traces with inexperienced checkmarks and inexperienced shading are coated. Traces with purple Xs and purple shading will not be coated. Supply: Siemens EDA
Referring to the simulation outcomes, once more you possibly can see that many of the design is roofed. Maybe the uncovered traces of code counsel that extra vectors or “unreachability” evaluation is required; however all of the inexperienced on this outcome gives the look that FSM design is in fine condition.
Clearly the formal evaluation outcomes paint a unique image. On this case, formal evaluation was centered on proving that the “mout” output is mutex, per spec. Therefore, formal used solely the logic that was wanted for the proof; on this case, not one of the FSM state bits have been wanted, solely a few of the sign assignments have been. Whereas this mutex requirement is required, its correctness doesn’t depend on a lot of the design. As such, the FSM design itself clearly wants a reexamination and additional testing. From a proper verification perspective, extra properties are wanted, or a property that exams increased order necessities to make fuller use of the FSM ought to be utilized.
It also needs to be famous that the formal protection proven right here is predicated on the logic wanted to show the property, generally known as a “proof core”. The same outcome can be achieved utilizing mutation protection evaluation, outlined in our upcoming Half 3 article.
A crude type of formal code protection is predicated on the structural cone of affect (COI) of the property, which might present protection from the property all the way in which again to the inputs, nearer to what’s proven above for the simulation outcome. Clearly, it’s not sufficient for signoff protection, and therefore helpful solely very early within the verification course of, when including properties (and in lots of instances, this step may be ignored fully).
Instance 3: Closing code protection holes
On this instance, think about the consumer is trying to shut protection through the use of formal evaluation to fill a protection “gap” present in simulation. For functions of illustration, we’ll present an preliminary failed try at this train—then the corrected model.
First, contemplate the next easy design coded in Verilog, the place outputs are once more mutex-based on an encoding of the three inputs. The simulation code protection outcomes for the next Verilog design are nearly good—just one gap stays.
Determine 4 Code protection outcomes from simulation exhibiting just one remaining gap; traces with inexperienced shading are coated and features with purple shading will not be coated. Supply: Siemens EDA
Relatively than spend time manipulating the parameters of the simulation check, the verification engineer elects to run a proper evaluation to attempt to cowl this gap. Therefore, the next property is written:
a_bogus: assert property (@(posedge clk) in1 |-> n1 );
When run in formal, this property yields the outcomes proven on the left of Determine 5.
Determine 5 Code protection outcomes with formal because of trivial property focusing on solely that code (proven on the left), and code protection from a property that’s checking the design’s necessities (proven on the precise). Supply: Siemens EDA
As per the instance, on the left of Determine 5, within the rush to plug the simulation gap within the “n1” sign path, the verification engineer tried to make use of a trivial property in formal that may plug the opening in simulation. Whereas combining this outcome with simulation would give 100% protection of this logic, this property is definitely ineffective. It exams nothing; it’s not tied to a check plan, or to the verification of any design necessities. Closing protection this manner is bogus, to make use of a colloquialism.
A greater method is to write down and apply a property tied to the check plan and the verification of design necessities. On this case the design requires the three outputs to be mutex, thus a extra helpful property, which checks the necessities, can be written as follows:
a_good: assert property (@(posedge clk) $onehot0({A,B,C}) );
This property, when confirmed in formal, yields the protection proven on the precise of Determine 5. Ideally, the simulation gap can be stuffed by operating a check that leveraged that logic; or if that logic was deemed unreachable, additional work wouldn’t be wanted. The opening factors to a weak spot within the check plan or testbench of the verification engine getting used. Whether it is deliberate—as a part of the check plan and the necessities needing verification—that a part of this design be verified with simulation and half with formal. Then, the above protection from every engine could possibly be safely merged to present an correct and full protection image for the general verification standing evaluation.
Watch out for false victory
The above examples present that easy mixing of simulation and formal protection knowledge on a one-to-one line/object/level foundation can mislead you—and your managers—to suppose that you’re finished; when the truth is that there are literally unverified areas of your code and check plan.
However it may get even worse. In Half 3 of this collection, we’ll reveal how even correctly merged 100% code and purposeful protection outcomes can nonetheless tempt you to prematurely conclude that your verification effort is over and it’s protected to declare victory. Particularly, with a easy FSM and supporting logic instance, we’ll present simply how straightforward it’s to fall into this entice; and the way to use mutation evaluation to exhaustively show each your DUT and your verification plan itself is bug free. Plus, we’ll summarize suggestions for correctly utilizing simulation and formal protection collectively.
Editor’s Be aware: The final article of this three-part collection on the pitfalls of blending formal and simulation protection will reveal how even correctly merged code and purposeful protection don’t imply that verification effort is over. The primary half outlined the dangers of arbitrarily merging simulation and formal protection.
Mark Eslinger is a product engineer within the IC Verification Methods division of Siemens EDA, the place he focuses on assertion-based strategies and formal verification.
Joe Hupcey III is product advertising supervisor for Siemens EDA’s Design & Verification Applied sciences formal product line of automated functions and superior property checking.
Nicolae Tusinschi is a product supervisor for formal verification options at Siemens EDA.
Associated Content material
