The pitfalls of blending formal and simulation: The place bother begins

May 28, 2022

1

The best useful verification environments make use of a number of evaluation applied sciences, the place the strengths of every are mixed to bolster one another to assist be certain that the machine beneath check (DUT) behaves as specified. Nevertheless, this creates an inherent problem of correctly evaluating—and mixing—the outcomes from every supply to present a succinct, correct image of the verification effort’s true standing.

The most typical downside we see is when design engineers wish to merge the outcomes from formal evaluation with the outcomes of RTL code and useful protection from their UVM testbench, but they don’t totally perceive what formal protection is offering. Therefore, we’ll begin on the acquainted floor of simulation-generated code and useful protection earlier than going into defining formal protection.

Simulation code and useful protection overview

Code protection is just the share of RTL code measuring the variety of statements in a physique of code that has been executed by way of a check run. Whereas it’s essential that the testbench can train the entire RTL code—for instance, there is no such thing as a useless code, implying a bug within the DUT—the design can nonetheless be lacking essential performance and/or the paths to key capabilities will be in violation of the specification.

Practical protection from RTL simulation is the metric of how a lot design performance has been exercised—as an example, “lined” by the testbench or verification setting. The quantity of useful protection that should be happy is explicitly outlined by the verification engineer within the type of a useful protection mannequin.

In its fundamental kind, it’s a user-defined mapping of every useful characteristic to be examined to a protection level, and these protection factors have sure circumstances (ranges, outlined transitions or crossings, and so on.) to meet earlier than they’re reported as 100% lined throughout simulation. All these circumstances for a canopy level are outlined within the type of bins. Plenty of cowl factors will be captured beneath one covergroup, and a set of a lot of covergroups is often referred to as a useful protection mannequin.

Throughout simulation, when sure circumstances of a canopy level are hit, these bins (circumstances) are being lined, and thus the variety of bins hit offers a measurement of verification progress. After executing a lot of testcases, a graphical report could also be generated to investigate the useful protection report and plans will be made to “cowl up” the “verification holes” by creating new assessments that may traverse the as-yet unexercised areas of the DUT code.

Word that it’s essential to understand that the protection rating can be a mirrored image of the standard of the testbench—for instance, whether or not the testbench is addressing all of the DUT circuitry that it’s essential confirm. Certainly, the protection rating and the variety of holes are a mirrored image of the standard of the verification check plan and its alignment with the unique specification itself.

Defining formal-based protection

Formal evaluation delivers the next sorts of protection not seen with simulation:

Reachability

Is there any mixture of enter alerts that may convey the state of the circuit to a specified node of significance? If there are none, the purpose is unreachable. Therefore, this yields comparable details about the presence of useless code as does simulation code protection.

Observability

What are all of the doable circuit and state area paths from a particular node to alerts laid out in an assertion? And are these the anticipated paths, and do they meet the design’s specification?

Structural cone of affect (COI)

Working forwards or backwards from a particular level within the circuit, what’s all of the logic that might probably have an effect on this node? This can be a tough type of protection and customarily not helpful for remaining signoff evaluation.

Mannequin-based mutation protection

Measures how nicely checkers cowl a design by inserting mutations electronically into the mannequin—the supply code itself will not be altered—and checking if any checker or assertion detects it. Mutation protection provides clear steerage on the place extra assertions are wanted and represents a stronger metric than some other kind of protection because it measures error detection, not simply train of design.

Whereas these measurements are basically totally different than simulation protection, the general utility is similar at a high-level: measure progress and concurrently consider the standard of the formal testbench (comprised of constraint and useful verification properties).

The place the difficulty begins

The most typical designer request is to “merge” simulation and formal-generated protection metrics corresponding to every specified protection level; the place usually the target is to allow verification groups to pick both formal or simulation to completely confirm a sub-set of a design. For instance, think about a given IP contained a design ingredient that was very nicely suited to formal evaluation—an arbiter is one instance—and the remainder of the design might be simply verified by both know-how.

Naturally, designers wish to allow formal to “take credit score” for the arbiter verification, and seamlessly merge this with the simulation outcomes on the remainder of the circuit. That is the place the difficulty can begin. The most important dangers to pay attention to are:

First, whatever the chosen know-how, simply because one thing is roofed doesn’t imply it’s correctly verified. Once more, think about the connection between code protection versus useful protection—simply because assessments can traverse 100% of the code doesn’t imply that the code can be functionally appropriate.
Simulation protection solely displays particular ahead paths the simulation has traversed from the inputs by way of the state area for a selected set of stimuli.
Some sorts of formal protection do replicate a ahead traversal, however simply because the formal evaluation may traverse among the identical states as a simulation, the logic lined is often higher. Moreover, formal evaluation has free-floating enter stimulus and is legitimate forever.
Different sorts of formal protection report how logic and signaling “work backwards” from an output.
Simulation is run on the cluster/system-on-chip (SoC) degree, whereas formal is usually run on the block degree. Therefore, code protection represents end-to-end testing versus a extra localized operate. How the protection information was generated will be misplaced as soon as the outcomes are logged into the protection database.

Backside-line: if you’re not cautious about understanding the variations between formal and simulation protection—in case you merely union the protection information on a 1-1 line/object/level foundation—you may incorrectly conclude that your verification high quality is greater than it truly is. For example, you possibly can mislead your self and your managers that you’re executed when the fact is that there are literally unverified areas of your code and check plan.

Editor’s Word: It’s a three-part collection on the pitfalls of blending formal and simulation protection. The second a part of this text collection will examine the outcomes from simulation and formal side-by-side with progressively advanced RTL DUT code examples.

Mark Eslinger is a product engineer within the IC Verification Methods division of Siemens EDA, the place he makes a speciality of assertion-based strategies and formal verification.

Joe Hupcey III is a product advertising and marketing supervisor for Siemens EDA’s Design & Verification Applied sciences formal product line of automated purposes and superior property checking.