International Symposium on System-on-Chip

International Symposium on System-on-Chip 2006

SoC 2006 Panel Discussion Summary

The Verification Gap: Will It Ever Close?

Chairman:
Prof. Jari Nurmi, Tampere University of Technology

Participants:
Doug Amos, Synplicity
Arvind, MIT
Gert Goossens, Target Compiler Technologies
Wolfgang Stronski, Cadence
Michael Siegel, OneSpin Solutions

Discussion

The chairman opened the panel discussion by presenting a few slides about the verification gap. The main questions were whether there are already too many gaps and what is exactly the verification gap? The first gap is a design gap, which means that we have too much capacity to implement such designs that we cannot verify any more. And this crisis gets worse. The more the design size and complexity increase, the more bugs exist. Moreover, the verification becomes more and more costly. Therefore, we can claim that design verification is the stopping issue of product development. The second gap is the verification gap, that is, we will not find all the design bugs.

The chairman introduced two main solutions for the gaps: first, the IP markets should provide the customers with reusable and pre-verified IP blocks. Second solutions is the EDA tool vendors' trend to model the hardware at Electronic System Level (ESL), which is higher level of abstraction than Register Transfer Level (RTL). However, the costly and complicated verification process also affects the quality of Intellectual Property (IP) blocks. A true, error free plug&play IP with a standard interface would be nice, but are Legos the only example of such IPs?

There is the need for full visibility in testing, but FPGAs do not offer that; we would need "true plug&play" and error-free IPs, with standard interfaces.

Steve Leibson from the audience had the opinion that if we should design cars using such IPs, the biggest verified unit would be a molecule. Wolfgang Stronski added that the industry moves more and more towards reuse because everything cannot be develop from scratch any more. So the industry needs verified and ready-to-connect components. However, Gert Goossens reminded that we cannot actually avoid the design of new blocks because there are no reusable IPs for new algorithms and solutions on the market. Nevertheless, we need to be sure that the new blocks work fine and if we deal more with the design gap than with the verification gap, we need tools to rapidly design error-free and easily integrated IPs. Also Arvind considered IP reuse important. There must be much more use of IPs, but they are hard to use. For instance, you cannot always know what do they do. Furthermore, we have not found the right abstraction level for IP blocks either, he added. Doug Amos then reminded that the market pressure is so high that there is no time to verify so much. Therefore, we have to rely on IP vendors that the IPs we buy work fine.

Siegel had the opinion that it is indeed possible to design error-free IPs. Furthermore, it is also possible to speed-up the verification process if we use parallel models instead of serial simulations. He continued that IP vendors should also use standard interfaces due to the high interfacing costs. One solution to increase the verification ability is the platform-based design, which also reduces the integration design costs. Stronski added that error-free IPs require a methodology change. If companies use manual testing, it is like fishing in the dark. Therefore, the testing environment should be affordable and appropriate. Amos then concluded that we obviously need to know a lot of people, who are using an IP, before we can buy it. He wondered, how could we get quality evaluations of IPs (such as we get at eBay) to avoid the various poor IP companies who design poor quality IPs.

How do you describe the IP block development in industry? Can we trust on IP blocks designed by small companies?

Arvind and Amos suggested that industry should use a neutral evaluator who rates IP vendors. But Amos was also wondering who is going the be the IP authority? Any journal, for instance, will not start criticizing IP vendor companies, since it could easily lose sponsors. However, if an IP vendor can prove it abilities, it will definitely have customers without eBay-style trading, added Siegel. Leibson from the audience commented that a company, who bought an IP, will not tell publicly, if it was good. The company will not even reveal that they are using an IP block of this manufacturer. Therefore, the eBay idea ("you judge mine and I will judge yours") will not work. Arvind asked if companies should admit, if they have bought a bad IP? Leibson strongly disagreed because this kind of evaluation would cause a conflict between the selling and buying companies.

Different views of an IP block differ a lot from each other in terms of behaviour and timing. Moreover, there is a gap between transaction and RTL levels.

Stronski suggested to use simulation to guarantee the IP behaviour but admitted that there is no straight comparison between abstraction levels. Companies are working on a link between C level and gate level description and simulation. Many tools are available today to verify IPs but we still need a final verification of the entire system. Siegel added that you must have a set of "integration conditions" when you have an RTL. We must use simulators to see if the IP is following the integration conditions when it is eventually placed inside a system. Now we develop IPs using SystemC, which is a much more powerful tool than the old ones because the abstraction level is raised, reminded Goossens.

Should we get a warranty or estimations of quality already when we design products or only when we verify them?

Goossens thought that the we can get warranty for quality at design time as soon as we bridge the gap between SystemC and RTL. Now there is something missing between them. Siegel's comment on verification was that if something goes wrong, we must be able to recover. We should not have to build a whole new SoC if for example software updates can fix it, he continued. Arvind reminded that we often (and wrongly) think that the verification gap is a different gap than the design gap.

Goossens emphasized the importance of software verification too. Hardware IP verification has already been discussed a lot, but systems contain more and more embedded software. Embedded software is a blessing for us because we can modify the system also after the chip is manufactured. Stronski continued that several companies offer platforms and we must be able to test the software as well. We also need methods to include the software in the design environment and test it. Moreover, we need accelerators to speed-up designs. Leibson then summarized that the good side of software is its flexibility and bad side is the shape of software engineers (they are even in a worse shape than RTL people) mostly due to the amount of code they must cope with. Moreover, software bugs may cause lives as well as hardware bugs.

Where do we use formal methods today?

Formal methods allow bug-free design already. However, we need to change the methodology when we introduce formal methods to industry. We cannot think that you need a PhD to do formal verification. We already use 60% of the employees only for verification. For instance, we could use very intelligent test-benches instead, commented Siegel. Stronski continued that there are indeed easy verification tools that allow to verify entire design at chip level but Arvind disagreed and claimed that formal methods are not at the right level yet. Then it must be possible to reuse also the test-benches once they are created instead of creating them from scratch every time, reminded Stronski.

Then why the IPs are not properly verified? For instance, considering TCP/IP, some errors have been found despite thorough verification.

Arvind reminded that this particular application type (TCP/IP) is very difficult to verify. Siegel added that it takes time, money, and a lot of expertise to verify IPs thoroughly and that is why they are not properly verified. According to Stronski, the key point is the reuse of as many components, which work fine, as possible.

Software designers can rely on standards to develop software, but why does companies not usually follow them?

You can integrate many different software components together. Therefore, the software designers cannot really imagine how their products will be used together. For this reason it is also easy to get bugs, said Arvind. Time to market constraints makes it nearly impossible to design totally bug-free consumer products.

Finally, the chairman asked whether we are able to shrink the verification gap within the next five years. Stronski's determined opinion was that we will close the gap within the next four weeks...