Code Coverage

Code coverage tells which parts of a design were exercised by the tests. OSVVMOpen Source VHDL Verification Methodology collects it per test case, merges it per test suite and per build, links the simulator’s HTMLHyperText Markup Language coverage report from the build report (see Reports), and can export it into a well-known data format, like Cobertura XML.

Code coverage is supported for the Aldec simulators (Active-HDL, Riviera-PRO, VSimSA), the Siemens simulators (ModelSim, QuestaSim, Visualizer) and NVC.

Turning on Code Coverage

Coverage is enabled separately for analysis and simulation:

• SetCoverageAnalyzeEnable instruments the files analyzed while it’s true. Turn it off before analyzing the testbench, so the testbench isn’t part of the coverage.

• SetCoverageSimulateEnable collects coverage in the simulations started while it’s true.

Both are false by default, so simulations run fast. SetCoverageEnable (default true) switches both off at once without changing them. All three accept any TclTool Command Language boolean (true, yes, on, 1, …).

NVC instruments a design at elaboration, not at analysis. With NVC, SetCoverageAnalyzeEnable has no effect: the whole design, including the testbench, is covered.

# File: Dut.pro
SetCoverageAnalyzeEnable true
analyze   Dut.vhd
SetCoverageAnalyzeEnable false

SetCoverageSimulateEnable true
analyze   TbDut.vhd
simulate  TbDut
SetCoverageSimulateEnable false

Kinds of Code Coverage

SetCoverageKinds selects the kinds of code coverage independent of the simulator; the vendor script translates them into the simulator’s options. all stands for all kinds, default for the kinds in the setting DefaultCoverageKinds: statement, branch and state machine coverage. A kind the simulator doesn’t support is left out; an unknown kind is an error. GetCoverageKinds returns the current kinds.

Kind	NVC (elaborate)	Siemens (analyze)	Aldec (analyze, simulate)
statement	statement	s	s
branch	branch	b	b
condition	expression	c	c
expression	expression	e	e
toggle	toggle	t
fsm	fsm-state	f	m
functional	functional

SetCoverageKinds all                  ;# all kinds
SetCoverageKinds {default toggle}     ;# statement branch fsm toggle
SetCoverageKinds                      ;# back to the default kinds

puts [GetCoverageKinds]
# statement branch fsm

Coverage Options

The options passed to the simulator are kept per step:

• SetCoverageAnalyzeOptions - used by analyze;

• SetCoverageElaborateOptions - used by the elaboration in simulate;

• SetCoverageSimulateOptions - used by simulate.

Their defaults come from the vendor script, built from the kinds. SetCoverageKinds sets all three to the vendor’s defaults for the new kinds, replacing options set before. Options set afterwards are passed to the simulator as given, so a simulator option without a kind is still possible. The Get commands (GetCoverageAnalyzeOptions, GetCoverageElaborateOptions, GetCoverageSimulateOptions) return the current options.

Active-HDL

# Defaults (default kinds)
SetCoverageAnalyzeOptions  "-coverage sbm"
SetCoverageSimulateOptions "-acdb -acdb_cov sbm -cc_all"

NVC

# Defaults (default kinds)
SetCoverageElaborateOptions "--cover=statement,branch,fsm-state"

# Own options instead of the kinds
SetCoverageElaborateOptions "--cover=statement"

QuestaSim

# Defaults (default kinds)
SetCoverageAnalyzeOptions  "+cover=sbf"
SetCoverageSimulateOptions "-coverage"

# Own options instead of the kinds
SetCoverageAnalyzeOptions  "+cover=bcst"

Riviera-PRO

# Defaults (default kinds)
SetCoverageAnalyzeOptions  "-coverage sbm"
SetCoverageSimulateOptions "-acdb_cov sbm -cc_all"

NVC’s Further Coverage Options

NVC has further code coverage options, which aren’t kinds, like fsm-no-default-enums or count-from-undefined. The variable NvcExtendedCoverageOptions holds them as a space separated list; they are appended to NVC’s --cover= option when the elaborate options are built from the kinds.

Set the variable in OsvvmSettingsLocal_NVC.tcl (see Configuration), then build the elaborate options again: with vendor_SetCoverageElaborateDefaults or with SetCoverageKinds.

# File: OsvvmSettingsLocal_NVC.tcl
namespace eval ::osvvm {
  variable NvcExtendedCoverageOptions "fsm-no-default-enums count-from-undefined"
  variable CoverageElaborateOptions   [vendor_SetCoverageElaborateDefaults]
}

# --cover=statement,branch,fsm-state,fsm-no-default-enums,count-from-undefined

Merging and Reports

With coverage on, each test case writes its coverage database. When a test suite completes, the databases of its test cases are merged; when the build completes, the databases of all test suites are merged and the simulator writes its HTML coverage report, which the build report links.

All of it is in <BuildName>/CodeCoverage/ of the simulation directory.

NVC

• ▾▸📁<BuildName>

  • ▾▸📁CodeCoverage

    • ▾▸📁<TestSuite>

      • 📄<TestCaseFileName>.ncdbper test case
    • ▾▸📁<BuildName>

      • 📄<TestSuite>.ncdbmerged per test suite
    • 📄<BuildName>.ncdbmerged for the build
    • ▾▸📁<BuildName>_code_cov

      • 📄index.htmlHTML report

QuestaSim

• ▾▸📁<BuildName>

  • ▾▸📁CodeCoverage

    • ▾▸📁<TestSuite>

      • 📄<TestCaseFileName>.ucdbper test case
    • ▾▸📁<BuildName>

      • 📄<TestSuite>.ucdbmerged per test suite
    • 📄<BuildName>.ucdbmerged for the build
    • 📁<BuildName>_code_covHTML report

Exporting Code Coverage

ExportCodeCoverage exports the code coverage of the last build that collected code coverage into a well-known data format, for CIcontinuous integration tools and coverage services. The file name is optional; options for this export are added with ExportOptions, written as an argument like generic.

SetCoverageExportEnable exports at the end of every build that collected code coverage (off by default). SetCoverageExportOptions sets the options of every export.

Simulator	Format	Default file in CodeCoverage/
NVC	Cobertura XMLExtensible Markup Language	<BuildName>_code_cov.cobertura.xml
Siemens	coverage report XML	<BuildName>_code_cov.questa.xml
Aldec	UCDBUnified Coverage Database XML	<BuildName>_code_cov.ucdb.xml

Other simulators print that the export isn’t supported yet.

# at the end of every build
SetCoverageExportEnable  true
SetCoverageExportOptions "--relative=.."

build ../OsvvmLibraries/RunDemoTestsWithCoverage.pro

# once more, into another file, with one more option
ExportCodeCoverage coverage.xml [ExportOptions "--relative=."]