yosys/docs/source/yosys_internals/extending_yosys/test_suites.rst

Testing Yosys
=============

.. todo:: adding tests (makefile-tests vs seed-tests)

.. todo:: pyosys tests

   (part of :ref:`yosys_internals/extending_yosys/ci:Automatic testing with Verific`)

.. TODO:: test-cells

   (run on pushes to main and PRs, under Linux)

The majority of the included test suite is run automatically on the `Yosys Git
repo`_.  For more about compiler tests and our CI process, check the :doc:`ci`
document. For the most up to date information, including OS versions of test
runners, refer to `the GitHub actions page`_.

.. _Yosys Git repo: https://github.com/YosysHQ/yosys
.. _the GitHub actions page: https://github.com/YosysHQ/yosys/actions

Running the included test suite
-------------------------------

The Yosys source comes with a test suite to avoid regressions and keep
everything working as expected.  Tests can be run by calling ``make test`` from
the root Yosys directory. By default, this runs vanilla and unit tests.

Vanilla tests
~~~~~~~~~~~~~

These make up the majority of our testing coverage.
They can be run with ``make vanilla-test`` and are based on calls to
make subcommands (``make makefile-tests``) and shell scripts
(``make seed-tests`` and ``make abcopt-tests``). Both use ``run-test.sh``
files, but make-based tests only call ``tests/gen-tests-makefile.sh``
to generate a makefile appropriate for the given directory, so only
afterwards when make is invoked do the tests actually run.

Usually their structure looks something like this:
you write a .ys file that gets automatically run,
which runs a frontend like ``read_verilog`` or ``read_rtlil`` with
a relative path or a heredoc, then runs some commands including the command
under test, and then uses :doc:`/using_yosys/more_scripting/selections`
with ``-assert-count``. Usually it's unnecessary to "register" the test anywhere
as if it's being added to an existing directory, depending
on how the ``run-test.sh`` in that directory works.

.. tip::

   Automatically run on pushes to main and PRs, under both macOS and Linux
   (``test-build.yml``)

.. TODO:: testing with sanitizers

   - Automatically run on pushes to main and during PR merges, under both macOS
     and Linux, using clang's UBSAN and ASAN (``test-sanitizers.yml``)
   - run locally with :makevar:`SANITIZER` (clean build first)

Unit tests
~~~~~~~~~~

Running the unit tests requires the following additional packages:

.. tab:: Ubuntu

   .. code:: console

      sudo apt-get install libgtest-dev

.. tab:: macOS

   No additional requirements.

Unit tests can be run with ``make unit-test``.

.. tip::

   Automatically run on pushes to main and PRs, under both macOS and Linux
   (``test-build.yml``)

Functional tests
~~~~~~~~~~~~~~~~

Testing functional backends (see
:doc:`/yosys_internals/extending_yosys/functional_ir`) has a few requirements in
addition to those listed in :ref:`getting_started/installation:Build
prerequisites`:

.. tab:: Ubuntu

   .. code:: console

      sudo apt-get install racket
      raco pkg install rosette
      pip install pytest-xdist pytest-xdist-gnumake

.. tab:: macOS

   .. code:: console

      brew install racket
      raco pkg install rosette
      pip install pytest-xdist pytest-xdist-gnumake

If you don't have one of the :ref:`getting_started/installation:CAD suite(s)`
installed, you should also install Z3 `following their
instructions <https://github.com/Z3Prover/z3>`_.

Then, set the :makevar:`ENABLE_FUNCTIONAL_TESTS` make variable when calling
``make test`` and the functional tests will be run as well.

.. tip::

   Automatically run as part of
   :ref:`yosys_internals/extending_yosys/ci:Automatic testing with Verific`

Docs tests
~~~~~~~~~~

There are some additional tests for checking examples included in the
documentation, which can be run by calling ``make test`` from the
:file:`yosys/docs` sub-directory (or ``make -C docs test`` from the root). This
also includes checking some macro commands to ensure that descriptions of them
are kept up to date, and is mostly intended for CI.

.. tip::

   Automatically run on pushes to main and PRs, under Linux (``test-build.yml``)


..
   How to add a unit test
   ----------------------

   Unit test brings some advantages, briefly, we can list some of them (reference
   [1](https://en.wikipedia.org/wiki/Unit_testing)):

   * Tests reduce bugs in new features;
   * Tests reduce bugs in existing features;
   * Tests are good documentation;
   * Tests reduce the cost of change;
   * Tests allow refactoring;

   With those advantages in mind, it was required to choose a framework which fits
   well with C/C++ code.  Hence, `google test`_ was chosen, because it is widely
   used and it is relatively easy learn.

   .. _google test: https://github.com/google/googletest

   Install and configure google test (manually)
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   In this section, you will see a brief description of how to install google test.
   However, it is strongly recommended that you take a look to the official
   repository (https://github.com/google/googletest) and refers to that if you have
   any problem to install it. Follow the steps below:

   * Install: cmake and pthread
   * Clone google test project from: https://github.com/google/googletest and enter
   in the project directory
   * Inside project directory, type:

   .. code-block:: console

      cmake -DBUILD_SHARED_LIBS=ON .
      make

   * After compilation, copy all ``*.so`` inside directory ``googlemock`` and
   ``googlemock/gtest`` to ``/usr/lib/``
   * Done! Now you can compile your tests.

   If you have any problem, go to the official repository to find help.

   Ps.: Some distros already have googletest packed. If your distro supports it,
   you can use it instead of compile.

   Create a new unit test
   ~~~~~~~~~~~~~~~~~~~~~~

   If you want to add new unit tests for Yosys, just follow the steps below:

   * Go to directory :file:`test/unit/`
   * In this directory you can find something similar Yosys's directory structure.
   To create your unit test file you have to follow this pattern:
   fileNameToImplementUnitTest + Test.cc. E.g.: if you want to implement the unit
   test for ``kernel/celledges.cc``, you will need to create a file like this:
   ``tests/unit/kernel/celledgesTest.cc``;
   * Implement your unit test

   Run unit tests
   ~~~~~~~~~~~~~~

   To compile and run all unit tests, just go to yosys root directory and type:

   .. code-block:: console

      make unit-test

   If you want to remove all unit test files, type:

   .. code-block:: console

      make clean-unit-test