Identifying the root cause of bugs ================================== This document references Yosys internals and is intended for people interested in solving or investigating Yosys bugs. This also applies if you are using a fuzzing tool; fuzzers have a tendency to find many variations of the same bug, so identifying the root cause is important for avoiding issue spam. If you're familiar with C/C++, you might try to have a look at the source code of the command that's failing. Even if you can't fix the problem yourself, it can be very helpful for anyone else investigating if you're able to identify where the issue is arising. Finding the failing command --------------------------- Using the ``-L`` flag can help here, allowing you to specify a file to log to, such as ``yosys -L out.log -s script.ys``. Most commands will print a header message when they begin; something like ``2.48. Executing HIERARCHY pass (managing design hierarchy).`` The last header message will usually be the failing command. There are some commands which don't print a header message, so you may want to add ``echo on`` to the start of your script. The `echo` command echoes each command executed, along with any arguments given to it. For the `hierarchy` example above this might be ``yosys> hierarchy -check``. .. note:: It may also be helpful to use the `log` command to add messages which you can then search for either in the terminal or the logfile. This can be quite useful if your script contains script-passes, like the :doc:`/using_yosys/synthesis/synth`, which call many sub-commands and you're not sure exactly which script-pass is calling the failing command. Minimizing scripts ------------------ .. warning:: This section is intended as **advanced usage**, and generally not necessary for normal bug reports. If you're using a command line prompt, such as ``yosys -p 'synth_xilinx' -o design.json design.v``, consider converting it to a script. It's generally much easier to iterate over changes to a script in a file rather than one on the command line, as well as being better for sharing with others. .. code-block:: yoscrypt :caption: example script, ``script.ys``, for prompt ``yosys -p 'synth_xilinx' -o design.json design.v`` read_verilog design.v synth_xilinx write_json design.json Next up you want to remove everything *after* the error occurs. If your final command calls sub-commands, replace it with its contents first. In the case of the :doc:`/using_yosys/synthesis/synth`, as well as certain other script-passes, you can use the ``-run`` option to simplify this. For example we can replace ``synth -top -lut`` with the :ref:`replace_synth`. The options ``-top -lut`` can be provided to each `synth` step, or to just the step(s) where it is relevant, as done here. .. code-block:: yoscrypt :caption: example replacement script for `synth` command :name: replace_synth synth -top -run :coarse synth -lut -run coarse:fine synth -lut -run fine:check synth -run check: Say we ran :ref:`replace_synth` and were able to remove the ``synth -run check:`` and still got our error, then we check the log and we see the last thing before the error was ``7.2. Executing MEMORY_MAP pass (converting memories to logic and flip-flops)``. By checking the output of ``yosys -h synth`` (or the `synth` help page) we can see that the `memory_map` pass is called in the ``fine`` step. We can then update our script to the following: .. code-block:: yoscrypt :caption: example replacement script for `synth` when `memory_map` is failing synth -top -run :fine opt -fast -full memory_map By giving `synth` the option ``-run :fine``, we are telling it to run from the beginning of the script until the ``fine`` step, where we then give it the exact commands to run. There are some cases where the commands given in the help output are not an exact match for what is being run, but are instead a simplification. If you find that replacing the script-pass with its contents causes the error to disappear, or change, try calling the script-pass with ``echo on`` to see exactly what commands are being called and what options are used. .. warning:: Before continuing further, *back up your code*. The following steps can remove context and lead to over-minimizing scripts, hiding underlying issues. Check out :ref:`yosys_internals/extending_yosys/advanced_bugpoint:Why context matters` to learn more. When a problem is occurring many steps into a script, minimizing the design at the start of the script isn't always enough to identify the cause of the issue. Each extra step of the script can lead to larger sections of the input design being needed for the specific problem to be preserved until it causes a crash. So to find the smallest possible reproducer it can sometimes be helpful to remove commands prior to the failure point. The simplest way to do this is by writing out the design, resetting the current state, and reading back the design: .. code-block:: yoscrypt write_rtlil ; design -reset; read_rtlil ; In most cases, this can be inserted immediately before the failing command while still producing the error, allowing you to :ref:`minimize your RTLIL` with the ```` output. For our previous example with `memory_map`, if :ref:`map_reset` still gives the same error, then we should now be able to call ``yosys design.il -p 'memory_map'`` to reproduce it. .. code-block:: yoscrypt :caption: resetting the design immediately before failure :name: map_reset synth -top -run :fine opt -fast -full write_rtlil design.il; design -reset; read_rtlil design.il; memory_map If that doesn't give the error (or doesn't give the same error), then you should try to move the write/reset/read earlier in the script until it does. If you have no idea where exactly you should put the reset, the best way is to use a "binary search" type approach, reducing the possible options by half after each attempt. .. note:: By default, `write_rtlil` doesn't include platform specific IP blocks and other primitive cell models which are typically loaded with a ``read_verilog -lib`` command at the start of the synthesis script. You may have to duplicate these commands *after* the call to ``design -reset``. It is also possible to write out *everything* with ``select =*; write_rtlil -selected ``. As an example, your script has 16 commands in it before failing on the 17th. If resetting immediately before the 17th doesn't reproduce the error, try between the 8th and 9th (8 is half of the total 16). If that produces the error then you can remove everything before the `read_rtlil` and try reset again in the middle of what's left, making sure to use a different name for the output file so that you don't overwrite what you've already got. If the error isn't produced then you need to go earlier still, so in this case you would do between the 4th and 5th (4 is half of the previous 8). Repeat this until you can't reduce the remaining commands any further. A more conservative, but more involved, method is to remove or comment out commands prior to the failing command. Each command, or group of commands, can be disabled one at a time while checking if the error still occurs, eventually giving the smallest subset of commands needed to take the original input through to the error. The difficulty with this method is that depending on your design, some commands may be necessary even if they aren't needed to reproduce the error. For example, if your design includes ``process`` blocks, many commands will fail unless you run the `proc` command. While this approach can do a better job of maintaining context, it is often easier to *recover* the context after the design has been minimized for producing the error. For more on recovering context, checkout :ref:`yosys_internals/extending_yosys/advanced_bugpoint:Why context matters`. Why context matters ------------------- Sometimes when a command is raising an error, you're seeing a symptom rather than the underlying issue. It's possible that an earlier command may be putting the design in an invalid state, which isn't picked up until the error is raised. This is particularly true for the pre-packaged :doc:`/using_yosys/synthesis/synth`, which rely on a combination of generic and architecture specific passes. As new features are added to Yosys and more designs are supported, the types of cells output by a pass can grow and change; and sometimes this leads to a mismatch in what a pass is intended to handle. If you minimized your script, and removed commands prior to the failure to get a smaller reproducer, try to work backwards and find which commands may have contributed to the design failing. From the minimized design you should have some understanding of the cell or cells which are producing the error; but where did those cells come from? The name and/or type of the cell can often point you in the right direction: .. code-block:: # internal cell types start with a $ # lowercase for word-level, uppercase for bit-level $and $_AND_ # cell types with $__ are typically intermediate cells used in techmapping $__MUL16X16 # cell types without a $ are either user-defined or architecture specific my_module SB_MAC16 # object names might give you the name of the pass that created them $procdff$1204 $memory\rom$rdmux[0][0][0]$a$1550 # or even the line number in the Yosys source $auto$muxcover.cc:557:implement_best_cover$2152 $auto$alumacc.cc:495:replace_alu$1209 Try running the unminimized script and search the log for the names of the objects in your minimized design. In the case of cells you can also search for the type of the cell. Remember that calling `stat` will list all the types of cells currently used in the design, and `select -list =*` will list the names of of all the current objects. You can add these commands to your script, or use an interactive terminal to run each command individually. Adding them to the script can be more repeatable, but if it takes a long time to run to the point you're interested in then an interactive shell session can give you more flexibility once you reach that point. You can also add a call to the `shell` command at any point in a script to start an interactive session at a given point; allowing you to script any preparation steps, then come back once it's done. A worked example ~~~~~~~~~~~~~~~~ Say you did all the minimization and found that an error in `synth_xilinx` occurs when a call to ``techmap -map +/xilinx/cells_map.v`` with ``MIN_MUX_INPUTS`` defined parses a `$_MUX16_` with all inputs set to ``1'x``. You could fix the bug in ``+/xilinx/cells_map.v``, but that might only solve this one case while leaving other problems that haven't been found yet. So you step through the original script, calling `stat` after each step to find when the `$_MUX16_` is added. You find that the `$_MUX16_` is introduced by a call to `muxcover`, but all the inputs are defined, so calling `techmap` now works as expected. From running `bugpoint` with the failing techmap you know that the cell with index ``2297`` will fail, so you call ``select top/*$2297`` to limit to just that cell. This can then be saved with ``design -save pre_bug`` or ``write_rtlil -selected pre_bug.il``, so that you don't have to re-run all the earlier steps to get back here. Next you step through the remaining commands and call `dump` after each to find when the inputs are disconnected. You find that ``opt -full`` has optimized away portions of the circuit, leading to `opt_expr` setting the undriven mux inputs to ``x``, but failing to remove the now unnecessary `$_MUX16_`. Now you've identified a problem in `opt_expr` that affects all of the wide muxes, and could happen in any synthesis flow, not just `synth_xilinx`. .. seealso:: This example is taken from `YosysHQ/yosys#4590 `_ and can be reproduced with a version of Yosys between 0.45 and 0.51.