Debugging
There are several sources of potential issues with DaCe programs:
Frontend failures during parsing programs to SDFGs
SDFG validity before or during optimization
Transformations that impair the correctness of the SDFG
Failures during the Code Generation process
Segmentation faults or errors in the generated code
In general, DaCe tries to raise a Python exception and clearly print the origin of the issue. However, to shed more light
into the origin of the problem, it can be useful to set the debugprint
configuration entry to 1
or verbose
.
There are several other important configuration entries: for frontend and debugging why Python functions become callbacks,
use frontend.verbose_errors
. For transformations that fail during matching, use optimizer.match_exception
.
For issues with Properties and Serialization, enable testing.serialization
and testing.deserialize_exception
.
Below we provide a more detailed methodology for debugging particular issues. You can find common errors and solutions here.
Graph Validation
SDFGs can be validated for soundness. This happens automatically during compilation, but can be triggered manually
using the dace.sdfg.sdfg.SDFG.validate()
method. It can be useful to detect issues in the graph, examples include
Memlets that mismatch their context, out of bounds access, undefined symbol use, scopes that are not properly closed,
and many more.
On validation failure, (unless specified) a copy of the failing SDFG will be saved in the current working directory,
under _dacegraphs/invalid.sdfg
, which includes the source of the error. Opening it in the Visual Studio Code
extension even zooms in on the issue automatically!
Debugging and Recompiling Generated Code
Note
For debugging the code generators and the generation process itself, see Debugging Code Generation.
If issues arise during compilation of the generated code, or the code is somehow incorrect, it can be useful to inspect and modify it.
The generated code of an SDFG is saved in the .dacecache
directory, under the name of the SDFG (which corresponds
to the function’s name in Python). You can inspect and modify the code by opening the file in your favorite text editor:
$ python my_program.py
-- [Some failure happens] --
$ cd .dacecache/myprogram/src
$ ls
cpu cuda
$ code cpu/myprogram.cpp
The generated code is organized in subdirectories for each target based on its code generator name, and in the above case there is both CPU and GPU code.
However, rerunning python my_program.py
will overwrite the generated code.
There are two ways to avoid this overwriting. The first is to set the regenerate_code
flag on the DaCe program to
False
:
@dace.program(regenerate_code=False)
def myprogram(...):
...
This will prevent the code from being regenerated, but it will cause DaCe to recompile the code. If you want to compile
the code yourself, you can set the recompile
flag to False
:
@dace.program(recompile=False)
def myprogram(...):
...
or set the configuration entry compiler.use_cache
to 1
to achieve the same effect globally (on each program).
Since this will prevent the code from being recompiled, you will need to manually go into the build directory and run
make
to recompile the code:
$ cd .dacecache/myprogram/build
$ make
$ cd ../../..
# If recompile=False is used, the below environment variable is not necessary.
$ DACE_compiler_use_cache=1 python my_program.py
# Program will not be regenerated nor recompiled.
Runtime compilation issues
If there are issues with the C++ Runtime Headers, you can find their location and edit them manually:
# Print out the runtime folder
$ python -c 'import dace; print(dace.__file__)'
/home/user/.local/lib/python3.8/site-packages/dace/__init__.py
# The files are in include/dace/*.h
$ cd /home/user/.local/lib/python3.8/site-packages/dace/runtime
It is, however, recommended to install DaCe in development mode, so that you can edit the files directly in the source folder.
Crashes in Compiled Programs
Compiled programs are compiled to a shared object (.so
/ .dll
file) that is linked to the host process. If using
a DaCe program within Python, debugging it requires simply calling any debugger (such as gdb
) on the Python process
and potentially setting breakpoints on the generated code (which can be found using the sdfg.build_folder
property).
For example:
gdb --args python myscript.py [args...]
In most cases, debugging in Release mode does not yield actionable results. To better debug compiled programs, set
the compiler.build_type
configuration entry to Debug
and rerun the program. The following example shows
a crashing program and how the process works:
import dace
import numpy as np
N = dace.symbol('N')
@dace.program
def example(a: dace.float32[N], b: dace.float32[N]):
b[5000000] = a[0]
n = 10
a = np.random.rand(n).astype(np.float32)
b = np.random.rand(n).astype(np.float32)
example(a, b) # Calling this function could trigger a segmentation fault
$ python example.py
...
sh: segmentation fault python example.py
$ gdb --args python example.py
...
(gdb) r
...
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffe7259186 in __program_example_internal(example_t*, float*, float*, int) () from /path/.dacecache/example/build/libexample.so
# No further information is given on the source of the issue. Below we set debug mode:
$ DACE_compiler_build_type=Debug gdb --args python example.py
...
(gdb) r
...
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffe7159186 in __program_example_internal (__state=0x5555574669a0, a=0x55555699efd0, b=0x555556f4c390, N=10)
--Type <RET> for more, q to quit, c to continue without paging--
at /path/.dacecache/example/src/cpu/example.cpp:27
27 b[5000000] = __out;
You can also use the Visual Studio Code extension to debug Python programs by using the DaCe debugger
debug provider.
It even supports mapping breakpoints from the Python code to the generated code.
For low-level access of the CMake configuration, you could also access the build folder, go to the build/
subdirectory, and call ccmake .
to modify it. After that run make
to rebuild.
GPU Debugging in DaCe
As GPU kernels cannot be debugged directly in gdb
, there are other tools that can be used to debug GPU programs.
The CUDA toolkit provides more tools to debug kernels: cuda-gdb
can break and debug CUDA kernels, and cuda-memcheck
can be used to track invalid memory accesses.
Additional debugging features in DaCe include GPU stream synchronization debugging. Since GPU toolkits (CUDA, HIP, OpenCL)
mostly run asynchronously using nonblocking calls, it is sometimes hard to pinpoint the source of an issue. Since GPU
programs can be large and run for a while, Debug
mode cannot always be enabled. For these reasons, DaCe provides
a mode that can run directly in Release
mode, called synchronous debugging. The mode inserts device-synchronization
calls after every GPU-related operation (kernel, library call) and checks for errors. This helps debug both crashes
and stream-related data races. Enable it by setting compiler.cuda.syncdebug
to True.
Debugging Transformations
Transformation debugging can be used for multiple purposes: it can be used to understand why transformations fail to match on a specific subgraph, debug exceptions on matching, and failures during application of transformations.
By default, exceptions during transformation matching emit a warning. To debugging exceptions on matching, enable the
optimizer.match_exception
configuration entry, which would turn them into errors.
If setting breakpoints, since transformations repeatedly try to apply on matching subgraphs on an SDFG, it is recommended to set conditional breakpoints including labels or any defining properties of the nodes/edges you want to debug the transformation for.
Another approach is to run the debugger on the Visual Studio Code extension’s optimizer daemon. The daemon is a Python script, so it can be debugged as such. Simply create a new debug configuration that starts the script (see Common issues with the Visual Studio Code extension on how to find the command) with the right port, kill the existing SDFG Optimizer, and debug the script. Breakpoints should now work inside DaCe or your custom transformations.
Debugging Frontend Issues
When debugging frontend issues, it is important to make the distinction between the frontend itself and transformations
applied on the initial SDFG. Thus, if there is a suspected issue in the frontend, first try disabling automatic simplification
(through the optimizer.automatic_simplification
config entry or the API, see below) and validating the initial
SDFG for soundness:
sdfg = bad_program.to_sdfg(simplify=False)
sdfg.validate()
If this works but some programs fail, it might be a serialization issue. Try a save/load roundtrip:
sdfg.save('test.sdfg')
sdfg = dace.SDFG.from_file('test.sdfg')
sdfg.validate()
# ...other validation methods...
Otherwise, the issue could be in the Simplify Pipeline. Try to simplify while validating every step:
sdfg.simplify(verbose=True, validate_all=True)
This helps understanding which component causes the issue.