deeplearning4j--deeplearning4j/troubleshooting/PREPROCESSOR_DEBUGGING.md

Using libnd4j.preprocess for Macro Debugging

Overview

The libnd4j.preprocess flag is a specialized debugging tool that runs only the C++ preprocessor, outputting the preprocessed source files before actual compilation. This is invaluable for:

• Debugging complex macro expansions
• Verifying include paths and header dependencies
• Understanding template instantiations
• Investigating conditional compilation issues

Usage

Basic Command

mvn clean install -Dlibnd4j.preprocess=ON

Output Location

Preprocessed files are created in the preprocessed directory with the following naming convention:

• Directory structure is flattened with underscores
• Extension changes to .i
• Example: include/ops/declarable/generic/transforms/reverse.cpp becomes preprocessed/ops_declarable_generic_transforms_reverse.i

What the Preprocessor Does

The preprocessor performs several transformations:

1. Macro expansion
2. Header file inclusion
3. Conditional compilation evaluation (#ifdef, #ifndef, etc.)
4. Line number and file name tracking
5. Comment removal
6. Macro constant substitution

Implementation Details

From the CMakeLists.txt configuration:

1. Directory Setup:

set(PREPROCESSED_DIR "${CMAKE_SOURCE_DIR}/preprocessed")
file(MAKE_DIRECTORY ${PREPROCESSED_DIR})

2. File Processing:

# Command format used internally
${compiler} -E ${include_dirs} "${src}" -o "${preprocessed_file}"

3. Language Detection:

if(src MATCHES "\\.c$")
    set(language "C")
elseif(src MATCHES "\\.cpp$|\\.cxx$|\\.cc$")
    set(language "CXX")
else()
    set(language "CXX")
endif()

Common Use Cases

1. Debugging Complex Macros

When dealing with complex macro definitions like BUILD_DOUBLE_TEMPLATE:

#if defined(SD_COMMON_TYPES_GEN) && defined(SD_COMMON_TYPES_@FL_TYPE_INDEX@)
BUILD_DOUBLE_TEMPLATE(template void someFunc, (arg_list,..),
                     SD_COMMON_TYPES_@FL_TYPE_INDEX@, SD_INDEXING_TYPES);
#endif

Using preprocessor output helps verify:

• Macro expansion correctness
• Template instantiation
• Type substitutions

2. Include Path Issues

When headers aren't found or wrong versions are included:

# Check full include path resolution
grep "#include" preprocessed/your_file.i

3. Conditional Compilation

For code with multiple #ifdef paths:

#ifdef SD_CUDA
    // CUDA-specific code
#else
    // CPU code
#endif

The preprocessed output shows exactly which path was taken.

Best Practices

1. Organized Investigation:

   # Create a directory for the specific issue
   mkdir debug_issue_123
   cd debug_issue_123
   
   # Run preprocessor and copy relevant files
   mvn clean install -Dlibnd4j.preprocess=ON
   cp ../preprocessed/relevant_file.i .
   

2. Diff Comparison:

   # Compare different configurations
   mv preprocessed/file.i file_config1.i
   # Change configuration
   mvn clean install -Dlibnd4j.preprocess=ON
   diff file_config1.i preprocessed/file.i
   

3. Macro Tracing:

   • Keep the original source file open
   • Search for specific macros in the preprocessed output
   • Use line directives (#line) to map back to source

Troubleshooting Common Issues

1. Missing Definitions

   • Problem: Macros not expanding as expected
   • Solution: Check preprocessed output for undefined macros

   grep -n "undefined" preprocessed/*.i
   

2. Include Order Issues

   • Problem: Header dependencies not resolving correctly
   • Solution: Examine the order of #include directives in preprocessed output

   grep -n "#include" preprocessed/your_file.i
   

3. Template Instantiation Problems

   • Problem: Templates not generating expected code
   • Solution: Search for template instantiations in preprocessed output

   grep -n "template" preprocessed/your_file.i
   

Tips for Large Codebases

1. Filtering Output:

   # Remove empty lines and comments
   grep -v '^$' preprocessed/file.i | grep -v '^//'
   
   # Focus on specific sections
   sed -n '/START_SECTION/,/END_SECTION/p' preprocessed/file.i
   

2. Finding Macro Expansions:

   # Search for specific macro expansions
   grep -A 5 -B 5 "MACRO_NAME" preprocessed/file.i
   

3. Managing Output Size:

   # Split large files for easier analysis
   split -l 1000 preprocessed/large_file.i split_
   
   # Create focused preprocessor outputs
   mv preprocessed better_name_preprocessed_$(date +%Y%m%d)