chore: import upstream snapshot with attribution

troubleshooting/BUILD_TROUBLESHOOTING.md

@@ -0,0 +1,73 @@
+# Debugging Build Flags for libnd4j
+
+This guide covers important build flags used for debugging crashes and performance issues in libnd4j.
+
+## Core Debug Flags
+
+### 1. libnd4j.calltrace
+- **Purpose**: Enables function call tracing for debugging segmentation faults and crashes
+- **Implementation**: Maps to `SD_GCC_FUNC_TRACE` cmake flag
+- **How to use**: 
+  ```bash
+  mvn -Dlibnd4j.calltrace=ON clean install
+  ```
+- **Technical details**: 
+  - Sets optimization level to 0 (`-O0`) to preserve debugging information
+  - Enables `-finstrument-functions` compiler flag
+  - Adds debugging symbols with `-g`
+  - Sets up function call tracing infrastructure
+
+### 2. libnd4j.printmath
+- **Purpose**: Prints individual math operations for debugging numerical issues
+- **How to use**:
+  ```bash
+  mvn -Dlibnd4j.printmath=ON clean install
+  ```
+- **Technical details**:
+  - Adds `SD_PRINT_MATH` compile definition
+  - Enables debugging output for operations in platformmath.h and templatemath.h
+  - Helps track calculation flow and intermediate results
+
+### 3. libnd4j.printindices
+- **Purpose**: Prints indices during array operations to detect out-of-bounds access
+- **How to use**:
+  ```bash
+  mvn -Dlibnd4j.printindices=ON clean install
+  ```
+- **Technical details**:
+  - Adds `PRINT_INDICES` compile definition
+  - Helps identify where array index violations occur
+  - Useful for tracking down segmentation faults related to memory access
+
+## Usage Scenarios
+
+### Debugging Segmentation Faults
+1. Enable calltrace to get function call history:
+```bash
+mvn -Dlibnd4j.calltrace=ON clean install
+```
+
+### Debugging Numerical Issues
+1. Enable math operation printing:
+```bash
+mvn -Dlibnd4j.printmath=ON clean install
+```
+
+### Finding Array Access Violations
+1. Enable index printing:
+```bash
+mvn -Dlibnd4j.printindices=ON clean install
+```
+
+### Combined Debugging
+For comprehensive debugging, you can combine multiple flags:
+```bash
+mvn -Dlibnd4j.calltrace=ON -Dlibnd4j.printmath=ON -Dlibnd4j.printindices=ON clean install
+```
+
+## Additional Notes
+
+- These flags will impact performance significantly and should only be used for debugging
+- The calltrace flag is particularly useful with tools like gdb and valgrind
+- When using printmath, focus on specific functions by setting PRINT_MATH_FUNCTION_NAME
+- For production builds, ensure all debug flags are turned OFF

troubleshooting/PREPROCESSOR_DEBUGGING.md

@@ -0,0 +1,169 @@
+# Using libnd4j.preprocess for Macro Debugging
+
+## Overview
+
+The `libnd4j.preprocess` flag is a specialized debugging tool that runs only the C++ preprocessor, outputting the preprocessed source files before actual compilation. This is invaluable for:
+- Debugging complex macro expansions
+- Verifying include paths and header dependencies
+- Understanding template instantiations
+- Investigating conditional compilation issues
+
+## Usage
+
+### Basic Command
+```bash
+mvn clean install -Dlibnd4j.preprocess=ON
+```
+
+### Output Location
+Preprocessed files are created in the `preprocessed` directory with the following naming convention:
+- Directory structure is flattened with underscores
+- Extension changes to `.i`
+- Example: `include/ops/declarable/generic/transforms/reverse.cpp` becomes `preprocessed/ops_declarable_generic_transforms_reverse.i`
+
+## What the Preprocessor Does
+
+The preprocessor performs several transformations:
+1. Macro expansion
+2. Header file inclusion
+3. Conditional compilation evaluation (#ifdef, #ifndef, etc.)
+4. Line number and file name tracking
+5. Comment removal
+6. Macro constant substitution
+
+## Implementation Details
+
+From the CMakeLists.txt configuration:
+
+1. Directory Setup:
+```cmake
+set(PREPROCESSED_DIR "${CMAKE_SOURCE_DIR}/preprocessed")
+file(MAKE_DIRECTORY ${PREPROCESSED_DIR})
+```
+
+2. File Processing:
+```cmake
+# Command format used internally
+${compiler} -E ${include_dirs} "${src}" -o "${preprocessed_file}"
+```
+
+3. Language Detection:
+```cmake
+if(src MATCHES "\\.c$")
+    set(language "C")
+elseif(src MATCHES "\\.cpp$|\\.cxx$|\\.cc$")
+    set(language "CXX")
+else()
+    set(language "CXX")
+endif()
+```
+
+## Common Use Cases
+
+### 1. Debugging Complex Macros
+When dealing with complex macro definitions like BUILD_DOUBLE_TEMPLATE:
+```cpp
+#if defined(SD_COMMON_TYPES_GEN) && defined(SD_COMMON_TYPES_@FL_TYPE_INDEX@)
+BUILD_DOUBLE_TEMPLATE(template void someFunc, (arg_list,..),
+                     SD_COMMON_TYPES_@FL_TYPE_INDEX@, SD_INDEXING_TYPES);
+#endif
+```
+
+Using preprocessor output helps verify:
+- Macro expansion correctness
+- Template instantiation
+- Type substitutions
+
+### 2. Include Path Issues
+When headers aren't found or wrong versions are included:
+```bash
+# Check full include path resolution
+grep "#include" preprocessed/your_file.i
+```
+
+### 3. Conditional Compilation
+For code with multiple #ifdef paths:
+```cpp
+#ifdef SD_CUDA
+    // CUDA-specific code
+#else
+    // CPU code
+#endif
+```
+The preprocessed output shows exactly which path was taken.
+
+## Best Practices
+
+1. **Organized Investigation**:
+   ```bash
+   # Create a directory for the specific issue
+   mkdir debug_issue_123
+   cd debug_issue_123
+   
+   # Run preprocessor and copy relevant files
+   mvn clean install -Dlibnd4j.preprocess=ON
+   cp ../preprocessed/relevant_file.i .
+   ```
+
+2. **Diff Comparison**:
+   ```bash
+   # Compare different configurations
+   mv preprocessed/file.i file_config1.i
+   # Change configuration
+   mvn clean install -Dlibnd4j.preprocess=ON
+   diff file_config1.i preprocessed/file.i
+   ```
+
+3. **Macro Tracing**:
+   - Keep the original source file open
+   - Search for specific macros in the preprocessed output
+   - Use line directives (#line) to map back to source
+
+## Troubleshooting Common Issues
+
+1. **Missing Definitions**
+   - Problem: Macros not expanding as expected
+   - Solution: Check preprocessed output for undefined macros
+   ```bash
+   grep -n "undefined" preprocessed/*.i
+   ```
+
+2. **Include Order Issues**
+   - Problem: Header dependencies not resolving correctly
+   - Solution: Examine the order of #include directives in preprocessed output
+   ```bash
+   grep -n "#include" preprocessed/your_file.i
+   ```
+
+3. **Template Instantiation Problems**
+   - Problem: Templates not generating expected code
+   - Solution: Search for template instantiations in preprocessed output
+   ```bash
+   grep -n "template" preprocessed/your_file.i
+   ```
+
+## Tips for Large Codebases
+
+1. **Filtering Output**:
+   ```bash
+   # Remove empty lines and comments
+   grep -v '^$' preprocessed/file.i | grep -v '^//'
+   
+   # Focus on specific sections
+   sed -n '/START_SECTION/,/END_SECTION/p' preprocessed/file.i
+   ```
+
+2. **Finding Macro Expansions**:
+   ```bash
+   # Search for specific macro expansions
+   grep -A 5 -B 5 "MACRO_NAME" preprocessed/file.i
+   ```
+
+3. **Managing Output Size**:
+   ```bash
+   # Split large files for easier analysis
+   split -l 1000 preprocessed/large_file.i split_
+   
+   # Create focused preprocessor outputs
+   mv preprocessed better_name_preprocessed_$(date +%Y%m%d)
+   ```

troubleshooting/README.md

@@ -0,0 +1,71 @@
+# Deeplearning4j Troubleshooting Documentation
+
+This directory contains comprehensive guides for debugging and troubleshooting libnd4j, particularly focusing on crashes, hangs, and build-related issues.
+
+## Available Documentation
+
+### 1. Build Debugging Flags (README.md)
+Covers the core debugging build flags for libnd4j including:
+- `libnd4j.calltrace` for function call tracing
+- `libnd4j.printmath` for debugging numerical operations
+- `libnd4j.printindices` for array bounds checking
+- Build configuration examples and usage scenarios
+
+### 2. Process Hang Troubleshooting (TROUBLESHOOTING_HANGS.md)
+Comprehensive guide for debugging hanging Java processes, including:
+- Using GDB with ptrace and direct process attachment
+- Valgrind integration with test suite
+- Address Sanitizer (ASAN) configuration and usage
+- CUDA Compute Sanitizer for GPU code
+- Best practices and common issues
+
+### 3. Preprocessor Debugging (PREPROCESSOR_DEBUGGING.md)
+Detailed guide for using the preprocessor debugging capabilities:
+- Using `libnd4j.preprocess` flag
+- Understanding preprocessor output
+- Debugging macro expansions
+- Troubleshooting include paths and dependencies
+- Tips for handling large codebases
+
+## Quick Start
+
+### For Build Issues:
+```bash
+# Enable all debugging flags
+mvn -Dlibnd4j.calltrace=ON -Dlibnd4j.printmath=ON -Dlibnd4j.printindices=ON clean install
+```
+
+### For Process Hangs:
+```bash
+# Attach to hanging process
+sudo gdb -p <process-id>
+thread apply all bt
+```
+
+### For Macro Issues:
+```bash
+# Generate preprocessor output
+mvn clean install -Dlibnd4j.preprocess=ON
+```
+
+## Common Use Cases
+
+1. **Debugging Crashes**
+   - Start with build flags from README.md
+   - Use generated debug information with GDB
+
+2. **Investigating Hangs**
+   - Follow TROUBLESHOOTING_HANGS.md
+   - Use appropriate tool based on symptom (GDB, Valgrind, ASAN)
+
+3. **Build/Macro Problems**
+   - Use PREPROCESSOR_DEBUGGING.md
+   - Examine preprocessor output for issues
+
+## Contributing
+
+When adding new troubleshooting documentation:
+1. Create a focused markdown file for the specific topic
+2. Update this README.md with a summary
+3. Include practical examples and command-line instructions
+4. Add cross-references to related documentation

troubleshooting/TROUBLESHOOTING_HANGS.md

@@ -0,0 +1,140 @@
+# Troubleshooting Hanging Java Processes with Native Code
+
+This guide covers various approaches to debugging Java processes that hang due to native code issues in libnd4j.
+
+## Understanding Process Hangs
+
+When a native crash occurs in JNI code, the Java process often appears to "hang" instead of crashing outright. This happens because:
+1. The JVM continues running even though the native code has encountered a fatal error
+2. The native crash may only affect one thread while others continue running
+3. The usual Java exception handling mechanisms don't catch native crashes
+
+## Debug Tools and Approaches
+
+### 1. GDB Debugging
+
+#### Using ptrace
+```bash
+# Attach to a running process
+sudo gdb -p <process-id>
+
+# Once in GDB
+(gdb) thread apply all bt
+```
+
+#### Direct Process Attachment
+```bash
+gdb -p <process-id>
+```
+
+Key GDB commands:
+- `thread apply all bt` - Get backtraces from all threads
+- `info threads` - List all threads
+- `thread <number>` - Switch to a specific thread
+- `bt` - Show backtrace of current thread
+
+### 2. Valgrind Integration
+
+#### Running Tests with Valgrind
+```bash
+mvn test -Dtest.prefix="valgrind --tool=memcheck"
+```
+
+The platform-tests/bin/java script provides special handling for Valgrind:
+- Automatically generates suppression files for JVM-related false positives
+- Adds important Valgrind flags:
+  - `--track-origins=yes`: Track the origins of uninitialized values
+  - `--keep-stacktraces=alloc-and-free`: Maintain allocation/free stacktraces
+  - `--error-limit=no`: Show all errors
+- Disables JIT compilation with `-Djava.compiler=NONE`
+
+### 3. Address Sanitizer (ASAN)
+
+#### Building with ASAN Support
+```bash
+mvn clean install -Dlibnd4j.sanitize=ON -Dlibnd4j.sanitizers="address,undefined,float-divide-by-zero,float-cast-overflow"
+```
+
+Key ASAN Features (from CMakeLists.txt):
+- Adds compilation flags:
+  - `-fsanitize=address`
+  - `-static-libasan`
+  - `-ftls-model=local-dynamic`
+- Requires preloading the ASAN library:
+  ```bash
+  export LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/*/libasan.so
+  ```
+
+Important Notes:
+- Cannot use thread and address sanitizer simultaneously
+- Address and undefined sanitizers must be used carefully together
+
+### 4. CUDA Compute Sanitizer
+
+For CUDA-specific issues:
+
+```bash
+compute-sanitizer --tool memcheck ./your-program
+```
+
+Or attach to running process:
+```bash
+compute-sanitizer --tool memcheck --attach-pid <process-id>
+```
+
+Features:
+- Memory access checking
+- Race condition detection
+- Leak detection
+- Initialization checking
+
+## Build Configurations
+
+### CPU Builds with Sanitizers
+```bash
+# Basic sanitizer build
+mvn clean install -Dlibnd4j.sanitize=ON
+
+# With specific sanitizers
+mvn clean install -Dlibnd4j.sanitize=ON -Dlibnd4j.sanitizers="address,undefined,float-divide-by-zero,float-cast-overflow"
+```
+
+### CUDA Builds with Debugging
+```bash
+# Enable CUDA debugging symbols
+mvn clean install -Dlibnd4j.chip=cuda -Dlibnd4j.cuda=cudnn -Dlibnd4j.build=debug
+```
+
+## Best Practices
+
+1. **Systematic Approach**:
+   - Start with ASAN/Valgrind for memory issues
+   - Use GDB for immediate investigation of hangs
+   - Use Compute Sanitizer for CUDA-specific problems
+
+2. **Log Collection**:
+   - Collect all thread dumps
+   - Save sanitizer outputs
+   - Keep core dumps if generated
+
+3. **Build Considerations**:
+   - Debug builds contain more information but run slower
+   - Sanitizer builds have significant overhead
+   - Consider using both debug symbols and sanitizers for thorough investigation
+
+## Common Issues and Solutions
+
+1. **Memory Access Violations**:
+   - Use ASAN or Valgrind to detect
+   - Check array bounds and pointer arithmetic
+   - Look for use-after-free scenarios
+
+2. **CUDA Synchronization Issues**:
+   - Use Compute Sanitizer's race detection
+   - Check for proper stream synchronization
+   - Verify kernel launch parameters
+
+3. **Resource Leaks**:
+   - Use Valgrind's memcheck
+   - Check CUDA memory management
+   - Verify native memory deallocations

troubleshooting/build-scripts-troubleshooting/README.md

@@ -0,0 +1,83 @@
+# Debug Build Scripts
+
+This directory contains specialized build scripts for debugging various aspects of libnd4j. Each script is configured with specific debug flags for different debugging scenarios.
+
+## Available Scripts
+
+### CPU Backend Scripts
+
+1. **build-cpu-backend-full-debug.sh**
+   - Full debug build with all debug flags enabled
+   - Includes calltrace, printmath, printindices, and address sanitizer
+   - Best for comprehensive debugging of CPU issues
+
+2. **build-cpu-backend-preprocessor-debug.sh**
+   - Enables preprocessor output for macro debugging
+   - Useful for investigating template and macro expansion issues
+
+3. **build-cpu-backend-valgrind-debug.sh**
+   - Optimized for Valgrind analysis
+   - Includes memory leak detection and call tracing
+   - Disables optimizations for better debugging
+
+4. **build-cpu-backend-onednn-debug.sh**
+   - Debug build with OneDNN support
+   - Includes all debug flags
+   - Useful for OneDNN-specific issues
+
+### CUDA Backend Scripts
+
+1. **build-cuda-backend-full-debug.sh**
+   - Full debug build for CUDA with all debug flags
+   - Includes NVCC output preservation
+   - Best for comprehensive debugging of CUDA issues
+
+2. **build-cuda-backend-compute-sanitizer.sh**
+   - Optimized for NVIDIA Compute Sanitizer
+   - Includes debug symbols and reduced optimization
+   - Best for CUDA memory and race condition debugging
+
+## Usage Notes
+
+1. **General Debug Builds**
+   ```bash
+   # For CPU debugging
+   ./build-cpu-backend-full-debug.sh
+   
+   # For CUDA debugging
+   ./build-cuda-backend-full-debug.sh
+   ```
+
+2. **Macro Debugging**
+   ```bash
+   ./build-cpu-backend-preprocessor-debug.sh
+   # Check preprocessed files in the preprocessed/ directory
+   ```
+
+3. **Memory Analysis**
+   ```bash
+   # For CPU with Valgrind
+   ./build-cpu-backend-valgrind-debug.sh
+   
+   # For CUDA with Compute Sanitizer
+   ./build-cuda-backend-compute-sanitizer.sh
+   ```
+
+## Flag Combinations
+
+The scripts use various combinations of these debug flags:
+
+- `libnd4j.build=debug`: Enables debug symbols
+- `libnd4j.calltrace=ON`: Enables function call tracing
+- `libnd4j.printmath=ON`: Prints math operations
+- `libnd4j.printindices=ON`: Prints array indices
+- `libnd4j.sanitize=ON`: Enables sanitizer support
+- `libnd4j.preprocess=ON`: Enables preprocessor output
+- `libnd4j.keepnvcc=ON`: Preserves NVCC output (CUDA)
+
+## Important Notes
+
+1. Debug builds are significantly slower than release builds
+2. Some flag combinations may be incompatible (e.g., different sanitizers)
+3. Ensure sufficient disk space for debug symbols and preprocessor output
+4. CUDA debugging requires appropriate NVIDIA tools installation

troubleshooting/build-scripts-troubleshooting/build-cpu-backend-full-debug.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+# Full debug build for CPU backend with all debug flags enabled
+
+cd ..
+mvn -Pcpu clean install -DskipTests \
+    -Dlibnd4j.build=debug \
+    -Dlibnd4j.calltrace=ON \
+    -Dlibnd4j.printmath=ON \
+    -Dlibnd4j.printindices=ON \
+    -Dlibnd4j.sanitize=ON \
+    -Dlibnd4j.sanitizers="address,undefined,float-divide-by-zero,float-cast-overflow" \
+    -pl :libnd4j,:nd4j-native-preset,:nd4j-native

troubleshooting/build-scripts-troubleshooting/build-cpu-backend-onednn-debug.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# Full debug build for CPU backend with OneDNN and debug flags enabled
+
+cd ..
+mvn -Pcpu clean install -DskipTests \
+    -Dlibnd4j.helper=onednn \
+    -Dlibnd4j.build=debug \
+    -Dlibnd4j.calltrace=ON \
+    -Dlibnd4j.printmath=ON \
+    -Dlibnd4j.printindices=ON \
+    -Dlibnd4j.sanitize=ON \
+    -Dlibnd4j.sanitizers="address,undefined,float-divide-by-zero,float-cast-overflow" \
+    -pl :libnd4j,:nd4j-native-preset,:nd4j-native

troubleshooting/build-scripts-troubleshooting/build-cpu-backend-preprocessor-debug.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+# CPU backend build with preprocessor output for macro debugging
+
+cd ..
+mvn -Pcpu clean install -DskipTests \
+    -Dlibnd4j.build=debug \
+    -Dlibnd4j.preprocess=ON \
+    -pl :libnd4j,:nd4j-native-preset,:nd4j-native

troubleshooting/build-scripts-troubleshooting/build-cpu-backend-valgrind-debug.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+# CPU backend build optimized for Valgrind analysis
+
+cd ..
+mvn -Pcpu clean install -DskipTests \
+    -Dlibnd4j.build=debug \
+    -Dlibnd4j.calltrace=ON \
+    -Dlibnd4j.optimization=0 \
+    -Dtest.prefix="valgrind --tool=memcheck --track-origins=yes --leak-check=full" \
+    -pl :libnd4j,:nd4j-native-preset,:nd4j-native

troubleshooting/build-scripts-troubleshooting/build-cuda-backend-compute-sanitizer.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# CUDA backend build optimized for Compute Sanitizer analysis
+
+cd ..
+mvn -Pcuda -Dlibnd4j.chip=cuda clean install -DskipTests \
+    -Dlibnd4j.build=debug \
+    -Dlibnd4j.calltrace=ON \
+    -Dlibnd4j.keepnvcc=ON \
+    -Dlibnd4j.optimization=0 \
+    -Dtest.runner="compute-sanitizer --tool memcheck" \
+    -pl :libnd4j,:nd4j-cuda-12.1-preset,:nd4j-cuda-12.1

troubleshooting/build-scripts-troubleshooting/build-cuda-backend-full-debug.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# Full debug build for CUDA backend with all debug flags enabled
+
+cd ..
+mvn -Pcuda -Dlibnd4j.chip=cuda clean install -DskipTests \
+    -Dlibnd4j.build=debug \
+    -Dlibnd4j.calltrace=ON \
+    -Dlibnd4j.printmath=ON \
+    -Dlibnd4j.printindices=ON \
+    -Dlibnd4j.sanitize=ON \
+    -Dlibnd4j.keepnvcc=ON \
+    -Dlibnd4j.sanitizers="address,undefined,float-divide-by-zero,float-cast-overflow" \
+    -pl :libnd4j,:nd4j-cuda-12.1-preset,:nd4j-cuda-12.1