chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,233 @@
|
||||
There's multiple different Ops designs supported in libND4j, and in this guide we'll try to explain how to build your very own operation.
|
||||
|
||||
## XYZ operations
|
||||
|
||||
This kind of operations is actually split into multiple subtypes, based on element-access and result type:
|
||||
- Transform operations: These operations typically take some NDArray in, and change each element independent of others.
|
||||
- Reduction operations: These operations take some NDArray and dimensions, and return reduced NDArray (or scalar) back. I.e. sum along dimension(s).
|
||||
- Scalar operations: These operations are similar to transforms, but they only do arithmetic operations, and second operand is scalar. I.e. each element in given NDArray will add given scalar value.
|
||||
- Pairwise operations: These operations are between regular transform operations and scalar operations. I.e. element-wise addition of two NDArrays.
|
||||
- Random operations: Most of these operations related to random numbers distributions: Uniform, Gauss, Bernoulli etc.
|
||||
|
||||
Despite differences between these operations, they are all using XZ/XYZ three-operand design, where X and Y are inputs, and Z is output.
|
||||
Data access in these operations is usually trivial, and loop based. I.e. most trivial loop for scalar transform will look like this:
|
||||
```c++
|
||||
for (sd::LongType i = start; i < end; i++) {
|
||||
result[i] = OpType::op(x[i], scalar, extraParams);
|
||||
}
|
||||
```
|
||||
|
||||
Operation used in this loop will be template-driven, and compiled statically. There are another loops implementation, depending on op group or strides within NDArrays, but idea will be the same all the time: each element of the NDArray will be accessed within loop.
|
||||
|
||||
Now, let's take a look into typical XYZ op implementation. Here's how `Add` operation will look like:
|
||||
|
||||
```c++
|
||||
|
||||
template<typename T>
|
||||
class Add {
|
||||
public:
|
||||
SD_OP_DEF static T op(T d1, T d2) {
|
||||
return d1 + d2;
|
||||
}
|
||||
|
||||
// this signature will be used in Scalar loops
|
||||
SD_OP_DEF static T op(T d1, T d2, T *params) {
|
||||
return d1 + d2;
|
||||
}
|
||||
|
||||
// this signature will be used in reductions
|
||||
SD_OP_DEF static T op(T d1) {
|
||||
return d1;
|
||||
}
|
||||
|
||||
// op for MetaOps
|
||||
SD_OP_DEF static T op(T d1, T *params) {
|
||||
return d1 + params[0];
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
This particular operation is used in different XYZ op groups, but you see the idea: element-wise operation, which is invoked on each element in given NDArray.
|
||||
So, if you want to add new XYZ operation to libnd4j, you should just add operation implementation to file `includes/ops/ops.h`, and assign it to specific ops group in file `includes/loops/legacy_ops.h` together with some number unique to this ops group, i.e.: `(21, simdOps::Add)`
|
||||
|
||||
After libnd4j is recompiled, this op will become available for legacy execution mechanism, NDArray wrappers, and `LegacyOp` wrappers (those are made to map legacy operations to CustomOps design for Graph).
|
||||
|
||||
|
||||
## Custom operations
|
||||
|
||||
Custom operations is a new concept, added recently and mostly suits SameDiff/Graph needs.
|
||||
For CustomOps we defined universal signature, with variable number of input/output NDArrays, and variable number of floating-point and integer arguments.
|
||||
However, there are some minor differences between various CustomOp declarations:
|
||||
- **DECLARE_OP**(string, int, int, bool): these operations take no fp/int arguments, and output shape equals to input shape.
|
||||
- **DECLARE_CONFIGURABLE_OP**(string, int, int, bool, int, int): these operations do take fp/int output arguments, and output shape equals to input shape.
|
||||
- **DECLARE_REDUCTION_OP**(string, int, int, bool, int, int): these operations do take fp/int output arguments, and output shape is calculated as Reduction.
|
||||
- **DECLARE_CUSTOM_OP**(string, int, int, bool, int, int): these operations return NDArray with custom shape, that usually depends on input and arguments.
|
||||
- **DECLARE_BOOLEAN_OP**(string, int, bool): these operations take some NDArrays and return scalar, where 0 is **False**, and other values are treated as **True**.
|
||||
|
||||
Let's take a look at example CustomOp:
|
||||
|
||||
```c++
|
||||
|
||||
CUSTOM_OP_IMPL(tear, 1, -1, false, 0, -1) {
|
||||
auto input = INPUT_VARIABLE(0);
|
||||
|
||||
REQUIRE_TRUE(!block.getIArguments()->empty(), 0, "At least 1 dimension should be specified for Tear");
|
||||
|
||||
std::vector<int> dims(*block.getIArguments());
|
||||
|
||||
for (auto &v: dims)
|
||||
REQUIRE_TRUE(v >= 0 && v < input->rankOf(), 0, "Tear dimensions should be non-negative values, and lower than input rank. Got %i instead", v);
|
||||
|
||||
auto tads = input->allTensorsAlongDimension(dims);
|
||||
for (int e = 0; e < tads->size(); e++) {
|
||||
auto outE = OUTPUT_VARIABLE(e);
|
||||
outE->assign(tads->at(e));
|
||||
|
||||
this->storeResult(block, e, *outE);
|
||||
}
|
||||
|
||||
delete tads;
|
||||
|
||||
return sd::Status::OK;
|
||||
}
|
||||
|
||||
DECLARE_SHAPE_FN(tear) {
|
||||
auto inShape = inputShape->at(0);
|
||||
|
||||
std::vector<int> dims(*block.getIArguments());
|
||||
|
||||
if (dims.size() > 1)
|
||||
std::sort(dims.begin(), dims.end());
|
||||
|
||||
shape::TAD tad(inShape, dims.data(), (int) dims.size());
|
||||
tad.createTadOnlyShapeInfo();
|
||||
sd::LongType numTads = shape::tadLength(inShape, dims.data(), (int) dims.size());
|
||||
|
||||
auto result = SHAPELIST();
|
||||
for (int e = 0; e < numTads; e++) {
|
||||
result->push_back(tad.tadOnlyShapeInfo);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
```
|
||||
|
||||
In the example above, we declare `tear` CustomOp implementation, and shape function for this op.
|
||||
So, at the moment of op execution, we assume that we will either have output array(s) provided by end-user, or they will be generated with shape function.
|
||||
|
||||
You can also see number of macros used, we'll cover those later as well. Beyond that - op execution logic is fairly simple & linear:
|
||||
Each new op implements protected member function `DeclarableOp<T>::validateAndExecute(Block<T>& block)`, and this method is eventually called either from GraphExecutioner, or via direct call, like `DeclarableOp<T>::execute(Block<T>& block)`.
|
||||
|
||||
Important part of op declaration is input/output description for the op. I.e. as shown above: `CUSTOM_OP_IMPL(tear, 1, -1, false, 0, -1)`.
|
||||
This declaration means:
|
||||
- Op name: `tear`
|
||||
- Op expects at least 1 NDArray as input
|
||||
- Op returns unknown positive number of NDArrays as output
|
||||
- Op can't be run in-place, so under any circumstances original NDArray will stay intact
|
||||
- Op doesn't expect any T (aka floating point) arguments
|
||||
- Op expects unknown positive number of integer arguments. In case of this op it's dimensions to split input NDArray.
|
||||
|
||||
Here's another example: `DECLARE_CUSTOM_OP(permute, 1, 1, true, 0, -2);`
|
||||
This declaration means:
|
||||
- Op name: `permute`
|
||||
- Op expects at least 1 NDArray as input
|
||||
- Op returns 1 NDArray as output
|
||||
- Op can be run in-place if needed (it means: input == output, and input is modified and returned as output)
|
||||
- Op doesn't expect any T arguments
|
||||
- Op expects unknown number of integer arguments OR no integer arguments at all.
|
||||
|
||||
## c++11 syntactic sugar
|
||||
|
||||
In ops you can easily use c++11 features, including lambdas. In some cases it might be easiest way to build your custom op (or some part of it) via `NDArray::applyLambda` or `NDArray::applyPairwiseLambda`:
|
||||
```c++
|
||||
auto lambda = LAMBDA_TT(_x, _y) {
|
||||
return (_x + _y) * 2;
|
||||
};
|
||||
|
||||
x.applyPairwiseLambda(&y, lambda);
|
||||
```
|
||||
|
||||
In this simple example, each element of NDArray `x` will get values set to `x[e] = (x[e] + y[e]) * 2`.
|
||||
|
||||
## Tests
|
||||
|
||||
For tests libnd4j uses Google Tests suit. All tests are located at `tests_cpu/layers_tests` folder. Here's a simple way to run those from command line:
|
||||
```
|
||||
cd tests_cpu
|
||||
cmake -G "Unix Makefiles"
|
||||
make -j 4
|
||||
./layers_tests/runtests
|
||||
```
|
||||
|
||||
You can also use your IDE (i.e. Jetbrains CLion) to run tests via GUI.
|
||||
|
||||
**PLEASE NOTE:** if you're considering submitting your new op to libnd4j repository via pull request - consider adding tests for it. Ops without tests won't be approved.
|
||||
|
||||
## Backend-specific operation
|
||||
|
||||
GPU/MPI/whatever to be added soon.
|
||||
|
||||
|
||||
## Utility macros
|
||||
We have number of utility macros, suitable for custom ops. Here they are:
|
||||
- **INPUT_VARIABLE**(int): this macro returns you NDArray at specified input index.
|
||||
- **OUTPUT_VARIABLE**(int): this macro returns you NDArray at specified output index.
|
||||
- **STORE_RESULT**(NDArray<T>): this macro stores result to VariableSpace.
|
||||
- **STORE_2_RESULTS**(NDArray<T>, NDArray<T>): this macro stores results according to VariableSpace.
|
||||
- **INT_ARG**(int): this macro returns you specific Integer argument passed to the given op.
|
||||
- **T_ARG**(int): this macro returns you specific T argument passed to the given op.
|
||||
- **ALLOCATE**(...): this macro check if Workspace is available, and either uses Workspace or direct memory allocation if Workspace isn't available.
|
||||
- **RELEASE**(...): this macro is made to release memory allocated with **ALLOCATE()** macro.
|
||||
- **REQUIRE_TRUE**(...): this macro takes condition, and evaluates it. If evaluation doesn't end up as True - exception is raised, and specified message is printed out.
|
||||
- **LAMBDA_T**(X) and **LAMBDA_TT**(X, Y): lambda declaration for `NDArray::applyLambda` and `NDArray::applyPairwiseLambda`
|
||||
- **COPY_SHAPE**(SRC, TGT): this macro allocates memory for TGT pointer and copies shape from SRC pointer
|
||||
- **ILAMBDA_T**(X) and **ILAMBDA_TT**(X, Y): lambda declaration for indexed lambdas, index argument is passed in as sd::LongType (aka **long long**)
|
||||
- **SD_INLINE**: platform-specific definition for functions inlining
|
||||
|
||||
|
||||
#### Explicit template instantiations in helper methods.
|
||||
We should explicitly instantiate template methods for different data types in libraries. Furthermore, to speed up parallel compilation we need to add those template instantiations in separate source files. Besides, another reason is that: some compilers are choked when these template instantiations are many in one translation unit.
|
||||
To ease this cumbersome operation we have Cmake helper and macros helpers.
|
||||
Example:
|
||||
Suppose we have such function:
|
||||
|
||||
template<typename X, typename Z>
|
||||
void argMin_(const NDArray& input, NDArray& output, const std::vector<sd::LongType>& dimensions);
|
||||
|
||||
We should write this to explicitly instantiate it.
|
||||
|
||||
BUILD_DOUBLE_TEMPLATE(template void argMin_, (const NDArray& input, NDArray& output, const std::vector<sd::LongType>& dimensions),
|
||||
SD_COMMON_TYPES, SD_INDEXING_TYPES);
|
||||
|
||||
Here:
|
||||
- ***SD_COMMON_TYPES*** means we want to use all types in the place of X
|
||||
- ***SD_INDEXING_TYPES*** means we will use index types ( int, int64_t) as Z type
|
||||
|
||||
But to speed up compilation process and also helping compilers we can further separate it into different source files. Firstly we rename the original template source with ***hpp*** extension:
|
||||
Secondly we add file with the suffix ***cpp.in*** (or ***cu.in*** for cuda) that will include that hpp header and place it in the appropriate compilation units folder. in our case it will be in **./libnd4j/include/ops/declarable/helpers/cpu/compilation_units** folder with the name ***argmax.cpp.in*** .
|
||||
Later we decide which type we want to separate into different sources. In our case we want to split ***SD_COMMON_TYPES*** (other ones: ***SD_INTEGER_TYPES , SD_FLOAT_TYPES, SD_PAIRWISE_TYPES*** ). We hint cmake that case with this (adding ***_GEN*** suffix):
|
||||
|
||||
#cmakedefine SD_COMMON_TYPES_GEN
|
||||
|
||||
Then we just add ***_@FL_TYPE_INDEX@*** as suffix in type name and it will split those types for us and generate cpp files inside ${CMAKE_BINARY_DIR}/compilation_units folder.
|
||||
|
||||
LIBND4J_TYPE_@FL_TYPE_INDEX@
|
||||
|
||||
Here is how the complete cpp.in file will look like:
|
||||
|
||||
#cmakedefine SD_COMMON_TYPES_GEN
|
||||
//this header is where our template functions resides
|
||||
#include <ops/declarable/helpers/cpu/indexReductions.hpp>
|
||||
|
||||
//guard against undefined cases
|
||||
#if defined(SD_COMMON_TYPES_GEN) && defined(SD_COMMON_TYPES_@FL_TYPE_INDEX@)
|
||||
namespace sd {
|
||||
namespace ops {
|
||||
namespace helpers {
|
||||
BUILD_DOUBLE_TEMPLATE(template void argMax_, (const NDArray& input, NDArray& output, const std::vector<sd::LongType>& dimensions),
|
||||
SD_COMMON_TYPES_@FL_TYPE_INDEX@, SD_INDEXING_TYPES);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#endif
|
||||
@@ -0,0 +1,56 @@
|
||||
# Debugging Practices with C++ Tests in libnd4j Library
|
||||
|
||||
In this document, we will walk through some general debugging practices while working with C++ tests in the libnd4j library. This involves the use of Google Test framework (gtest), building with debug flags, and employing memory checking tools like Valgrind.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Ensure that you have the following installed on your system:
|
||||
|
||||
- Google Test framework (gtest)
|
||||
- CMake (for building the project)
|
||||
- Valgrind (for memory leak checking)
|
||||
- A modern C++ compiler that supports C++11 or later
|
||||
|
||||
## Building with Debug Flags
|
||||
|
||||
In order to debug effectively, you will need to compile your project with debugging information. This can be accomplished by adding debug flags during the build process.
|
||||
|
||||
If you're using CMake, you can do this by setting the `CMAKE_BUILD_TYPE` variable to `Debug`. This can be done through the command line as follows:
|
||||
|
||||
```
|
||||
cmake -DCMAKE_BUILD_TYPE=Debug ..
|
||||
```
|
||||
|
||||
When you're building your project with debug flags, the compiler will include extra information in the resulting executable that can be used to help find bugs.
|
||||
|
||||
## Debugging with Google Test (gtest)
|
||||
|
||||
Google Test is a powerful framework for writing C++ tests. It provides a simple and flexible way to write reliable tests, including features like test suites and assertions that make it easy to structure your tests and verify your code.
|
||||
|
||||
You can run a specific test using the `--gtest_filter` flag. The value passed to `--gtest_filter` is a colon-separated list of wildcard patterns (they may include `*` as a wildcard). Only the tests whose full names match one of the patterns will be executed.
|
||||
|
||||
For example, to run the `BasicTest_Scatter_1` test of the `ListOperationsTests` test case, you can use:
|
||||
|
||||
```
|
||||
./blasbuild/cpu/tests_cpu/layers_tests/runtests --gtest_filter=ListOperationsTests.BasicTest_Scatter_1
|
||||
```
|
||||
|
||||
## Debugging with Valgrind
|
||||
|
||||
Valgrind is an open-source tool that can be used for memory debugging, memory leak detection, and profiling. If your program is crashing or behaving unexpectedly, Valgrind can help identify memory leaks, uninitialized memory, and other related problems.
|
||||
|
||||
To run your tests under Valgrind, use the following command:
|
||||
|
||||
```
|
||||
valgrind --track-origins=yes -v ./blasbuild/cpu/tests_cpu/layers_tests/runtests --gtest_filter=ListOperationsTests.BasicTest_Scatter_1
|
||||
```
|
||||
|
||||
Here's what the flags do:
|
||||
|
||||
- `--track-origins=yes`: This tells Valgrind to track the origin of uninitialized values. This can be useful for finding the source of uninitialized value errors.
|
||||
- `-v`: This increases the verbosity of Valgrind's output, giving you more information about what's happening.
|
||||
- `--gtest_filter=ListOperationsTests.BasicTest_Scatter_1`: This is telling gtest to run only the `BasicTest_Scatter_1` test from the `ListOperationsTests` test suite.
|
||||
|
||||
## Conclusion
|
||||
|
||||
This should give you a solid foundation for debugging C++ tests in the libnd4j library. Remember, the key to effective debugging is to take a systematic approach, carefully observing the behavior of your program and making hypotheses about what could be causing any issues.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Deeplearning4j: Enhanced Stack Trace Feature Overview
|
||||
|
||||
## Introduction
|
||||
|
||||
For developers who are knee-deep in troubleshooting, understanding where a problem originated can be invaluable. In line with that, Deeplearning4j now introduces an advanced feature that provides an insightful fusion of Java and C++ stack traces. This is especially useful when debugging issues related to memory allocation and deallocation.
|
||||
|
||||
## Feature: SD_GCC_FUNCTRACE
|
||||
|
||||
When you build Deeplearning4j with the `SD_GCC_FUNCTRACE` option turned on, it activates the ability to display C++ stack traces. This powerful feature, however, comes with a caveat: it requires numerous platform-specific dependencies to function seamlessly.
|
||||
|
||||
### What's New?
|
||||
|
||||
When the aforementioned feature is active, developers can now enable a fresh capability that showcases both Java and C++ stack traces at every instance of memory allocation and deallocation in the Deeplearning4j codebase.
|
||||
|
||||
Here's the crux of this new feature:
|
||||
|
||||
1. **Allocation and Deallocation Triggers**: The stack traces will be printed just as a buffer is about to be deallocated.
|
||||
2. **Crash Insights**: Typically, the last deallocation that took place will pinpoint the site of the crash.
|
||||
3. **Full Problem Context**: By analyzing Java and C++ stack traces side by side, developers can derive a comprehensive understanding of the issue at hand.
|
||||
4. **Enhancement Over Sanitizers**: This feature is a supplement to sanitizers, which occasionally falter in showing internal stack traces instead of the real underlying problem.
|
||||
|
||||
## Enabling the Feature
|
||||
|
||||
Activating this feature is straightforward. Here's a snippet to do just that:
|
||||
|
||||
```java
|
||||
Nd4j.getEnvironment().setFuncTraceForAllocate(true);
|
||||
Nd4j.getEnvironment().setFuncTraceForDeallocate(true);
|
||||
```
|
||||
|
||||
With these lines of code:
|
||||
|
||||
- The first line will enable the printing of stack traces during memory allocation.
|
||||
- The second line will do the same for deallocation.
|
||||
|
||||
## Conclusion
|
||||
|
||||
By leveraging this new feature, developers can achieve a granular understanding of memory-related issues in Deeplearning4j's operations. This comprehensive insight into both Java and C++ realms will significantly streamline the debugging process and enhance code reliability.
|
||||
|
||||
_Remember, while powerful, this feature can also be verbose. Hence, it's recommended to use it judiciously, primarily when deep troubleshooting is necessary._
|
||||
@@ -0,0 +1,172 @@
|
||||
## Helpers
|
||||
|
||||
### Requirements Helper
|
||||
|
||||
Requirements helper was introduced to replace plain checks for making them output informative messages (Debug and Verbose mode) and also replace macros REQUIRE_TRUE.
|
||||
|
||||
- it will lazily evaluate values and messages if the type wrapped and has` getValue` and `getMsg` methods
|
||||
- it is implicit bool. this makes it usable with logical operators and also inside if conditions. Besides it will benefit from shortcircuit nature of those operators.
|
||||
- it has the following check methods
|
||||
```cpp
|
||||
Requirements& expect(const T& expVar,const T1& reqVar, Op comparison, const char *first_half="")
|
||||
Requirements& expectEq(const T& exp,const T1& req)
|
||||
Requirements& expectNotEq(const T& exp,const T1& req)
|
||||
Requirements& expectLess(const T& exp,const T1& req)
|
||||
Requirements& expectLessEq(const T& exp,const T1& req)
|
||||
Requirements& expectGreater(T exp, T1 req)
|
||||
Requirements& expectGreaterEq(const T& exp,const T1& req
|
||||
Requirements& expectTrue(const T& expVar, const char *msg=)
|
||||
Requirements& expectFalse(const T& expVar, const char *msg=)
|
||||
```
|
||||
- you can either log the success case or throw error on the failure
|
||||
- it can use plain types for checks.
|
||||
- if value has stream operator it will be used to output its value. for custom types you may need add that by yourself
|
||||
`ostream& operator<<(ostream& os, const CustomUserType& dt)`
|
||||
- there is generic template `InfoVariable` wrapper for types to make it informative. you can use lambda operators with them as well to make it lazily evaluated
|
||||
- we added custom `ShapeInfoVariable` wrapper for the NDArray and vector<> shapes to make them informative
|
||||
- one can use `expect` to add its own proper comparison. simple lambda for that will be like this:
|
||||
```cpp
|
||||
[](const decltype(expType)& l, const decltype(reqType)& r){
|
||||
//compare and return
|
||||
return ....;
|
||||
}
|
||||
```
|
||||
|
||||
#### Examples:
|
||||
|
||||
firstly, we should enable logging
|
||||
```cpp
|
||||
sd::Environment::getInstance().setDebug(true);
|
||||
sd::Environment::getInstance().setVerbose(true);
|
||||
```
|
||||
|
||||
1. simple case
|
||||
|
||||
```cpp
|
||||
Requirements req1("Requirement Helper Example#1");
|
||||
int x = 20;
|
||||
req1.expectLess(x, 22);
|
||||
req1.expectEq(x, 21); //should fail
|
||||
```
|
||||
|
||||
|
||||
Output:
|
||||
``` Requirement Helper Example#1: {20} expected to be equal to 21```
|
||||
|
||||
2. using InfoVariable wrapper
|
||||
```cpp
|
||||
int age = 15;
|
||||
Requirements req2("Requirement Helper Example#2");
|
||||
req2.expectGreaterEq(makeInfoVariable(age, "the user's age"), 18);
|
||||
```
|
||||
Output:
|
||||
```
|
||||
Requirement Helper Example#2: the user's age {15} expected to be greater than or equal 18
|
||||
```
|
||||
|
||||
3. helper behavior while using many checks in one block
|
||||
```cpp
|
||||
int getAge(){
|
||||
std::cout<<"getAge() was called"<<std::endl;
|
||||
return 15;
|
||||
}
|
||||
....
|
||||
Requirements req3("Requirement Helper Example#3");
|
||||
int z = 20;
|
||||
req3.expectEq(z, 21);
|
||||
req3.expectGreaterEq(makeInfoVariable(getAge(), "the user's age"), 18);
|
||||
```
|
||||
Output:
|
||||
```
|
||||
Requirement Helper Example#3: {20} expected to be equal to 21
|
||||
getAge() was called
|
||||
```
|
||||
|
||||
As it is seen the second check did not happen as the previous failed. But still ```getAge()``` method was called as its function argument.
|
||||
|
||||
4. using **shortcircuit** to avoid Requirement call at all if the previous one was failed
|
||||
```cpp
|
||||
Requirements req4("Requirement Helper Example#4");
|
||||
int zz = 20;
|
||||
req4.expectEq(zz, 21) && //shortcircuit And
|
||||
req4.expectGreaterEq(makeInfoVariable(getAge(), "the user's age"), 18);
|
||||
```
|
||||
Output:
|
||||
```
|
||||
Requirement Helper Example#4: {20} expected to be equal to 21
|
||||
```
|
||||
5. using lambdas with InfoVariable. it will make it lazily evaluated
|
||||
```cpp
|
||||
Requirements req5("Requirement Helper Example#5");
|
||||
req5.expectEq( 21,
|
||||
makeInfoVariable(21, []{
|
||||
std::cout<<"lambda call#1"<<std::endl;
|
||||
return "twenty one";
|
||||
}));
|
||||
req5.expectEq(makeInfoVariable([]{ return 20;}, []{return "twenty";}),
|
||||
makeInfoVariable(21, []{
|
||||
std::cout<<"lambda call#2"<<std::endl;
|
||||
return "twenty one";
|
||||
}));
|
||||
req5.expectGreaterEq(makeInfoVariable([]{
|
||||
std::cout<<"lambda call#3" <<std::endl;
|
||||
return 15;
|
||||
},
|
||||
[]{ return "the user's age";}),
|
||||
makeInfoVariable([]{return 18;}, []{return "the allowed age";})
|
||||
);
|
||||
```
|
||||
Output:
|
||||
```
|
||||
lambda call#2
|
||||
Requirement Helper Example#5: twenty {20} expected to be equal to twenty one 21
|
||||
|
||||
```
|
||||
|
||||
6. use bool nature and also log the success case
|
||||
```cpp
|
||||
Requirements req6("Requirement Helper Example#6");
|
||||
NDArray * arr= nullptr;
|
||||
arr !=nullptr && req6.expectEq(arr->rankOf(), 3) ;
|
||||
req6.logTheSuccess();
|
||||
```
|
||||
Output:
|
||||
```
|
||||
Requirement Helper Example#6: meets the requirements
|
||||
```
|
||||
|
||||
7. custom comparision lambda and also another usage of the custom wrapper written by us ```ShapeInfoVariable```. Note: we will use ```std::vector<int>```. this wrapper can be used with ```NDArray``` as well.
|
||||
```cpp
|
||||
Requirements req7("Requirement Helper Example#7");
|
||||
req7.expect(makeShapeInfoVariable(std::vector<sd::LongType>{2,3,4,5}, SHAPE_MSG_INPUT0), makeShapeInfoVariable(std::vector<sd::LongType>{2,3,4,7}, SHAPE_MSG_INPUT1),
|
||||
[](const std::vector<sd::LongType>& l, const std::vector<sd::LongType>& r){
|
||||
return l == r;
|
||||
}
|
||||
, EXPECTED_EQ_MSG);
|
||||
}
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Requirement Helper Example#7: the Shape of the Input NDArray#0 {[2, 3, 4, 5]} expected to be equal to the Shape of the Input NDArray#1 [2, 3, 4, 7]
|
||||
```
|
||||
|
||||
8. throw error when there is failure
|
||||
```cpp
|
||||
Requirements req8("Requirement Helper Example#8");
|
||||
req8.expectEq(6,6) &&
|
||||
req8.expectIn(6, {1,2,3,7,8,9});
|
||||
req8.throws();
|
||||
```
|
||||
Output:
|
||||
```
|
||||
terminate called after throwing an instance of 'std::invalid_argument'
|
||||
what(): Op validation failed
|
||||
...
|
||||
Requirement Helper Example#8: {6} expected to be one of these {[1, 2, 3, 7, 8, 9, ]}
|
||||
```
|
||||
|
||||
|
||||
##### Here is live example:
|
||||
**Note:** some classes were mocked there and do not represent the exact implementations in libnd4j.
|
||||
https://godbolt.org/z/sq98vchs5
|
||||
@@ -0,0 +1,32 @@
|
||||
# libnd4j CUDA Kernel Launch Configuration
|
||||
|
||||
This part of the libnd4j codebase provides a centralized way to handle CUDA launch configurations. It allows customization of launch dimensions using environment variables.
|
||||
|
||||
## Conceptual Overview
|
||||
|
||||
The code sets up CUDA launch configurations for a wide range of algorithms used in the libnd4j codebase. These configurations are expressed as `dim3` structures, representing grid and block dimensions for launching CUDA kernels. The configurations are stored in a map, improving performance via a caching mechanism.
|
||||
|
||||
### Key Feature
|
||||
|
||||
A significant feature of this code is the flexibility it offers, allowing users to specify custom launch dimensions through environment variables. This enables modifications of these configurations at both compile-time and runtime.
|
||||
|
||||
## Examples of Usage
|
||||
|
||||
```cpp
|
||||
// Fetching Launch Dimensions
|
||||
dim3 launchDims = getLaunchDims("matrixMultiply");
|
||||
//invoke your cuda function
|
||||
yourKernel<<<launchDims.y,launchDims.x,launchDims.z>>>
|
||||
//This returns the blocks, threads and shared memory used for the kernel.
|
||||
```
|
||||
|
||||
|
||||
The launch configuration can be found in
|
||||
```bash
|
||||
../include/execution/cuda/LaunchDims.h
|
||||
../include/execution/cuda/LaunchDims.cu
|
||||
```
|
||||
|
||||
Every kernel has environment variables that work at build time and runtime.
|
||||
Build time is used to modify defaults. Runtime is used to override buildtime and defaults.
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
Please follow following instructions to build nd4j for raspberry PI:
|
||||
|
||||
1. download cross compilation tools for Raspberry PI
|
||||
|
||||
```
|
||||
$ apt-get/yum install git cmake
|
||||
(You may substitute any path you prefer instead of $HOME/raspberrypi in the following two steps)
|
||||
$ mkdir $HOME/raspberrypi
|
||||
$ export RPI_HOME=$HOME/raspberrypi
|
||||
$ cd $RPI_HOME
|
||||
$ git clone git://github.com/raspberrypi/tools.git
|
||||
$ export PATH=$PATH:$RPI_HOME/tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin
|
||||
```
|
||||
|
||||
2. download deeplearning4j:
|
||||
|
||||
```
|
||||
$ cd $HOME
|
||||
$ git clone https://github.com/eclipse/deeplearning4j.git
|
||||
```
|
||||
|
||||
3. build libnd4j:
|
||||
|
||||
```
|
||||
$ cd deeplearning4j/libnd4j
|
||||
$ ./buildnativeoperations.sh -o linux-armhf
|
||||
```
|
||||
|
||||
4. build nd4j
|
||||
|
||||
```
|
||||
$ export LIBND4J_HOME=<pathTond4JNI>
|
||||
$ cd $HOME/deeplearning4j/nd4j
|
||||
$ mvn clean install -Djavacpp.platform=linux-armhf -Djavacpp.platform.compiler=$HOME/raspberrypi/tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin/arm-linux-gnueabihf-g++ -DskipTests -Dmaven.javadoc.skip=true -pl '!:nd4j-cuda-9.1,!:nd4j-cuda-9.1-platform,!:nd4j-tests'
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
Due to the macros and sheer number of type combinations we often run in to undefined symbols when linking.
|
||||
|
||||
We typically use this command on linux to find issues:
|
||||
nm -uC libnd4j/blasbuild/cpu/blas/libnd4jcpu.so
|
||||
|
||||
or cuda:
|
||||
nm -uC libnd4j/blasbuild/cuda/blas/libnd4jcuda.so
|
||||
|
||||
|
||||
You can also do the same for *just* undefined:
|
||||
nm -C --undefined-only libnd4jcpu.so
|
||||
nm -C --undefined-only libnd4jcuda.so
|
||||
@@ -0,0 +1,145 @@
|
||||
# Graph
|
||||
|
||||
### Basic idea
|
||||
libnd4j contains Directed Acyclic Graph execution engine, suited for both local and remote execution. However, main goal here is execution of externally originated graphs, serialized into FlatBuffers and provided either via pointer, or file.
|
||||
|
||||
|
||||
This basic example shows execution of graph loaded from file:
|
||||
```c++
|
||||
auto graph = GraphExecutioner<float>::importFromFlatBuffers("./some_file.fb");
|
||||
GraphExecutioner<float>::execute(graph);
|
||||
// ... do something with results ...
|
||||
delete graph;
|
||||
```
|
||||
|
||||
### FlatBuffers schemas
|
||||
You can find scheme files [here](https://github.com/eclipse/deeplearning4j/tree/master/libnd4j/include/graph/scheme).
|
||||
|
||||
At this moment libnd4j repo contains compiled definitions for C++, Python, Java, and JSON, but FlatBuffers can be compiled for PHP, C#, JavaScript, TypeScript and Go as well. Please refer to `flatc` instructions to do that.
|
||||
|
||||
Such bindings allow you to build FlatBuffers files/buffers suitable for remote execution of your graph and obtaining results back. I.e. you can use JavaScript to build graph (or just update variables/placeholders), send them to remote RPC server powered by libnd4j, and get results back.
|
||||
|
||||
Use flatc-generate.sh to generate the relevant source files if you need to change anything related to flatbuffers such
|
||||
as the samediff file format or the UI.
|
||||
|
||||
### Graph execution logic
|
||||
No matter how graph is represented on the front-end, on backend it's rather simple: topologically sorted list of operations executed sequentially if there's shared dependencies, or (optionally) in parallel, if there's no shared dependencies for current graph nodes.
|
||||
|
||||
Each node in the graph represents single linear algebra operation applied to input(s) of the node. For example: `z = Add(x, y)` is operation that takes 2 NDArrays as input, and produes 1 NDArray as output. So, graph is built of such primitive operations, which are executed sequentially.
|
||||
|
||||
### Memory management within graph
|
||||
Everything that happens within graph during execution, stays within VariableSpace. It acts as storage for Variables and NDArrays produced during graph execution. On top of that, there's an option to use pre-allocated Workspaces for allocation of NDArrays.
|
||||
|
||||
|
||||
### Current graph limitations
|
||||
There are some limitations. Some of them will be lifted eventually, others won't be. Here's the list:
|
||||
- Graph has single data type. I.e. Graph<float> or Graph<float16> or Graph<double> _This limitation will be lifted soon._
|
||||
- On some platforms, like Java, single Variable/Placeholder size is limited to 2GB buffer size. However, on libnd4j side there's no such limitation.
|
||||
- Variable size/dimensionality has limitations: max NDArray rank is limited to 32 at this moment, and any single dimension is limited to SD_MAX_INT size.
|
||||
- Recursion isn't directly supported at this moment.
|
||||
- CUDA isn't supported at this moment. _This limitation will be lifted soon._
|
||||
- When used from C++, Graph only supports FeedForward mode. _This limitation will be lifted soon._
|
||||
|
||||
|
||||
### Documentation
|
||||
Documentation for individual operations, and basic classes (like NDArray, Graph etc) is available as part of Nd4j javadoc: https://javadoc.io/doc/org.nd4j/nd4j-api/latest/index.html
|
||||
|
||||
### Embedded profiling
|
||||
If you're adding new ops, and want to make sure they run ok on your specific device - you might want to give a shot to embedded Graph profiling helper.
|
||||
Despite being simple - it still provides you with time spent in various parts of Graph.
|
||||
|
||||
```c++
|
||||
Environment::getInstance().setProfiling(true);
|
||||
auto graph = GraphExecutioner::importFromFlatBuffers("./resources/ae_00.fb");
|
||||
|
||||
auto profile = GraphProfilingHelper::profile(graph, 1000);
|
||||
profile->printOut();
|
||||
|
||||
delete graph;
|
||||
```
|
||||
|
||||
1000 iterations laterm you'll get statistics printed out. Statistics basically includes time spent in various parts of code and memory allocation details.
|
||||
|
||||
Here's how it'll look like:
|
||||
```
|
||||
Printing out Graph...
|
||||
8. matmul; Inputs: [{1:0}, {2:0}];
|
||||
9. biasadd; Inputs: [{8:0}, {3:0}];
|
||||
10. TRANSFORM:{15}; Inputs: [{9:0}];
|
||||
11. rank; Inputs: [{2:0}];
|
||||
12. subtract; Inputs: [{11:0}, {4:0}];
|
||||
13. range; Inputs: [{5:0}, {11:0}, {6:0}];
|
||||
14. subtract; Inputs: [{12:0}, {13:0}];
|
||||
15. transpose; Inputs: [{2:0}, {14:0}];
|
||||
16. matmul; Inputs: [{10:0}, {15:0}];
|
||||
17. biasadd; Inputs: [{16:0}, {7:0}];
|
||||
18. TRANSFORM:{15}; Inputs: [{17:0}];
|
||||
|
||||
Printing out Scopes...
|
||||
Graph profile: 1000 executions
|
||||
|
||||
Memory:
|
||||
ACT: 0; TMP: 0; OBJ: 0; TTL: 1788;
|
||||
|
||||
Time:
|
||||
Construction time: 2135 ns;
|
||||
Execution time: 41820 ns;
|
||||
|
||||
Per-node reports:
|
||||
Node: <8:MatMul>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 200;
|
||||
Time: PREP: 1160 ns; EXEC: 3167 ns; TTL: 5929 ns;
|
||||
PREP: INPUT: 251 ns; SHAPE: 382 ns; ARRAY: 217 ns;
|
||||
Node: <9:BiasAdd>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 104;
|
||||
Time: PREP: 917 ns; EXEC: 3580 ns; TTL: 5957 ns;
|
||||
PREP: INPUT: 220 ns; SHAPE: 213 ns; ARRAY: 217 ns;
|
||||
Node: <10:Tanh>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 104;
|
||||
Time: PREP: 756 ns; EXEC: 241 ns; TTL: 1927 ns;
|
||||
PREP: INPUT: 140 ns; SHAPE: 195 ns; ARRAY: 205 ns;
|
||||
Node: <11:transpose/Rank>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 36;
|
||||
Time: PREP: 522 ns; EXEC: 119 ns; TTL: 1403 ns;
|
||||
PREP: INPUT: 109 ns; SHAPE: 69 ns; ARRAY: 171 ns;
|
||||
Node: <12:transpose/sub>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 36;
|
||||
Time: PREP: 666 ns; EXEC: 185 ns; TTL: 1684 ns;
|
||||
PREP: INPUT: 192 ns; SHAPE: 94 ns; ARRAY: 168 ns;
|
||||
Node: <13:transpose/Range>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 556;
|
||||
Time: PREP: 808 ns; EXEC: 647 ns; TTL: 2416 ns;
|
||||
PREP: INPUT: 297 ns; SHAPE: 228 ns; ARRAY: 181 ns;
|
||||
Node: <14:transpose/sub_1>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 56;
|
||||
Time: PREP: 721 ns; EXEC: 541 ns; TTL: 2205 ns;
|
||||
PREP: INPUT: 23 ns; SHAPE: 92 ns; ARRAY: 165 ns;
|
||||
Node: <15:transpose>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 96;
|
||||
Time: PREP: 3936 ns; EXEC: 602 ns; TTL: 5811 ns;
|
||||
PREP: INPUT: 194 ns; SHAPE: 3241 ns; ARRAY: 257 ns;
|
||||
Node: <16:MatMul_1>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 312;
|
||||
Time: PREP: 970 ns; EXEC: 3565 ns; TTL: 6066 ns;
|
||||
PREP: INPUT: 203 ns; SHAPE: 320 ns; ARRAY: 193 ns;
|
||||
Node: <17:BiasAdd_1>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 144;
|
||||
Time: PREP: 914 ns; EXEC: 3528 ns; TTL: 5870 ns;
|
||||
PREP: INPUT: 231 ns; SHAPE: 191 ns; ARRAY: 223 ns;
|
||||
Node: <18:output>
|
||||
Memory: ACT: 0; TMP: 0; OBJ: 0; TTL: 144;
|
||||
Time: PREP: 805 ns; EXEC: 285 ns; TTL: 1928 ns;
|
||||
PREP: INPUT: 157 ns; SHAPE: 192 ns; ARRAY: 232 ns;
|
||||
|
||||
Special timers:
|
||||
No special timers were set
|
||||
```
|
||||
|
||||
|
||||
### Roadmap
|
||||
In short-to-medium term following improvements are expected:
|
||||
- CUDA support for all new ops
|
||||
- Additional data types support: int, long long, q types, bool
|
||||
- Sparse tensors support
|
||||
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
### Note to iOS build
|
||||
|
||||
I used LLVM 4.0 to build `ios-arm`, `ios-x86`, and `ios-x86_64`. LLVM clang seems to have a bug in recognizing arm64, I used Xcode 9.0 to build `ios-arm64` without `-fopenmp`.
|
||||
This is for build only, though. Since I have not built nd4j yet, it is to be decided later if MLPMnistSingleLayerExample and MLPMnistTwoLayerExample will run on iOS.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Build DL4J for Linux on Power
|
||||
|
||||
This document explains how to build DL4J for Linux on Power
|
||||
|
||||
## Check pre-requirements
|
||||
|
||||
The following tools are necessary to build DL4J for Linux on Power
|
||||
|
||||
1. gcc version 4.9.X (older or newer version may cause build errors.)
|
||||
Check the version by "gcc --version", and install it from https://gcc.gnu.org/ if necessary.
|
||||
Look at Apppendix of this document to install gcc 4.9.
|
||||
|
||||
2. OpenJDK java 1.8
|
||||
Check the version is OpenJDK and 1.8 by "java -version".
|
||||
|
||||
3. Apache Maven 3.3 or later
|
||||
Check the version by "mvn -v", and install it from https://maven.apache.org/ if necessary.
|
||||
|
||||
4. CUDA7.5 (CUDA7.0 is not supported)
|
||||
Check the version by "nvcc -V"
|
||||
|
||||
5. cmake version 3.5.0 or later
|
||||
|
||||
## Set Environment Variables
|
||||
|
||||
Edit the CUDA, JAVA_HOME, CC, CXX environment variables according to your system
|
||||
|
||||
```
|
||||
export CUDA=/usr/local/cuda-7.5 # CUDA Directory
|
||||
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el # JAVA Directory
|
||||
export CC=/path/to/gcc # gcc command for build
|
||||
export CXX=/path/to/g++ # g++ command for build
|
||||
export MAVEN_OPTS='-Xmx4096M -Dos.arch=ppc64le'
|
||||
export _JAVA_OPTIONS=-Dos.arch=ppc64le
|
||||
export LIBND4J_HOME=`/bin/pwd`/libnd4j
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
```
|
||||
|
||||
## Clone and build DL4J, and execute a sample program
|
||||
|
||||
1. Clone necessary modules from github as follows.
|
||||
```
|
||||
git clone https://github.com/eclipse/deeplearning4j.git
|
||||
```
|
||||
|
||||
2. Modify the setting file to enable GPU. (Skip this step if you do not use GPU)
|
||||
Modify the following line in dl4j-0.4-examples/pom.xml as follows.
|
||||
```
|
||||
Before: <nd4j.backend>nd4j-native</nd4j.backend>
|
||||
After: <nd4j.backend>nd4j-cuda-7.5</nd4j.backend>
|
||||
```
|
||||
|
||||
3. Build modules as follows. (Make sure you follow the instructions in order.)
|
||||
```
|
||||
(cd javacpp/; mvn clean install -DskipTests)
|
||||
(cd libnd4j; ./buildnativeoperations.sh)
|
||||
(cd libnd4j; ./buildnativeoperations.sh -c cuda)
|
||||
(cd nd4j; mvn -e clean install -DskipTests -DskipTests -Djavacpp.platform.dependency=false -Dmaven.javadoc.skip=true)
|
||||
(cd Canova/; mvn clean install -DskipTests -Djavacpp.platform.dependency=false)
|
||||
(cd deeplearning4j; mvn clean package -DskipTests -Djavacpp.platform.dependency=false)
|
||||
(cd dl4j-0.4-examples; mvn clean package -DskipTests)
|
||||
```
|
||||
|
||||
4. Test the module by running the LenetMnist example
|
||||
```
|
||||
(cd dl4j-0.4-examples; java -cp target/deeplearning4j-examples-0.4-rc0-SNAPSHOT-bin.jar org.deeplearning4j.examples.convolution.LenetMnistExample)
|
||||
```
|
||||
|
||||
## Appendix
|
||||
|
||||
How to build gcc 4.9.3 on Linux on Power
|
||||
|
||||
```
|
||||
$ tar xvfz gcc-4.9.3.tar.gz
|
||||
$ cd gcc-4.9.3
|
||||
$ mkdir -p build
|
||||
$ (cd build; ../configure --enable-languages=c,c++ --prefix=<install path> --disable-bootstrap --disable-multilib)
|
||||
$ (cd build; make)
|
||||
$ (cd build; make install)
|
||||
# You need to add <install path>/lib64 in LD_LIBRRY_PATH
|
||||
# BLAS need to be specified in LD_LIBRARY_PATH to run CPU(native) version
|
||||
```
|
||||
@@ -0,0 +1,52 @@
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
The nd4j and libnd4j code base can be difficult to debug due to the lack of tools for tracing program execution. We have implemented function tracing using the `-finstrument-functions` flag provided by GCC to trace function calls and exits in the code base. This feature will help developers better understand the flow of the program and identify potential issues.
|
||||
|
||||
## How to Enable Function Tracing
|
||||
|
||||
To enable function tracing, follow these steps:
|
||||
|
||||
1. Build the code base with `-Dlibnd4j.functrace=ON`.
|
||||
2. Add the correct compiler flags to the javacpp plugin.
|
||||
3. Add the correct flags to the cmake build.
|
||||
4. Add a maven profile to each nd4j backend containing the correct compiler flags to work with `-finstrument-functions`.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Compiler Flags
|
||||
|
||||
We have used the following compiler flags to enable function tracing:
|
||||
|
||||
- `-Bsymbolic`: Bind references to global symbols at link time, reducing the runtime overhead.
|
||||
- `-rdynamic`: Export all dynamic symbols to the dynamic symbol table, making them available for backtracing.
|
||||
- `-fno-omit-frame-pointer`: Do not omit the frame pointer, allowing for accurate backtraces.
|
||||
- `-fno-optimize-sibling-calls`: Disable sibling call optimization to maintain the correct call stack.
|
||||
- `-finstrument-functions`: Enable instrumentation of function entry and exit points.
|
||||
- `-g`: Enable debugging information.
|
||||
- `-O0`: Set the optimization level to zero for easier debugging.
|
||||
|
||||
### Setting the Output File
|
||||
|
||||
We have implemented a method in each backend to set the output file for tracing results. For example, in the Nd4jCpu class:
|
||||
|
||||
```java
|
||||
Nd4jCpu nd4jCpu = (Nd4jCpu) NativeOpsHolder.getInstance().getDeviceNativeOps();
|
||||
nd4jCpu.setInstrumentOut("profilerout.txt");
|
||||
```
|
||||
|
||||
This calls a C++ method that sets the appropriate file to use. Each backend will have this call. Note that we don't put this in NativeOps (the parent backend agnostic interface for this) because this normally should not be included in any builds due to overhead.
|
||||
|
||||
### LD_PRELOAD and LD_DEBUG
|
||||
|
||||
To ensure the correct implementation of the enter/exit functions is used, `LD_PRELOAD` is utilized to preload the libnd4j binary generated during the build. The built-in libc implementation of these functions is a no-op, so preloading the libnd4j binary is necessary.
|
||||
|
||||
`LD_DEBUG` can be used to verify that the correct implementation is being used by showing the symbols being loaded and their origin.
|
||||
|
||||
## Sample Log Output
|
||||
|
||||
```
|
||||
g long> > >::end() (/home/agibsonccc/Documents/GitHub/deeplearning4j/libnd4j/blasbuild/cpu/blas/libnd4jcpu.so)
|
||||
enter std::operator==(std::_Rb_tree_iterator<std::pair<int const, long long> > const&, std::_Rb_tree_iterator<std::pair<int const, long long> > const&) (/home/agibsonccc/Documents/GitHub/deeplearning4j/libnd4j/blasbuild/cpu/blas/libnd4jcpu.so)
|
||||
exit std::operator==(std::_Rb_tree_iterator<std::pair<int const, long long> > const&, std::_Rb_tree_iterator<std::pair<int const, long long> > const&) (/home/agibsonccc/Documents/GitHub/deeplearning4j/libnd4j/blasbuild/cpu/blas/libnd4
|
||||
Reference in New Issue
Block a user