chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,197 @@
|
||||
Introduction into Android Development {#tutorial_android_dev_intro}
|
||||
=====================================
|
||||
|
||||
@prev_tutorial{tutorial_clojure_dev_intro}
|
||||
@next_tutorial{tutorial_dev_with_OCV_on_Android}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Rostislav Vasilikhin |
|
||||
| Compatibility | OpenCV >= 4.0 |
|
||||
|
||||
@tableofcontents
|
||||
|
||||
This guide was designed to help you in learning Android development basics and setting up your
|
||||
working environment quickly. It was tested with Ubuntu 22.04 and Windows 10.
|
||||
|
||||
If you encounter any error after thoroughly following these steps, feel free to contact us via
|
||||
OpenCV [Forum](https://forum.opencv.org). We'll do our best to help you out.
|
||||
|
||||
Preface
|
||||
-------
|
||||
|
||||
Android is a Linux-based, open source mobile operating system developed by Open Handset Alliance led
|
||||
by Google. See the [Android home site](http://www.android.com/about/) for general details.
|
||||
|
||||
Development for Android significantly differs from development for other platforms. So before
|
||||
starting programming for Android we recommend you make sure that you are familiar with the following
|
||||
key topics:
|
||||
|
||||
-# [Java](http://en.wikipedia.org/wiki/Java_(programming_language)) programming language that is
|
||||
the primary development technology for Android OS. Also, you can find [Oracle docs on
|
||||
Java](http://docs.oracle.com/javase/) useful.
|
||||
-# [Java Native Interface (JNI)](http://en.wikipedia.org/wiki/Java_Native_Interface) that is a
|
||||
technology of running native code in Java virtual machine. Also, you can find [Oracle docs on
|
||||
JNI](http://docs.oracle.com/javase/7/docs/technotes/guides/jni/) useful.
|
||||
-# [Android
|
||||
Activity](http://developer.android.com/training/basics/activity-lifecycle/starting.html) and its
|
||||
life-cycle, that is an essential Android API class.
|
||||
-# OpenCV development will certainly require some knowledge of the [Android
|
||||
Camera](http://developer.android.com/guide/topics/media/camera.html) specifics.
|
||||
|
||||
Manual environment setup for Android development
|
||||
------------------------------------------------
|
||||
|
||||
In this tutorial we're gonna use an official Android Studio IDE and a set of other freely available tools.
|
||||
|
||||
### Get tools and dependencies
|
||||
|
||||
Here's how to get a ready to work environment:
|
||||
|
||||
1. Download and install Android Studio:
|
||||
* Ubuntu:
|
||||
1. Download Android Studio: https://developer.android.com/studio
|
||||
2. Extract the tar.gz archive
|
||||
3. Follow the instructions in `Install-Linux-tar.txt`: open `android-studio/bin` folder in terminal and run `./studio.sh`
|
||||
4. Perform standard installation through GUI
|
||||
5. Optionally you can add a shortcut on a desktop for a quick access by clicking menu ***Tools -> Create desktop entry***. The menu appears after any project is created or opened.
|
||||
* Windows:
|
||||
Just download Android Studio from the official site and run installer.
|
||||
|
||||
2. Install fresh Android SDK and NDK:
|
||||
1. Open SDK manager in Android Studio (***Customize -> All Settings -> Languages & Frameworks -> Android SDK***)
|
||||
2. Enable "Show Package Details" checkbox
|
||||
|
||||
3. Check SDK and NDK of the latest versions and press OK
|
||||
4. Make sure that your device support the chosen SDK versions
|
||||
|
||||
3. Install all the necessary packages for the build:
|
||||
- `sudo apt install git cmake ninja-build openjdk-17-jdk openjdk-17-jre`
|
||||
- the rest required packages are dependencies and should be installed automatically
|
||||
|
||||
### Check OpenCV examples
|
||||
|
||||
1. Download OpenCV from Android SDK from official [release page on Github](https://github.com/opencv/opencv/releases)
|
||||
or [SourceForge](https://sourceforge.net/projects/opencvlibrary/).
|
||||
2. Extract zip archive with your OS tools.
|
||||
3. Open the project `<YOUR_OPENCV_BUILD_FOLDER>/OpenCV-android-sdk/samples` in Android Studio.
|
||||
4. Connect your device
|
||||
* Debugging should be enabled on a device, you can find an instruction about it across the web
|
||||
* Alternatively you can use a virtual device that comes with the Android studio
|
||||
|
||||
5. Choose a sample from the drop-down menu (for example, `15-puzzle`) and run it.
|
||||
|
||||
Setup Device for Testing and Debugging
|
||||
--------------------------------------
|
||||
|
||||
Usually the recipe above works as expected, but in some cases there are additional actions that must
|
||||
be performed. In this section we'll cover some cases.
|
||||
|
||||
### Windows host computer
|
||||
|
||||
If you have Windows 10 or higher then you don't have to do additional actions to connect
|
||||
a phone and run samples on it. However, earlier Windows versions require a longer procedure:
|
||||
|
||||
-# Enable USB debugging on the Android device (via Settings menu).
|
||||
-# Attach the Android device to your PC with a USB cable.
|
||||
-# Go to Start Menu and **right-click** on Computer. Select Manage in the context menu. You may be
|
||||
asked for Administrative permissions.
|
||||
-# Select Device Manager in the left pane and find an unknown device in the list. You may try
|
||||
unplugging it and then plugging back in order to check whether it's your exact equipment appears
|
||||
in the list.
|
||||
|
||||
|
||||
|
||||
-# Try your luck installing Google USB drivers without any modifications: **right-click** on the
|
||||
unknown device, select Properties menu item --\> Details tab --\> Update Driver button.
|
||||
|
||||
|
||||
|
||||
-# Select Browse computer for driver software.
|
||||
|
||||
|
||||
|
||||
-# Specify the path to `<Android SDK folder>/extras/google/usb_driver/` folder.
|
||||
|
||||
|
||||
|
||||
-# If you get the prompt to install unverified drivers and report about success - you've finished
|
||||
with USB driver installation.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
-# Otherwise (getting the failure like shown below) follow the next steps.
|
||||
|
||||
|
||||
|
||||
-# Again **right-click** on the unknown device, select Properties --\> Details --\> Hardware Ids
|
||||
and copy the line like `USB\VID_XXXX&PID_XXXX&MI_XX`.
|
||||
|
||||
|
||||
|
||||
-# Now open file `<Android SDK folder>/extras/google/usb_driver/android_winusb.inf`. Select either
|
||||
Google.NTx86 or Google.NTamd64 section depending on your host system architecture.
|
||||
|
||||
|
||||
|
||||
-# There should be a record like existing ones for your device and you need to add one manually.
|
||||
|
||||
|
||||
|
||||
-# Save the `android_winusb.inf` file and try to install the USB driver again.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
-# This time installation should go successfully.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
-# And an unknown device is now recognized as an Android phone.
|
||||
|
||||
|
||||
|
||||
-# Successful device USB connection can be verified in console via adb devices command.
|
||||
|
||||
|
||||
|
||||
-# Now, in Eclipse go Run -\> Run/Debug to run your application in regular or debugging mode.
|
||||
Device Chooser will let you choose among the devices.
|
||||
|
||||
### Linux host computer
|
||||
|
||||
While the latest Ubuntu versions work well with connected Android devices, there can be issues on older versions.
|
||||
However, most of them can be fixed easily. You have to create a new **/etc/udev/rules.d/51-android.rules** configuration file that contains
|
||||
information about your Android device. You may find some Vendor ID's
|
||||
[here](http://developer.android.com/tools/device.html#VendorIds) or execute lsusb command to view
|
||||
VendorID of plugged Android device. Here is an example of such file for LG device:
|
||||
@code{.guess}
|
||||
SUBSYSTEM=="usb", ATTR{idVendor}=="1004", MODE="0666", GROUP="plugdev"
|
||||
@endcode
|
||||
Then restart your adb server (even better to restart the system), plug in your Android device and
|
||||
execute adb devices command. You will see the list of attached devices:
|
||||
|
||||
```
|
||||
savuor@rostislav-laptop:~/Android/Sdk/platform-tools$ ./adb devices
|
||||
List of devices attached
|
||||
R58MB40Q3VP device
|
||||
|
||||
savuor@rostislav-laptop:~/Android/Sdk/platform-tools$
|
||||
```
|
||||
|
||||
### Mac OS host computer
|
||||
|
||||
No actions are required, just connect your device via USB and run adb devices to check connection.
|
||||
|
||||
What's next
|
||||
-----------
|
||||
|
||||
Now, when you have your instance of OpenCV4Adroid SDK set up and configured, you may want to proceed
|
||||
to using OpenCV in your own application. You can learn how to do that in a separate @ref tutorial_dev_with_OCV_on_Android tutorial.
|
||||
@@ -0,0 +1,87 @@
|
||||
# How to run deep networks on Android device {#tutorial_android_dnn_intro}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dev_with_OCV_on_Android}
|
||||
@next_tutorial{tutorial_android_ocl_intro}
|
||||
|
||||
@see @ref tutorial_table_of_content_dnn
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Dmitry Kurtaev |
|
||||
| Compatibility | OpenCV >= 4.9 |
|
||||
|
||||
@tableofcontents
|
||||
|
||||
## Introduction
|
||||
In this tutorial you'll know how to run deep learning networks on Android device
|
||||
using OpenCV deep learning module.
|
||||
Tutorial was written for Android Studio 2022.2.1.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Download and install Android Studio from https://developer.android.com/studio.
|
||||
|
||||
- Get the latest pre-built OpenCV for Android release from https://github.com/opencv/opencv/releases
|
||||
and unpack it (for example, `opencv-4.X.Y-android-sdk.zip`, minimum version 4.9 is required).
|
||||
|
||||
- Download MobileNet object detection model from https://github.com/chuanqi305/MobileNet-SSD.
|
||||
Configuration file `MobileNetSSD_deploy.prototxt` and model weights `MobileNetSSD_deploy.caffemodel`
|
||||
are required.
|
||||
|
||||
## Create an empty Android Studio project and add OpenCV dependency
|
||||
|
||||
Use @ref tutorial_dev_with_OCV_on_Android tutorial to initialize your project and add OpenCV.
|
||||
|
||||
## Make an app
|
||||
|
||||
Our sample will takes pictures from a camera, forwards it into a deep network and
|
||||
receives a set of rectangles, class identifiers and confidence values in range [0, 1].
|
||||
|
||||
- First of all, we need to add a necessary widget which displays processed
|
||||
frames. Modify `app/src/main/res/layout/activity_main.xml`:
|
||||
@include android/mobilenet-objdetect/res/layout/activity_main.xml
|
||||
|
||||
- Modify `/app/src/main/AndroidManifest.xml` to enable full-screen mode, set up
|
||||
a correct screen orientation and allow to use a camera.
|
||||
@code{.xml}
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
|
||||
|
||||
<application
|
||||
android:label="@string/app_name">
|
||||
@endcode
|
||||
@snippet android/mobilenet-objdetect/gradle/AndroidManifest.xml mobilenet_tutorial
|
||||
|
||||
- Replace content of `app/src/main/java/com/example/myapplication/MainActivity.java` and set a custom package name if necessary:
|
||||
|
||||
@snippet android/mobilenet-objdetect/src/org/opencv/samples/opencv_mobilenet/MainActivity.java mobilenet_tutorial_package
|
||||
@snippet android/mobilenet-objdetect/src/org/opencv/samples/opencv_mobilenet/MainActivity.java mobilenet_tutorial
|
||||
|
||||
- Put downloaded `deploy.prototxt` and `mobilenet_iter_73000.caffemodel`
|
||||
into `app/src/main/res/raw` folder. OpenCV DNN model is mainly designed to load ML and DNN models
|
||||
from file. Modern Android does not allow it without extra permissions, but provides Java API to load
|
||||
bytes from resources. The sample uses alternative DNN API that initializes a model from in-memory
|
||||
buffer rather than a file. The following function reads model file from resources and converts it to
|
||||
`MatOfBytes` (analog of `std::vector<char>` in C++ world) object suitable for OpenCV Java API:
|
||||
|
||||
@snippet android/mobilenet-objdetect/src/org/opencv/samples/opencv_mobilenet/MainActivity.java mobilenet_tutorial_resource
|
||||
|
||||
And then the network initialization is done with the following lines:
|
||||
|
||||
@snippet android/mobilenet-objdetect/src/org/opencv/samples/opencv_mobilenet/MainActivity.java init_model_from_memory
|
||||
|
||||
See also [Android documentation on resources](https://developer.android.com/guide/topics/resources/providing-resources.html)
|
||||
|
||||
- Take a look how DNN model input is prepared and inference result is interpreted:
|
||||
|
||||
@snippet android/mobilenet-objdetect/src/org/opencv/samples/opencv_mobilenet/MainActivity.java mobilenet_handle_frame
|
||||
|
||||
`Dnn.blobFromImage` converts camera frame to neural network input tensor. Resize and statistical
|
||||
normalization are applied. Each line of network output tensor contains information on one detected
|
||||
object in the following order: confidence in range [0, 1], class id, left, top, right, bottom box
|
||||
coordinates. All coordinates are in range [0, 1] and should be scaled to image size before rendering.
|
||||
|
||||
- Launch an application and make a fun!
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
Use OpenCL in Android camera preview based CV application {#tutorial_android_ocl_intro}
|
||||
=====================================
|
||||
|
||||
@prev_tutorial{tutorial_android_dnn_intro}
|
||||
@next_tutorial{tutorial_macos_install}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Andrey Pavlenko, Alexander Panov |
|
||||
| Compatibility | OpenCV >= 4.9 |
|
||||
|
||||
@tableofcontents
|
||||
|
||||
This guide was designed to help you in use of [OpenCL ™](https://www.khronos.org/opencl/) in Android camera preview based CV application.
|
||||
Tutorial was written for [Android Studio](http://developer.android.com/tools/studio/index.html) 2022.2.1. It was tested with Ubuntu 22.04.
|
||||
|
||||
This tutorial assumes you have the following installed and configured:
|
||||
|
||||
- Android Studio (2022.2.1.+)
|
||||
- JDK 17
|
||||
- Android SDK
|
||||
- Android NDK (25.2.9519653+)
|
||||
- download OpenCV source code from [github](git@github.com:opencv/opencv.git) or from [releases](https://opencv.org/releases/) and build by [instruction on wiki](https://github.com/opencv/opencv/wiki/Custom-OpenCV-Android-SDK-and-AAR-package-build).
|
||||
|
||||
It also assumes that you are familiar with Android Java and JNI programming basics.
|
||||
If you need help with anything of the above, you may refer to our @ref tutorial_android_dev_intro guide.
|
||||
|
||||
This tutorial also assumes you have an Android operated device with OpenCL enabled.
|
||||
|
||||
The related source code is located within OpenCV samples at
|
||||
[opencv/samples/android/tutorial-4-opencl](https://github.com/opencv/opencv/tree/4.x/samples/android/tutorial-4-opencl/) directory.
|
||||
|
||||
How to build custom OpenCV Android SDK with OpenCL
|
||||
--------------------------------------------------
|
||||
|
||||
1. __Assemble and configure Android OpenCL SDK.__
|
||||
The JNI part of the sample depends on standard Khornos OpenCL headers, and C++ wrapper for OpenCL and libOpenCL.so.
|
||||
The standard OpenCL headers may be copied from 3rdparty directory in OpenCV repository or you Linux distribution package.
|
||||
C++ wrapper is available in [official Khronos reposiotry on Github](https://github.com/KhronosGroup/OpenCL-CLHPP).
|
||||
Copy the header files to didicated directory in the following way:
|
||||
@code{.bash}
|
||||
cd your_path/ && mkdir ANDROID_OPENCL_SDK && mkdir ANDROID_OPENCL_SDK/include && cd ANDROID_OPENCL_SDK/include
|
||||
cp -r path_to_opencv/opencv/3rdparty/include/opencl/1.2/CL . && cd CL
|
||||
wget https://github.com/KhronosGroup/OpenCL-CLHPP/raw/main/include/CL/opencl.hpp
|
||||
wget https://github.com/KhronosGroup/OpenCL-CLHPP/raw/main/include/CL/cl2.hpp
|
||||
@endcode
|
||||
libOpenCL.so may be provided with BSP or just downloaded from any OpenCL-cabaple Android device with relevant arhitecture.
|
||||
@code{.bash}
|
||||
cd your_path/ANDROID_OPENCL_SDK && mkdir lib && cd lib
|
||||
adb pull /system/vendor/lib64/libOpenCL.so
|
||||
@endcode
|
||||
System verison of libOpenCL.so may have a lot of platform specific dependencies. `-Wl,--allow-shlib-undefined` flag allows
|
||||
to ignore 3rdparty symbols if they are not used during the build.
|
||||
The following CMake line allows to link the JNI part against standard OpenCL, but not include the loadLibrary into
|
||||
application package. System OpenCL API is used in run-time.
|
||||
@code
|
||||
target_link_libraries(${target} -lOpenCL)
|
||||
@endcode
|
||||
|
||||
|
||||
2. __Build custom OpenCV Android SDK with OpenCL.__
|
||||
OpenCL support (T-API) is disabled in OpenCV builds for Android OS by default.
|
||||
but it's possible to rebuild locally OpenCV for Android with OpenCL/T-API enabled: use `-DWITH_OPENCL=ON` option for CMake.
|
||||
You also need to specify the path to the Android OpenCL SDK: use `-DANDROID_OPENCL_SDK=path_to_your_Android_OpenCL_SDK` option for CMake.
|
||||
If you are building OpenCV using `build_sdk.py` please follow [instruction on wiki](https://github.com/opencv/opencv/wiki/Custom-OpenCV-Android-SDK-and-AAR-package-build).
|
||||
Set these CMake parameters in your `.config.py`, e.g. `ndk-18-api-level-21.config.py`:
|
||||
@code{.py}
|
||||
ABI("3", "arm64-v8a", None, 21, cmake_vars=dict('WITH_OPENCL': 'ON', 'ANDROID_OPENCL_SDK': 'path_to_your_Android_OpenCL_SDK'))
|
||||
@endcode
|
||||
If you are building OpenCV using cmake/ninja, use this bash script (set your NDK_VERSION and your paths instead of examples of paths):
|
||||
@code{.bash}
|
||||
cd path_to_opencv && mkdir build && cd build
|
||||
export NDK_VERSION=25.2.9519653
|
||||
export ANDROID_SDK=/home/user/Android/Sdk/
|
||||
export ANDROID_OPENCL_SDK=/path_to_ANDROID_OPENCL_SDK/
|
||||
export ANDROID_HOME=$ANDROID_SDK
|
||||
export ANDROID_NDK_HOME=$ANDROID_SDK/ndk/$NDK_VERSION/
|
||||
cmake -GNinja -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake -DANDROID_STL=c++_shared -DANDROID_NATIVE_API_LEVEL=24
|
||||
-DANDROID_SDK=$ANDROID_SDK -DANDROID_NDK=$ANDROID_NDK_HOME -DBUILD_JAVA=ON -DANDROID_HOME=$ANDROID_SDK -DBUILD_ANDROID_EXAMPLES=ON
|
||||
-DINSTALL_ANDROID_EXAMPLES=ON -DANDROID_ABI=arm64-v8a -DWITH_OPENCL=ON -DANDROID_OPENCL_SDK=$ANDROID_OPENCL_SDK ..
|
||||
@endcode
|
||||
|
||||
Preface
|
||||
-------
|
||||
|
||||
Using [GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units)
|
||||
via OpenCL for applications performance enhancements is quite a modern trend now.
|
||||
Some CV algo-s (e.g. image filtering) run much faster on a GPU than on a CPU.
|
||||
Recently it has become possible on Android OS.
|
||||
|
||||
The most popular CV application scenario for an Android operated device is starting camera in preview mode, applying some CV algo to every frame
|
||||
and displaying the preview frames modified by that CV algo.
|
||||
|
||||
Let's consider how we can use OpenCL in this scenario. In particular let's try two ways: direct calls to OpenCL API and recently introduced OpenCV T-API
|
||||
(aka [Transparent API](https://docs.google.com/presentation/d/1qoa29N_B-s297-fp0-b3rBirvpzJQp8dCtllLQ4DVCY/present)) - implicit OpenCL accelerations of some OpenCV algo-s.
|
||||
|
||||
Application structure
|
||||
---------------------
|
||||
|
||||
Starting Android API level 11 (Android 3.0) [Camera API](http://developer.android.com/reference/android/hardware/Camera.html)
|
||||
allows use of OpenGL texture as a target for preview frames.
|
||||
Android API level 21 brings a new [Camera2 API](http://developer.android.com/reference/android/hardware/camera2/package-summary.html)
|
||||
that provides much more control over the camera settings and usage modes,
|
||||
it allows several targets for preview frames and OpenGL texture in particular.
|
||||
|
||||
Having a preview frame in an OpenGL texture is a good deal for using OpenCL because there is an
|
||||
[OpenGL-OpenCL Interoperability API (cl_khr_gl_sharing)](https://www.khronos.org/registry/cl/sdk/1.2/docs/man/xhtml/cl_khr_gl_sharing.html),
|
||||
allowing sharing OpenGL texture data with OpenCL functions without copying (with some restrictions of course).
|
||||
|
||||
Let's create a base for our application that just configures Android camera to send preview frames to OpenGL texture and displays these frames
|
||||
on display without any processing.
|
||||
|
||||
A minimal `Activity` class for that purposes looks like following:
|
||||
|
||||
@code{.java}
|
||||
public class Tutorial4Activity extends Activity {
|
||||
|
||||
private MyGLSurfaceView mView;
|
||||
|
||||
@Override
|
||||
public void onCreate(Bundle savedInstanceState) {
|
||||
super.onCreate(savedInstanceState);
|
||||
requestWindowFeature(Window.FEATURE_NO_TITLE);
|
||||
getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,
|
||||
WindowManager.LayoutParams.FLAG_FULLSCREEN);
|
||||
getWindow().setFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON,
|
||||
WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
|
||||
setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE);
|
||||
|
||||
mView = new MyGLSurfaceView(this);
|
||||
setContentView(mView);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void onPause() {
|
||||
mView.onPause();
|
||||
super.onPause();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void onResume() {
|
||||
super.onResume();
|
||||
mView.onResume();
|
||||
}
|
||||
}
|
||||
@endcode
|
||||
|
||||
And a minimal `View` class respectively:
|
||||
|
||||
@snippet samples/android/tutorial-4-opencl/src/org/opencv/samples/tutorial4/MyGLSurfaceView.java minimal_surface_view
|
||||
|
||||
@note we use two renderer classes: one for legacy [Camera](http://developer.android.com/reference/android/hardware/Camera.html) API
|
||||
and another for modern [Camera2](http://developer.android.com/reference/android/hardware/camera2/package-summary.html).
|
||||
|
||||
A minimal `Renderer` class can be implemented in Java (OpenGL ES 2.0 [available](http://developer.android.com/reference/android/opengl/GLES20.html) in Java),
|
||||
but since we are going to modify the preview texture with OpenCL let's move OpenGL stuff to JNI.
|
||||
Here is a simple Java wrapper for our JNI stuff:
|
||||
|
||||
@snippet samples/android/tutorial-4-opencl/src/org/opencv/samples/tutorial4/NativePart.java native_part
|
||||
|
||||
Since `Camera` and `Camera2` APIs differ significantly in camera setup and control, let's create a base class for the two corresponding renderers:
|
||||
|
||||
@code{.java}
|
||||
public abstract class MyGLRendererBase implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener {
|
||||
protected final String LOGTAG = "MyGLRendererBase";
|
||||
|
||||
protected SurfaceTexture mSTex;
|
||||
protected MyGLSurfaceView mView;
|
||||
|
||||
protected boolean mGLInit = false;
|
||||
protected boolean mTexUpdate = false;
|
||||
|
||||
MyGLRendererBase(MyGLSurfaceView view) {
|
||||
mView = view;
|
||||
}
|
||||
|
||||
protected abstract void openCamera();
|
||||
protected abstract void closeCamera();
|
||||
protected abstract void setCameraPreviewSize(int width, int height);
|
||||
|
||||
public void onResume() {
|
||||
Log.i(LOGTAG, "onResume");
|
||||
}
|
||||
|
||||
public void onPause() {
|
||||
Log.i(LOGTAG, "onPause");
|
||||
mGLInit = false;
|
||||
mTexUpdate = false;
|
||||
closeCamera();
|
||||
if(mSTex != null) {
|
||||
mSTex.release();
|
||||
mSTex = null;
|
||||
NativeGLRenderer.closeGL();
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void onFrameAvailable(SurfaceTexture surfaceTexture) {
|
||||
//Log.i(LOGTAG, "onFrameAvailable");
|
||||
mTexUpdate = true;
|
||||
mView.requestRender();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onDrawFrame(GL10 gl) {
|
||||
//Log.i(LOGTAG, "onDrawFrame");
|
||||
if (!mGLInit)
|
||||
return;
|
||||
|
||||
synchronized (this) {
|
||||
if (mTexUpdate) {
|
||||
mSTex.updateTexImage();
|
||||
mTexUpdate = false;
|
||||
}
|
||||
}
|
||||
NativeGLRenderer.drawFrame();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onSurfaceChanged(GL10 gl, int surfaceWidth, int surfaceHeight) {
|
||||
Log.i(LOGTAG, "onSurfaceChanged("+surfaceWidth+"x"+surfaceHeight+")");
|
||||
NativeGLRenderer.changeSize(surfaceWidth, surfaceHeight);
|
||||
setCameraPreviewSize(surfaceWidth, surfaceHeight);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
|
||||
Log.i(LOGTAG, "onSurfaceCreated");
|
||||
String strGLVersion = GLES20.glGetString(GLES20.GL_VERSION);
|
||||
if (strGLVersion != null)
|
||||
Log.i(LOGTAG, "OpenGL ES version: " + strGLVersion);
|
||||
|
||||
int hTex = NativeGLRenderer.initGL();
|
||||
mSTex = new SurfaceTexture(hTex);
|
||||
mSTex.setOnFrameAvailableListener(this);
|
||||
openCamera();
|
||||
mGLInit = true;
|
||||
}
|
||||
}
|
||||
@endcode
|
||||
|
||||
As you can see, inheritors for `Camera` and `Camera2` APIs should implement the following abstract methods:
|
||||
@code{.java}
|
||||
protected abstract void openCamera();
|
||||
protected abstract void closeCamera();
|
||||
protected abstract void setCameraPreviewSize(int width, int height);
|
||||
@endcode
|
||||
|
||||
Let's leave the details of their implementation beyond of this tutorial, please refer the
|
||||
[source code](https://github.com/opencv/opencv/tree/4.x/samples/android/tutorial-4-opencl/) to see them.
|
||||
|
||||
Preview Frames modification
|
||||
---------------------------
|
||||
|
||||
The details OpenGL ES 2.0 initialization are also quite straightforward and noisy to be quoted here,
|
||||
but the important point here is that the OpeGL texture to be the target for camera preview should be of type `GL_TEXTURE_EXTERNAL_OES`
|
||||
(not `GL_TEXTURE_2D`), internally it keeps picture data in _YUV_ format.
|
||||
That makes unable sharing it via CL-GL interop (`cl_khr_gl_sharing`) and accessing its pixel data via C/C++ code.
|
||||
To overcome this restriction we have to perform an OpenGL rendering from this texture to another regular `GL_TEXTURE_2D` one
|
||||
using _FrameBuffer Object_ (aka FBO).
|
||||
|
||||
### C/C++ code
|
||||
|
||||
After that we can read (_copy_) pixel data from C/C++ via `glReadPixels()` and write them back to texture after modification via `glTexSubImage2D()`.
|
||||
|
||||
### Direct OpenCL calls
|
||||
|
||||
Also that `GL_TEXTURE_2D` texture can be shared with OpenCL without copying, but we have to create OpenCL context with special way for that:
|
||||
|
||||
@snippet samples/android/tutorial-4-opencl/jni/CLprocessor.cpp init_opencl
|
||||
|
||||
Then the texture can be wrapped by a `cl::ImageGL` object and processed via OpenCL calls:
|
||||
|
||||
@snippet samples/android/tutorial-4-opencl/jni/CLprocessor.cpp process_pure_opencl
|
||||
|
||||
### OpenCV T-API
|
||||
|
||||
But instead of writing OpenCL code by yourselves you may want to use __OpenCV T-API__ that calls OpenCL implicitly.
|
||||
All that you need is to pass the created OpenCL context to OpenCV (via `cv::ocl::attachContext()`) and somehow wrap OpenGL texture with `cv::UMat`.
|
||||
Unfortunately `UMat` keeps OpenCL _buffer_ internally, that can't be wrapped over either OpenGL _texture_ or OpenCL _image_ - so we have to copy image data here:
|
||||
|
||||
@snippet samples/android/tutorial-4-opencl/jni/CLprocessor.cpp process_tapi
|
||||
|
||||
@note We have to make one more image data copy when placing back the modified image to the original OpenGL texture via OpenCL image wrapper.
|
||||
|
||||
Performance notes
|
||||
-----------------
|
||||
|
||||
To compare the performance we measured FPS of the same preview frames modification (_Laplacian_) done by C/C++ code (call to `cv::Laplacian` with `cv::Mat`),
|
||||
by direct OpenCL calls (using OpenCL _images_ for input and output), and by OpenCV _T-API_ (call to `cv::Laplacian` with `cv::UMat`) on _Sony Xperia Z3_ with 720p camera resolution:
|
||||
* __C/C++ version__ shows __3-4 fps__
|
||||
* __direct OpenCL calls__ shows __25-27 fps__
|
||||
* __OpenCV T-API__ shows __11-13 fps__ (due to extra copying from `cl_image` to `cl_buffer` and back)
|
||||
@@ -0,0 +1,236 @@
|
||||
Android Development with OpenCV {#tutorial_dev_with_OCV_on_Android}
|
||||
===============================
|
||||
|
||||
@prev_tutorial{tutorial_android_dev_intro}
|
||||
@next_tutorial{tutorial_android_dnn_intro}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original authors | Alexander Panov, Rostislav Vasilikhin |
|
||||
| Compatibility | OpenCV >= 4.9.0 |
|
||||
|
||||
@tableofcontents
|
||||
|
||||
This tutorial has been created to help you use OpenCV library within your Android project.
|
||||
|
||||
This guide was checked on Ubuntu but contains no platform-dependent parts, therefore should be compatible with any OS supported by Android Studio and OpenCV4Android SDK.
|
||||
|
||||
This tutorial assumes you have the following installed and configured:
|
||||
|
||||
- Android Studio
|
||||
- JDK
|
||||
- Android SDK and NDK
|
||||
- Optional: OpenCV for Android SDK from official [release page on Github](https://github.com/opencv/opencv/releases)
|
||||
or [SourceForge](https://sourceforge.net/projects/opencvlibrary/). Advanced: as alternative the SDK may be
|
||||
built from source code by [instruction on wiki](https://github.com/opencv/opencv/wiki/Custom-OpenCV-Android-SDK-and-AAR-package-build).
|
||||
|
||||
If you need help with anything of the above, you may refer to our @ref tutorial_android_dev_intro guide.
|
||||
|
||||
If you encounter any error after thoroughly following these steps, feel free to contact us via OpenCV [forum](https://forum.opencv.org). We'll do our best to help you out.
|
||||
|
||||
|
||||
Hello OpenCV sample with SDK
|
||||
----------------------------
|
||||
|
||||
In this section we're gonna create a simple app that does nothing but OpenCV loading. In next section we'll extend it to support camera.
|
||||
|
||||
In addition to this instruction you can use some video guide, for example [this one](https://www.youtube.com/watch?v=bR7lL886-uc&ab_channel=ProgrammingHut)
|
||||
|
||||
1. Open Android Studio and create empty project by choosing ***Empty Views Activity***
|
||||
|
||||
|
||||
|
||||
2. Setup the project:
|
||||
- Choose ***Java*** language
|
||||
- Choose ***Groovy DSL*** build configuration language
|
||||
- Choose ***Minumum SDK*** with the version number not less than was used during OpenCV 4 Android build
|
||||
- If you don't know it, you can find it in file `OpenCV-android-sdk/sdk/build.gradle` at `android -> defaultConfig -> minSdkVersion`
|
||||
|
||||
|
||||
|
||||
|
||||
3. Click ***File -> New -> Import module...*** and select OpenCV SDK path
|
||||
|
||||
|
||||
|
||||
4. Set module name as `OpenCV` and press `Finish`
|
||||
|
||||
|
||||
|
||||
5. OpenCV also provides experiemental Kotlin support. Please add Android Kotlin plugin to `MyApplication/OpenCV/build.gradle` file:
|
||||
@code{.gradle}
|
||||
plugins {
|
||||
id 'org.jetbrains.kotlin.android' version '1.7.10' #version may differ for your setup
|
||||
}
|
||||
@endcode
|
||||
Like this:
|
||||
|
||||
If you don't do this, you may get an error:
|
||||
@code
|
||||
Task failed with an exception.
|
||||
-----------
|
||||
* Where:
|
||||
Build file '/home/alexander/AndroidStudioProjects/MyApplication/opencv/build.gradle' line: 4
|
||||
|
||||
* What went wrong:
|
||||
A problem occurred evaluating project ':opencv'.
|
||||
> Plugin with id 'kotlin-android' not found.
|
||||
@endcode
|
||||
The fix was found [here](https://stackoverflow.com/questions/73225714/import-opencv-sdk-to-android-studio-chipmunk)
|
||||
|
||||
6. OpenCV project uses `buildConfig` feature. Please enable it in
|
||||
`MyApplication/OpenCV/build.gradle` file to `android` block:
|
||||
@code{.gradle}
|
||||
buildFeatures{
|
||||
buildConfig true
|
||||
}
|
||||
|
||||
@endcode
|
||||
Like this:
|
||||
|
||||
If you don't do this, you may get an error:
|
||||
@code
|
||||
JavaCameraView.java:15: error: cannot find symbol import org.opencv.BuildConfig; ^ symbol: class BuildConfig location: package org.opencv
|
||||
@endcode
|
||||
The fix was found [here](https://stackoverflow.com/questions/76374886/error-cannot-find-symbol-import-org-opencv-buildconfig-android-studio) and [here](https://forum.opencv.org/t/task-compiledebugjavawithjavac-failed/13667/4)
|
||||
|
||||
7. Add the module to the project:
|
||||
- Click ***File -> Project structure... -> Dependencies -> All modules -> + (Add Dependency button) -> Module dependency***
|
||||
|
||||
|
||||
|
||||
- Choose `app`
|
||||
|
||||
|
||||
|
||||
- Select `OpenCV`
|
||||
|
||||
|
||||
|
||||
8. Before using any OpenCV function you have to load the library first. If you application includes other OpenCV-dependent native libraries you should load them ***after*** OpenCV initialization.
|
||||
Add the folowing code to load the library at app start:
|
||||
@snippet samples/android/tutorial-1-camerapreview/src/org/opencv/samples/tutorial1/Tutorial1Activity.java ocv_loader_init
|
||||
Like this:
|
||||
|
||||
|
||||
9. Choose a device to check the sample on and run the code by pressing `run` button
|
||||
|
||||
|
||||
|
||||
Hello OpenCV sample with Maven Central
|
||||
--------------------------------------
|
||||
|
||||
Since OpenCV 4.9.0 OpenCV for Android package is available with Maven Central and may be installed
|
||||
automatically as Gradle dependency. In this section we're gonna create a simple app that does nothing
|
||||
but OpenCV loading with Maven Central.
|
||||
|
||||
1. Open Android Studio and create empty project by choosing ***Empty Views Activity***
|
||||
|
||||
|
||||
|
||||
2. Setup the project:
|
||||
- Choose ***Java*** language
|
||||
- Choose ***Groovy DSL*** build configuration language
|
||||
- Choose ***Minumum SDK*** with the version number not less than OpenCV supports. For 4.9.0 minimal SDK version is 21.
|
||||
|
||||
|
||||
|
||||
3. Edit `build.gradle` and add OpenCV library to Dependencies list like this:
|
||||
@code{.gradle}
|
||||
dependencies {
|
||||
implementation 'org.opencv:opencv:4.9.0'
|
||||
}
|
||||
@endcode
|
||||
`4.9.0` may be replaced by any version available as [official release](https://central.sonatype.com/artifact/org.opencv/opencv).
|
||||
|
||||
4. Before using any OpenCV function you have to load the library first. If you application includes other
|
||||
OpenCV-dependent native libraries you should load them ***after*** OpenCV initialization. Add the folowing
|
||||
code to load the library at app start:
|
||||
@snippet samples/android/tutorial-1-camerapreview/src/org/opencv/samples/tutorial1/Tutorial1Activity.java ocv_loader_init
|
||||
Like this:
|
||||
|
||||
|
||||
5. Choose a device to check the sample on and run the code by pressing `run` button
|
||||
|
||||
|
||||
|
||||
Camera view sample
|
||||
------------------
|
||||
|
||||
In this section we'll extend our empty OpenCV app created in the previous section to support camera. We'll take camera frames and display them on the screen.
|
||||
|
||||
1. Tell a system that we need camera permissions.
|
||||
Add the following code to the file `MyApplication/app/src/main/AndroidManifest.xml`:
|
||||
@snippet samples/android/tutorial-1-camerapreview/gradle/AndroidManifest.xml camera_permissions
|
||||
Like this:
|
||||
|
||||
|
||||
2. Go to `activity_main.xml` layout and delete TextView with text "Hello World!"
|
||||
|
||||
|
||||
|
||||
This can also be done in Code or Split mode by removing the `TextView` block from XML file.
|
||||
|
||||
3. Add camera view to the layout:
|
||||
1. Add a scheme into layout description:
|
||||
@code{.xml}
|
||||
xmlns:opencv="http://schemas.android.com/apk/res-auto"
|
||||
@endcode
|
||||
|
||||
2. Replace `TextView` with `org.opencv.android.JavaCameraView` widget:
|
||||
@snippet /samples/android/tutorial-1-camerapreview/res/layout/tutorial1_surface_view.xml camera_view
|
||||
|
||||
3. If you get a layout warning replace `fill_parent` values by `match_parent` for `android:layout_width` and `android:layout_height` properties
|
||||
|
||||
You'll get a code like this:
|
||||
|
||||
@include /samples/android/tutorial-1-camerapreview/res/layout/tutorial1_surface_view.xml
|
||||
|
||||
4. Inherit the main class from `org.opencv.android.CameraActivity`. CameraActivity implements
|
||||
camera perimission requiest and some other utilities needed for CV application. Methods we're
|
||||
interested in to override are `onCreate`, `onDestroy`, `onPause`, `onResume` and `getCameraViewList`
|
||||
|
||||
5. Implement the interface `org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2`
|
||||
`onCameraFrame` method should return the `Mat` object with content for render.
|
||||
The sample just returns camera frame for preview: `return inputFrame.rgba();`
|
||||
|
||||
6. Allocate `org.opencv.android.CameraBridgeViewBase` object:
|
||||
- It should be created at app start (`onCreate` method) and this class should be set as a listener
|
||||
- At pause/resume (`onPause`, `onResume` methods) it should be disabled/enabled
|
||||
- Should be disabled at app finish (`onDestroy` method)
|
||||
- Should be returned in `getCameraViewList`
|
||||
|
||||
7. Optionally you can forbid the phone to dim screen or lock:
|
||||
|
||||
@snippet samples/android/tutorial-1-camerapreview/src/org/opencv/samples/tutorial1/Tutorial1Activity.java keep_screen
|
||||
|
||||
Finally you'll get source code similar to this:
|
||||
|
||||
@include samples/android/tutorial-1-camerapreview/src/org/opencv/samples/tutorial1/Tutorial1Activity.java
|
||||
|
||||
This is it! Now you can run the code on your device to check it.
|
||||
|
||||
|
||||
Let's discuss some most important steps
|
||||
---------------------------------------
|
||||
|
||||
Every Android application with UI must implement Activity and View. By the first steps we create blank
|
||||
activity and default view layout. The simplest OpenCV-centric application must perform OpenCV
|
||||
initialization, create a view to show preview from camera and implement `CvCameraViewListener2` interface
|
||||
to get frames from camera and process them.
|
||||
|
||||
First of all we create our application view using XML layout. Our layout consists of the only one
|
||||
full screen component of class `org.opencv.android.JavaCameraView`. This OpenCV class is inherited from
|
||||
`CameraBridgeViewBase` that extends `SurfaceView` and under the hood uses standard Android camera API.
|
||||
|
||||
The `CvCameraViewListener2` interface lets you add some processing steps after the frame is grabbed from
|
||||
the camera and before it's rendered on the screen. The most important method is `onCameraFrame`. This is
|
||||
a callback function and it's called on retrieving frame from camera. It expects that `onCameraFrame`
|
||||
function returns RGBA frame that will be drawn on the screen.
|
||||
|
||||
The callback passes a frame from camera to our class as an object of `CvCameraViewFrame` class.
|
||||
This object has `rgba()` and `gray()` methods that let a user get colored or one-channel grayscale
|
||||
frame as a `Mat` class object.
|
||||
|
||||
@note Do not save or use `CvCameraViewFrame` object out of `onCameraFrame` callback. This object does
|
||||
not have its own state and its behavior outside the callback is unpredictable!
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 118 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 243 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 181 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 235 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 106 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 205 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 300 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 232 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 103 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 141 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 131 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 217 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 294 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.7 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 8.8 KiB |
Reference in New Issue
Block a user