chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,123 @@
|
||||
Adding (blending) two images using OpenCV {#tutorial_adding_images}
|
||||
=========================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_mat_operations}
|
||||
@next_tutorial{tutorial_basic_linear_transform}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
We will learn how to blend two images!
|
||||
Goal
|
||||
----
|
||||
|
||||
In this tutorial you will learn:
|
||||
|
||||
- what is *linear blending* and why it is useful;
|
||||
- how to add two images using **addWeighted()**
|
||||
|
||||
Theory
|
||||
------
|
||||
|
||||
@note
|
||||
The explanation below belongs to the book [Computer Vision: Algorithms and
|
||||
Applications](https://szeliski.org/Book/) by Richard Szeliski
|
||||
|
||||
From our previous tutorial, we already know a bit of *Pixel operators*. An interesting dyadic
|
||||
(two-input) operator is the *linear blend operator*:
|
||||
|
||||
\f[g(x) = (1 - \alpha)f_{0}(x) + \alpha f_{1}(x)\f]
|
||||
|
||||
By varying \f$\alpha\f$ from \f$0 \rightarrow 1\f$, this operator can be used to perform a temporal
|
||||
*cross-dissolve* between two images or videos, as seen in slide shows and film productions (cool,
|
||||
eh?)
|
||||
|
||||
Source Code
|
||||
-----------
|
||||
|
||||
@add_toggle_cpp
|
||||
Download the source code from
|
||||
[here](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/cpp/tutorial_code/core/AddingImages/AddingImages.cpp).
|
||||
@include cpp/tutorial_code/core/AddingImages/AddingImages.cpp
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
Download the source code from
|
||||
[here](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/java/tutorial_code/core/AddingImages/AddingImages.java).
|
||||
@include java/tutorial_code/core/AddingImages/AddingImages.java
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
Download the source code from
|
||||
[here](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/python/tutorial_code/core/AddingImages/adding_images.py).
|
||||
@include python/tutorial_code/core/AddingImages/adding_images.py
|
||||
@end_toggle
|
||||
|
||||
Explanation
|
||||
-----------
|
||||
|
||||
Since we are going to perform:
|
||||
|
||||
\f[g(x) = (1 - \alpha)f_{0}(x) + \alpha f_{1}(x)\f]
|
||||
|
||||
We need two source images (\f$f_{0}(x)\f$ and \f$f_{1}(x)\f$). So, we load them in the usual way:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/AddingImages/AddingImages.cpp load
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/AddingImages/AddingImages.java load
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/AddingImages/adding_images.py load
|
||||
@end_toggle
|
||||
|
||||
We used the following images: [LinuxLogo.jpg](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/data/LinuxLogo.jpg) and [WindowsLogo.jpg](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/data/WindowsLogo.jpg)
|
||||
|
||||
@warning Since we are *adding* *src1* and *src2*, they both have to be of the same size
|
||||
(width and height) and type.
|
||||
|
||||
Now we need to generate the `g(x)` image. For this, the function **addWeighted()** comes quite handy:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/AddingImages/AddingImages.cpp blend_images
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/AddingImages/AddingImages.java blend_images
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/AddingImages/adding_images.py blend_images
|
||||
Numpy version of above line (but cv function is around 2x faster):
|
||||
\code{.py}
|
||||
dst = np.uint8(alpha*(img1)+beta*(img2))
|
||||
\endcode
|
||||
@end_toggle
|
||||
|
||||
since **addWeighted()** produces:
|
||||
\f[dst = \alpha \cdot src1 + \beta \cdot src2 + \gamma\f]
|
||||
In this case, `gamma` is the argument \f$0.0\f$ in the code above.
|
||||
|
||||
Create windows, show the images and wait for the user to end the program.
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/AddingImages/AddingImages.cpp display
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/AddingImages/AddingImages.java display
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/AddingImages/adding_images.py display
|
||||
@end_toggle
|
||||
|
||||
Result
|
||||
------
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 6.4 KiB |
@@ -0,0 +1,325 @@
|
||||
Changing the contrast and brightness of an image! {#tutorial_basic_linear_transform}
|
||||
=================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_adding_images}
|
||||
@next_tutorial{tutorial_discrete_fourier_transform}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
In this tutorial you will learn how to:
|
||||
|
||||
- Access pixel values
|
||||
- Initialize a matrix with zeros
|
||||
- Learn what @ref cv::saturate_cast does and why it is useful
|
||||
- Get some cool info about pixel transformations
|
||||
- Improve the brightness of an image on a practical example
|
||||
|
||||
Theory
|
||||
------
|
||||
|
||||
@note
|
||||
The explanation below belongs to the book [Computer Vision: Algorithms and
|
||||
Applications](https://szeliski.org/Book/) by Richard Szeliski
|
||||
|
||||
### Image Processing
|
||||
|
||||
- A general image processing operator is a function that takes one or more input images and
|
||||
produces an output image.
|
||||
- Image transforms can be seen as:
|
||||
- Point operators (pixel transforms)
|
||||
- Neighborhood (area-based) operators
|
||||
|
||||
### Pixel Transforms
|
||||
|
||||
- In this kind of image processing transform, each output pixel's value depends on only the
|
||||
corresponding input pixel value (plus, potentially, some globally collected information or
|
||||
parameters).
|
||||
- Examples of such operators include *brightness and contrast adjustments* as well as color
|
||||
correction and transformations.
|
||||
|
||||
### Brightness and contrast adjustments
|
||||
|
||||
- Two commonly used point processes are *multiplication* and *addition* with a constant:
|
||||
|
||||
\f[g(x) = \alpha f(x) + \beta\f]
|
||||
|
||||
- The parameters \f$\alpha > 0\f$ and \f$\beta\f$ are often called the *gain* and *bias* parameters;
|
||||
sometimes these parameters are said to control *contrast* and *brightness* respectively.
|
||||
- You can think of \f$f(x)\f$ as the source image pixels and \f$g(x)\f$ as the output image pixels. Then,
|
||||
more conveniently we can write the expression as:
|
||||
|
||||
\f[g(i,j) = \alpha \cdot f(i,j) + \beta\f]
|
||||
|
||||
where \f$i\f$ and \f$j\f$ indicates that the pixel is located in the *i-th* row and *j-th* column.
|
||||
|
||||
Code
|
||||
----
|
||||
|
||||
@add_toggle_cpp
|
||||
- **Downloadable code**: Click
|
||||
[here](https://github.com/opencv/opencv/tree/4.x/samples/cpp/tutorial_code/ImgProc/BasicLinearTransforms.cpp)
|
||||
|
||||
- The following code performs the operation \f$g(i,j) = \alpha \cdot f(i,j) + \beta\f$ :
|
||||
@include samples/cpp/tutorial_code/ImgProc/BasicLinearTransforms.cpp
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
- **Downloadable code**: Click
|
||||
[here](https://github.com/opencv/opencv/tree/4.x/samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/BasicLinearTransformsDemo.java)
|
||||
|
||||
- The following code performs the operation \f$g(i,j) = \alpha \cdot f(i,j) + \beta\f$ :
|
||||
@include samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/BasicLinearTransformsDemo.java
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
- **Downloadable code**: Click
|
||||
[here](https://github.com/opencv/opencv/tree/4.x/samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/BasicLinearTransforms.py)
|
||||
|
||||
- The following code performs the operation \f$g(i,j) = \alpha \cdot f(i,j) + \beta\f$ :
|
||||
@include samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/BasicLinearTransforms.py
|
||||
@end_toggle
|
||||
|
||||
Explanation
|
||||
-----------
|
||||
|
||||
- We load an image using @ref cv::imread and save it in a Mat object:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/ImgProc/BasicLinearTransforms.cpp basic-linear-transform-load
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/BasicLinearTransformsDemo.java basic-linear-transform-load
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/BasicLinearTransforms.py basic-linear-transform-load
|
||||
@end_toggle
|
||||
|
||||
- Now, since we will make some transformations to this image, we need a new Mat object to store
|
||||
it. Also, we want this to have the following features:
|
||||
|
||||
- Initial pixel values equal to zero
|
||||
- Same size and type as the original image
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/ImgProc/BasicLinearTransforms.cpp basic-linear-transform-output
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/BasicLinearTransformsDemo.java basic-linear-transform-output
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/BasicLinearTransforms.py basic-linear-transform-output
|
||||
@end_toggle
|
||||
|
||||
We observe that @ref cv::Mat::zeros returns a Matlab-style zero initializer based on
|
||||
*image.size()* and *image.type()*
|
||||
|
||||
- We ask now the values of \f$\alpha\f$ and \f$\beta\f$ to be entered by the user:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/ImgProc/BasicLinearTransforms.cpp basic-linear-transform-parameters
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/BasicLinearTransformsDemo.java basic-linear-transform-parameters
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/BasicLinearTransforms.py basic-linear-transform-parameters
|
||||
@end_toggle
|
||||
|
||||
- Now, to perform the operation \f$g(i,j) = \alpha \cdot f(i,j) + \beta\f$ we will access to each
|
||||
pixel in image. Since we are operating with BGR images, we will have three values per pixel (B,
|
||||
G and R), so we will also access them separately. Here is the piece of code:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/ImgProc/BasicLinearTransforms.cpp basic-linear-transform-operation
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/BasicLinearTransformsDemo.java basic-linear-transform-operation
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/BasicLinearTransforms.py basic-linear-transform-operation
|
||||
@end_toggle
|
||||
|
||||
Notice the following (**C++ code only**):
|
||||
- To access each pixel in the images we are using this syntax: *image.at\<Vec3b\>(y,x)[c]*
|
||||
where *y* is the row, *x* is the column and *c* is B, G or R (0, 1 or 2).
|
||||
- Since the operation \f$\alpha \cdot p(i,j) + \beta\f$ can give values out of range or not
|
||||
integers (if \f$\alpha\f$ is float), we use cv::saturate_cast to make sure the
|
||||
values are valid.
|
||||
|
||||
- Finally, we create windows and show the images, the usual way.
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/ImgProc/BasicLinearTransforms.cpp basic-linear-transform-display
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/BasicLinearTransformsDemo.java basic-linear-transform-display
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/BasicLinearTransforms.py basic-linear-transform-display
|
||||
@end_toggle
|
||||
|
||||
@note
|
||||
Instead of using the **for** loops to access each pixel, we could have simply used this command:
|
||||
|
||||
@add_toggle_cpp
|
||||
@code{.cpp}
|
||||
image.convertTo(new_image, -1, alpha, beta);
|
||||
@endcode
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@code{.java}
|
||||
image.convertTo(newImage, -1, alpha, beta);
|
||||
@endcode
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@code{.py}
|
||||
new_image = cv.convertScaleAbs(image, alpha=alpha, beta=beta)
|
||||
@endcode
|
||||
@end_toggle
|
||||
|
||||
where @ref cv::Mat::convertTo would effectively perform *new_image = a*image + beta\*. However, we
|
||||
wanted to show you how to access each pixel. In any case, both methods give the same result but
|
||||
convertTo is more optimized and works a lot faster.
|
||||
|
||||
Result
|
||||
------
|
||||
|
||||
- Running our code and using \f$\alpha = 2.2\f$ and \f$\beta = 50\f$
|
||||
@code{.bash}
|
||||
$ ./BasicLinearTransforms lena.jpg
|
||||
Basic Linear Transforms
|
||||
-------------------------
|
||||
* Enter the alpha value [1.0-3.0]: 2.2
|
||||
* Enter the beta value [0-100]: 50
|
||||
@endcode
|
||||
|
||||
- We get this:
|
||||
|
||||
|
||||
|
||||
Practical example
|
||||
----
|
||||
|
||||
In this paragraph, we will put into practice what we have learned to correct an underexposed image by adjusting the brightness
|
||||
and the contrast of the image. We will also see another technique to correct the brightness of an image called
|
||||
gamma correction.
|
||||
|
||||
### Brightness and contrast adjustments
|
||||
|
||||
Increasing (/ decreasing) the \f$\beta\f$ value will add (/ subtract) a constant value to every pixel. Pixel values outside of the [0 ; 255]
|
||||
range will be saturated (i.e. a pixel value higher (/ lesser) than 255 (/ 0) will be clamped to 255 (/ 0)).
|
||||
|
||||
|
||||
|
||||
The histogram represents for each color level the number of pixels with that color level. A dark image will have many pixels with
|
||||
low color value and thus the histogram will present a peak in its left part. When adding a constant bias, the histogram is shifted to the
|
||||
right as we have added a constant bias to all the pixels.
|
||||
|
||||
The \f$\alpha\f$ parameter will modify how the levels spread. If \f$ \alpha < 1 \f$, the color levels will be compressed and the result
|
||||
will be an image with less contrast.
|
||||
|
||||
|
||||
|
||||
Note that these histograms have been obtained using the Brightness-Contrast tool in the Gimp software. The brightness tool should be
|
||||
identical to the \f$\beta\f$ bias parameters but the contrast tool seems to differ to the \f$\alpha\f$ gain where the output range
|
||||
seems to be centered with Gimp (as you can notice in the previous histogram).
|
||||
|
||||
It can occur that playing with the \f$\beta\f$ bias will improve the brightness but in the same time the image will appear with a
|
||||
slight veil as the contrast will be reduced. The \f$\alpha\f$ gain can be used to diminue this effect but due to the saturation,
|
||||
we will lose some details in the original bright regions.
|
||||
|
||||
### Gamma correction
|
||||
|
||||
[Gamma correction](https://en.wikipedia.org/wiki/Gamma_correction) can be used to correct the brightness of an image by using a non
|
||||
linear transformation between the input values and the mapped output values:
|
||||
|
||||
\f[O = \left( \frac{I}{255} \right)^{\gamma} \times 255\f]
|
||||
|
||||
As this relation is non linear, the effect will not be the same for all the pixels and will depend to their original value.
|
||||
|
||||
|
||||
|
||||
When \f$ \gamma < 1 \f$, the original dark regions will be brighter and the histogram will be shifted to the right whereas it will
|
||||
be the opposite with \f$ \gamma > 1 \f$.
|
||||
|
||||
### Correct an underexposed image
|
||||
|
||||
The following image has been corrected with: \f$ \alpha = 1.3 \f$ and \f$ \beta = 40 \f$.
|
||||
|
||||
![By Visem (Own work) [CC BY-SA 3.0], via Wikimedia Commons](images/Basic_Linear_Transform_Tutorial_linear_transform_correction.jpg) { width=90% }
|
||||
|
||||
The overall brightness has been improved but you can notice that the clouds are now greatly saturated due to the numerical saturation
|
||||
of the implementation used ([highlight clipping](https://en.wikipedia.org/wiki/Clipping_(photography)) in photography).
|
||||
|
||||
The following image has been corrected with: \f$ \gamma = 0.4 \f$.
|
||||
|
||||
![By Visem (Own work) [CC BY-SA 3.0], via Wikimedia Commons](images/Basic_Linear_Transform_Tutorial_gamma_correction.jpg) { width=90% }
|
||||
|
||||
The gamma correction should tend to add less saturation effect as the mapping is non linear and there is no numerical saturation possible as in the previous method.
|
||||
|
||||
|
||||
|
||||
The previous figure compares the histograms for the three images (the y-ranges are not the same between the three histograms).
|
||||
You can notice that most of the pixel values are in the lower part of the histogram for the original image. After \f$ \alpha \f$,
|
||||
\f$ \beta \f$ correction, we can observe a big peak at 255 due to the saturation as well as a shift in the right.
|
||||
After gamma correction, the histogram is shifted to the right but the pixels in the dark regions are more shifted
|
||||
(see the gamma curves [figure](Basic_Linear_Transform_Tutorial_gamma.png)) than those in the bright regions.
|
||||
|
||||
In this tutorial, you have seen two simple methods to adjust the contrast and the brightness of an image. **They are basic techniques
|
||||
and are not intended to be used as a replacement of a raster graphics editor!**
|
||||
|
||||
### Code
|
||||
|
||||
@add_toggle_cpp
|
||||
Code for the tutorial is [here](https://github.com/opencv/opencv/blob/4.x/samples/cpp/tutorial_code/ImgProc/changing_contrast_brightness_image/changing_contrast_brightness_image.cpp).
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
Code for the tutorial is [here](https://github.com/opencv/opencv/blob/4.x/samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/ChangingContrastBrightnessImageDemo.java).
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
Code for the tutorial is [here](https://github.com/opencv/opencv/blob/4.x/samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/changing_contrast_brightness_image.py).
|
||||
@end_toggle
|
||||
|
||||
Code for the gamma correction:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/ImgProc/changing_contrast_brightness_image/changing_contrast_brightness_image.cpp changing-contrast-brightness-gamma-correction
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/ImgProc/changing_contrast_brightness_image/ChangingContrastBrightnessImageDemo.java changing-contrast-brightness-gamma-correction
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/imgProc/changing_contrast_brightness_image/changing_contrast_brightness_image.py changing-contrast-brightness-gamma-correction
|
||||
@end_toggle
|
||||
|
||||
A look-up table is used to improve the performance of the computation as only 256 values needs to be calculated once.
|
||||
|
||||
### Additional resources
|
||||
|
||||
- [Gamma correction in graphics rendering](https://learnopengl.com/#!Advanced-Lighting/Gamma-Correction)
|
||||
- [Gamma correction and images displayed on CRT monitors](http://www.graphics.cornell.edu/~westin/gamma/gamma.html)
|
||||
- [Digital exposure techniques](http://www.cambridgeincolour.com/tutorials/digital-exposure-techniques.htm)
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 90 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 270 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 3.1 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 3.4 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 1.4 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 222 KiB |
@@ -0,0 +1,245 @@
|
||||
Discrete Fourier Transform {#tutorial_discrete_fourier_transform}
|
||||
==========================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_basic_linear_transform}
|
||||
@next_tutorial{tutorial_file_input_output_with_xml_yml}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
We'll seek answers for the following questions:
|
||||
|
||||
- What is a Fourier transform and why use it?
|
||||
- How to do it in OpenCV?
|
||||
- Usage of functions such as: **copyMakeBorder()** , **merge()** , **dft()** ,
|
||||
**getOptimalDFTSize()** , **log()** and **normalize()** .
|
||||
|
||||
Source code
|
||||
-----------
|
||||
|
||||
@add_toggle_cpp
|
||||
You can [download this from here
|
||||
](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp) or
|
||||
find it in the
|
||||
`samples/cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp` of the
|
||||
OpenCV source code library.
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
You can [download this from here
|
||||
](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java) or
|
||||
find it in the
|
||||
`samples/java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java` of the
|
||||
OpenCV source code library.
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
You can [download this from here
|
||||
](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py) or
|
||||
find it in the
|
||||
`samples/python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py` of the
|
||||
OpenCV source code library.
|
||||
@end_toggle
|
||||
|
||||
Here's a sample usage of **dft()** :
|
||||
|
||||
@add_toggle_cpp
|
||||
@include cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@include java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@include python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py
|
||||
@end_toggle
|
||||
|
||||
Explanation
|
||||
-----------
|
||||
|
||||
The Fourier Transform will decompose an image into its sinus and cosines components. In other words,
|
||||
it will transform an image from its spatial domain to its frequency domain. The idea is that any
|
||||
function may be approximated exactly with the sum of infinite sinus and cosines functions. The
|
||||
Fourier Transform is a way how to do this. Mathematically a two dimensional images Fourier transform
|
||||
is:
|
||||
|
||||
\f[F(k,l) = \displaystyle\sum\limits_{i=0}^{N-1}\sum\limits_{j=0}^{N-1} f(i,j)e^{-i2\pi(\frac{ki}{N}+\frac{lj}{N})}\f]\f[e^{ix} = \cos{x} + i\sin {x}\f]
|
||||
|
||||
Here f is the image value in its spatial domain and F in its frequency domain. The result of the
|
||||
transformation is complex numbers. Displaying this is possible either via a *real* image and a
|
||||
*complex* image or via a *magnitude* and a *phase* image. However, throughout the image processing
|
||||
algorithms only the *magnitude* image is interesting as this contains all the information we need
|
||||
about the images geometric structure. Nevertheless, if you intend to make some modifications of the
|
||||
image in these forms and then you need to retransform it you'll need to preserve both of these.
|
||||
|
||||
In this sample I'll show how to calculate and show the *magnitude* image of a Fourier Transform. In
|
||||
case of digital images are discrete. This means they may take up a value from a given domain value.
|
||||
For example in a basic gray scale image values usually are between zero and 255. Therefore the
|
||||
Fourier Transform too needs to be of a discrete type resulting in a Discrete Fourier Transform
|
||||
(*DFT*). You'll want to use this whenever you need to determine the structure of an image from a
|
||||
geometrical point of view. Here are the steps to follow (in case of a gray scale input image *I*):
|
||||
|
||||
### Expand the image to an optimal size
|
||||
|
||||
The performance of a DFT is dependent of the image
|
||||
size. It tends to be the fastest for image sizes that are multiple of the numbers two, three and
|
||||
five. Therefore, to achieve maximal performance it is generally a good idea to pad border values
|
||||
to the image to get a size with such traits. The **getOptimalDFTSize()** returns this
|
||||
optimal size and we can use the **copyMakeBorder()** function to expand the borders of an
|
||||
image (the appended pixels are initialized with zero):
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp expand
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java expand
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py expand
|
||||
@end_toggle
|
||||
|
||||
### Make place for both the complex and the real values
|
||||
|
||||
The result of a Fourier Transform is
|
||||
complex. This implies that for each image value the result is two image values (one per
|
||||
component). Moreover, the frequency domains range is much larger than its spatial counterpart.
|
||||
Therefore, we store these usually at least in a *float* format. Therefore we'll convert our
|
||||
input image to this type and expand it with another channel to hold the complex values:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp complex_and_real
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java complex_and_real
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py complex_and_real
|
||||
@end_toggle
|
||||
|
||||
### Make the Discrete Fourier Transform
|
||||
It's possible an in-place calculation (same input as
|
||||
output):
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp dft
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java dft
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py dft
|
||||
@end_toggle
|
||||
|
||||
### Transform the real and complex values to magnitude
|
||||
A complex number has a real (*Re*) and a
|
||||
complex (imaginary - *Im*) part. The results of a DFT are complex numbers. The magnitude of a
|
||||
DFT is:
|
||||
|
||||
\f[M = \sqrt[2]{ {Re(DFT(I))}^2 + {Im(DFT(I))}^2}\f]
|
||||
|
||||
Translated to OpenCV code:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp magnitude
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java magnitude
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py magnitude
|
||||
@end_toggle
|
||||
|
||||
### Switch to a logarithmic scale
|
||||
It turns out that the dynamic range of the Fourier
|
||||
coefficients is too large to be displayed on the screen. We have some small and some high
|
||||
changing values that we can't observe like this. Therefore the high values will all turn out as
|
||||
white points, while the small ones as black. To use the gray scale values to for visualization
|
||||
we can transform our linear scale to a logarithmic one:
|
||||
|
||||
\f[M_1 = \log{(1 + M)}\f]
|
||||
|
||||
Translated to OpenCV code:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp log
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java log
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py log
|
||||
@end_toggle
|
||||
|
||||
### Crop and rearrange
|
||||
Remember, that at the first step, we expanded the image? Well, it's time
|
||||
to throw away the newly introduced values. For visualization purposes we may also rearrange the
|
||||
quadrants of the result, so that the origin (zero, zero) corresponds with the image center.
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp crop_rearrange
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java crop_rearrange
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py crop_rearrange
|
||||
@end_toggle
|
||||
|
||||
### Normalize
|
||||
This is done again for visualization purposes. We now have the magnitudes,
|
||||
however this are still out of our image display range of zero to one. We normalize our values to
|
||||
this range using the @ref cv::normalize() function.
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.cpp normalize
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet java/tutorial_code/core/discrete_fourier_transform/DiscreteFourierTransform.java normalize
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/discrete_fourier_transform/discrete_fourier_transform.py normalize
|
||||
@end_toggle
|
||||
|
||||
Result
|
||||
------
|
||||
|
||||
An application idea would be to determine the geometrical orientation present in the image. For
|
||||
example, let us find out if a text is horizontal or not? Looking at some text you'll notice that the
|
||||
text lines sort of form also horizontal lines and the letters form sort of vertical lines. These two
|
||||
main components of a text snippet may be also seen in case of the Fourier transform. Let us use
|
||||
[this horizontal ](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/data/imageTextN.png) and [this rotated](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/data/imageTextR.png)
|
||||
image about a text.
|
||||
|
||||
In case of the horizontal text:
|
||||
|
||||
|
||||
|
||||
In case of a rotated text:
|
||||
|
||||
|
||||
|
||||
You can see that the most influential components of the frequency domain (brightest dots on the
|
||||
magnitude image) follow the geometric rotation of objects on the image. From this we may calculate
|
||||
the offset and perform an image rotation to correct eventual miss alignments.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
+297
@@ -0,0 +1,297 @@
|
||||
File Input and Output using XML / YAML / JSON files {#tutorial_file_input_output_with_xml_yml}
|
||||
==============================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_discrete_fourier_transform}
|
||||
@next_tutorial{tutorial_how_to_use_OpenCV_parallel_for_}
|
||||
@next_tutorial{tutorial_how_to_use_OpenCV_parallel_for_new}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
You'll find answers to the following questions:
|
||||
|
||||
- How do you print and read text entries to a file in OpenCV using YAML, XML, or JSON files?
|
||||
- How can you perform the same operations for OpenCV data structures?
|
||||
- How can this be done for your custom data structures?
|
||||
- How do you use OpenCV data structures, such as @ref cv::FileStorage , @ref cv::FileNode or @ref
|
||||
cv::FileNodeIterator .
|
||||
|
||||
Source code
|
||||
-----------
|
||||
@add_toggle_cpp
|
||||
You can [download this from here
|
||||
](https://github.com/opencv/opencv/tree/4.x/samples/cpp/tutorial_code/core/file_input_output/file_input_output.cpp) or find it in the
|
||||
`samples/cpp/tutorial_code/core/file_input_output/file_input_output.cpp` of the OpenCV source code
|
||||
library.
|
||||
|
||||
Here's a sample code of how to achieve all the stuff enumerated at the goal list.
|
||||
|
||||
@include cpp/tutorial_code/core/file_input_output/file_input_output.cpp
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
You can [download this from here
|
||||
](https://github.com/opencv/opencv/tree/4.x/samples/python/tutorial_code/core/file_input_output/file_input_output.py) or find it in the
|
||||
`samples/python/tutorial_code/core/file_input_output/file_input_output.py` of the OpenCV source code
|
||||
library.
|
||||
|
||||
Here's a sample code of how to achieve all the stuff enumerated at the goal list.
|
||||
|
||||
@include python/tutorial_code/core/file_input_output/file_input_output.py
|
||||
@end_toggle
|
||||
|
||||
Explanation
|
||||
-----------
|
||||
|
||||
Here we talk only about XML, YAML and JSON file inputs. Your output (and its respective input) file may
|
||||
have only one of these extensions and the structure coming from this. They are two kinds of data
|
||||
structures you may serialize: *mappings* (like the STL map and the Python dictionary) and *element sequence* (like the STL
|
||||
vector). The difference between these is that in a map every element has a unique name through what
|
||||
you may access it. For sequences you need to go through them to query a specific item.
|
||||
|
||||
-# **XML/YAML/JSON File Open and Close.** Before you write any content to such file you need to open it
|
||||
and at the end to close it. The XML/YAML/JSON data structure in OpenCV is @ref cv::FileStorage . To
|
||||
specify that this structure to which file binds on your hard drive you can use either its
|
||||
constructor or the *open()* function of this:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp open
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py open
|
||||
@end_toggle
|
||||
Either one of this you use the second argument is a constant specifying the type of operations
|
||||
you'll be able to on them: WRITE, READ or APPEND. The extension specified in the file name also
|
||||
determinates the output format that will be used. The output may be even compressed if you
|
||||
specify an extension such as *.xml.gz*.
|
||||
|
||||
The file automatically closes when the @ref cv::FileStorage objects is destroyed. However, you
|
||||
may explicitly call for this by using the *release* function:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp close
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py close
|
||||
@end_toggle
|
||||
-# **Input and Output of text and numbers.** In C++, the data structure uses the \<\< output
|
||||
operator in the STL library. In Python, @ref cv::FileStorage.write() is used instead. For
|
||||
outputting any type of data structure we need first to specify its name. We do this by just
|
||||
simply pushing the name of this to the stream in C++. In Python, the first parameter for the
|
||||
write function is the name. For basic types you may follow this with the print of the value :
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp writeNum
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py writeNum
|
||||
@end_toggle
|
||||
Reading in is a simple addressing (via the [] operator) and casting operation or a read via
|
||||
the \>\> operator. In Python, we address with getNode() and use real() :
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp readNum
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp readNum
|
||||
@end_toggle
|
||||
-# **Input/Output of OpenCV Data structures.** Well these behave exactly just as the basic C++
|
||||
and Python types:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp iomati
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp iomatw
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp iomat
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py iomati
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py iomatw
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py iomat
|
||||
@end_toggle
|
||||
-# **Input/Output of vectors (arrays) and associative maps.** As I mentioned beforehand, we can
|
||||
output maps and sequences (array, vector) too. Again we first print the name of the variable and
|
||||
then we have to specify if our output is either a sequence or map.
|
||||
|
||||
For sequence before the first element print the "[" character and after the last one the "]"
|
||||
character. With Python, call `FileStorage.startWriteStruct(structure_name, struct_type)`,
|
||||
where `struct_type` is `cv2.FileNode_MAP` or `cv2.FileNode_SEQ` to start writing the structure.
|
||||
Call `FileStorage.endWriteStruct()` to finish the structure:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp writeStr
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py writeStr
|
||||
@end_toggle
|
||||
For maps the drill is the same however now we use the "{" and "}" delimiter characters:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp writeMap
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py writeMap
|
||||
@end_toggle
|
||||
To read from these we use the @ref cv::FileNode and the @ref cv::FileNodeIterator data
|
||||
structures. The [] operator of the @ref cv::FileStorage class (or the getNode() function in Python) returns a @ref cv::FileNode data
|
||||
type. If the node is sequential we can use the @ref cv::FileNodeIterator to iterate through the
|
||||
items. In Python, the at() function can be used to address elements of the sequence and the
|
||||
size() function returns the length of the sequence:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp readStr
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py readStr
|
||||
@end_toggle
|
||||
For maps you can use the [] operator (at() function in Python) again to access the given item (or the \>\> operator too):
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp readMap
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py readMap
|
||||
@end_toggle
|
||||
-# **Read and write your own data structures.** Suppose you have a data structure such as:
|
||||
@add_toggle_cpp
|
||||
@code{.cpp}
|
||||
class MyData
|
||||
{
|
||||
public:
|
||||
MyData() : A(0), X(0), id() {}
|
||||
public: // Data Members
|
||||
int A;
|
||||
double X;
|
||||
string id;
|
||||
};
|
||||
@endcode
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@code{.py}
|
||||
class MyData:
|
||||
def __init__(self):
|
||||
self.A = self.X = 0
|
||||
self.name = ''
|
||||
@endcode
|
||||
@end_toggle
|
||||
In C++, it's possible to serialize this through the OpenCV I/O XML/YAML interface (just as
|
||||
in case of the OpenCV data structures) by adding a read and a write function inside and outside of your
|
||||
class. In Python, you can get close to this by implementing a read and write function inside
|
||||
the class. For the inside part:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp inside
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py inside
|
||||
@end_toggle
|
||||
@add_toggle_cpp
|
||||
In C++, you need to add the following functions definitions outside the class:
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp outside
|
||||
@end_toggle
|
||||
Here you can observe that in the read section we defined what happens if the user tries to read
|
||||
a non-existing node. In this case we just return the default initialization value, however a
|
||||
more verbose solution would be to return for instance a minus one value for an object ID.
|
||||
|
||||
Once you added these four functions use the \>\> operator for write and the \<\< operator for
|
||||
read (or the defined input/output functions for Python):
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp customIOi
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp customIOw
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp customIO
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py customIOi
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py customIOw
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py customIO
|
||||
@end_toggle
|
||||
Or to try out reading a non-existing read:
|
||||
@add_toggle_cpp
|
||||
@snippet cpp/tutorial_code/core/file_input_output/file_input_output.cpp nonexist
|
||||
@end_toggle
|
||||
@add_toggle_python
|
||||
@snippet python/tutorial_code/core/file_input_output/file_input_output.py nonexist
|
||||
@end_toggle
|
||||
|
||||
Result
|
||||
------
|
||||
|
||||
Well mostly we just print out the defined numbers. On the screen of your console you could see:
|
||||
@code{.bash}
|
||||
Write Done.
|
||||
|
||||
Reading:
|
||||
100image1.jpg
|
||||
Awesomeness
|
||||
baboon.jpg
|
||||
Two 2; One 1
|
||||
|
||||
|
||||
R = [1, 0, 0;
|
||||
0, 1, 0;
|
||||
0, 0, 1]
|
||||
T = [0; 0; 0]
|
||||
|
||||
MyData =
|
||||
{ id = mydata1234, X = 3.14159, A = 97}
|
||||
|
||||
Attempt to read NonExisting (should initialize the data structure with its default).
|
||||
NonExisting =
|
||||
{ id = , X = 0, A = 0}
|
||||
|
||||
Tip: Open up output.xml with a text editor to see the serialized data.
|
||||
@endcode
|
||||
Nevertheless, it's much more interesting what you may see in the output xml file:
|
||||
@code{.xml}
|
||||
<?xml version="1.0"?>
|
||||
<opencv_storage>
|
||||
<iterationNr>100</iterationNr>
|
||||
<strings>
|
||||
image1.jpg Awesomeness baboon.jpg</strings>
|
||||
<Mapping>
|
||||
<One>1</One>
|
||||
<Two>2</Two></Mapping>
|
||||
<R type_id="opencv-matrix">
|
||||
<rows>3</rows>
|
||||
<cols>3</cols>
|
||||
<dt>u</dt>
|
||||
<data>
|
||||
1 0 0 0 1 0 0 0 1</data></R>
|
||||
<T type_id="opencv-matrix">
|
||||
<rows>3</rows>
|
||||
<cols>1</cols>
|
||||
<dt>d</dt>
|
||||
<data>
|
||||
0. 0. 0.</data></T>
|
||||
<MyData>
|
||||
<A>97</A>
|
||||
<X>3.1415926535897931e+000</X>
|
||||
<id>mydata1234</id></MyData>
|
||||
</opencv_storage>
|
||||
@endcode
|
||||
Or the YAML file:
|
||||
@code{.yaml}
|
||||
%YAML:1.0
|
||||
iterationNr: 100
|
||||
strings:
|
||||
- "image1.jpg"
|
||||
- Awesomeness
|
||||
- "baboon.jpg"
|
||||
Mapping:
|
||||
One: 1
|
||||
Two: 2
|
||||
R: !!opencv-matrix
|
||||
rows: 3
|
||||
cols: 3
|
||||
dt: u
|
||||
data: [ 1, 0, 0, 0, 1, 0, 0, 0, 1 ]
|
||||
T: !!opencv-matrix
|
||||
rows: 3
|
||||
cols: 1
|
||||
dt: d
|
||||
data: [ 0., 0., 0. ]
|
||||
MyData:
|
||||
A: 97
|
||||
X: 3.1415926535897931e+000
|
||||
id: mydata1234
|
||||
@endcode
|
||||
You may observe a runtime instance of this on the [YouTube
|
||||
here](https://www.youtube.com/watch?v=A4yqVnByMMM) .
|
||||
|
||||
@youtube{A4yqVnByMMM}
|
||||
@@ -0,0 +1,227 @@
|
||||
How to scan images, lookup tables and time measurement with OpenCV {#tutorial_how_to_scan_images}
|
||||
==================================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_mat_the_basic_image_container}
|
||||
@next_tutorial{tutorial_mat_mask_operations}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
We'll seek answers for the following questions:
|
||||
|
||||
- How to go through each and every pixel of an image?
|
||||
- How are OpenCV matrix values stored?
|
||||
- How to measure the performance of our algorithm?
|
||||
- What are lookup tables and why use them?
|
||||
|
||||
Our test case
|
||||
-------------
|
||||
|
||||
Let us consider a simple color reduction method. By using the unsigned char C and C++ type for
|
||||
matrix item storing, a channel of pixel may have up to 256 different values. For a three channel
|
||||
image this can allow the formation of way too many colors (16 million to be exact). Working with so
|
||||
many color shades may give a heavy blow to our algorithm performance. However, sometimes it is
|
||||
enough to work with a lot less of them to get the same final result.
|
||||
|
||||
In this cases it's common that we make a *color space reduction*. This means that we divide the
|
||||
color space current value with a new input value to end up with fewer colors. For instance every
|
||||
value between zero and nine takes the new value zero, every value between ten and nineteen the value
|
||||
ten and so on.
|
||||
|
||||
When you divide an *uchar* (unsigned char - aka values between zero and 255) value with an *int*
|
||||
value the result will be also *char*. These values may only be char values. Therefore, any fraction
|
||||
will be rounded down. Taking advantage of this fact the upper operation in the *uchar* domain may be
|
||||
expressed as:
|
||||
|
||||
\f[I_{new} = (\frac{I_{old}}{10}) * 10\f]
|
||||
|
||||
A simple color space reduction algorithm would consist of just passing through every pixel of an
|
||||
image matrix and applying this formula. It's worth noting that we do a divide and a multiplication
|
||||
operation. These operations are bloody expensive for a system. If possible it's worth avoiding them
|
||||
by using cheaper operations such as a few subtractions, addition or in best case a simple
|
||||
assignment. Furthermore, note that we only have a limited number of input values for the upper
|
||||
operation. In case of the *uchar* system this is 256 to be exact.
|
||||
|
||||
Therefore, for larger images it would be wise to calculate all possible values beforehand and during
|
||||
the assignment just make the assignment, by using a lookup table. Lookup tables are simple arrays
|
||||
(having one or more dimensions) that for a given input value variation holds the final output value.
|
||||
Its strength is that we do not need to make the calculation, we just need to read the result.
|
||||
|
||||
Our test case program (and the code sample below) will do the following: read in an image passed
|
||||
as a command line argument (it may be either color or grayscale) and apply the reduction
|
||||
with the given command line argument integer value. In OpenCV, at the moment there are
|
||||
three major ways of going through an image pixel by pixel. To make things a little more interesting
|
||||
we'll make the scanning of the image using each of these methods, and print out how long it took.
|
||||
|
||||
You can download the full source code [here
|
||||
](https://github.com/opencv/opencv/tree/4.x/samples/cpp/tutorial_code/core/how_to_scan_images/how_to_scan_images.cpp) or look it up in
|
||||
the samples directory of OpenCV at the cpp tutorial code for the core section. Its basic usage is:
|
||||
@code{.bash}
|
||||
how_to_scan_images imageName.jpg intValueToReduce [G]
|
||||
@endcode
|
||||
The final argument is optional. If given the image will be loaded in grayscale format, otherwise
|
||||
the BGR color space is used. The first thing is to calculate the lookup table.
|
||||
|
||||
@snippet how_to_scan_images.cpp dividewith
|
||||
|
||||
Here we first use the C++ *stringstream* class to convert the third command line argument from text
|
||||
to an integer format. Then we use a simple look and the upper formula to calculate the lookup table.
|
||||
No OpenCV specific stuff here.
|
||||
|
||||
Another issue is how do we measure time? Well OpenCV offers two simple functions to achieve this
|
||||
cv::getTickCount() and cv::getTickFrequency() . The first returns the number of ticks of
|
||||
your systems CPU from a certain event (like since you booted your system). The second returns how
|
||||
many times your CPU emits a tick during a second. So, measuring amount of time elapsed between
|
||||
two operations is as easy as:
|
||||
@code{.cpp}
|
||||
double t = (double)getTickCount();
|
||||
// do something ...
|
||||
t = ((double)getTickCount() - t)/getTickFrequency();
|
||||
cout << "Times passed in seconds: " << t << endl;
|
||||
@endcode
|
||||
|
||||
@anchor tutorial_how_to_scan_images_storing
|
||||
How is the image matrix stored in memory?
|
||||
-----------------------------------------
|
||||
|
||||
As you could already read in my @ref tutorial_mat_the_basic_image_container tutorial the size of the matrix
|
||||
depends on the color system used. More accurately, it depends on the number of channels used. In
|
||||
case of a grayscale image we have something like:
|
||||
|
||||
|
||||
|
||||
For multichannel images the columns contain as many sub columns as the number of channels. For
|
||||
example in case of an BGR color system:
|
||||
|
||||
|
||||
|
||||
Note that the order of the channels is inverse: BGR instead of RGB. Because in many cases the memory
|
||||
is large enough to store the rows in a successive fashion the rows may follow one after another,
|
||||
creating a single long row. Because everything is in a single place following one after another this
|
||||
may help to speed up the scanning process. We can use the cv::Mat::isContinuous() function to *ask*
|
||||
the matrix if this is the case. Continue on to the next section to find an example.
|
||||
|
||||
The efficient way
|
||||
-----------------
|
||||
|
||||
When it comes to performance you cannot beat the classic C style operator[] (pointer) access.
|
||||
Therefore, the most efficient method we can recommend for making the assignment is:
|
||||
|
||||
@snippet how_to_scan_images.cpp scan-c
|
||||
|
||||
Here we basically just acquire a pointer to the start of each row and go through it until it ends.
|
||||
In the special case that the matrix is stored in a continuous manner we only need to request the
|
||||
pointer a single time and go all the way to the end. We need to look out for color images: we have
|
||||
three channels so we need to pass through three times more items in each row.
|
||||
|
||||
There's another way of this. The *data* data member of a *Mat* object returns the pointer to the
|
||||
first row, first column. If this pointer is null you have no valid input in that object. Checking
|
||||
this is the simplest method to check if your image loading was a success. In case the storage is
|
||||
continuous we can use this to go through the whole data pointer. In case of a grayscale image this
|
||||
would look like:
|
||||
@code{.cpp}
|
||||
uchar* p = I.data;
|
||||
|
||||
for( unsigned int i = 0; i < ncol*nrows; ++i)
|
||||
*p++ = table[*p];
|
||||
@endcode
|
||||
You would get the same result. However, this code is a lot harder to read later on. It gets even
|
||||
harder if you have some more advanced technique there. Moreover, in practice I've observed you'll
|
||||
get the same performance result (as most of the modern compilers will probably make this small
|
||||
optimization trick automatically for you).
|
||||
|
||||
The iterator (safe) method
|
||||
--------------------------
|
||||
|
||||
In case of the efficient way making sure that you pass through the right amount of *uchar* fields
|
||||
and to skip the gaps that may occur between the rows was your responsibility. The iterator method is
|
||||
considered a safer way as it takes over these tasks from the user. All you need to do is to ask the
|
||||
begin and the end of the image matrix and then just increase the begin iterator until you reach the
|
||||
end. To acquire the value *pointed* by the iterator use the \* operator (add it before it).
|
||||
|
||||
@snippet how_to_scan_images.cpp scan-iterator
|
||||
|
||||
In case of color images we have three uchar items per column. This may be considered a short vector
|
||||
of uchar items, that has been baptized in OpenCV with the *Vec3b* name. To access the n-th sub
|
||||
column we use simple operator[] access. It's important to remember that OpenCV iterators go through
|
||||
the columns and automatically skip to the next row. Therefore in case of color images if you use a
|
||||
simple *uchar* iterator you'll be able to access only the blue channel values.
|
||||
|
||||
On-the-fly address calculation with reference returning
|
||||
-------------------------------------------------------
|
||||
|
||||
The final method isn't recommended for scanning. It was made to acquire or modify somehow random
|
||||
elements in the image. Its basic usage is to specify the row and column number of the item you want
|
||||
to access. During our earlier scanning methods you could already notice that it is important through
|
||||
what type we are looking at the image. It's no different here as you need to manually specify what
|
||||
type to use at the automatic lookup. You can observe this in case of the grayscale images for the
|
||||
following source code (the usage of the + cv::Mat::at() function):
|
||||
|
||||
@snippet how_to_scan_images.cpp scan-random
|
||||
|
||||
The function takes your input type and coordinates and calculates the address of the
|
||||
queried item. Then returns a reference to that. This may be a constant when you *get* the value and
|
||||
non-constant when you *set* the value. As a safety step in **debug mode only**\* there is a check
|
||||
performed that your input coordinates are valid and do exist. If this isn't the case you'll get a
|
||||
nice output message of this on the standard error output stream. Compared to the efficient way in
|
||||
release mode the only difference in using this is that for every element of the image you'll get a
|
||||
new row pointer for what we use the C operator[] to acquire the column element.
|
||||
|
||||
If you need to do multiple lookups using this method for an image it may be troublesome and time
|
||||
consuming to enter the type and the at keyword for each of the accesses. To solve this problem
|
||||
OpenCV has a cv::Mat_ data type. It's the same as Mat with the extra need that at definition
|
||||
you need to specify the data type through what to look at the data matrix, however in return you can
|
||||
use the operator() for fast access of items. To make things even better this is easily convertible
|
||||
from and to the usual cv::Mat data type. A sample usage of this you can see in case of the
|
||||
color images of the function above. Nevertheless, it's important to note that the same operation
|
||||
(with the same runtime speed) could have been done with the cv::Mat::at function. It's just a less
|
||||
to write for the lazy programmer trick.
|
||||
|
||||
The Core Function
|
||||
-----------------
|
||||
|
||||
This is a bonus method of achieving lookup table modification in an image. In image
|
||||
processing it's quite common that you want to modify all of a given image values to some other value.
|
||||
OpenCV provides a function for modifying image values, without the need to write the scanning logic
|
||||
of the image. We use the cv::LUT() function of the core module. First we build a Mat type of the
|
||||
lookup table:
|
||||
|
||||
@snippet how_to_scan_images.cpp table-init
|
||||
|
||||
Finally call the function (I is our input image and J the output one):
|
||||
|
||||
@snippet how_to_scan_images.cpp table-use
|
||||
|
||||
Performance Difference
|
||||
----------------------
|
||||
|
||||
For the best result compile the program and run it yourself. To make the differences more
|
||||
clear, I've used a quite large (2560 X 1600) image. The performance presented here are for
|
||||
color images. For a more accurate value I've averaged the value I got from the call of the function
|
||||
for hundred times.
|
||||
|
||||
Method | Time
|
||||
--------------- | ----------------------
|
||||
Efficient Way | 79.4717 milliseconds
|
||||
Iterator | 83.7201 milliseconds
|
||||
On-The-Fly RA | 93.7878 milliseconds
|
||||
LUT function | 32.5759 milliseconds
|
||||
|
||||
We can conclude a couple of things. If possible, use the already made functions of OpenCV (instead
|
||||
of reinventing these). The fastest method turns out to be the LUT function. This is because the OpenCV
|
||||
library is multi-thread enabled via Intel Threaded Building Blocks. However, if you need to write a
|
||||
simple image scan prefer the pointer method. The iterator is a safer bet, however quite slower.
|
||||
Using the on-the-fly reference access method for full image scan is the most costly in debug mode.
|
||||
In the release mode it may beat the iterator approach or not, however it surely sacrifices for this
|
||||
the safety trait of iterators.
|
||||
|
||||
Finally, you may watch a sample run of the program on the [video posted](https://www.youtube.com/watch?v=fB3AN5fjgwc) on our YouTube channel.
|
||||
|
||||
@youtube{fB3AN5fjgwc}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.8 KiB |
+201
@@ -0,0 +1,201 @@
|
||||
How to use the OpenCV parallel_for_ function to parallelize your code (Mandelbrot set example) {#tutorial_how_to_use_OpenCV_parallel_for_}
|
||||
==================================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_file_input_output_with_xml_yml}
|
||||
@next_tutorial{tutorial_how_to_use_OpenCV_parallel_for_new}
|
||||
@next_tutorial{tutorial_univ_intrin}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
|
||||
@note See this [tuturial](@ref tutorial_how_to_use_OpenCV_parallel_for_new) for a `parallel_for_` usage applied to image convolution.
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
The goal of this tutorial is to show you how to use the OpenCV `parallel_for_` framework to easily
|
||||
parallelize your code. To illustrate the concept, we will write a program to draw a Mandelbrot set
|
||||
exploiting almost all the CPU load available.
|
||||
The full tutorial code is [here](https://github.com/opencv/opencv/blob/4.x/samples/cpp/tutorial_code/core/how_to_use_OpenCV_parallel_for_/how_to_use_OpenCV_parallel_for_.cpp).
|
||||
If you want more information about multithreading, you will have to refer to a reference book or course as this tutorial is intended
|
||||
to remain simple.
|
||||
|
||||
Precondition
|
||||
------------
|
||||
|
||||
The first precondition is to have OpenCV built with a parallel framework.
|
||||
In OpenCV 4, the following parallel frameworks are available in that order:
|
||||
|
||||
1. Intel Threading Building Blocks (3rdparty library, should be explicitly enabled)
|
||||
2. OpenMP (integrated to compiler, should be explicitly enabled)
|
||||
3. APPLE GCD (system wide, used automatically (APPLE only))
|
||||
4. Windows RT concurrency (system wide, used automatically (Windows RT only))
|
||||
5. Windows concurrency (part of runtime, used automatically (Windows only - MSVC++ >= 10))
|
||||
6. Pthreads (if available)
|
||||
|
||||
As you can see, several parallel frameworks can be used in the OpenCV library. Some parallel libraries
|
||||
are third party libraries and have to be explicitly built and enabled in CMake (e.g. TBB), others are
|
||||
automatically available with the platform (e.g. APPLE GCD) but chances are that you should be enable to
|
||||
have access to a parallel framework either directly or by enabling the option in CMake and rebuild the library.
|
||||
|
||||
The second (weak) precondition is more related to the task you want to achieve as not all computations
|
||||
are suitable / can be adapted to be run in a parallel way. To remain simple, tasks that can be split
|
||||
into multiple elementary operations with no memory dependency (no possible race condition) are easily
|
||||
parallelizable. Computer vision processing are often easily parallelizable as most of the time the processing of
|
||||
one pixel does not depend to the state of other pixels.
|
||||
|
||||
Simple example: drawing a Mandelbrot set
|
||||
----
|
||||
|
||||
We will use the example of drawing a Mandelbrot set to show how from a regular sequential code you can easily adapt
|
||||
the code to parallelize the computation.
|
||||
|
||||
Theory
|
||||
------
|
||||
|
||||
The Mandelbrot set definition has been named in tribute to the mathematician Benoit Mandelbrot by the mathematician
|
||||
Adrien Douady. It has been famous outside of the mathematics field as the image representation is an example of a
|
||||
class of fractals, a mathematical set that exhibits a repeating pattern displayed at every scale (even more, a
|
||||
Mandelbrot set is self-similar as the whole shape can be repeatedly seen at different scale). For a more in-depth
|
||||
introduction, you can look at the corresponding [Wikipedia article](https://en.wikipedia.org/wiki/Mandelbrot_set).
|
||||
Here, we will just introduce the formula to draw the Mandelbrot set (from the mentioned Wikipedia article).
|
||||
|
||||
> The Mandelbrot set is the set of values of \f$ c \f$ in the complex plane for which the orbit of 0 under iteration
|
||||
> of the quadratic map
|
||||
> \f[\begin{cases} z_0 = 0 \\ z_{n+1} = z_n^2 + c \end{cases}\f]
|
||||
> remains bounded.
|
||||
> That is, a complex number \f$ c \f$ is part of the Mandelbrot set if, when starting with \f$ z_0 = 0 \f$ and applying
|
||||
> the iteration repeatedly, the absolute value of \f$ z_n \f$ remains bounded however large \f$ n \f$ gets.
|
||||
> This can also be represented as
|
||||
> \f[\limsup_{n\to\infty}|z_{n+1}|\leqslant2\f]
|
||||
|
||||
Pseudocode
|
||||
----------
|
||||
|
||||
A simple algorithm to generate a representation of the Mandelbrot set is called the
|
||||
["escape time algorithm"](https://en.wikipedia.org/wiki/Mandelbrot_set#Escape_time_algorithm).
|
||||
For each pixel in the rendered image, we test using the recurrence relation if the complex number is bounded or not
|
||||
under a maximum number of iterations. Pixels that do not belong to the Mandelbrot set will escape quickly whereas
|
||||
we assume that the pixel is in the set after a fixed maximum number of iterations. A high value of iterations will
|
||||
produce a more detailed image but the computation time will increase accordingly. We use the number of iterations
|
||||
needed to "escape" to depict the pixel value in the image.
|
||||
|
||||
```
|
||||
For each pixel (Px, Py) on the screen, do:
|
||||
{
|
||||
x0 = scaled x coordinate of pixel (scaled to lie in the Mandelbrot X scale (-2, 1))
|
||||
y0 = scaled y coordinate of pixel (scaled to lie in the Mandelbrot Y scale (-1, 1))
|
||||
x = 0.0
|
||||
y = 0.0
|
||||
iteration = 0
|
||||
max_iteration = 1000
|
||||
while (x*x + y*y < 2*2 AND iteration < max_iteration) {
|
||||
xtemp = x*x - y*y + x0
|
||||
y = 2*x*y + y0
|
||||
x = xtemp
|
||||
iteration = iteration + 1
|
||||
}
|
||||
color = palette[iteration]
|
||||
plot(Px, Py, color)
|
||||
}
|
||||
```
|
||||
|
||||
To relate between the pseudocode and the theory, we have:
|
||||
* \f$ z = x + iy \f$
|
||||
* \f$ z^2 = x^2 + i2xy - y^2 \f$
|
||||
* \f$ c = x_0 + iy_0 \f$
|
||||
|
||||
|
||||
|
||||
On this figure, we recall that the real part of a complex number is on the x-axis and the imaginary part on the y-axis.
|
||||
You can see that the whole shape can be repeatedly visible if we zoom at particular locations.
|
||||
|
||||
Implementation
|
||||
--------------
|
||||
|
||||
Escape time algorithm implementation
|
||||
------------------------------------
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_.cpp mandelbrot-escape-time-algorithm
|
||||
|
||||
Here, we used the [`std::complex`](https://en.cppreference.com/cpp/numeric/complex) template class to represent a
|
||||
complex number. This function performs the test to check if the pixel is in set or not and returns the "escaped" iteration.
|
||||
|
||||
Sequential Mandelbrot implementation
|
||||
------------------------------------
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_.cpp mandelbrot-sequential
|
||||
|
||||
In this implementation, we sequentially iterate over the pixels in the rendered image to perform the test to check if the
|
||||
pixel is likely to belong to the Mandelbrot set or not.
|
||||
|
||||
Another thing to do is to transform the pixel coordinate into the Mandelbrot set space with:
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_.cpp mandelbrot-transformation
|
||||
|
||||
Finally, to assign the grayscale value to the pixels, we use the following rule:
|
||||
* a pixel is black if it reaches the maximum number of iterations (pixel is assumed to be in the Mandelbrot set),
|
||||
* otherwise we assign a grayscale value depending on the escaped iteration and scaled to fit the grayscale range.
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_.cpp mandelbrot-grayscale-value
|
||||
|
||||
Using a linear scale transformation is not enough to perceive the grayscale variation. To overcome this, we will boost
|
||||
the perception by using a square root scale transformation (borrowed from Jeremy D. Frens in his
|
||||
[blog post](https://web.archive.org/web/20250419124416/http://www.programming-during-recess.net/2016/06/26/color-schemes-for-mandelbrot-sets/)):
|
||||
\f$ f \left( x \right) = \sqrt{\frac{x}{\text{maxIter}}} \times 255 \f$
|
||||
|
||||
|
||||
|
||||
The green curve corresponds to a simple linear scale transformation, the blue one to a square root scale transformation
|
||||
and you can observe how the lowest values will be boosted when looking at the slope at these positions.
|
||||
|
||||
Parallel Mandelbrot implementation
|
||||
----------------------------------
|
||||
|
||||
When looking at the sequential implementation, we can notice that each pixel is computed independently. To optimize the
|
||||
computation, we can perform multiple pixel calculations in parallel, by exploiting the multi-core architecture of modern
|
||||
processor. To achieve this easily, we will use the OpenCV @ref cv::parallel_for_ framework.
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_.cpp mandelbrot-parallel
|
||||
|
||||
The first thing is to declare a custom class that inherits from @ref cv::ParallelLoopBody and to override the
|
||||
`virtual void operator ()(const cv::Range& range) const`.
|
||||
|
||||
The range in the `operator ()` represents the subset of pixels that will be treated by an individual thread.
|
||||
This splitting is done automatically to distribute equally the computation load. We have to convert the pixel index coordinate
|
||||
to a 2D `[row, col]` coordinate. Also note that we have to keep a reference on the mat image to be able to modify in-place
|
||||
the image.
|
||||
|
||||
The parallel execution is called with:
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_.cpp mandelbrot-parallel-call
|
||||
|
||||
Here, the range represents the total number of operations to be executed, so the total number of pixels in the image.
|
||||
To set the number of threads, you can use: @ref cv::setNumThreads. You can also specify the number of splitting using the
|
||||
nstripes parameter in @ref cv::parallel_for_. For instance, if your processor has 4 threads, setting `cv::setNumThreads(2)`
|
||||
or setting `nstripes=2` should be the same as by default it will use all the processor threads available but will split the
|
||||
workload only on two threads.
|
||||
|
||||
@note
|
||||
C++ 11 standard allows simplifying the parallel implementation by get rid of the `ParallelMandelbrot` class and replacing it with lambda expression:
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_.cpp mandelbrot-parallel-call-cxx11
|
||||
|
||||
Results
|
||||
-------
|
||||
|
||||
You can find the full tutorial code [here](https://github.com/opencv/opencv/blob/4.x/samples/cpp/tutorial_code/core/how_to_use_OpenCV_parallel_for_/how_to_use_OpenCV_parallel_for_.cpp).
|
||||
The performance of the parallel implementation depends of the type of CPU you have. For instance, on 4 cores / 8 threads
|
||||
CPU, you can expect a speed-up of around 6.9X. There are many factors to explain why we do not achieve a speed-up of almost 8X.
|
||||
Main reasons should be mostly due to:
|
||||
* the overhead to create and manage the threads,
|
||||
* background processes running in parallel,
|
||||
* the difference between 4 hardware cores with 2 logical threads for each core and 8 hardware cores.
|
||||
|
||||
The resulting image produced by the tutorial code (you can modify the code to use more iterations and assign a pixel color
|
||||
depending on the escaped iteration and using a color palette to get more aesthetic images):
|
||||
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
+167
@@ -0,0 +1,167 @@
|
||||
How to use the OpenCV parallel_for_ function to parallelize your code (convolution example) {#tutorial_how_to_use_OpenCV_parallel_for_new}
|
||||
==================================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_file_input_output_with_xml_yml}
|
||||
@prev_tutorial{tutorial_how_to_use_OpenCV_parallel_for_}
|
||||
@next_tutorial{tutorial_univ_intrin}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
The goal of this tutorial is to demonstrate the use of the OpenCV `parallel_for_` framework to easily parallelize your code. To illustrate the concept, we will write a program to perform convolution operation over an image.
|
||||
The full tutorial code is [here](https://github.com/opencv/opencv/blob/4.x/samples/cpp/tutorial_code/core/how_to_use_OpenCV_parallel_for_/how_to_use_OpenCV_parallel_for_new.cpp).
|
||||
|
||||
Precondition
|
||||
----
|
||||
|
||||
### Parallel Frameworks
|
||||
The first precondition is to have OpenCV built with a parallel framework.
|
||||
In OpenCV 4.5, the following parallel frameworks are available in that order:
|
||||
|
||||
* Intel Threading Building Blocks (3rdparty library, should be explicitly enabled)
|
||||
* OpenMP (integrated to compiler, should be explicitly enabled)
|
||||
* APPLE GCD (system wide, used automatically (APPLE only))
|
||||
* Windows RT concurrency (system wide, used automatically (Windows RT only))
|
||||
* Windows concurrency (part of runtime, used automatically (Windows only - MSVC++ >= 10))
|
||||
* Pthreads
|
||||
|
||||
As you can see, several parallel frameworks can be used in the OpenCV library. Some parallel libraries are third party libraries and have to be explicitly enabled in CMake before building, while others are automatically available with the platform (e.g. APPLE GCD).
|
||||
|
||||
### Race Conditions
|
||||
Race conditions occur when more than one thread try to write *or* read and write to a particular memory location simultaneously.
|
||||
Based on that, we can broadly classify algorithms into two categories:-
|
||||
1. Algorithms in which only a single thread writes data to a particular memory location.
|
||||
* In *convolution*, for example, even though multiple threads may read from a pixel at a particular time, only a single thread *writes* to a particular pixel.
|
||||
|
||||
2. Algorithms in which multiple threads may write to a single memory location.
|
||||
* Finding contours, features, etc. Such algorithms may require each thread to add data to a global variable simultaneously. For example, when detecting features, each thread will add features of their respective parts of the image to a common vector, thus creating a race condition.
|
||||
|
||||
Convolution
|
||||
-----------
|
||||
|
||||
We will use the example of performing a convolution to demonstrate the use of `parallel_for_` to parallelize the computation. This is an example of an algorithm which does not lead to a race condition.
|
||||
|
||||
Theory
|
||||
------
|
||||
Convolution is a simple mathematical operation widely used in image processing. Here, we slide a smaller matrix, called the *kernel*, over an image and a sum of the product of pixel values and corresponding values in the kernel gives us the value of the particular pixel in the output (called the anchor point of the kernel). Based on the values in the kernel, we get different results.
|
||||
In the example below, we use a 3x3 kernel (anchored at its center) and convolve over a 5x5 matrix to produce a 3x3 matrix. The size of the output can be altered by padding the input with suitable values.
|
||||
|
||||
|
||||
For more information about different kernels and what they do, look [here](https://en.wikipedia.org/wiki/Kernel_(image_processing))
|
||||
|
||||
For the purpose of this tutorial, we will implement the simplest form of the function which takes a grayscale image (1 channel) and an odd length square kernel and produces an output image.
|
||||
The operation will not be performed in-place.
|
||||
@note We can store a few of the relevant pixels temporarily to make sure we use the original values during the convolution and then do it in-place. However, the purpose of this tutorial is to introduce parallel_for_ function and an inplace implementation may be too complicated.
|
||||
|
||||
Pseudocode
|
||||
-----------
|
||||
|
||||
InputImage src, OutputImage dst, kernel(size n)
|
||||
makeborder(src, n/2)
|
||||
for each pixel (i, j) strictly inside borders, do:
|
||||
{
|
||||
value := 0
|
||||
for k := -n/2 to n/2, do:
|
||||
for l := -n/2 to n/2, do:
|
||||
value += kernel[n/2 + k][n/2 + l]*src[i + k][j + l]
|
||||
|
||||
dst[i][j] := value
|
||||
}
|
||||
|
||||
For an *n-sized kernel*, we will add a border of size *n/2* to handle edge cases.
|
||||
We then run two loops to move along the kernel and add the products to sum
|
||||
|
||||
Implementation
|
||||
--------------
|
||||
|
||||
### Sequential implementation
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp convolution-sequential
|
||||
|
||||
We first make an output matrix(dst) with the same size as src and add borders to the src image(to handle edge cases).
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp convolution-make-borders
|
||||
|
||||
We then sequentially iterate over the pixels in the src image and compute the value over the kernel and the neighbouring pixel values.
|
||||
We then fill value to the corresponding pixel in the dst image.
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp convolution-kernel-loop
|
||||
|
||||
### Parallel implementation
|
||||
|
||||
When looking at the sequential implementation, we can notice that each pixel depends on multiple neighbouring pixels but only one pixel is edited at a time. Thus, to optimize the computation, we can split the image into stripes and parallelly perform convolution on each, by exploiting the multi-core architecture of modern processor. The OpenCV @ref cv::parallel_for_ framework automatically decides how to split the computation efficiently and does most of the work for us.
|
||||
|
||||
@note Although values of a pixel in a particular stripe may depend on pixel values outside the stripe, these are only read only operations and hence will not cause undefined behaviour.
|
||||
|
||||
|
||||
We first declare a custom class that inherits from @ref cv::ParallelLoopBody and override the `virtual void operator ()(const cv::Range& range) const`.
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp convolution-parallel
|
||||
|
||||
The range in the `operator ()` represents the subset of values that will be treated by an individual thread. Based on the requirement, there may be different ways of splitting the range which in turn changes the computation.
|
||||
|
||||
For example, we can either
|
||||
1. Split the entire traversal of the image and obtain the [row, col] coordinate in the following way (as shown in the above code):
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp overload-full
|
||||
|
||||
We would then call the parallel_for_ function in the following way:
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp convolution-parallel-function
|
||||
<br>
|
||||
|
||||
2. Split the rows and compute for each row:
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp overload-row-split
|
||||
|
||||
In this case, we call the parallel_for_ function with a different range:
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp convolution-parallel-function-row
|
||||
|
||||
@note In our case, both implementations perform similarly. Some cases may allow better memory access patterns or other performance benefits.
|
||||
|
||||
To set the number of threads, you can use: @ref cv::setNumThreads. You can also specify the number of splitting using the nstripes parameter in @ref cv::parallel_for_. For instance, if your processor has 4 threads, setting `cv::setNumThreads(2)` or setting `nstripes=2` should be the same as by default it will use all the processor threads available but will split the workload only on two threads.
|
||||
|
||||
@note C++ 11 standard allows simplifying the parallel implementation by getting rid of the `parallelConvolution` class and replacing it with lambda expression:
|
||||
|
||||
@snippet how_to_use_OpenCV_parallel_for_new.cpp convolution-parallel-cxx11
|
||||
|
||||
Results
|
||||
-----------
|
||||
|
||||
The resulting time taken for execution of the two implementations on a
|
||||
* *512x512 input* with a *5x5 kernel*:
|
||||
|
||||
This program shows how to use the OpenCV parallel_for_ function and
|
||||
compares the performance of the sequential and parallel implementations for a
|
||||
convolution operation
|
||||
Usage:
|
||||
./a.out [image_path -- default lena.jpg]
|
||||
|
||||
Sequential Implementation: 0.0953564s
|
||||
Parallel Implementation: 0.0246762s
|
||||
Parallel Implementation(Row Split): 0.0248722s
|
||||
|
||||
<br>
|
||||
|
||||
* *512x512 input with a 3x3 kernel*
|
||||
|
||||
This program shows how to use the OpenCV parallel_for_ function and
|
||||
compares the performance of the sequential and parallel implementations for a
|
||||
convolution operation
|
||||
Usage:
|
||||
./a.out [image_path -- default lena.jpg]
|
||||
|
||||
Sequential Implementation: 0.0301325s
|
||||
Parallel Implementation: 0.0117053s
|
||||
Parallel Implementation(Row Split): 0.0117894s
|
||||
|
||||
The performance of the parallel implementation depends on the type of CPU you have. For instance, on 4 cores - 8 threads CPU, runtime may be 6x to 7x faster than a sequential implementation. There are many factors to explain why we do not achieve a speed-up of 8x:
|
||||
* the overhead to create and manage the threads,
|
||||
* background processes running in parallel,
|
||||
* the difference between 4 hardware cores with 2 logical threads for each core and 8 hardware cores.
|
||||
|
||||
In the tutorial, we used a horizontal gradient filter(as shown in the animation above), which produces an image highlighting the vertical edges.
|
||||
|
||||
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 147 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 64 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
@@ -0,0 +1,201 @@
|
||||
Mask operations on matrices {#tutorial_mat_mask_operations}
|
||||
===========================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_how_to_scan_images}
|
||||
@next_tutorial{tutorial_mat_operations}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Mask operations on matrices are quite simple. The idea is that we recalculate each pixel's value in
|
||||
an image according to a mask matrix (also known as kernel). This mask holds values that will adjust
|
||||
how much influence neighboring pixels (and the current pixel) have on the new pixel value. From a
|
||||
mathematical point of view we make a weighted average, with our specified values.
|
||||
|
||||
Our test case
|
||||
-------------
|
||||
|
||||
Let's consider the issue of an image contrast enhancement method. Basically we want to apply for
|
||||
every pixel of the image the following formula:
|
||||
|
||||
\f[I(i,j) = 5*I(i,j) - [ I(i-1,j) + I(i+1,j) + I(i,j-1) + I(i,j+1)]\f]\f[\iff I(i,j)*M, \text{where }
|
||||
M = \bordermatrix{ _i\backslash ^j & -1 & 0 & +1 \cr
|
||||
-1 & 0 & -1 & 0 \cr
|
||||
0 & -1 & 5 & -1 \cr
|
||||
+1 & 0 & -1 & 0 \cr
|
||||
}\f]
|
||||
|
||||
The first notation is by using a formula, while the second is a compacted version of the first by
|
||||
using a mask. You use the mask by putting the center of the mask matrix (in the upper case noted by
|
||||
the zero-zero index) on the pixel you want to calculate and sum up the pixel values multiplied with
|
||||
the overlapped matrix values. It's the same thing, however in case of large matrices the latter
|
||||
notation is a lot easier to look over.
|
||||
|
||||
Code
|
||||
----
|
||||
|
||||
@add_toggle_cpp
|
||||
You can download this source code from [here
|
||||
](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp) or look in the
|
||||
OpenCV source code libraries sample directory at
|
||||
`samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp`.
|
||||
@include samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
You can download this source code from [here
|
||||
](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java) or look in the
|
||||
OpenCV source code libraries sample directory at
|
||||
`samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java`.
|
||||
@include samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
You can download this source code from [here
|
||||
](https://raw.githubusercontent.com/opencv/opencv/4.x/samples/python/tutorial_code/core/mat_mask_operations/mat_mask_operations.py) or look in the
|
||||
OpenCV source code libraries sample directory at
|
||||
`samples/python/tutorial_code/core/mat_mask_operations/mat_mask_operations.py`.
|
||||
@include samples/python/tutorial_code/core/mat_mask_operations/mat_mask_operations.py
|
||||
@end_toggle
|
||||
|
||||
The Basic Method
|
||||
----------------
|
||||
|
||||
Now let us see how we can make this happen by using the basic pixel access method or by using the
|
||||
**filter2D()** function.
|
||||
|
||||
Here's a function that will do this:
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp basic_method
|
||||
|
||||
At first we make sure that the input images data is in unsigned char format. For this we use the
|
||||
@ref CV_Assert function (macro) that throws an error when the expression inside it is false.
|
||||
@snippet samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp 8_bit
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java basic_method
|
||||
|
||||
At first we make sure that the input images data in unsigned 8 bit format.
|
||||
@snippet samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java 8_bit
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_mask_operations/mat_mask_operations.py basic_method
|
||||
|
||||
At first we make sure that the input images data in unsigned 8 bit format.
|
||||
@code{.py}
|
||||
my_image = cv.cvtColor(my_image, cv.CV_8U)
|
||||
@endcode
|
||||
|
||||
@end_toggle
|
||||
|
||||
We create an output image with the same size and the same type as our input. As you can see in the
|
||||
@ref tutorial_how_to_scan_images_storing "storing" section, depending on the number of channels we may have one or more
|
||||
subcolumns.
|
||||
|
||||
@add_toggle_cpp
|
||||
We will iterate through them via pointers so the total number of elements depends on
|
||||
this number.
|
||||
@snippet samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp create_channels
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java create_channels
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@code{.py}
|
||||
height, width, n_channels = my_image.shape
|
||||
result = np.zeros(my_image.shape, my_image.dtype)
|
||||
@endcode
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_cpp
|
||||
We'll use the plain C [] operator to access pixels. Because we need to access multiple rows at the
|
||||
same time we'll acquire the pointers for each of them (a previous, a current and a next line). We
|
||||
need another pointer to where we're going to save the calculation. Then simply access the right
|
||||
items with the [] operator. For moving the output pointer ahead we simply increase this (with one
|
||||
byte) after each operation:
|
||||
@snippet samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp basic_method_loop
|
||||
|
||||
On the borders of the image the upper notation results inexistent pixel locations (like minus one -
|
||||
minus one). In these points our formula is undefined. A simple solution is to not apply the kernel
|
||||
in these points and, for example, set the pixels on the borders to zeros:
|
||||
|
||||
@snippet samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp borders
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
We need to access multiple rows and columns which can be done by adding or subtracting 1 to the current center (i,j).
|
||||
Then we apply the sum and put the new value in the Result matrix.
|
||||
@snippet samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java basic_method_loop
|
||||
|
||||
On the borders of the image the upper notation results in inexistent pixel locations (like (-1,-1)).
|
||||
In these points our formula is undefined. A simple solution is to not apply the kernel
|
||||
in these points and, for example, set the pixels on the borders to zeros:
|
||||
|
||||
@snippet samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java borders
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
We need to access multiple rows and columns which can be done by adding or subtracting 1 to the current center (i,j).
|
||||
Then we apply the sum and put the new value in the Result matrix.
|
||||
@snippet samples/python/tutorial_code/core/mat_mask_operations/mat_mask_operations.py basic_method_loop
|
||||
@end_toggle
|
||||
|
||||
The filter2D function
|
||||
---------------------
|
||||
|
||||
Applying such filters are so common in image processing that in OpenCV there is a function that
|
||||
will take care of applying the mask (also called a kernel in some places). For this you first need
|
||||
to define an object that holds the mask:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp kern
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java kern
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_mask_operations/mat_mask_operations.py kern
|
||||
@end_toggle
|
||||
|
||||
Then call the **filter2D()** function specifying the input, the output image and the kernel to
|
||||
use:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_mask_operations/mat_mask_operations.cpp filter2D
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_mask_operations/MatMaskOperations.java filter2D
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_mask_operations/mat_mask_operations.py filter2D
|
||||
@end_toggle
|
||||
|
||||
The function even has a fifth optional argument to specify the center of the kernel, a sixth
|
||||
for adding an optional value to the filtered pixels before storing them in K and a seventh one
|
||||
for determining what to do in the regions where the operation is undefined (borders).
|
||||
|
||||
This function is shorter, less verbose and, because there are some optimizations, it is usually faster
|
||||
than the *hand-coded method*. For example in my test while the second one took only 13
|
||||
milliseconds the first took around 31 milliseconds. Quite some difference.
|
||||
|
||||
For example:
|
||||
|
||||
|
||||
|
||||
@add_toggle_cpp
|
||||
Check out an instance of running the program on our [YouTube
|
||||
channel](http://www.youtube.com/watch?v=7PF1tAU9se4) .
|
||||
@youtube{7PF1tAU9se4}
|
||||
@end_toggle
|
||||
@@ -0,0 +1,270 @@
|
||||
Operations with images {#tutorial_mat_operations}
|
||||
======================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_mat_mask_operations}
|
||||
@next_tutorial{tutorial_adding_images}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Input/Output
|
||||
------------
|
||||
|
||||
### Images
|
||||
|
||||
Load an image from a file:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Load an image from a file
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Load an image from a file
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Load an image from a file
|
||||
@end_toggle
|
||||
|
||||
If you read a jpg file, a 3 channel image is created by default. If you need a grayscale image, use:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Load an image from a file in grayscale
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Load an image from a file in grayscale
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Load an image from a file in grayscale
|
||||
@end_toggle
|
||||
|
||||
@note Format of the file is determined by its content (first few bytes). To save an image to a file:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Save image
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Save image
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Save image
|
||||
@end_toggle
|
||||
|
||||
@note Format of the file is determined by its extension.
|
||||
|
||||
@note Use cv::imdecode and cv::imencode to read and write an image from/to memory rather than a file.
|
||||
|
||||
Basic operations with images
|
||||
----------------------------
|
||||
|
||||
### Accessing pixel intensity values
|
||||
|
||||
In order to get pixel intensity value, you have to know the type of an image and the number of
|
||||
channels. Here is an example for a single channel grey scale image (type 8UC1) and pixel coordinates
|
||||
x and y:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Pixel access 1
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Pixel access 1
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Pixel access 1
|
||||
@end_toggle
|
||||
|
||||
C++ version only:
|
||||
intensity.val[0] contains a value from 0 to 255. Note the ordering of x and y. Since in OpenCV
|
||||
images are represented by the same structure as matrices, we use the same convention for both
|
||||
cases - the 0-based row index (or y-coordinate) goes first and the 0-based column index (or
|
||||
x-coordinate) follows it. Alternatively, you can use the following notation (**C++ only**):
|
||||
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Pixel access 2
|
||||
|
||||
Now let us consider a 3 channel image with BGR color ordering (the default format returned by
|
||||
imread):
|
||||
|
||||
**C++ code**
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Pixel access 3
|
||||
|
||||
**Python Python**
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Pixel access 3
|
||||
|
||||
You can use the same method for floating-point images (for example, you can get such an image by
|
||||
running Sobel on a 3 channel image) (**C++ only**):
|
||||
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Pixel access 4
|
||||
|
||||
The same method can be used to change pixel intensities:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Pixel access 5
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Pixel access 5
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Pixel access 5
|
||||
@end_toggle
|
||||
|
||||
There are functions in OpenCV, especially from calib3d module, such as cv::projectPoints, that take an
|
||||
array of 2D or 3D points in the form of Mat. Matrix should contain exactly one column, each row
|
||||
corresponds to a point, matrix type should be 32FC2 or 32FC3 correspondingly. Such a matrix can be
|
||||
easily constructed from `std::vector` (**C++ only**):
|
||||
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Mat from points vector
|
||||
|
||||
One can access a point in this matrix using the same method `Mat::at` (**C++ only**):
|
||||
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Point access
|
||||
|
||||
### Memory management and reference counting
|
||||
|
||||
Mat is a structure that keeps matrix/image characteristics (rows and columns number, data type etc)
|
||||
and a pointer to data. So nothing prevents us from having several instances of Mat corresponding to
|
||||
the same data. A Mat keeps a reference count that tells if data has to be deallocated when a
|
||||
particular instance of Mat is destroyed. Here is an example of creating two matrices without copying
|
||||
data (**C++ only**):
|
||||
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Reference counting 1
|
||||
|
||||
As a result, we get a 32FC1 matrix with 3 columns instead of 32FC3 matrix with 1 column. `pointsMat`
|
||||
uses data from points and will not deallocate the memory when destroyed. In this particular
|
||||
instance, however, developer has to make sure that lifetime of `points` is longer than of `pointsMat`
|
||||
If we need to copy the data, this is done using, for example, cv::Mat::copyTo or cv::Mat::clone:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Reference counting 2
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Reference counting 2
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Reference counting 2
|
||||
@end_toggle
|
||||
|
||||
An empty output Mat can be supplied to each function.
|
||||
Each implementation calls Mat::create for a destination matrix.
|
||||
This method allocates data for a matrix if it is empty.
|
||||
If it is not empty and has the correct size and type, the method does nothing.
|
||||
If however, size or type are different from the input arguments, the data is deallocated (and lost) and a new data is allocated.
|
||||
For example:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Reference counting 3
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Reference counting 3
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Reference counting 3
|
||||
@end_toggle
|
||||
|
||||
### Primitive operations
|
||||
|
||||
There is a number of convenient operators defined on a matrix. For example, here is how we can make
|
||||
a black image from an existing greyscale image `img`
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Set image to black
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Set image to black
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Set image to black
|
||||
@end_toggle
|
||||
|
||||
Selecting a region of interest:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Select ROI
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Select ROI
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Select ROI
|
||||
@end_toggle
|
||||
|
||||
Conversion from color to greyscale:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp BGR to Gray
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java BGR to Gray
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py BGR to Gray
|
||||
@end_toggle
|
||||
|
||||
Change image type from 8UC1 to 32FC1:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp Convert to CV_32F
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java Convert to CV_32F
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py Convert to CV_32F
|
||||
@end_toggle
|
||||
|
||||
### Visualizing images
|
||||
|
||||
It is very useful to see intermediate results of your algorithm during development process. OpenCV
|
||||
provides a convenient way of visualizing images. A 8U image can be shown using:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp imshow 1
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java imshow 1
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py imshow 1
|
||||
@end_toggle
|
||||
|
||||
A call to waitKey() starts a message passing cycle that waits for a key stroke in the "image"
|
||||
window. A 32F image needs to be converted to 8U type. For example:
|
||||
|
||||
@add_toggle_cpp
|
||||
@snippet samples/cpp/tutorial_code/core/mat_operations/mat_operations.cpp imshow 2
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_java
|
||||
@snippet samples/java/tutorial_code/core/mat_operations/MatOperations.java imshow 2
|
||||
@end_toggle
|
||||
|
||||
@add_toggle_python
|
||||
@snippet samples/python/tutorial_code/core/mat_operations/mat_operations.py imshow 2
|
||||
@end_toggle
|
||||
|
||||
@note Here cv::namedWindow is not necessary since it is immediately followed by cv::imshow.
|
||||
Nevertheless, it can be used to change the window properties or when using cv::createTrackbar
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 2.3 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.7 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 5.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.7 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.3 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.3 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.5 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.7 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 21 KiB |
+276
@@ -0,0 +1,276 @@
|
||||
Mat - The Basic Image Container {#tutorial_mat_the_basic_image_container}
|
||||
===============================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@next_tutorial{tutorial_how_to_scan_images}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
We have multiple ways to acquire digital images from the real world: digital cameras, scanners,
|
||||
computed tomography, and magnetic resonance imaging to name a few. In every case what we (humans)
|
||||
see are images. However, when transforming this to our digital devices what we record are numerical
|
||||
values for each of the points of the image.
|
||||
|
||||
|
||||
|
||||
For example in the above image you can see that the mirror of the car is nothing more than a matrix
|
||||
containing all the intensity values of the pixel points. How we get and store the pixels values may
|
||||
vary according to our needs, but in the end all images inside a computer world may be reduced to
|
||||
numerical matrices and other information describing the matrix itself. *OpenCV* is a computer vision
|
||||
library whose main focus is to process and manipulate this information. Therefore, the first thing
|
||||
you need to be familiar with is how OpenCV stores and handles images.
|
||||
|
||||
Mat
|
||||
---
|
||||
|
||||
OpenCV has been around since 2001. In those days the library was built around a *C* interface and to
|
||||
store the image in the memory they used a C structure called *IplImage*. This is the one you'll see
|
||||
in most of the older tutorials and educational materials. The problem with this is that it brings to
|
||||
the table all the minuses of the C language. The biggest issue is the manual memory management. It
|
||||
builds on the assumption that the user is responsible for taking care of memory allocation and
|
||||
deallocation. While this is not a problem with smaller programs, once your code base grows it will
|
||||
be more of a struggle to handle all this rather than focusing on solving your development goal.
|
||||
|
||||
Luckily C++ came around and introduced the concept of classes making easier for the user through
|
||||
automatic memory management (more or less). The good news is that C++ is fully compatible with C so
|
||||
no compatibility issues can arise from making the change. Therefore, OpenCV 2.0 introduced a new C++
|
||||
interface which offered a new way of doing things which means you do not need to fiddle with memory
|
||||
management, making your code concise (less to write, to achieve more). The main downside of the C++
|
||||
interface is that many embedded development systems at the moment support only C. Therefore, unless
|
||||
you are targeting embedded platforms, there's no point to using the *old* methods (unless you're a
|
||||
masochist programmer and you're asking for trouble).
|
||||
|
||||
The first thing you need to know about *Mat* is that you no longer need to manually allocate its
|
||||
memory and release it as soon as you do not need it. While doing this is still a possibility, most
|
||||
of the OpenCV functions will allocate its output data automatically. As a nice bonus if you pass on
|
||||
an already existing *Mat* object, which has already allocated the required space for the matrix,
|
||||
this will be reused. In other words we use at all times only as much memory as we need to perform
|
||||
the task.
|
||||
|
||||
*Mat* is basically a class with two data parts: the matrix header (containing information such as
|
||||
the size of the matrix, the method used for storing, at which address is the matrix stored, and so
|
||||
on) and a pointer to the matrix containing the pixel values (taking any dimensionality depending on
|
||||
the method chosen for storing) . The matrix header size is constant, however the size of the matrix
|
||||
itself may vary from image to image and usually is larger by orders of magnitude.
|
||||
|
||||
OpenCV is an image processing library. It contains a large collection of image processing functions.
|
||||
To solve a computational challenge, most of the time you will end up using multiple functions of the
|
||||
library. Because of this, passing images to functions is a common practice. We should not forget
|
||||
that we are talking about image processing algorithms, which tend to be quite computational heavy.
|
||||
The last thing we want to do is further decrease the speed of your program by making unnecessary
|
||||
copies of potentially *large* images.
|
||||
|
||||
To tackle this issue OpenCV uses a reference counting system. The idea is that each *Mat* object has
|
||||
its own header, however a matrix may be shared between two *Mat* objects by having their matrix
|
||||
pointers point to the same address. Moreover, the copy operators **will only copy the headers** and
|
||||
the pointer to the large matrix, not the data itself.
|
||||
|
||||
@code{.cpp}
|
||||
Mat A, C; // creates just the header parts
|
||||
A = imread(argv[1], IMREAD_COLOR); // here we'll know the method used (allocate matrix)
|
||||
|
||||
Mat B(A); // Use the copy constructor
|
||||
|
||||
C = A; // Assignment operator
|
||||
@endcode
|
||||
|
||||
All the above objects, in the end, point to the same single data matrix and making a modification
|
||||
using any of them will affect all the other ones as well. In practice the different objects just
|
||||
provide different access methods to the same underlying data. Nevertheless, their header parts are
|
||||
different. The real interesting part is that you can create headers which refer to only a subsection
|
||||
of the full data. For example, to create a region of interest (*ROI*) in an image you just create
|
||||
a new header with the new boundaries:
|
||||
@code{.cpp}
|
||||
Mat D (A, Rect(10, 10, 100, 100) ); // using a rectangle
|
||||
Mat E = A(Range::all(), Range(1,3)); // using row and column boundaries
|
||||
@endcode
|
||||
Now you may ask -- if the matrix itself may belong to multiple *Mat* objects, who takes responsibility
|
||||
for cleaning it up when it's no longer needed? The short answer is: the last object that used it.
|
||||
This is handled by using a reference counting mechanism. Whenever somebody copies a header of a
|
||||
*Mat* object, a counter is increased for the matrix. Whenever a header is cleaned, this counter
|
||||
is decreased. When the counter reaches zero the matrix is freed. Sometimes you will want to copy
|
||||
the matrix itself too, so OpenCV provides @ref cv::Mat::clone() and @ref cv::Mat::copyTo() functions.
|
||||
@code{.cpp}
|
||||
Mat F = A.clone();
|
||||
Mat G;
|
||||
A.copyTo(G);
|
||||
@endcode
|
||||
Now modifying *F* or *G* will not affect the matrix pointed to by the *A*'s header. What you need to
|
||||
remember from all this is that:
|
||||
|
||||
- Output image allocation for OpenCV functions is automatic (unless specified otherwise).
|
||||
- You do not need to think about memory management with OpenCV's C++ interface.
|
||||
- The assignment operator and the copy constructor only copy the header.
|
||||
- The underlying matrix of an image may be copied using the @ref cv::Mat::clone() and @ref cv::Mat::copyTo()
|
||||
functions.
|
||||
|
||||
Storing methods
|
||||
-----------------
|
||||
|
||||
This is about how you store the pixel values. You can select the color space and the data type used.
|
||||
The color space refers to how we combine color components in order to code a given color. The
|
||||
simplest one is the grayscale where the colors at our disposal are black and white. The combination
|
||||
of these allows us to create many shades of gray.
|
||||
|
||||
For *colorful* ways we have a lot more methods to choose from. Each of them breaks it down to three
|
||||
or four basic components and we can use the combination of these to create the others. The most
|
||||
popular one is RGB, mainly because this is also how our eye builds up colors. Its base colors are
|
||||
red, green and blue. To code the transparency of a color sometimes a fourth element, alpha (A), is
|
||||
added.
|
||||
|
||||
There are, however, many other color systems, each with their own advantages:
|
||||
|
||||
- RGB is the most common as our eyes use something similar, however keep in mind that OpenCV standard display
|
||||
system composes colors using the BGR color space (red and blue channels are swapped places).
|
||||
- The HSV and HLS decompose colors into their hue, saturation and value/luminance components,
|
||||
which is a more natural way for us to describe colors. You might, for example, dismiss the last
|
||||
component, making your algorithm less sensible to the light conditions of the input image.
|
||||
- YCrCb is used by the popular JPEG image format.
|
||||
- CIE L\*a\*b\* is a perceptually uniform color space, which comes in handy if you need to measure
|
||||
the *distance* of a given color to another color.
|
||||
|
||||
Each of the building components has its own valid domains. This leads to the data type used. How
|
||||
we store a component defines the control we have over its domain. The smallest data type possible is
|
||||
*char*, which means one byte or 8 bits. This may be unsigned (so can store values from 0 to 255) or
|
||||
signed (values from -127 to +127). Although this width, in the case of three components (like RGB), already gives 16
|
||||
million possible colors to represent, we may acquire an even finer control by
|
||||
using the float (4 byte = 32 bit) or double (8 byte = 64 bit) data types for each component.
|
||||
Nevertheless, remember that increasing the size of a component also increases the size of the whole
|
||||
picture in memory.
|
||||
|
||||
Creating a Mat object explicitly
|
||||
----------------------------------
|
||||
|
||||
In the @ref tutorial_display_image tutorial you have already learned how to write a matrix to an image
|
||||
file by using the @ref cv::imwrite() function. However, for debugging purposes it's much more
|
||||
convenient to see the actual values. You can do this using the \<\< operator of *Mat*. Be aware that
|
||||
this only works for two dimensional matrices.
|
||||
|
||||
Although *Mat* works really well as an image container, it is also a general matrix class.
|
||||
Therefore, it is possible to create and manipulate multidimensional matrices. You can create a Mat
|
||||
object in multiple ways:
|
||||
|
||||
- @ref cv::Mat::Mat Constructor
|
||||
|
||||
@snippet mat_the_basic_image_container.cpp constructor
|
||||
|
||||
|
||||
|
||||
For two dimensional and multichannel images we first define their size: row and column count wise.
|
||||
|
||||
Then we need to specify the data type to use for storing the elements and the number of channels
|
||||
per matrix point. To do this we have multiple definitions constructed according to the following
|
||||
convention:
|
||||
@code
|
||||
CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The channel number]
|
||||
@endcode
|
||||
For instance, *CV_8UC3* means we use unsigned char types that are 8 bit long and each pixel has
|
||||
three of these to form the three channels. There are types predefined for up to four channels. The
|
||||
@ref cv::Scalar is four element short vector. Specify it and you can initialize all matrix
|
||||
points with a custom value. If you need more you can create the type with the upper macro, setting
|
||||
the channel number in parenthesis as you can see below.
|
||||
|
||||
- Use C/C++ arrays and initialize via constructor
|
||||
|
||||
@snippet mat_the_basic_image_container.cpp init
|
||||
|
||||
The upper example shows how to create a matrix with more than two dimensions. Specify its
|
||||
dimension, then pass a pointer containing the size for each dimension and the rest remains the
|
||||
same.
|
||||
|
||||
- @ref cv::Mat::create function:
|
||||
|
||||
@snippet mat_the_basic_image_container.cpp create
|
||||
|
||||
|
||||
|
||||
You cannot initialize the matrix values with this construction. It will only reallocate its matrix
|
||||
data memory if the new size will not fit into the old one.
|
||||
|
||||
- MATLAB style initializer: @ref cv::Mat::zeros , @ref cv::Mat::ones , @ref cv::Mat::eye . Specify size and
|
||||
data type to use:
|
||||
|
||||
@snippet mat_the_basic_image_container.cpp matlab
|
||||
|
||||
|
||||
|
||||
- For small matrices you may use initializer lists:
|
||||
|
||||
@snippet mat_the_basic_image_container.cpp list
|
||||
|
||||
|
||||
|
||||
- Create a new header for an existing *Mat* object and @ref cv::Mat::clone or @ref cv::Mat::copyTo it.
|
||||
|
||||
@snippet mat_the_basic_image_container.cpp clone
|
||||
|
||||
|
||||
|
||||
@note
|
||||
You can fill out a matrix with random values using the @ref cv::randu() function. You need to
|
||||
give a lower and upper limit for the random values:
|
||||
@snippet mat_the_basic_image_container.cpp random
|
||||
|
||||
|
||||
Output formatting
|
||||
-----------------
|
||||
|
||||
In the above examples you could see the default formatting option. OpenCV, however, allows you to
|
||||
format your matrix output:
|
||||
|
||||
- Default
|
||||
@snippet mat_the_basic_image_container.cpp out-default
|
||||
|
||||
|
||||
- Python
|
||||
@snippet mat_the_basic_image_container.cpp out-python
|
||||
|
||||
|
||||
- Comma separated values (CSV)
|
||||
@snippet mat_the_basic_image_container.cpp out-csv
|
||||
|
||||
|
||||
- Numpy
|
||||
@snippet mat_the_basic_image_container.cpp out-numpy
|
||||
|
||||
|
||||
- C
|
||||
@snippet mat_the_basic_image_container.cpp out-c
|
||||
|
||||
|
||||
Output of other common items
|
||||
----------------------------
|
||||
|
||||
OpenCV offers support for output of other common OpenCV data structures too via the \<\< operator:
|
||||
|
||||
- 2D Point
|
||||
@snippet mat_the_basic_image_container.cpp out-point2
|
||||
|
||||
|
||||
- 3D Point
|
||||
@snippet mat_the_basic_image_container.cpp out-point3
|
||||
|
||||
|
||||
- std::vector via cv::Mat
|
||||
@snippet mat_the_basic_image_container.cpp out-vector
|
||||
|
||||
|
||||
- std::vector of points
|
||||
@snippet mat_the_basic_image_container.cpp out-vector-points
|
||||
|
||||
|
||||
Most of the samples here have been included in a small console application. You can download it from
|
||||
[here](https://github.com/opencv/opencv/tree/4.x/samples/cpp/tutorial_code/core/mat_the_basic_image_container/mat_the_basic_image_container.cpp)
|
||||
or in the core section of the cpp samples.
|
||||
|
||||
You can also find a quick video demonstration of this on
|
||||
[YouTube](https://www.youtube.com/watch?v=1tibU7vGWpk).
|
||||
|
||||
@youtube{1tibU7vGWpk}
|
||||
@@ -0,0 +1,19 @@
|
||||
The Core Functionality (core module) {#tutorial_table_of_content_core}
|
||||
=====================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
##### Basic
|
||||
- @subpage tutorial_mat_the_basic_image_container
|
||||
- @subpage tutorial_how_to_scan_images
|
||||
- @subpage tutorial_mat_mask_operations
|
||||
- @subpage tutorial_mat_operations
|
||||
- @subpage tutorial_adding_images
|
||||
- @subpage tutorial_basic_linear_transform
|
||||
|
||||
##### Advanced
|
||||
- @subpage tutorial_discrete_fourier_transform
|
||||
- @subpage tutorial_file_input_output_with_xml_yml
|
||||
- @subpage tutorial_how_to_use_OpenCV_parallel_for_
|
||||
- @subpage tutorial_how_to_use_OpenCV_parallel_for_new
|
||||
- @subpage tutorial_univ_intrin
|
||||
@@ -0,0 +1,333 @@
|
||||
Vectorizing your code using Universal Intrinsics {#tutorial_univ_intrin}
|
||||
==================================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_how_to_use_OpenCV_parallel_for_new}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Compatibility | OpenCV >= 4.11 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
The goal of this tutorial is to provide a guide to using the @ref core_hal_intrin feature to vectorize your C++ code for a faster runtime.
|
||||
We'll briefly look into _SIMD intrinsics_ and how to work with wide _registers_, followed by a tutorial on the basic operations using wide registers.
|
||||
|
||||
Theory
|
||||
------
|
||||
|
||||
In this section, we will briefly look into a few concepts to better help understand the functionality.
|
||||
|
||||
### Intrinsics
|
||||
Intrinsics are functions which are separately handled by the compiler. These functions are often optimized to perform in the most efficient ways possible and hence run faster than normal implementations. However, since these functions depend on the compiler, it makes it difficult to write portable applications.
|
||||
|
||||
### SIMD
|
||||
SIMD stands for **Single Instruction, Multiple Data**. SIMD Intrinsics allow the processor to vectorize calculations. The data is stored in what are known as *registers*. A *register* may be *128-bits*, *256-bits* or *512-bits* wide. Each *register* stores **multiple values** of the **same data type**. The size of the register and the size of each value determines the number of values stored in total.
|
||||
|
||||
Depending on what *Instruction Sets* your CPU supports, you may be able to use the different registers. To learn more, look [here](https://en.wikipedia.org/wiki/Instruction_set_architecture)
|
||||
|
||||
### VLA
|
||||
VLA stands for **Vector Length Agnostic** .
|
||||
A mechanism where the register width is determined by the hardware at runtime rather than being fixed at compile time.
|
||||
This allows a single binary to scale its performance across different CPUs within the same architecture (e.g., RVV or SVE).
|
||||
|
||||
Universal Intrinsics
|
||||
--------------------
|
||||
|
||||
OpenCV's universal intrinsics provides an abstraction to SIMD and VLA vectorization methods and allows the user to use intrinsics without the need to write system specific code.
|
||||
Supported SIMD/VLA technologies are detailed in @ref core_hal_intrin .
|
||||
|
||||
**We will now introduce the available structures and functions:**
|
||||
* Register structures
|
||||
* Load and store
|
||||
* Mathematical Operations
|
||||
* Reduce and Mask
|
||||
|
||||
### Register Structures
|
||||
|
||||
The Universal Intrinsics set implements every register as a structure based on the particular SIMD register.
|
||||
All types contain the `nlanes` enumeration which gives the exact number of values that the type can hold. This eliminates the need to hardcode the number of values during implementations.
|
||||
|
||||
@note Each register structure is under the `cv` namespace.
|
||||
|
||||
There are **two types** of registers:
|
||||
|
||||
* **Variable sized registers**: These structures do not have a fixed size and their exact bit length is deduced during compilation, based on the available SIMD capabilities. Consequently, the value of the `nlanes` enum is determined in compile time.
|
||||
<br>
|
||||
|
||||
Each structure follows the following convention:
|
||||
|
||||
v_[type of value][size of each value in bits]
|
||||
|
||||
For instance, **v_uint8 holds 8-bit unsigned integers** and **v_float32 holds 32-bit floating point values**. We then declare a register like we would declare any object in C++
|
||||
|
||||
Based on the available SIMD instruction set, a particular register will hold different number of values.
|
||||
For example: If your computer supports a maximum of 256bit registers,
|
||||
* *v_uint8* will hold 32 8-bit unsigned integers
|
||||
* *v_float64* will hold 4 64-bit floats (doubles)
|
||||
|
||||
v_uint8 a; // a is a register supporting uint8(char) data
|
||||
int n = a.nlanes; // n holds 32
|
||||
|
||||
Available data type and sizes:
|
||||
|Type|Size in bits|
|
||||
|-:|:-|
|
||||
|uint| 8, 16, 32, 64|
|
||||
|int | 8, 16, 32, 64|
|
||||
|float | 32, 64|
|
||||
|
||||
* **Constant sized registers**: These structures have a fixed bit size and hold a constant number of values. We need to know what SIMD instruction set is supported by the system and select compatible registers. Use these only if exact bit length is necessary.
|
||||
<br>
|
||||
|
||||
Each structure follows the convention:
|
||||
|
||||
v_[type of value][size of each value in bits]x[number of values]
|
||||
|
||||
Suppose we want to store
|
||||
* 32-bit(*size in bits*) signed integers in a **128 bit register**. Since the register size is already known, we can find out the *number of data points in register* (*128/32 = 4*):
|
||||
|
||||
v_int32x8 reg1 // holds 8 32-bit signed integers.
|
||||
|
||||
* 64-bit floats in 512 bit register:
|
||||
|
||||
v_float64x8 reg2 // reg2.nlanes = 8
|
||||
|
||||
### Load and Store operations
|
||||
|
||||
Now that we know how registers work, let us look at the functions used for filling these registers with values.
|
||||
|
||||
* **Load**: Load functions allow you to *load* values into a register.
|
||||
* *Constructors* - When declaring a register structure, we can either provide a memory address from where the register will pick up contiguous values, or provide the values explicitly as multiple arguments (Explicit multiple arguments is available only for Constant Sized Registers):
|
||||
|
||||
float ptr[32] = {1, 2, 3 ..., 32}; // ptr is a pointer to a contiguous memory block of 32 floats
|
||||
|
||||
// Variable Sized Registers //
|
||||
int x = v_float32().nlanes; // set x as the number of values the register can hold
|
||||
|
||||
v_float32 reg1(ptr); // reg1 stores first x values according to the maximum register size available.
|
||||
v_float32 reg2(ptr + x); // reg stores the next x values
|
||||
|
||||
// Constant Sized Registers //
|
||||
v_float32x4 reg1(ptr); // reg1 stores the first 4 floats (1, 2, 3, 4)
|
||||
v_float32x4 reg2(ptr + 4); // reg2 stores the next 4 floats (5, 6, 7, 8)
|
||||
|
||||
// Or we can explicitly write down the values.
|
||||
v_float32x4(1, 2, 3, 4);
|
||||
|
||||
<br>
|
||||
* *Load Function* - We can use the load method and provide the memory address of the data:
|
||||
|
||||
float ptr[32] = {1, 2, 3, ..., 32};
|
||||
v_float32 reg_var;
|
||||
reg_var = vx_load(ptr); // loads values from ptr[0] upto ptr[reg_var.nlanes - 1]
|
||||
|
||||
v_float32x4 reg_128;
|
||||
reg_128 = v_load(ptr); // loads values from ptr[0] upto ptr[3]
|
||||
|
||||
v_float32x8 reg_256;
|
||||
reg_256 = v256_load(ptr); // loads values from ptr[0] upto ptr[7]
|
||||
|
||||
v_float32x16 reg_512;
|
||||
reg_512 = v512_load(ptr); // loads values from ptr[0] upto ptr[15]
|
||||
|
||||
@note The load function assumes data is unaligned. If your data is aligned, you may use the `vx_load_aligned()` function.
|
||||
<br>
|
||||
|
||||
* **Store**: Store functions allow you to *store* the values from a register into a particular memory location.
|
||||
* To store values from a register into a memory location, you may use the *v_store()* function:
|
||||
|
||||
float ptr[4];
|
||||
v_store(ptr, reg); // store the first 128 bits(interpreted as 4x32-bit floats) of reg into ptr.
|
||||
<br>
|
||||
@note Ensure **ptr** has the same type as register. You can also cast the register into the proper type before carrying out operations. Simply typecasting the pointer to a particular type will lead wrong interpretation of data.
|
||||
|
||||
### Binary and Unary Operators
|
||||
|
||||
The universal intrinsics set provides element wise binary and unary operations.
|
||||
|
||||
@note Since OpenCV 4.11, C++ operator overloading (e.g., +, ) in Universal Intrinsics has been deprecated in favor of explicit wrapper functions (e.g., v_add, v_mul) to ensure compatibility with VLA architectures.
|
||||
See also: https://github.com/opencv/opencv/issues/27267
|
||||
|
||||
* **Arithmetics**: We can add, subtract, multiply and divide two registers element-wise. The registers must be of the same width and hold the same type. To multiply two registers, for example:
|
||||
|
||||
v_float32 a, b; // {a1, ..., an}, {b1, ..., bn}
|
||||
v_float32 c = v_add(a, b); // {a1 + b1, ..., an + bn}
|
||||
v_flaot32 d = v_mul(a, b); // {a1 * b1, ..., an * bn}
|
||||
|
||||
<br>
|
||||
|
||||
* **Bitwise Logic and Shifts**: We can left shift or right shift the bits of each element of the register. We can also apply bitwise and, or, xor and not operators between two registers element-wise:
|
||||
|
||||
v_int32 as; // {a1, ..., an}
|
||||
v_int32 al = v_shl(as, 2); // {a1 << 2, ..., an << 2}
|
||||
v_int32 bl = v_shr(as, 2); // {a1 >> 2, ..., an >> 2}
|
||||
|
||||
v_int32 a, b;
|
||||
v_int32 a_and_b = v_and(a, b); // {a1 & b1, ..., an & bn}
|
||||
|
||||
<br>
|
||||
|
||||
* **Comparison Operators**: We can compare values between two registers using the v_lt(<), v_gt(>), v_le(<=) , v_ge(>=), v_eq(==) and v_ne(!=). Since each register contains multiple values, we don't get a single bool for these operations. Instead, for true values, all bits are converted to one (0xff for 8 bits, 0xffff for 16 bits, etc), while false values return bits converted to zero.
|
||||
|
||||
// let us consider the following code is run in a 128-bit register
|
||||
v_uint8 a; // a = {0, 1, 2, ..., 13, 14, 15}
|
||||
v_uint8 b; // b = {15, 14, 13, ..., 2, 1, 0}
|
||||
|
||||
v_uint8 c = v_lt(a, b); // c = {255, 255, 255, ..., 0, 0, 0}
|
||||
|
||||
/*
|
||||
let us look at the first 4 values in binary
|
||||
|
||||
a = |00000000|00000001|00000010|00000011|
|
||||
b = |00001111|00001110|00001101|00001100|
|
||||
c = |11111111|11111111|11111111|11111111|
|
||||
|
||||
If we store the values of c and print them as integers, we will get 255 for true values and 0 for false values.
|
||||
*/
|
||||
---
|
||||
// In a computer supporting 256-bit registers
|
||||
v_int32 a; // a = {1, 2, 3, 4, 5, 6, 7, 8}
|
||||
v_int32 b; // b = {8, 7, 6, 5, 4, 3, 2, 1}
|
||||
|
||||
v_int32 c = v_lt(a, b); // c = {-1, -1, -1, -1, 0, 0, 0, 0}
|
||||
|
||||
/*
|
||||
The true values are 0xffffffff, which in signed 32-bit integer representation is equal to -1.
|
||||
*/
|
||||
<br>
|
||||
|
||||
* **Min/Max operations**: We can use the *v_min()* and *v_max()* functions to return registers containing element-wise min, or max, of the two registers:
|
||||
|
||||
v_int32 a; // {a1, ..., an}
|
||||
v_int32 b; // {b1, ..., bn}
|
||||
|
||||
v_int32 mn = v_min(a, b); // {min(a1, b1), ..., min(an, bn)}
|
||||
v_int32 mx = v_max(a, b); // {max(a1, b1), ..., max(an, bn)}
|
||||
<br>
|
||||
|
||||
@note Comparison and Min/Max operators are not available for 64 bit integers. Bitwise shift and logic operators are available only for integer values. Bitwise shift is available only for 16, 32 and 64 bit registers.
|
||||
|
||||
### Reduce and Mask
|
||||
|
||||
* **Reduce Operations**: The *v_reduce_min()*, *v_reduce_max()* and *v_reduce_sum()* return a single value denoting the min, max or sum of the entire register:
|
||||
|
||||
v_int32 a; // a = {a1, ..., a4}
|
||||
int mn = v_reduce_min(a); // mn = min(a1, ..., an)
|
||||
int sum = v_reduce_sum(a); // sum = a1 + ... + an
|
||||
<br>
|
||||
|
||||
* **Mask Operations**: Mask operations allow us to replicate conditionals in wide registers. These include:
|
||||
* *v_check_all()* - Returns a bool, which is true if all the values in the register are less than zero.
|
||||
* *v_check_any()* - Returns a bool, which is true if any value in the register is less than zero.
|
||||
* *v_select()* - Returns a register, which blends two registers, based on a mask.
|
||||
|
||||
v_uint8 a; // {a1, .., an}
|
||||
v_uint8 b; // {b1, ..., bn}
|
||||
|
||||
v_int32x4 mask: // {0xff, 0, 0, 0xff, ..., 0xff, 0}
|
||||
|
||||
v_uint8 Res = v_select(mask, a, b) // {a1, b2, b3, a4, ..., an-1, bn}
|
||||
|
||||
/*
|
||||
"Res" will contain the value from "a" if mask is true (all bits set to 1),
|
||||
and value from "b" if mask is false (all bits set to 0)
|
||||
|
||||
We can use comparison operators to generate mask and v_select to obtain results based on conditionals.
|
||||
It is common to set all values of b to 0. Thus, v_select will give values of "a" or 0 based on the mask.
|
||||
*/
|
||||
|
||||
## Demonstration
|
||||
In the following section, we will vectorize a simple convolution function for single channel and compare the results to a scalar implementation.
|
||||
@note Not all algorithms are improved by manual vectorization. In fact, in certain cases, the compiler may *autovectorize* the code, thus producing faster results for scalar implementations.
|
||||
|
||||
You may learn more about convolution from the previous tutorial. We use the same naive implementation from the previous tutorial and compare it to the vectorized version.
|
||||
|
||||
The full tutorial code is [here](https://github.com/opencv/opencv/tree/4.x/samples/cpp/tutorial_code/core/univ_intrin/univ_intrin.cpp).
|
||||
|
||||
### Vectorizing Convolution
|
||||
|
||||
We will first implement a 1-D convolution and then vectorize it. The 2-D vectorized convolution will perform 1-D convolution across the rows to produce the correct results.
|
||||
|
||||
#### 1-D Convolution: Scalar
|
||||
@snippet univ_intrin.cpp convolution-1D-scalar
|
||||
|
||||
1. We first set up variables and make a border on both sides of the src matrix, to take care of edge cases.
|
||||
@snippet univ_intrin.cpp convolution-1D-border
|
||||
|
||||
2. For the main loop, we select an index *i* and offset it on both sides along with the kernel, using the k variable. We store the value in *value* and add it to the *dst* matrix.
|
||||
@snippet univ_intrin.cpp convolution-1D-scalar-main
|
||||
|
||||
#### 1-D Convolution: Vector
|
||||
|
||||
We will now look at the vectorized version of 1-D convolution.
|
||||
@snippet univ_intrin.cpp convolution-1D-vector
|
||||
|
||||
1. In our case, the kernel is a float. Since the kernel's datatype is the largest, we convert src to float32, forming *src_32*. We also make a border like we did for the naive case.
|
||||
@snippet univ_intrin.cpp convolution-1D-convert
|
||||
|
||||
2. Now, for each column in the *kernel*, we calculate the scalar product of the value with all *window* vectors of length `step`. We add these values to the already stored values in ans
|
||||
@snippet univ_intrin.cpp convolution-1D-main
|
||||
|
||||
* We declare a pointer to the src_32 and kernel and run a loop for each kernel element
|
||||
@snippet univ_intrin.cpp convolution-1D-main-h1
|
||||
|
||||
* We load a register with the current kernel element. A window is shifted from *0* to *len - step* and its product with the kernel_wide array is added to the values stored in *ans*. We store the values back into *ans*
|
||||
@snippet univ_intrin.cpp convolution-1D-main-h2
|
||||
|
||||
* Since the length might not be divisible by steps, we take care of the remaining values directly. The number of *tail* values will always be less than *step* and will not affect the performance significantly. We store all the values to *ans* which is a float pointer. We can also directly store them in a `Mat` object
|
||||
@snippet univ_intrin.cpp convolution-1D-main-h3
|
||||
|
||||
* Here is an iterative example:
|
||||
|
||||
For example:
|
||||
kernel: {k1, k2, k3}
|
||||
src: ...|a1|a2|a3|a4|...
|
||||
|
||||
|
||||
iter1:
|
||||
for each idx i in (0, len), 'step' idx at a time
|
||||
kernel_wide: |k1|k1|k1|k1|
|
||||
window: |a0|a1|a2|a3|
|
||||
ans: ...| 0| 0| 0| 0|...
|
||||
sum = ans + window * kernel_wide
|
||||
= |a0 * k1|a1 * k1|a2 * k1|a3 * k1|
|
||||
|
||||
iter2:
|
||||
kernel_wide: |k2|k2|k2|k2|
|
||||
window: |a1|a2|a3|a4|
|
||||
ans: ...|a0 * k1|a1 * k1|a2 * k1|a3 * k1|...
|
||||
sum = ans + window * kernel_wide
|
||||
= |a0 * k1 + a1 * k2|a1 * k1 + a2 * k2|a2 * k1 + a3 * k2|a3 * k1 + a4 * k2|
|
||||
|
||||
iter3:
|
||||
kernel_wide: |k3|k3|k3|k3|
|
||||
window: |a2|a3|a4|a5|
|
||||
ans: ...|a0 * k1 + a1 * k2|a1 * k1 + a2 * k2|a2 * k1 + a3 * k2|a3 * k1 + a4 * k2|...
|
||||
sum = sum + window * kernel_wide
|
||||
= |a0*k1 + a1*k2 + a2*k3|a1*k1 + a2*k2 + a3*k3|a2*k1 + a3*k2 + a4*k3|a3*k1 + a4*k2 + a5*k3|
|
||||
|
||||
|
||||
@note The function parameters also include *row*, *rowk* and *len*. These values are used when using the function as an intermediate step of 2-D convolution
|
||||
|
||||
#### 2-D Convolution
|
||||
|
||||
Suppose our kernel has *ksize* rows. To compute the values for a particular row, we compute the 1-D convolution of the previous *ksize/2* and the next *ksize/2* rows, with the corresponding kernel row. The final values is simply the sum of the individual 1-D convolutions
|
||||
@snippet univ_intrin.cpp convolution-2D
|
||||
|
||||
1. We first initialize variables and make a border above and below the *src* matrix. The left and right sides are handled by the 1-D convolution function.
|
||||
@snippet univ_intrin.cpp convolution-2D-init
|
||||
|
||||
2. For each row, we calculate the 1-D convolution of the rows above and below it. we then add the values to the *dst* matrix.
|
||||
@snippet univ_intrin.cpp convolution-2D-main
|
||||
|
||||
3. We finally convert the *dst* matrix to a *8-bit* `unsigned char` matrix
|
||||
@snippet univ_intrin.cpp convolution-2D-conv
|
||||
|
||||
Results
|
||||
-------
|
||||
|
||||
In the tutorial, we used a horizontal gradient kernel. We obtain the same output image for both methods.
|
||||
|
||||
Improvement in runtime varies and will depend on the SIMD capabilities available in your CPU.
|
||||
Reference in New Issue
Block a user