chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,224 @@
|
||||
---
|
||||
id: doris-init
|
||||
title: Use Apache Doris to Store Metrics and Logs Data (Optional)
|
||||
sidebar_label: Metrics/Logs Store Doris
|
||||
---
|
||||
|
||||
Apache HertzBeat's historical data storage relies on the time series database, you can choose one of them to install and initialize, or not to install (note ⚠️ but it is strongly recommended to configure in the production environment)
|
||||
|
||||
> It is recommended to use Greptime as metrics storage.
|
||||
|
||||
Apache Doris is an MPP-based real-time analytics database. In HertzBeat, Doris can be used to store:
|
||||
|
||||
- Metrics history data (`hzb_history`)
|
||||
- Log data (`hzb_log`)
|
||||
|
||||
**⚠️ If you do not configure a time-series database, only the last hour of historical data is retained.**
|
||||
|
||||
> If you already have a Doris cluster, skip directly to the YML configuration section.
|
||||
|
||||
## Install Doris (Optional)
|
||||
|
||||
You can deploy Doris by package or Docker. For production, follow the official deployment guide:
|
||||
|
||||
- Doris docs: [Quick Start](https://doris.apache.org/docs/4.x/gettingStarted/quick-start/)
|
||||
|
||||
For HertzBeat integration, ensure at least:
|
||||
|
||||
- FE MySQL service port is reachable (default `9030`)
|
||||
- FE HTTP service port is reachable (default `8030`)
|
||||
|
||||
### Note: Install MySQL JDBC Driver Jar
|
||||
|
||||
- Download MySQL JDBC driver jar, for example mysql-connector-java-8.1.0.jar. [https://mvnrepository.com/artifact/com.mysql/mysql-connector-j/8.1.0](https://mvnrepository.com/artifact/com.mysql/mysql-connector-j/8.1.0)
|
||||
- Copy this jar to the `ext-lib` directory in HertzBeat installation directory.
|
||||
- Restart HertzBeat service.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. Doris FE and BE are running normally.
|
||||
2. HertzBeat can access:
|
||||
- FE MySQL port (default `9030`) for metadata/query
|
||||
- FE HTTP port (default `8030`) for Stream Load
|
||||
3. The configured Doris user has permission to create database/table and insert/query data.
|
||||
|
||||
### Configure Doris in HertzBeat `application.yml`
|
||||
|
||||
1. Edit `hertzbeat/config/application.yml`.
|
||||
|
||||
For Docker deployment, mount the config file from host.
|
||||
For package deployment, modify `hertzbeat/config/application.yml` directly.
|
||||
|
||||
2. Configure `warehouse.store.doris` (Production Environment Recommended using Stream Load Mode):
|
||||
|
||||
```yaml
|
||||
warehouse:
|
||||
store:
|
||||
doris:
|
||||
enabled: true
|
||||
# FE MySQL endpoint
|
||||
url: jdbc:mysql://127.0.0.1:9030
|
||||
username: root
|
||||
password:
|
||||
|
||||
table-config:
|
||||
# Enable dynamic partition for automatic expiration
|
||||
enable-partition: true
|
||||
# HOUR / DAY / MONTH
|
||||
partition-time-unit: DAY
|
||||
# Number of history partitions to keep
|
||||
partition-retention-days: 30
|
||||
# Number of future partitions to pre-create
|
||||
partition-future-days: 3
|
||||
buckets: 8
|
||||
replication-num: 3
|
||||
|
||||
pool-config:
|
||||
minimum-idle: 5
|
||||
maximum-pool-size: 20
|
||||
connection-timeout: 30000
|
||||
|
||||
write-config:
|
||||
# Strongly recommend stream mode in production for high throughput
|
||||
write-mode: stream
|
||||
batch-size: 1000
|
||||
flush-interval: 5
|
||||
stream-load-config:
|
||||
# FE HTTP port
|
||||
http-port: ":8030"
|
||||
timeout: 60
|
||||
max-bytes-per-batch: 10485760
|
||||
# For complex networks (K8s/cross-domain): direct / public / private
|
||||
redirect-policy: ""
|
||||
```
|
||||
|
||||
### Switching to Stream Load Mode
|
||||
|
||||
#### Production Environment Configuration
|
||||
|
||||
For production deployments, **strongly recommend using Stream Load mode** to ensure high-performance large-scale writes. Stream Load writes directly to Doris storage layer, providing better throughput improvement compared to JDBC mode.
|
||||
|
||||
#### Pre-Switch Checklist
|
||||
|
||||
1. **Network Reachability**
|
||||
- Ensure HertzBeat can access Doris FE HTTP port (default `8030`)
|
||||
- If direct connection is not possible, configure BE endpoint labels in Doris
|
||||
|
||||
2. **Special Configuration for Complex Network Scenarios**
|
||||
|
||||
In K8s, cross-domain, or load-balanced environments, Stream Load's redirect mechanism requires special attention:
|
||||
- FE redirects requests to an available BE, which must be reachable from HertzBeat
|
||||
- Control returned BE address type via `redirect-policy`:
|
||||
- `direct`: Direct BE IP connection
|
||||
- `public`: Use public IP (cloud environments)
|
||||
- `private`: Use private IP (private networks)
|
||||
- Leave empty to use Doris default policy
|
||||
|
||||
Reference: [Doris Stream Load in Complex Networks](https://doris.apache.org/zh-CN/docs/4.x/data-operate/import/load-internals/stream-load-in-complex-network)
|
||||
|
||||
#### Switching Steps
|
||||
|
||||
1. **Modify Configuration File**
|
||||
|
||||
Edit `hertzbeat/config/application.yml` and change `write-mode` to `stream`:
|
||||
|
||||
```yaml
|
||||
warehouse:
|
||||
store:
|
||||
doris:
|
||||
write-config:
|
||||
write-mode: stream # Change here: from jdbc to stream
|
||||
stream-load-config:
|
||||
http-port: ":8030"
|
||||
timeout: 60
|
||||
max-bytes-per-batch: 10485760
|
||||
redirect-policy: "" # Configure if complex network
|
||||
```
|
||||
|
||||
2. **Restart HertzBeat Service**
|
||||
|
||||
3. **Verify Successful Switch**
|
||||
|
||||
Check HertzBeat logs for Stream Load messages
|
||||
|
||||
#### Common Switching Questions
|
||||
|
||||
**Q: Do I need to rebuild tables after switching?**
|
||||
|
||||
A: No. Stream Load and JDBC modes use the same table structure, fully compatible.
|
||||
|
||||
**Q: Will data be lost when switching from JDBC to Stream Load?**
|
||||
|
||||
A: No. Both write modes are independent, historical data remains unchanged.
|
||||
|
||||
**Q: How do I rollback if Stream Load fails?**
|
||||
|
||||
A: If the stream processing fails, it will automatically try to use the jdbc mode for fallback writing
|
||||
|
||||
**Q: Still getting timeouts in cross-network setup with redirect-policy configured?**
|
||||
|
||||
A: Possible causes:
|
||||
|
||||
- Returned BE address under current `redirect-policy` setting is unreachable
|
||||
- Try different `redirect-policy` values (`direct` / `public` / `private`)
|
||||
- Contact Doris admin to verify BE endpoint label configuration
|
||||
|
||||
### Parameter Notes
|
||||
|
||||
| Parameter | Description |
|
||||
| --- | --- |
|
||||
| `enabled` | Enable/disable Doris storage |
|
||||
| `url` | Doris FE MySQL JDBC endpoint |
|
||||
| `table-config.enable-partition` | Enable dynamic partition and automatic expiration |
|
||||
| `table-config.partition-time-unit` | Partition granularity: `HOUR` / `DAY` / `MONTH` |
|
||||
| `table-config.partition-retention-days` | Number of partitions retained |
|
||||
| `table-config.partition-future-days` | Number of future partitions pre-created |
|
||||
| `table-config.buckets` | Bucket count for table distribution |
|
||||
| `table-config.replication-num` | Replica count |
|
||||
| `write-config.write-mode` | `jdbc` or `stream` |
|
||||
| `write-config.batch-size` | Write batch size |
|
||||
| `write-config.flush-interval` | Flush interval in seconds |
|
||||
| `stream-load-config.http-port` | Doris FE HTTP port for Stream Load |
|
||||
| `stream-load-config.timeout` | Stream Load timeout in seconds |
|
||||
| `stream-load-config.max-bytes-per-batch` | Max bytes per stream-load batch |
|
||||
| `stream-load-config.redirect-policy` | Redirect policy for FE->BE endpoint selection: `direct` / `public` / `private` |
|
||||
|
||||
### Restart HertzBeat
|
||||
|
||||
After configuration changes, restart HertzBeat to apply Doris storage settings.
|
||||
|
||||
### Verify Doris Storage Is Working
|
||||
|
||||
1. Check HertzBeat logs for Stream Load success messages.
|
||||
2. Verify database and tables:
|
||||
|
||||
```sql
|
||||
SHOW CREATE TABLE hertzbeat.hzb_history;
|
||||
SHOW CREATE TABLE hertzbeat.hzb_log;
|
||||
```
|
||||
|
||||
1. If partition is enabled, check dynamic partition state:
|
||||
|
||||
```sql
|
||||
SHOW DYNAMIC PARTITION TABLES FROM hertzbeat;
|
||||
SHOW PARTITIONS FROM hertzbeat.hzb_history;
|
||||
SHOW PARTITIONS FROM hertzbeat.hzb_log;
|
||||
```
|
||||
|
||||
### FAQ
|
||||
|
||||
1. Do I need to enable partition to use bucket distribution?
|
||||
|
||||
> No. Buckets work with or without dynamic partition. `enable-partition` only controls dynamic partition and automatic expiration.
|
||||
|
||||
2. Can I use Doris for both metrics and logs at the same time?
|
||||
|
||||
> Yes. HertzBeat writes metrics into `hzb_history` and logs into `hzb_log` with the same Doris datasource configuration.
|
||||
|
||||
3. If I change partition/bucket settings in `application.yml`, will existing tables auto-update?
|
||||
|
||||
> No. Existing Doris table DDL is not automatically altered. For schema-level changes, apply DDL manually or recreate tables.
|
||||
|
||||
4. Is stream load compression enabled?
|
||||
|
||||
> Current implementation uses JSON stream load by default.
|
||||
Reference in New Issue
Block a user