wehub-resource-sync
138 lines
13 KiB
Markdown
138 lines
13 KiB
Markdown
# Flink On Yarn Monitoring
|
|
|
|
> Measurement and monitoring of general metrics for Flink stream engine in Yarn running mode.
|
|
|
|
## Configuration Parameters
|
|
|
|
| Parameter Name | Parameter Help Description |
|
|
|---------------------|---------------------------------------------------------------------------------------------------------------------|
|
|
| Monitoring Host | The monitored peer's IPV4, IPV6, or domain name. Note ⚠️ do not include protocol headers (e.g., https://, http://). |
|
|
| Task Name | The name identifying this monitoring task. The name must be unique. |
|
|
| Yarn Port | The Yarn port, corresponding to the port in `yarn.resourcemanager.webapp.address`. |
|
|
| Query Timeout | The timeout for JVM connections, in milliseconds, default is 3000 ms. |
|
|
| Enable SSL | Whether to enable SSL |
|
|
| Username | Connection username |
|
|
| Password | Connection password |
|
|
| Monitoring Interval | Interval for periodic data collection, in seconds, minimum interval is 30 seconds. |
|
|
| Tags | Used for categorizing and managing monitoring resources. |
|
|
| Description | Additional notes and descriptions for this monitoring task. Users can add notes here. |
|
|
|
|
### Collected Metrics
|
|
|
|
#### Metrics Set: JobManager Metrics
|
|
|
|
| Metric Name | Metric Unit | Metric Help Description |
|
|
|-------------------------------------------------------|-------------|----------------------------------------------------|
|
|
| Status.JVM.Memory.NonHeap.Committed | Bytes | Non-heap memory committed |
|
|
| Status.JVM.Memory.Mapped.TotalCapacity | Bytes | Total capacity of mapped memory |
|
|
| Status.JVM.Memory.NonHeap.Used | Bytes | Non-heap memory used |
|
|
| Status.JVM.Memory.Metaspace.Max | Bytes | Maximum capacity of metaspace |
|
|
| Status.JVM.GarbageCollector.G1_Old_Generation.Count | Count | Count of old generation garbage collections |
|
|
| Status.JVM.Memory.Direct.MemoryUsed | Bytes | Direct memory used |
|
|
| Status.JVM.Memory.Mapped.MemoryUsed | Bytes | Mapped memory used |
|
|
| Status.JVM.GarbageCollector.G1_Young_Generation.Count | Count | Count of young generation garbage collections |
|
|
| Status.JVM.Memory.Direct.TotalCapacity | Bytes | Total capacity of direct memory |
|
|
| Status.JVM.GarbageCollector.G1_Old_Generation.Time | ms | Time spent on old generation garbage collections |
|
|
| Status.JVM.Memory.Heap.Committed | Bytes | Heap memory committed |
|
|
| Status.JVM.Memory.Mapped.Count | Count | Count of mapped memory |
|
|
| Status.JVM.Memory.Metaspace.Used | Bytes | Metaspace memory used |
|
|
| Status.JVM.Memory.Direct.Count | Count | Count of direct memory |
|
|
| Status.JVM.Memory.Heap.Used | Bytes | Heap memory used |
|
|
| Status.JVM.Memory.Heap.Max | Bytes | Maximum capacity of heap memory |
|
|
| Status.JVM.GarbageCollector.G1_Young_Generation.Time | ms | Time spent on young generation garbage collections |
|
|
| Status.JVM.Memory.NonHeap.Max | Bytes | Maximum capacity of non-heap memory |
|
|
|
|
#### Metrics Set: JobManager Config
|
|
|
|
| Metric Name | Metric Unit | Metric Help Description |
|
|
|--------------------------------------------|-------------|----------------------------------------------------|
|
|
| internal.jobgraph-path | - | Internal job graph path |
|
|
| env.java.home | - | Java environment path |
|
|
| classloader.check-leaked-classloader | - | Whether to check for leaked class loaders |
|
|
| env.java.opts | - | Java options |
|
|
| high-availability.cluster-id | - | High availability cluster ID |
|
|
| jobmanager.rpc.address | - | JobManager's RPC address |
|
|
| jobmanager.memory.jvm-overhead.min | Bytes | Minimum JVM overhead for JobManager |
|
|
| jobmanager.web.port | Port | JobManager's Web port |
|
|
| webclient.port | Port | Web client port |
|
|
| execution.savepoint.ignore-unclaimed-state | - | Whether to ignore unclaimed state |
|
|
| io.tmp.dirs | Path | Temporary file directories |
|
|
| parallelism.default | - | Default parallelism |
|
|
| taskmanager.memory.fraction | - | TaskManager memory fraction |
|
|
| taskmanager.numberOfTaskSlots | - | Number of task slots for TaskManager |
|
|
| yarn.application.name | - | Yarn application name |
|
|
| taskmanager.heap.mb | MB | Heap memory size for TaskManager |
|
|
| taskmanager.memory.process.size | GB | Process memory size for TaskManager |
|
|
| web.port | Port | Web port |
|
|
| classloader.resolve-order | - | Class loader resolve order |
|
|
| jobmanager.heap.mb | MB | Heap memory size for JobManager |
|
|
| jobmanager.memory.off-heap.size | Bytes | Off-heap memory size for JobManager |
|
|
| state.backend.incremental | - | Whether the state backend is incremental |
|
|
| execution.target | - | Execution target |
|
|
| jobmanager.memory.process.size | GB | Process memory size for JobManager |
|
|
| web.tmpdir | Path | Web temporary directory |
|
|
| yarn.ship-files | Path | Yarn shipped files |
|
|
| jobmanager.rpc.port | Port | JobManager's RPC port |
|
|
| internal.io.tmpdirs.use-local-default | - | Whether to use local default temporary directories |
|
|
| execution.checkpointing.interval | ms | Checkpointing interval |
|
|
| execution.attached | - | Whether to execute attached |
|
|
| internal.cluster.execution-mode | - | Internal cluster execution mode |
|
|
| execution.shutdown-on-attached-exit | - | Whether to shutdown on attached exit |
|
|
| pipeline.jars | Path | Pipeline JAR files |
|
|
| rest.address | - | REST address |
|
|
| state.backend | - | State backend type |
|
|
| jobmanager.memory.jvm-metaspace.size | Bytes | JVM metaspace size for JobManager |
|
|
| $internal.deployment.config-dir | Path | Internal deployment configuration directory |
|
|
| $internal.yarn.log-config-file | Path | Internal Yarn log configuration file path |
|
|
| jobmanager.memory.heap.size | Bytes | Heap memory size for JobManager |
|
|
| state.checkpoints.dir | Path | State checkpoints directory |
|
|
| jobmanager.memory.jvm-overhead.max | Bytes | Maximum JVM overhead for JobManager |
|
|
|
|
#### TaskManager Metrics
|
|
|
|
| Metric Name | Metric Unit | Metric Help Description |
|
|
|---------------------------------------|-------------|---------------------------------------------------|
|
|
| Container ID | - | Container ID for uniquely identifying a container |
|
|
| Path | - | Container path |
|
|
| Data Port | Port | Data transmission port |
|
|
| JMX Port | Port | JMX (Java Management Extensions) port |
|
|
| Last Heartbeat | Timestamp | Last heartbeat time |
|
|
| All Slots | Count | Total number of task slots in the container |
|
|
| Free Slots | Count | Number of free task slots in the container |
|
|
| totalResourceCpuCores | Cores | Total number of CPU cores in the container |
|
|
| totalResourceTaskHeapMemory | MB | Total task heap memory size in the container |
|
|
| totalResourceManagedMemory | MB | Total managed memory size in the container |
|
|
| totalResourceNetworkMemory | MB | Total network memory size in the container |
|
|
| freeResourceCpuCores | Cores | Number of free CPU cores in the container |
|
|
| freeResourceTaskHeapMemory | MB | Free task heap memory size in the container |
|
|
| freeResourceTaskOffHeapMemory | MB | Free task off-heap memory size in the container |
|
|
| freeResourceManagedMemory | MB | Free managed memory size in the container |
|
|
| freeResourceNetworkMemory | MB | Free network memory size in the container |
|
|
| CPU Cores | Cores | Number of CPU cores |
|
|
| Physical MEM | MB | Size of physical memory |
|
|
| JVM Heap Size | MB | Size of JVM heap memory |
|
|
| Flink Managed MEM | MB | Size of Flink managed memory |
|
|
| Framework Heap | MB | Size of framework heap memory |
|
|
| Task Heap | MB | Size of task heap memory |
|
|
| Framework Off-Heap | MB | Size of framework off-heap memory |
|
|
| memoryConfigurationTaskOffHeap | Bytes | Task off-heap memory configuration |
|
|
| Network | MB | Network memory configuration |
|
|
| Managed Memory | MB | Managed memory configuration |
|
|
| JVM Metaspace | MB | Size of JVM metaspace |
|
|
| JVM Overhead | MB | JVM overhead |
|
|
| memoryConfigurationTotalFlinkMemory | Bytes | Total Flink memory configuration |
|
|
| memoryConfigurationTotalProcessMemory | Bytes | Total process memory configuration |
|
|
|
|
#### TaskManager Status Metrics
|
|
|
|
| Metric Name | Metric Unit | Metric Help Description |
|
|
|-----------------------------------|-------------|------------------------------------|
|
|
| Status.Shuffle.Netty.TotalMemory | MB | Total memory used by Netty Shuffle |
|
|
| Status.Flink.Memory.Managed.Used | MB | Managed memory used by Flink |
|
|
| Status.JVM.Memory.Metaspace.Used | MB | Used JVM metaspace memory |
|
|
| Status.JVM.Memory.Metaspace.Max | MB | Maximum JVM metaspace memory |
|
|
| Status.JVM.Memory.Heap.Used | MB | Used JVM heap memory |
|
|
| Status.JVM.Memory.Heap.Max | MB | Maximum JVM heap memory |
|
|
| Status.Flink.Memory.Managed.Total | MB | Total managed memory by Flink |
|
|
| Status.Shuffle.Netty.UsedMemory | MB | Used memory by Netty Shuffle |
|