gitleaks--gitleaks/README.md

Note

本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
English · 原始项目 · 上游 README
原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。

Gitleaks

┌─○───┐
│ │╲  │
│ │ ○ │
│ ○ ░ │
└─░───┘

Warning

Gitleaks 功能已完备。我不会再将新功能合并进 Gitleaks。未来版本将仅提供安全补丁。我正在将精力转向 Betterleaks

Gitleaks 是一款用于检测 Git 仓库、文件以及通过 stdin 等方式提交给它的其他内容中密码、API 密钥和令牌等机密信息的工具。若想进一步了解检测引擎的工作原理，请参阅这篇博客：Regex is (almost) all you need.

➜  ~/code(master) gitleaks git -v

    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks


Finding:     "export BUNDLE_ENTERPRISE__CONTRIBSYS__COM=cafebabe:deadbeef",
Secret:      cafebabe:deadbeef
RuleID:      sidekiq-secret
Entropy:     2.609850
File:        cmd/generate/config/rules/sidekiq.go
Line:        23
Commit:      cd5226711335c68be1e720b318b7bc3135a30eb2
Author:      John
Email:       john@users.noreply.github.com
Date:        2022-08-03T12:31:40Z
Fingerprint: cd5226711335c68be1e720b318b7bc3135a30eb2:cmd/generate/config/rules/sidekiq.go:sidekiq-secret:23

入门

可通过 Homebrew、Docker 或 Go 安装 Gitleaks。许多主流平台和操作系统也可在 发布页面. 获取其二进制版本。此外，Gitleaks 可直接在仓库中配置为 pre-commit 钩子，也可通过 Gitleaks-Action. 实现为 GitHub Action。

安装

# MacOS
brew install gitleaks

# Docker (DockerHub)
docker pull zricethezav/gitleaks:latest
docker run -v ${path_to_host_folder_to_scan}:/path zricethezav/gitleaks:latest [COMMAND] [OPTIONS] [SOURCE_PATH]

# Docker (ghcr.io)
docker pull ghcr.io/gitleaks/gitleaks:latest
docker run -v ${path_to_host_folder_to_scan}:/path ghcr.io/gitleaks/gitleaks:latest [COMMAND] [OPTIONS] [SOURCE_PATH]

# From Source (make sure `go` is installed)
git clone https://github.com/gitleaks/gitleaks.git
cd gitleaks
make build

Pre-Commit

1. 从 https://pre-commit.com/#install 安装 pre-commit

2. 在仓库根目录创建 .pre-commit-config.yaml 文件，内容如下：

   repos:
     - repo: https://github.com/gitleaks/gitleaks
       rev: v8.24.2
       hooks:
         - id: gitleaks
   

   若要 原生运行 gitleaks，或使用 gitleaks-docker pre-commit ID 通过 官方 Docker 镜像 运行 gitleaks

3. 执行 pre-commit autoupdate 将配置自动更新到最新仓库版本

4. 使用 pre-commit install 完成安装

5. 一切就绪！

➜ git commit -m "this commit contains a secret"
Detect hardcoded secrets.................................................Failed

注意：若要禁用 gitleaks 的 pre-commit 钩子，可在 commit 命令前加上 SKIP=gitleaks，这样会跳过 gitleaks 的执行

➜ SKIP=gitleaks git commit -m "skip gitleaks check"
Detect hardcoded secrets................................................Skipped

用法

Gitleaks scans code, past or present, for secrets

Usage:
  gitleaks [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  dir         scan directories or files for secrets
  git         scan git repositories for secrets
  help        Help about any command
  stdin       detect secrets from stdin
  version     display gitleaks version

Flags:
  -b, --baseline-path string          path to baseline with issues that can be ignored
  -c, --config string                 config file path
                                      order of precedence:
                                      1. --config/-c
                                      2. env var GITLEAKS_CONFIG
                                      3. env var GITLEAKS_CONFIG_TOML with the file content
                                      4. (target path)/.gitleaks.toml
                                      If none of the four options are used, then gitleaks will use the default config
      --diagnostics string            enable diagnostics (http OR comma-separated list: cpu,mem,trace). cpu=CPU prof, mem=memory prof, trace=exec tracing, http=serve via net/http/pprof
      --diagnostics-dir string        directory to store diagnostics output files when not using http mode (defaults to current directory)
      --enable-rule strings           only enable specific rules by id
      --exit-code int                 exit code when leaks have been encountered (default 1)
  -i, --gitleaks-ignore-path string   path to .gitleaksignore file or folder containing one (default ".")
  -h, --help                          help for gitleaks
      --ignore-gitleaks-allow         ignore gitleaks:allow comments
  -l, --log-level string              log level (trace, debug, info, warn, error, fatal) (default "info")
      --max-archive-depth int         allow scanning into nested archives up to this depth (default "0", no archive traversal is done)
      --max-decode-depth int          allow recursive decoding up to this depth (default "0", no decoding is done)
      --max-target-megabytes int      files larger than this will be skipped
      --no-banner                     suppress banner
      --no-color                      turn off color for verbose output
      --redact uint[=100]             redact secrets from logs and stdout. To redact only parts of the secret just apply a percent value from 0..100. For example --redact=20 (default 100%)
  -f, --report-format string          output format (json, csv, junit, sarif, template)
  -r, --report-path string            report file
      --report-template string        template file used to generate the report (implies --report-format=template)
      --timeout int                   set a timeout for gitleaks commands in seconds (default "0", no timeout is set)
  -v, --verbose                       show verbose output from scan
      --version                       version for gitleaks

Use "gitleaks [command] --help" for more information about a command.

命令

⚠️ v8.19.0 引入了一项变更，弃用了 detect 和 protect。这些命令仍可使用，但已隐藏在 --help 菜单中。请参阅此 gist 获取便捷的命令对照说明。 若发现 v8.19.0 破坏了现有命令（detect/protect），请提交 issue。

共有三种扫描模式：git、dir 和 stdin。

Git

git 命令可用于扫描本地 Git 仓库。在底层，gitleaks 使用 git log -p 命令扫描补丁（patch）。 可使用 log-opts 选项配置 git log -p 的行为。 例如，若要对一段提交范围运行 gitleaks，可使用以下 命令：gitleaks git -v --log-opts="--all commitA..commitB" path_to_repo。更多信息请参阅 git log 文档。 若未以位置参数指定目标，gitleaks 将尝试将当前工作目录作为 Git 仓库进行扫描。

Dir

dir（别名包括 files、directory）命令可用于扫描目录和文件。示例：gitleaks dir -v path_to_directory_or_file。 若未通过位置参数指定目标，gitleaks 将扫描当前工作目录。

Stdin

你也可以使用 stdin 命令将数据流式传输给 gitleaks。示例：cat some_file | gitleaks -v stdin

创建基线（baseline）

在扫描大型仓库或历史较长的仓库时，使用基线会十分方便。使用基线时，gitleaks 会忽略基线中已存在的旧发现项。基线可以是任意 gitleaks 报告。要创建 gitleaks 报告，请在使用 --report-path 参数的情况下运行 gitleaks。

gitleaks git --report-path gitleaks-report.json # This will save the report in a file called gitleaks-report.json

基线创建完成后，可在再次运行 detect 命令时应用：

gitleaks git --baseline-path gitleaks-report.json --report-path findings.json

使用 --baseline-path 参数运行 detect 命令后，报告输出（findings.json）将仅包含新问题。

Pre-Commit 钩子

你可以将示例 pre-commit.py 脚本复制到 .git/hooks/ 目录中，将 Gitleaks 作为 pre-commit 钩子运行。

加载配置

优先级顺序如下：

1. --config/-c 选项：

   gitleaks git --config /home/dev/customgitleaks.toml .
   

2. 环境变量 GITLEAKS_CONFIG（文件路径）：

   export GITLEAKS_CONFIG="/home/dev/customgitleaks.toml"
   gitleaks git .
   

3. 环境变量 GITLEAKS_CONFIG_TOML（文件内容）：

   export GITLEAKS_CONFIG_TOML=`cat customgitleaks.toml`
   gitleaks git .
   

4. 目标路径内的 .gitleaks.toml 文件：

   gitleaks git .
   

若上述四种方式均未使用，gitleaks 将使用默认配置。

配置

Gitleaks 提供了一种配置格式，你可据此编写自己的密钥检测规则：

# Title for the gitleaks configuration file.
title = "Custom Gitleaks configuration"

# You have basically two options for your custom configuration:
#
# 1. define your own configuration, default rules do not apply
#
#    use e.g., the default configuration as starting point:
#    https://github.com/gitleaks/gitleaks/blob/master/config/gitleaks.toml
#
# 2. extend a configuration, the rules are overwritten or extended
#
#    When you extend a configuration the extended rules take precedence over the
#    default rules. I.e., if there are duplicate rules in both the extended
#    configuration and the default configuration the extended rules or
#    attributes of them will override the default rules.
#    Another thing to know with extending configurations is you can chain
#    together multiple configuration files to a depth of 2. Allowlist arrays are
#    appended and can contain duplicates.

# useDefault and path can NOT be used at the same time. Choose one.
[extend]
# useDefault will extend the default gitleaks config built in to the binary
# the latest version is located at:
# https://github.com/gitleaks/gitleaks/blob/master/config/gitleaks.toml
useDefault = true
# or you can provide a path to a configuration to extend from.
# The path is relative to where gitleaks was invoked,
# not the location of the base config.
# path = "common_config.toml"
# If there are any rules you don't want to inherit, they can be specified here.
disabledRules = [ "generic-api-key"]

# An array of tables that contain information that define instructions
# on how to detect secrets
[[rules]]
# Unique identifier for this rule
id = "awesome-rule-1"

# Short human-readable description of the rule.
description = "awesome rule 1"

# Golang regular expression used to detect secrets. Note Golang's regex engine
# does not support lookaheads.
regex = '''one-go-style-regex-for-this-rule'''

# Int used to extract secret from regex match and used as the group that will have
# its entropy checked if `entropy` is set.
secretGroup = 3

# Float representing the minimum shannon entropy a regex group must have to be considered a secret.
entropy = 3.5

# Golang regular expression used to match paths. This can be used as a standalone rule or it can be used
# in conjunction with a valid `regex` entry.
path = '''a-file-path-regex'''

# Keywords are used for pre-regex check filtering. Rules that contain
# keywords will perform a quick string compare check to make sure the
# keyword(s) are in the content being scanned. Ideally these values should
# either be part of the identiifer or unique strings specific to the rule's regex
# (introduced in v8.6.0)
keywords = [
  "auth",
  "password",
  "token",
]

# Array of strings used for metadata and reporting purposes.
tags = ["tag","another tag"]

    # ⚠️ In v8.21.0 `[rules.allowlist]` was replaced with `[[rules.allowlists]]`.
    # This change was backwards-compatible: instances of `[rules.allowlist]` still  work.
    #
    # You can define multiple allowlists for a rule to reduce false positives.
    # A finding will be ignored if _ANY_ `[[rules.allowlists]]` matches.
    [[rules.allowlists]]
    description = "ignore commit A"
    # When multiple criteria are defined the default condition is "OR".
    # e.g., this can match on |commits| OR |paths| OR |stopwords|.
    condition = "OR"
    commits = [ "commit-A", "commit-B"]
    paths = [
      '''go\.mod''',
      '''go\.sum'''
    ]
    # note: stopwords targets the extracted secret, not the entire regex match
    # like 'regexes' does. (stopwords introduced in 8.8.0)
    stopwords = [
      '''client''',
      '''endpoint''',
    ]

    [[rules.allowlists]]
    # The "AND" condition can be used to make sure all criteria match.
    # e.g., this matches if |regexes| AND |paths| are satisfied.
    condition = "AND"
    # note: |regexes| defaults to check the _Secret_ in the finding.
    # Acceptable values for |regexTarget| are "secret" (default), "match", and "line".
    regexTarget = "match"
    regexes = [ '''(?i)parseur[il]''' ]
    paths = [ '''package-lock\.json''' ]

# You can extend a particular rule from the default config. e.g., gitlab-pat
# if you have defined a custom token prefix on your GitLab instance
[[rules]]
id = "gitlab-pat"
# all the other attributes from the default rule are inherited

    [[rules.allowlists]]
    regexTarget = "line"
    regexes = [ '''MY-glpat-''' ]


# ⚠️ In v8.25.0 `[allowlist]` was replaced with `[[allowlists]]`.
#
# Global allowlists have a higher order of precedence than rule-specific allowlists.
# If a commit listed in the `commits` field below is encountered then that commit will be skipped and no
# secrets will be detected for said commit. The same logic applies for regexes and paths.
[[allowlists]]
description = "global allow list"
commits = [ "commit-A", "commit-B", "commit-C"]
paths = [
  '''gitleaks\.toml''',
  '''(.*?)(jpg|gif|doc)'''
]
# note: (global) regexTarget defaults to check the _Secret_ in the finding.
# Acceptable values for regexTarget are "match" and "line"
regexTarget = "match"
regexes = [
  '''219-09-9999''',
  '''078-05-1120''',
  '''(9[0-9]{2}|666)-\d{2}-\d{4}''',
]
# note: stopwords targets the extracted secret, not the entire regex match
# like 'regexes' does. (stopwords introduced in 8.8.0)
stopwords = [
  '''client''',
  '''endpoint''',
]

# ⚠️ In v8.25.0, `[[allowlists]]` have a new field called |targetRules|.
#
# Common allowlists can be defined once and assigned to multiple rules using |targetRules|.
# This will only run on the specified rules, not globally.
[[allowlists]]
targetRules = ["awesome-rule-1", "awesome-rule-2"]
description = "Our test assets trigger false-positives in a couple rules."
paths = ['''tests/expected/._\.json$''']

请参考默认 gitleaks 配置 查看示例，或若希望为默认配置做贡献，请遵循 贡献指南。此外，你还可以阅读 这篇 gitleaks 博客文章，其中介绍了高级配置方案。

其他配置

复合规则（多部分或 required 规则）

在 v8.28.0 中，Gitleaks 引入了复合规则（composite rules），由一条“主”规则和一条或多条辅助或 required 规则组成。要创建复合规则，请在主规则中添加 [[rules.required]] 表，指定 id，并可选择设置 withinLines 和/或 withinColumns 邻近约束。片段（fragment）是 Gitleaks 一次处理的一块内容（通常是文件、文件的一部分或 git diff），邻近匹配会指示主规则：仅当辅助 required 规则也在片段的指定区域内找到匹配时，才报告发现项。

邻近匹配（Proximity matching）： 使用 withinLines 和 withinColumns 字段会指示主规则：仅当辅助 required 规则也在指定邻近范围内找到匹配时，才报告发现项。你可以设置：

• withinLines: N - 必需发现项必须在 N 行以内（纵向）
• withinColumns: N - 必需发现项必须在 N 个字符以内（横向）
• 两者同时设置 - 创建矩形搜索区域（两个约束都必须满足）
• 两者都不设置 - 片段级匹配（必需发现项可在同一片段中的任意位置）

下图说明了各种邻近行为：

p = primary captured secret
a = auxiliary (required) captured secret
fragment = section of data gitleaks is looking at


    *Fragment-level proximity*
    Any required finding in the fragment
          ┌────────┐
   ┌──────┤fragment├─────┐
   │      └──────┬─┤     │ ┌───────┐
   │             │a│◀────┼─│✓ MATCH│
   │          ┌─┐└─┘     │ └───────┘
   │┌─┐       │p│        │
   ││a│    ┌─┐└─┘        │ ┌───────┐
   │└─┘    │a│◀──────────┼─│✓ MATCH│
   └─▲─────┴─┴───────────┘ └───────┘
     │    ┌───────┐
     └────│✓ MATCH│
          └───────┘


   *Column bounded proximity*
   `withinColumns = 3`
          ┌────────┐
   ┌────┬─┤fragment├─┬───┐
   │      └──────┬─┤     │ ┌───────────┐
   │    │        │a│◀┼───┼─│+1C ✓ MATCH│
   │          ┌─┐└─┘     │ └───────────┘
   │┌─┐ │     │p│    │   │
┌──▶│a│  ┌─┐  └─┘        │ ┌───────────┐
│  │└─┘ ││a│◀────────┼───┼─│-2C ✓ MATCH│
│  │       ┘             │ └───────────┘
│  └── -3C ───0C─── +3C ─┘
│  ┌─────────┐
│  │ -4C ✗ NO│
└──│  MATCH  │
   └─────────┘


   *Line bounded proximity*
   `withinLines = 4`
         ┌────────┐
   ┌─────┤fragment├─────┐
  +4L─ ─ ┴────────┘─ ─ ─│
   │                    │
   │              ┌─┐   │ ┌────────────┐
   │         ┌─┐  │a│◀──┼─│+1L ✓ MATCH │
   0L  ┌─┐   │p│  └─┘   │ ├────────────┤
   │   │a│◀──┴─┴────────┼─│-1L ✓ MATCH │
   │   └─┘              │ └────────────┘
   │                    │ ┌─────────┐
  -4L─ ─ ─ ─ ─ ─ ─ ─┌─┐─│ │-5L ✗ NO │
   │                │a│◀┼─│  MATCH  │
   └────────────────┴─┴─┘ └─────────┘


   *Line and column bounded proximity*
   `withinLines = 4`
   `withinColumns = 3`
         ┌────────┐
   ┌─────┤fragment├─────┐
  +4L   ┌└────────┴ ┐   │
   │            ┌─┐     │ ┌───────────────┐
   │    │       │a│◀┼───┼─│+2L/+1C ✓ MATCH│
   │         ┌─┐└─┘     │ └───────────────┘
   0L   │    │p│    │   │
   │         └─┘        │
   │    │           │   │ ┌────────────┐
  -4L    ─ ─ ─ ─ ─ ─┌─┐ │ │-5L/+3C ✗ NO│
   │                │a│◀┼─│   MATCH    │
   └───-3C────0L───+3C┴─┘ └────────────┘

关于复合规则的一些最后想法。

这是一项实验性功能！它可能会变更，所以别基于这项功能去卖新的 B2B SaaS 功能。基于扫描类型（git 与 dir）的上下文很有意思。我正在关注相关情况。复合规则对 git 扫描可能不是超级有用，因为 gitleaks 只查看 git 历史中的新增内容。扫描 git 历史中的非新增内容，对 `required` 规则可能有用。哦对了，这是 readme，我这就闭嘴。

gitleaks:allow

如果你明知要提交一个 gitleaks 会捕获的测试密钥，可以在该行添加 gitleaks:allow 注释，指示 gitleaks 忽略该密钥。例如：

class CustomClass:
    discord_client_secret = '8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ'  #gitleaks:allow

.gitleaksignore

你可以在仓库根目录创建 .gitleaksignore 文件来忽略特定发现项。在 v8.10.0 版本中，Gitleaks 向 Gitleaks 报告添加了 Fingerprint 值。每个泄露项或发现项都有一个可唯一标识密钥的 Fingerprint。将该 fingerprint 添加到 .gitleaksignore 文件即可忽略该特定密钥。示例见 Gitleaks 的 .gitleaksignore。注意：此功能为实验性功能，未来可能会变更。

解码

有时密钥会以某种编码方式存储，仅用正则表达式难以发现。 现在你可以让 gitleaks 自动查找并解码 编码文本。标志 --max-decode-depth 可启用此功能（默认值 "0" 表示默认禁用）。

自解码后的文本也可能包含编码 文本以来，支持递归解码。标志 --max-decode-depth 设置递归深度限制。当没有新的编码文本段需要解码时，递归会停止，因此将最大深度设得很高并不意味着会进行那么多次遍历。它只会按需进行所需次数的遍历。总体而言，解码只会略微增加 扫描时间。

编码文本的发现项与普通发现项在以下 方面有所不同：

• 位置指向编码文本的边界
  • 如果规则在编码文本之外匹配，边界会相应调整以 包含该匹配
• match 和 secret 包含解码后的值
• 会添加两个标签 decoded:<encoding> 和 decode-depth:<depth>

当前支持的编码：

• percent - 任何可打印 ASCII 的 percent 编码值
• hex - 任何长度 >= 32 个字符的可打印 ASCII hex 编码值
• base64 - 任何长度 >= 16 个字符的可打印 ASCII base64 编码值

归档扫描

有时密钥会打包在 zip 文件或 tarball 等归档文件中， 难以发现。现在你可以让 gitleaks 自动 提取并扫描归档内容。标志 --max-archive-depth 可为 dir 和 git 两种扫描类型启用此功能。默认值 "0" 表示默认禁用。

由于归档中也可能包含其他归档，支持递归扫描。 --max-archive-depth 标志设置递归深度限制。当没有新的归档需要提取时，递归会停止，因此将最大深度设得很高只是 设置了可能达到该深度的上限。实际深度只会按需达到所需程度。

位于归档内的密钥发现项会包含归档内文件的路径。内部路径以 ! 分隔。

发现项示例（为简洁起见已缩短）：

Finding:     DB_PASSWORD=8ae31cacf141669ddfb5da
...
File:        testdata/archives/nested.tar.gz!archives/files.tar!files/.env.prod
Line:        4
Commit:      6e6ee6596d337bb656496425fb98644eb62b4a82
...
Fingerprint: 6e6ee6596d337bb656496425fb98644eb62b4a82:testdata/archives/nested.tar.gz!archives/files.tar!files/.env.prod:generic-api-key:4
Link:        https://github.com/leaktk/gitleaks/blob/6e6ee6596d337bb656496425fb98644eb62b4a82/testdata/archives/nested.tar.gz

这表示在 files/.env.prod. 的第 4 行检测到密钥，该文件位于 archives/files.tar，而它又位于 testdata/archives/nested.tar.gz 中。

当前支持的格式：

支持 mholt 的 archives package 所支持的 compression 和 archive 格式。

报告

Gitleaks 内置支持多种报告格式：json, csv, junit, 和 sarif.

如果这些格式都不符合你的需求，你可以使用 Go text/template .tmpl 文件 和 --report-template 标志创建自己的报告格式。该模板可以使用 Masterminds/sprig 模板库提供的扩展功能.

例如，以下模板可提供自定义 JSON 输出：

# jsonextra.tmpl
[{{ $lastFinding := (sub (len . ) 1) }}
{{- range $i, $finding := . }}{{with $finding}}
    {
        "Description": {{ quote .Description }},
        "StartLine": {{ .StartLine }},
        "EndLine": {{ .EndLine }},
        "StartColumn": {{ .StartColumn }},
        "EndColumn": {{ .EndColumn }},
        "Line": {{ quote .Line }},
        "Match": {{ quote .Match }},
        "Secret": {{ quote .Secret }},
        "File": "{{ .File }}",
        "SymlinkFile": {{ quote .SymlinkFile }},
        "Commit": {{ quote .Commit }},
        "Entropy": {{ .Entropy }},
        "Author": {{ quote .Author }},
        "Email": {{ quote .Email }},
        "Date": {{ quote .Date }},
        "Message": {{ quote .Message }},
        "Tags": [{{ $lastTag := (sub (len .Tags ) 1) }}{{ range $j, $tag := .Tags }}{{ quote . }}{{ if ne $j $lastTag }},{{ end }}{{ end }}],
        "RuleID": {{ quote .RuleID }},
        "Fingerprint": {{ quote .Fingerprint }}
    }{{ if ne $i $lastFinding }},{{ end }}
{{- end}}{{ end }}
]

用法：

$ gitleaks dir ~/leaky-repo/ --report-path "report.json" --report-format template --report-template testdata/report/jsonextra.tmpl

赞助

coderabbit.ai

退出码

遇到泄漏时，你始终可以使用 --exit-code 标志设置退出码。默认退出码如下：

0 - no leaks present
1 - leaks or error encountered
126 - unknown flag

加入 Discord！