Add limit_commit_ids_to_related setting.

Each template received `commit_ids`, a list of ALL commit hashes known
to Itsy-Gitsy.  For large repos with hundreds of thousands or millions
of commits, this huge list uses lots of memory and takes a long time
to search.

`limit_commit_ids_to_related` reduces the list to just a handful of
relevant commits for certain templates, which can give a massive
performance boost.
This commit is contained in:
Trevor Bentley
2023-01-20 02:26:16 +01:00
parent 5f0f9978c9
commit c257ba12f0
4 changed files with 57 additions and 1 deletions
+1 -1
View File
@@ -183,7 +183,7 @@ Syntax highlighting uses syntect's pure-Rust implementation by default, to avoid
All metadata of all repositories, except for file contents, is held in memory. Large repositories can easily exhaust memory, and disk usage can also get quite high. There are several `limit_*` settings available in the configuration for restricting the amount of data held in memory, with the tradeoff of reducing the amount of data available for the generated output. `limit_context` and `limit_diffs` are particularly important restrictions to set on repositories with thousands of commits.
Small repositories with dozens to hundreds of commits can be generated on the order of a few seconds or less. Large repositories take *considerably* longer; parsing 1,115,000 commits from the Linux kernel repository with `limit_tree_depth = 3`, `limit_context = 100` and `limit_diffs = 0` took ~25 minutes on a fast laptop, and produced a ~5.6GB website. Peak memory usage was 8GB.
Small repositories with dozens to hundreds of commits can be generated on the order of a few seconds or less. Large repositories take *considerably* longer and significant amounts of RAM, but aggressively applying limits in the configuration can help; parsing 1,115,000 commits from the Linux kernel repository with syntax highlighting disabled, `limit_commit_ids_to_related = true`, `limit_tree_depth = 3`, `limit_context = 100` and `limit_diffs = 0` took ~7 minutes on a fast laptop, and produced a ~5.7GB website. Peak memory usage was ~8GB.
## Other Considerations