lusk/itsy-gitsy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

refactor: support parallel execution

Browse Source 
* parallel processing of multiple repositories
* parallel parsing of git history
* parallel template output
* configuration option for number of threads
* improved detection of exceeded size limits

Side effects of parallel execution are out-of-order output messages,
and larger potential overshoot in both processing and detecting when
limits are exceeded.  It will stop if it exceeds the limits, but it
can exceed them by more before stopping.

Processing of extremely large repositories is several times faster.

The order of commits in the history output might not match the output
of `git log`, since the order of commits in the log can change when
splitting it up into smaller pieces.
This commit is contained in:
Trevor Bentley
2023-01-19 20:55:26 +01:00
parent 48b6dd056a
commit d83f682aa2
6 changed files with 490 additions and 300 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -4
View File
@@ -177,15 +177,13 @@ As always, if security is a concern, best practice is to follow the rules of lea
## Performance
High performance is not a primary goal of Itsy-Gitsy, since it is mostly intended for indexing small personal projects, but various settings are provided to allow it to handle large repositories.
The majority of parsing and generation is linear and single-threaded, except for rendering file content output. File contents are rendered in parallel, including syntax highlighting and Markdown rendering.
High performance is not a primary goal of Itsy-Gitsy, since it is mostly intended for indexing small personal projects, but various settings are provided to allow it to handle large repositories. Most operations take advantage of parallelism. By default, Itsy-Gitsy splits the tasks across all cores, but the number of parallel threads can be reduced in the configuration file.
Syntax highlighting uses syntect's pure-Rust implementation by default, to avoid an extra dependency. This implementation is quite slow, and performance can be greatly improved by using syntect's `onig` mode, which uses the faster Oniguruma C library for highlighting. This can be enabled at build time with `cargo build --features highlight_fast`.
All metadata of all repositories, except for file contents, is held in memory. Large repositories can easily exhaust memory, and disk usage can also get quite high. There are several `limit_*` settings available in the configuration for restricting the amount of data held in memory, with the tradeoff of reducing the amount of data available for the generated output. `limit_context` and `limit_diffs` are particularly important restrictions to set on repositories with thousands of commits.
Small repositories with dozens to hundreds of commits can be generated on the order of a few seconds or less. Large repositories take *considerably* longer; parsing 1,000,000 commits from the Linux kernel repository with `limit_tree_depth = 3`, `limit_context = 10` and `limit_diffs = 100` took ~30 minutes on a fast laptop, and produced a ~2GB website.
Small repositories with dozens to hundreds of commits can be generated on the order of a few seconds or less. Large repositories take *considerably* longer; parsing 1,115,000 commits from the Linux kernel repository with `limit_tree_depth = 3`, `limit_context = 100` and `limit_diffs = 0` took ~25 minutes on a fast laptop, and produced a ~5.6GB website. Peak memory usage was 8GB.
## Other Considerations