ingest

README

Ingest

Ingest is a tool I've written to make my life easier when preparing content for LLMs.

It parses directories of plain text files, such as source code, into a single markdown file suitable for ingestion by AI/LLMs.

Ingest can also pass the prompt directly to an LLM such as Ollama for processing.

Features

• Traverse directory structures and generate a tree view
• Include/exclude files based on glob patterns
• Parse output directly to LLMs such as Ollama or any OpenAI compatible API for processing
• Generate and include git diffs and logs
• Count approximate tokens for LLM compatibility
• Customisable output templates
• Copy output to clipboard (when available)
• Export to file or print to console
• Optional JSON output

Installation

go install (recommended)

Make sure you have Go installed on your system, then run:

go install github.com/sammcj/ingest@HEAD

Manual install

1. Download the latest release from the releases page
2. Move the binary to a directory in your PATH, e.g. mv ingest* /usr/local/bin/ingest

Usage

Basic usage:

ingest [flags] <paths>

ingest will default the current working directory, if no path is provided, e.g:

$ ingest

⠋ Traversing directory and building tree...  [0s] [i] 9741 Tokens (Approximate)
[✓] Copied to clipboard successfully.

Generate a prompt from a directory, including only Python files:

ingest -i "**/*.py" /path/to/project

Generate a prompt with git diff and copy to clipboard:

ingest -d /path/to/project

Generate a prompt for multiple files/directories:

ingest /path/to/project /path/to/other/project

Generate a prompt and save to a file:

ingest -o output.md /path/to/project

LLM Integration

Ingest can pass the generated prompt to LLMs that have an OpenAI compatible API such as Ollama for processing.

ingest --llm /path/to/project

By default this will use any prompt suffix from your configuration file:

./ingest utils.go --llm
⠋ Traversing directory and building tree...  [0s]
This is Go code for a file named `utils.go`. It contains various utility functions for
handling terminal output, clipboard operations, and configuration directories.
...

You can provide a prompt suffix to append to the generated prompt:

ingest --llm -p "explain this code" /path/to/project

Configuration

Ingest uses a configuration file located at ~/.config/ingest/config.json.

You can make Ollama processing run without prompting setting "llm_auto_run": true in the config file.

The config file also contains:

• llm_model: The model to use for processing the prompt, e.g. "llama3.1:8b-q5_k_m".
• llm_prompt_prefix: An optional prefix to prepend to the prompt, e.g. "This is my application."
• llm_prompt_suffix: An optional suffix to append to the prompt, e.g. "explain this code"

Ingest uses the following directories for user-specific configuration:

• ~/.config/ingest/patterns/exclude: Add .glob files here to exclude additional patterns.
• ~/.config/ingest/patterns/templates: Add custom .tmpl files here for different output formats.

These directories will be created automatically on first run, along with README files explaining their purpose.

Flags

• -i, --include: Patterns to include (can be used multiple times)
• -e, --exclude: Patterns to exclude (can be used multiple times)
• --include-priority: Include files in case of conflict between include and exclude patterns
• --exclude-from-tree: Exclude files/folders from the source tree based on exclude patterns
• --tokens: Display the token count of the generated prompt
• -c, --encoding: Optional tokeniser to use for token count
• -o, --output: Optional output file path
• --llm: Send the generated prompt to an OpenAI compatible LLM server (such as Ollama) for processing
• -p, --prompt: Optional prompt suffix to append to the generated prompt
• -d, --diff: Include git diff
• --git-diff-branch: Generate git diff between two branches
• --git-log-branch: Retrieve git log between two branches
• -l, --line-number: Add line numbers to the source code
• --no-codeblock: Disable wrapping code inside markdown code blocks
• --relative-paths: Use relative paths instead of absolute paths
• -n, --no-clipboard: Disable copying to clipboard
• -t, --template: Path to a custom Handlebars template
• --json: Print output as JSON
• --pattern-exclude: Path to a specific .glob file for exclude patterns
• --print-default-excludes: Print the default exclude patterns
• --print-default-template: Print the default template
• --report: Print the largest parsed files
• --verbose: Print verbose output
• -V, --version: Print the version number (WIP - still trying to get this to work nicely)

Excludes

You can get a list of the default excludes by parsing --print-default-excludes to ingest. These are defined in defaultExcludes.go.

To override the default excludes, create a default.glob file in ~/.config/ingest/patterns/exclude with the patterns you want to exclude.

Templates

Templates are written in standard go templating syntax.

You can get a list of the default templates by parsing --print-default-template to ingest. These are defined in template.go.

To override the default templates, create a default.tmpl file in ~/.config/ingest/patterns/templates with the template you want to use by default.

Contributing

Contributions are welcome, Please feel free to submit a Pull Request.

License

• Copyright 2024 Sam McLeod
• This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgements

• Initially inspired by mufeedvh/code2prompt