Feat: Introduce .indexerignore for flexible project specific file exclusion

[marksolly]

Introduces support for a .indexerignore file, allowing users to have fine-grained control over which files and directories are excluded from the indexing process. This feature uses the same gitwildmatch patterns as .gitignore, providing a familiar and powerful way to customize file discovery.

Problem Solved:

Different projects have unique needs for what should be included in a code index. While the default exclusion list is a good starting point, it cannot cover all use cases. Projects may contain large build artifacts, generated code, or other files that are not relevant for code analysis and can slow down indexing. The .indexerignore file provides a simple and standard way for developers to exclude these project-specific files.

Implementation Details:

1. .indexerignore Support:

   • The ProjectScanner in src/code_index_mcp/indexing/scanner.py now checks for a .indexerignore file in the project's root directory.
   • If found, it uses pathspec to parse the file and exclude any matching files and directories from the scan.

2. Debugging with Optional Logging:

   • To help users debug their .indexerignore rules, the set_project_path tool has been updated with a new optional boolean parameter: generate_log_file.
   • When generate_log_file is set to true, a .indexer.log file is created in the project root. This log details which files and directories were included or excluded during the scan, making it easy to verify that the ignore patterns are working as expected.

How to Use:

1. Create a .indexerignore file in the root of your project.

2. Add glob patterns for files and directories you wish to exclude, just like a .gitignore file. For example:

   # Ignore all test data
   test/data/
   
   # Ignore generated files
   *.generated.js

3. To debug your ignore patterns, call the set_project_path tool with generate_log_file set to true:

   {
     "tool": "set_project_path",
     "path": "/path/to/your/project",
     "generate_log_file": true
   }