[ENH]: Skip caching for output collections in frontend

[tanujnay112]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Check collection metadata to see if a collection is an output collection and prevent caching its CollectionAndSegments in the frontend if it's true.
  • Get rid of sleep(60) instances in function end to end test cases.
• New functionality
  • ...

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?

Observability plan

What is the plan to instrument and monitor this change?

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[tanujnay112]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [ENH]: Delete attached function when output collection is deleted #5957
• [ENH]: Skip caching for output collections in frontend #5956 👈 (View in Graphite)
• [ENH]: Add collection metadata to output collection to identify source attached function #5955
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[propel-code-bot]

Skip caching output collections and streamline tests

Introduces metadata-based detection of output collections so the frontend cache avoids storing their CollectionAndSegments. Also removes long sleep delays from distributed statistics and task API tests, relying solely on wait_for_version_increase, and updates the Go proto generation target to include heapservice.proto.

Key Changes

• Added SOURCE_ATTACHED_FUNCTION_ID_KEY constant and Collection::is_output_collection() helper in rust/types/src/collection.rs.
• Updated CollectionsWithSegmentsProvider::set_collection_with_segments() to skip caching when collection.is_output_collection() returns true.
• Removed time.sleep() calls from chromadb/test/distributed/test_statistics_wrapper.py and chromadb/test/distributed/test_task_api.py, keeping only version-wait helpers.
• Included chromadb/proto/heapservice.proto in the Go proto generation rule in idl/makefile.

Affected Areas

• rust/types/src/collection.rs
• rust/frontend/src/get_collection_with_segments_provider.rs
• chromadb/test/distributed/test_statistics_wrapper.py
• chromadb/test/distributed/test_task_api.py
• idl/makefile

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[Maintainability] For consistency with how logservice.proto's generated files are handled, it seems heapservice.proto should also have its generated files moved to a dedicated directory (e.g., ../go/pkg/proto/heapservicepb/).

Currently, the generated heapservice*.go files will be moved to the generic coordinatorpb directory by the wildcard mv command on the next line. Adding a specific move command for heapservice would align with the pattern set by logservice.

Suggested change

	@mv ../go/pkg/proto/coordinatorpb/chromadb/proto/logservice*.go ../go/pkg/proto/logservicepb/
	@mv ../go/pkg/proto/coordinatorpb/chromadb/proto/logservice*.go ../go/pkg/proto/logservicepb/
	@mv ../go/pkg/proto/coordinatorpb/chromadb/proto/heapservice*.go ../go/pkg/proto/heapservicepb/

Context for Agents

For consistency with how `logservice.proto`'s generated files are handled, it seems `heapservice.proto` should also have its generated files moved to a dedicated directory (e.g., `../go/pkg/proto/heapservicepb/`).

Currently, the generated `heapservice*.go` files will be moved to the generic `coordinatorpb` directory by the wildcard `mv` command on the next line. Adding a specific move command for `heapservice` would align with the pattern set by `logservice`.

```suggestion
	@mv ../go/pkg/proto/coordinatorpb/chromadb/proto/logservice*.go ../go/pkg/proto/logservicepb/
	@mv ../go/pkg/proto/coordinatorpb/chromadb/proto/heapservice*.go ../go/pkg/proto/heapservicepb/
```

File: idl/makefile
Line: 31

[HammadB]

I am a little bit on the fence about this conditional caching. The intent of the cache is to remove the sysdb from hot path situations.