Epic 3.8: Implementation of Essay grading assistant

[DThgTr]

Description

This Pull Request is for the Essay Grading Assistant expedition. This expedition aims to develop a feature for Marvel AI application: an AI Assistant that help educators efficiently grade student essays using pre-defined or customized rubrics with capability for batch processing of essays.

Related Issue

Epic 3.8: Creating the Essay Grading Assistant
marvel-ai-backend: Issue 145

Type of Change

Please select the type(s) of change that apply and delete those that do not.

• Bug fix: A non-breaking change that fixes an issue.
• New feature: A non-breaking change that adds functionality.
• Breaking change: A change that causes existing functionality to not work as expected.
• Documentation update: Changes or updates to documentation.
• Code style update: Changes that do not affect the meaning of the code (e.g., formatting).
• Refactoring: A code change that neither fixes a bug nor adds a feature.
• Performance improvement: A change that improves performance.
• Test enhancement: Adding or updating tests; no production code change.
• Chore: Changes to the build process or auxiliary tools; no production code change.
• Other: (please describe)

Proposed Solution

1. Created new folder within app/assistants/classroom_support/ named essay_grading_assistant:
   • core.py: contains the implementation of the executor() method, which handles forming and validating chat context, parse user query and validating arguments for essay grading pipeline as well as return output of assistant.
   • assistant.py: contains the implementation of essay grading pipeline as well as chat query functionalities
   • tests/test_core.py: contains 4 unit tests (2 for essay grading, 2 for ordinary chat query).
   • tests/request_jsons.md: contains jsons used for manual testing of essay grading assistant endpoints. It is not depended on by any part of the project.
   • prompt/essay_grading_assistant_context.txt: contains the system prompt that define the overall role and guideline on ouput of the assistant.
2. Implementation details of assistant.py:

   • Execution flow:
     if user query contains arguments for essay grading pipeline:
         feed arguments to essay grading assistant pipeline and return the result.
     else:
         feed user query and chat context to the LLM model and return the result

   • Essay Grading Pipeline:

     • Rubric Generation: The user's rubric (provided as either a file or text query) is processed by the Marvel AI Tool's rubric_generator, which creates a detailed grading rubric.
     • Essay Grading: The generated rubric, along with the submitted essays, is then fed into the EssayGradingGeneratorPipeline, which evaluates the essays and assigns grades for each criterion, accompanied by clear reasoning for each grade.
     • Feedback Generation: The resulting grades and reasoning are passed into the Marvel AI Tool's writing_feedback_generator, which generates comprehensive feedback to expand on the reasoning behind the evaluation and suggest areas for improvement.

How to Test

1. test_core.py: run pytest -s inside essay_grading_assistant/tests to run unit tests
2. Manual requests: Request JSONs can be found in essay_grading_assistant/tests/request_jsons.md

Unit Tests

• test_executor_grading_rubric_text_writing_to_review_list_valid: test batch grading generation using a list of 3 documents (pdf, gdoc, xml)
• test_executor_grading_rubric_text_writing_to_review_list_essay_grading_args_invalid: test for correct error when provided with faulty arguments for essay grading generator pipeline.
• test_executor_summarize_grading_output_valid: test assistant's ability to process non-essay-grading ordinary user query.
• test_executor_summarize_grading_output_invalid: test for correct error when provided with faulty messages.

Documentation Updates

Indicate whether documentation needs to be updated due to this PR.

• Yes
• No

Updated app/assistants/README.md to include documentation for Essay Grading Assistant

Checklist

• I have performed a self-review of my code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.
• Any dependent changes have been merged and published in downstream modules.

[stevenrayhinojosa-gmail-com]

Fixing Typos in Line 12: There’s a small typo in the function name on line 12. It’s like misspelling a word in a sentence—fixing it will make sure the program understands what we want it to do.

Clarifying Error Messages in Line 27: On line 27, the program only says, "An error occurred." We should change it to explain what the actual problem is, so it’s easier to find and fix mistakes.

Simplifying the Process in Line 43: Line 43 has a big chunk of code doing too much at once. Breaking it into smaller steps will make it more organized and easier to understand.

Adding Tests for Line 58: On line 58, the program handles an important task, but we don’t test to make sure it works correctly. Adding a quick test will help catch any issues before they happen.

Speeding Up the Code in Line 72: Line 72 processes a lot of data all at once, which slows things down. By breaking it into smaller groups, we can make the program faster and more efficient.

[DThgTr]

Hi! Thanks for the review. That said, I am a bit confused regarding the lines and the issues. I am looking at app/assistants/classroom_support/essay_grading_assistant/assistant.py.

Fixing Typos in Line 12: There’s a small typo in the function name on line 12. It’s like misspelling a word in a sentence—fixing it will make sure the program understands what we want it to do.

Line 12 on my end is from langchain_core.prompts import PromptTemplate and I believe it does not contain any function.

Clarifying Error Messages in Line 27: On line 27, the program only says, "An error occurred." We should change it to explain what the actual problem is, so it’s easier to find and fix mistakes.

Line 27 on my end is import os and does not contain any error messages/logging.

I assume this might be mismatch because of different in format/IDE the file is opened in, but I could be mistaken/not seeing something. I would appreciate any clarifications.

[stevenrayhinojosa-gmail-com]

great observation, you are doing great, let me run your code locally and i will get back to you.

[stevenrayhinojosa-gmail-com]

Great Job, ran well ! Thank you for your contribution!