Add a token limit flat to the command.

[shanecp]

• Add token limit flag and file limit, so output can be split by tokens
• Update docs to support
• Add requirements file.

This gives a solution for #35, #37. Perhaps #26

[davidvasandani]

Tested with a huge repo using the following params and output:

-t 62000000 -m 20

Writing to file output_1.txt
Written 61963289 tokens to file.
Writing to file output_2.txt
Written 61914604 tokens to file.
Writing to file output_3.txt
Written 23705417 tokens to file.
Writing to file output_4.txt
Written 1191179680 tokens to file.
Writing to file output_5.txt
Repository contents written to 5 file(s).

output_4.txt is 1191179680 tokens which is greater than the specified 62000000 tokens.

[shanecp]

If a source file is larger than the given output token limit, there are only a few options.

• Ignore the token limit, and output the complete file (This is the current method chosen. That's why you might be seeing a file going above the token limit)
• Split input file to multiple output files
• Ignore that file