Skip to content

Add more data to the dev seeds #481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 61 commits into
base: master
Choose a base branch
from

Conversation

MareStare
Copy link
Contributor

@MareStare MareStare commented Mar 27, 2025

Before you begin

  • I understand my contributions may be rejected for any reason
  • I understand my contributions are for the benefit of Derpibooru and/or the Philomena software
  • I understand my contributions are licensed under the GNU AGPLv3
  • I understand all of the above

This PR cherry-picks two commits from the redesign branch (2d5da61 and 16978b3).

Note that with all this new seed data the initial startup now takes several minutes (haven't measured exactly), but I think it's worth it.

Improved logging

Added command logging with color highlight to the dev scripts, so now the logs look like this:

image

I also noticed this warning in the seeding logs:


app-1  | [info] Elixir.Philomena.IndexWorker[3a594501-b02a-4ef8-875c-6a4511db8149] done: 24ms
app-1  | identify: unknown image property "%[profile:icc]" @ warning/property.c/InterpretImageProperties/4238.

Plus this warning:

Logger.warning("Failed to operate on all backends")

And, there was also the following error:

app-1  | [error] Task #PID<0.98808.0> started from #PID<0.98805.0> terminating
app-1  | ** (KeyError) key :body not found in: nil
app-1  |
app-1  | If you are using the dot syntax, such as map.field, make sure the left-hand side of the dot is a map
app-1  |     (philomena 1.1.0) lib/philomena_media/objects.ex:72: PhilomenaMedia.Objects.download_file/2
app-1  |     (philomena 1.1.0) lib/philomena/images/thumbnailer.ex:147: Philomena.Images.Thumbnailer.download_image_file/1
app-1  |     (philomena 1.1.0) lib/philomena/images/thumbnailer.ex:78: Philomena.Images.Thumbnailer.generate_thumbnails/1
app-1  |     (philomena 1.1.0) lib/philomena/workers/thumbnail_worker.ex:6: Philomena.ThumbnailWorker.perform/1
app-1  |     (exq 0.19.0) lib/exq/worker/server.ex:175: anonymous fn/4 in Exq.Worker.Server.dispatch_work/3
app-1  |     (elixir 1.18.1) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
app-1  | Function: #Function<0.120145007/0 in Exq.Worker.Server.dispatch_work/3>
app-1  |     Args: []
app-1  | [info] Queueing job ef191eea-be47-4f95-bf28-1f861390a939 to retry in 85.0 seconds
app-1  | [warning] {{:badkey, :body, nil},
app-1  |  [
app-1  |    {PhilomenaMedia.Objects, :download_file, 2,
app-1  |     [file: ~c"lib/philomena_media/objects.ex", line: 72]},
app-1  |    {Philomena.Images.Thumbnailer, :download_image_file, 1,
app-1  |     [file: ~c"lib/philomena/images/thumbnailer.ex", line: 147]},
app-1  |    {Philomena.Images.Thumbnailer, :generate_thumbnails, 1,
app-1  |     [file: ~c"lib/philomena/images/thumbnailer.ex", line: 78]},
app-1  |    {Philomena.ThumbnailWorker, :perform, 1,
app-1  |     [file: ~c"lib/philomena/workers/thumbnail_worker.ex", line: 6]},
app-1  |    {Exq.Worker.Server, :"-dispatch_work/3-fun-0-", 4,
app-1  |     [file: ~c"lib/exq/worker/server.ex", line: 175]},
app-1  |    {Task.Supervised, :invoke_mfa, 2,
app-1  |     [file: ~c"lib/task/supervised.ex", line: 101]}
app-1  |  ]}
app-1  | [warning] Elixir.Philomena.ThumbnailWorker[ef191eea-be47-4f95-bf28-1f861390a939] fail: 6ms

I'll try to take a closer look at those, but even with these issues I could still open the site and see the new data.

@MareStare MareStare force-pushed the feat/import-seeds-from-redesign-branch branch from c207961 to 1bebe12 Compare March 30, 2025 16:40
@MareStare MareStare force-pushed the feat/import-seeds-from-redesign-branch branch from 1bebe12 to 4b9b9cc Compare March 30, 2025 20:15
@MareStare MareStare force-pushed the feat/import-seeds-from-redesign-branch branch from 818aede to 8ee8d96 Compare April 1, 2025 01:47
@Meow
Copy link
Member

Meow commented Apr 3, 2025

I have a few suggestions, since I'm pretty much the original author of a lot of this PRs stuff:

  • Do not seed automatically, instead move it to be a process that can be triggered manually via a mix command (so, by default, only seed 1 user and basic search indices, stuff crucial to site operation)
  • Store just a basic set of sample images (e.g. one of each format, extreme resolutions, dimensions)
  • Generate 1k or so random image uploads with randomized tags, re-use the same images over and over, simply just bypassing duplication checks
  • Generate a few dupe reports as samples, enough to trigger their pagination
  • Generate reports, enough to go into full pagination
  • Generate galleries, also enough to go into pagination
  • Send the "Administrator" account a bunch of PMs from various accounts
  • Generate more users, each with uniquely filled in profile
  • Randomly fill in user profiles, e.g. randomly assign an artist tag, randomly fill in a profile, or just decide to leave it completely blank)

MareStare and others added 18 commits May 12, 2025 08:29
I want to share the same context, but make a separate test for the
edge-case scenario.
Looks like package was renamed in NPM registry to
`@vitest/eslint-plugin`. This package is from same exact repository.
Resolves issue with conflicting dependencies when trying to install
`@vitest/eslint-plugin`:

npm error Could not resolve dependency:
npm error dev @vitest/eslint-plugin@"*" from the root project
npm error
npm error Conflicting peer dependency: @typescript-eslint/[email protected]
npm error node_modules/@typescript-eslint/utils
npm error   peer @typescript-eslint/utils@"^8.24.0" from @vitest/[email protected]
npm error   node_modules/@vitest/eslint-plugin
npm error     dev @vitest/eslint-plugin@"*" from the root project
MareStare added 29 commits May 12, 2025 08:57
@MareStare MareStare force-pushed the feat/import-seeds-from-redesign-branch branch from 4582f70 to 2b12042 Compare May 12, 2025 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants