Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heavy CPU load and very long runtime when task page_thumbnails for a *.7z with many (>1000) images #1059

Open
LFsWang opened this issue Aug 18, 2024 · 2 comments

Comments

@LFsWang
Copy link

LFsWang commented Aug 18, 2024

LRR Version and OS
v0.9.21
Docker

Bug Details
Put a 7z file with 1000 images into content directory, and open that file in the website to start the page_thumbnails task, and it will use about 2 hours with 100% cpu usage (1 core) to generate 1000 thumbnails .

image

CPU: i5-1340P
RAM: 16GB DDR4 3200Mhz
Disk: NVMe SSD

Matching Logs

---
args:
- ce4dbc07a118e1a10ecb366fc4094ea56635e855
- 0
attempts: '3'
children: []
created: 2024-08-18T09:23:26.21037Z
delayed: 2024-08-18T09:23:26.21037Z
finished: 2024-08-18T11:04:26.66756Z
id: '751'
notes:
  pages: 1116
  progress: 1116
parents: []
priority: '0'
queue: default
result: ~
retried: ~
retries: '0'
started: 2024-08-18T09:23:26.22036Z
state: finished
task: page_thumbnails
worker: '28'

https://github.com/Difegue/LANraragi/blob/v.0.9.21/lib/LANraragi/Utils/Minion.pm#L75
https://github.com/Difegue/LANraragi/blob/v.0.9.21/lib/LANraragi/Utils/Archive.pm#L155

Maybe it need a cache for extract_thumbnail ?

@Difegue
Copy link
Owner

Difegue commented Aug 18, 2024

The page thumbnail mechanism has been reworked to run sequentially on a single job as of the current version in order to avoid spamming all cores and filling the runner with 1 job per page when a lot of pages need to be thumbnailed.
I'd have to reintroduce some small multithreading within the single job to improve this.

However, there is caching in place, so even if the job is halted or restarted, it'll pick back up wherever it stopped creating thumbnails.

@Ayazk726
Copy link

Ayazk726 commented Aug 29, 2024

If you have lots of images (>2000),you need about 2s to create a thumbnail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants