Feature Request | 功能需求

[tuchief]

1、Timed task
2、Online packaging and deployment
3、Log monitoring alarm

[my8100]

@tuchief

2、Online packaging and deployment

Have you tried the Projects > Deploy page?

[tuchief]

I know what you said, but this method should be manually packaged into egg, and then uploaded, and can not directly package the source project into egg, automatically upload it?

[my8100]

I know what you said, but this method should be manually packaged into egg, and then uploaded, and can not directly package the source project into egg, automatically upload it?

Ok, I would try to figure out a better way to eggify local projects.

[tuchief]

You can refer to https://github.com/Gerapy/Gerapy

[my8100]

v0.9.9: Add auto eggifying

[tuchief]

Wow, the response is fast! Expecting the same timing tasks

[wymen2018]

1. You can select multiple crawlers when you want to set a timed task. You can select multiple crawlers to start at each time period after selecting the time point.
   Imagine that I have 100 crawlers in a project. This feature does provide convenience and management.
   2, as above, because there are a hundred crawlers, so can you have the function of custom label, that is, I can use the label screen and view the running status of the specified category of crawlers, and timed tasks
   Thanks!

[my8100]

You can select multiple crawlers when you want to set a timed task. You can select multiple crawlers to start at each time period after selecting the time point.
Imagine that I have 100 crawlers in a project. This feature does provide convenience and management.

You mean there are 100 spiders in a project and you want to schedule some of them to run periodically?

as above, because there are a hundred crawlers, so can you have the function of custom label, that is, I can use the label screen and view the running status of the specified category of crawlers, and timed tasks

What about labeling some related jobs with a specific jobid?

[wymen2018]

You mean there are 100 spiders in a project and you want to schedule some of them to run periodically?

No, Simultaneously select multiple spiders to set one timed task.

What about labeling some related jobs with a specific jobid?

Labels will be better because they can be more visualized according to their own classification

[my8100]

OK, I would take it into account when implementing this feature. Thanks for your advice!

[my8100]

v1.0.0rc1: Add Email Notice, with multi-triggers provided, including:

• ON_JOB_RUNNING_INTERVAL

• ON_JOB_FINISHED

• When reaching the threshold of specific kind of log: ['CRITICAL', 'ERROR', 'WARNING', 'REDIRECT', 'RETRY', 'IGNORE'], in the meanwhile, you can ask ScrapydWeb to stop/forcestop current job automatically.

Get it via the pip install scrapydweb==1.0.0rc1 command.

Email sample:

[LWsmile]

ERROR in utils: !!!!! ConnectionError HTTPConnectionPool(host='127.0.0.1', port=6800): Max retries exceeded with url: /jobs (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6111cae9b0>: Failed to establish a new connection: [Errno 111] Connection refused',))

[my8100]

ERROR in utils: !!!!! ConnectionError HTTPConnectionPool(host='127.0.0.1', port=6800): Max retries exceeded with url: /jobs (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6111cae9b0>: Failed to establish a new connection: [Errno 111] Connection refused',))

Please new an issue with details.

[JerryChenn07]

This project is pretty good. If you can join Timed task, it will be a more complete project.

Come on!

[lidonghe]

timed task +1

[my8100]

v1.2.0: Support Timer Tasks to schedule a spider run periodically

[lidonghe]

While adding a timer task, I have to choose a version, but actually I only want to use the latest version so that if the project is updated, I don't need to update my task accordingly.

[my8100]

OK, it would be fixed in the future release.
For the time being, go to the Jobs page and click either multinode or Start button to work around.

[my8100]

Or modify the code below to vm.versions = ['default: the latest version'].concat(obj.versions);

scrapydweb/scrapydweb/templates/scrapydweb/schedule.html

Line 826 in 560e998

vm.versions = obj.versions;

[lidonghe]

Or modify the code below to vm.versions = ['default: the latest version'].concat(obj.versions);
scrapydweb/scrapydweb/templates/scrapydweb/schedule.html

Line 826 in 560e998

                         vm.versions = obj.versions;

It works, thanks

[my8100]

The modification has been commited.

[heave-Rother]

For Logs categorization, can the same spiders distributed on different scrapyd be aggregated? @my8100

[my8100]

@heave-Rother
For the time being, you can switch to a specific page of the neighboring node
with the help of Node Scroller and Node Skipping.

If that cannot satisfy your need, could you draw a picture to show me your idea?

[heave-Rother]

@my8100 ok,

[my8100]

@heave-Rother

1. What would you use the checkboxes in the drop-down list for?
   Did you notice the checkboxes for nodes in the Servers page?
2. Stats aggregation would be followed up in PR Support aggregating stats of the same job from selected Scrapyd servers #72.

[heave-Rother]

@my8100
yes, I want to choose the server I want to join the statistics.

[my8100]

@heave-Rother
I see. Thanks for your suggestion.

[seozed]

add a image of docker, please

[my8100]

add a image of docker, please

@seozed
Check out the docker image created by @luzihang123.

my8100/logparser#15 (comment)

[devxiaosong]

感谢作者，这是我找到的最好的爬虫集群操作平台。提几个需求：
1，给每个node加描述，方便自己看。
2，通过手机短信发送报警信息。
3，如何支持基于scrapy-redis的分布式爬虫的配置、启动？

[my8100]

@devxiaosong
Replied in #107.

[Tobeyforce]

Please add a short tutorial on how to switch from the flask development server to a production server with https enabled using letsencrypt. It would be much appreciated.

[my8100]

Please add a short tutorial on how to switch from the flask development server to a production server with https enabled using letsencrypt. It would be much appreciated.

Please try it out and share your result.

scrapydweb/scrapydweb/run.py

Line 116 in 6b9663b

logger.info("For running Flask in production, check out http://flask.pocoo.org/docs/1.0/deploying/")

scrapydweb/scrapydweb/default_settings.py

Lines 82 to 92 in 6b9663b

	############################## ScrapydWeb #####################################
	# The default is False, set it to True and add both CERTIFICATE_FILEPATH and PRIVATEKEY_FILEPATH
	# to run ScrapydWeb in HTTPS mode.
	# Note that this feature is not fully tested, please leave your comment here if ScrapydWeb
	# raises any excepion at startup: https://github.com/my8100/scrapydweb/issues/18
	ENABLE_HTTPS = False
	# e.g. '/home/username/cert.pem'
	CERTIFICATE_FILEPATH = ''
	# e.g. '/home/username/cert.key'
	PRIVATEKEY_FILEPATH = ''

[Jasonjk3]

可以在页面上动态添加/删除 scrapyd 节点吗

[my8100]

可以在页面上动态添加/删除 scrapyd 节点吗

Editing Scrapyd servers via GUI is not supported.