How to Stop the crawler without pressing CTRL + C?

[spiderseo]

Hello guys, I need a way to stop the specific instance of the advertools crawler with Python instead of using the CTRL + C, I need to do it through Python script, is there a way?Thanks!

[eliasdabbas]

Short answer, run this from the command line:

kill -9 $(ps -A | grep 'scrapy runspider' | cut -f 2 -d ' ' | head -n 1)

Long answer:

kill -9: stop a process using it's process ID (which is a number)
The number is the result of running the following expression.

ps -A: lists processes with a special option -A
grep 'scrapy runspider': in the list of processes search for a line that contains 'scrapy runspider' (the one we are interested in)
cut -f 2 -d ' ': this would "cut" the output into columns, takes the second column (defined as anything delimited with a space)
head -n 1: take the first line of the output

The result is the process ID we are looking for.

Then kill does the job.

[spiderseo]

I found the solution, add this to Advertools my friend:

def stop_crawler():

# Find the process ID (PID) of the 'scrapy runspider' command
process = subprocess.Popen(['tasklist'], stdout=subprocess.PIPE)
output, _ = process.communicate()

# Extract the PID from the output using findstr command
pid_process = subprocess.Popen(
    ['findstr', 'scrapy runspider'], stdin=subprocess.PIPE, stdout=subprocess.PIPE
)
pid_output, _ = pid_process.communicate(input=output)

pid = pid_output.decode().split()[1]  # Extract the PID

# Kill the process using the obtained PID
subprocess.call(['taskkill', '/F', '/PID', pid])

[eliasdabbas]

Great!

Similar solution for Linux/MacOS systems:

import subprocess
import re

ps_A = subprocess.run(['ps', 'A'], capture_output=True)
pid_to_kill = re.findall(
    '.*scrapy runspider.*',
    ps_A.stdout.decode(),
    re.MULTILINE)[0].strip().split()[0]

subprocess.run(['kill', '-9', pid_to_kill])

[eliasdabbas]

The challenge remains: which spider are you stopping?

What if you are running multiple spiders at the same time?

[spiderseo]

Yes! that's what I am concerned about now... I am testing with multiple spiders, and the only way that It stops is when I stop all the spiders....

[spiderseo]

Hey Elias! This is the solution to track the specific Spider on Windows, I built script:

import platform, psutil
import pandas as pd

class SpiderTracker:

def __init__(self) ->:

    self.operating_system:str = platform.system()
    self.process_name:str = {"Linux":"scrapy", "Windows":'scrapy.exe'}[self.operating_system]
    self.process_args_to_kil:list[str] = {"Linux":['kill', '-9'], "Windows":['taskkill', '/F', '/PID']}[self.operating_system]

def get_running_spiders(self) -> list[dict]:

    """Returns as list of dictionaries containing running scrapy spiders, where the cmdline key has the spider data and the pid has the process id of that specific spider."""
    running_spiders:list = []
    scrapy_processes:list = []

    # Get the list of all running processes
    for process in psutil.process_iter(['pid', 'name', 'cmdline']):
        if process.info['name'] == self.process_name:
            scrapy_processes.append(process)

    # Iterate through the scrapy processes
    for process in scrapy_processes:
        cmdline, pid = ' '.join(process.info['cmdline']), process.info['pid']
        running_spiders.append({"pid":pid, "cmdline":cmdline})

    return running_spiders


 def stop_spider_id(self, url):

    running_spiders:list[dict] = self.get_running_spiders()
    df:object = pd.DataFrame(running_spiders)
    pid_to_kill:str = df[df['cmdline'].str.contains(url, regex=True)]['pid'].to_numpy(dtype=str)[0]
    process_args_to_kil:list[str] = self.process_args_to_kil + [pid_to_kill]
    subprocess.call(process_args_to_kil)

[eliasdabbas]

Thanks a lot!