python中子进程读取线上的超时

timeout on subprocess readline in python

我有一个小问题,我不太确定如何解决。下面是一个最小的例子:

我所拥有的

1
2
3
4
scan_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(some_criterium):
    line = scan_process.stdout.readline()
    some_criterium = do_something(line)

我想要什么

1
2
3
4
5
6
7
scan_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(some_criterium):
    line = scan_process.stdout.readline()
    if nothing_happens_after_10s:
        break
    else:
        some_criterium = do_something(line)

我从一个子流程中读取一行并对它做一些事情。我想要的是,如果在固定的时间间隔后没有线路到达,我就退出。有什么建议吗?


谢谢你的回答!我找到了一种方法来解决我的问题,只需使用select.poll查看stdout。

1
2
3
4
5
6
7
8
9
10
11
import select
...
scan_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
poll_obj = select.poll()
poll_obj.register(scan_process.stdout, select.POLLIN)  
while(some_criterium and not time_limit):
    poll_result = poll_obj.poll(0)
    if poll_result:
        line = scan_process.stdout.readline()
        some_criterium = do_something(line)
    update(time_limit)


这里有一个可移植的解决方案,它强制执行使用asyncio读取单行的超时:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/usr/bin/env python3
import asyncio
import sys
from asyncio.subprocess import PIPE, STDOUT

async def run_command(*args, timeout=None):
    # start child process
    # NOTE: universal_newlines parameter is not supported
    process = await asyncio.create_subprocess_exec(*args,
            stdout=PIPE, stderr=STDOUT)

    # read line (sequence of bytes ending with b'
') asynchronously
    while True:
        try:
            line = await asyncio.wait_for(process.stdout.readline(), timeout)
        except asyncio.TimeoutError:
            pass
        else:
            if not line: # EOF
                break
            elif do_something(line):
                continue # while some criterium is satisfied
        process.kill() # timeout or some criterium is not satisfied
        break
    return await process.wait() # wait for the child process to exit


if sys.platform =="win32":
    loop = asyncio.ProactorEventLoop() # for subprocess'
pipes on Windows
    asyncio.set_event_loop(loop)
else:
    loop = asyncio.get_event_loop()

returncode = loop.run_until_complete(run_command("cmd","arg 1","arg 2",
                                                 timeout=10))
loop.close()


我在python中使用了一些更通用的东西(IIRC也从so问题中拼凑而成,但我不记得是哪一个问题)。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import thread
from threading import Timer

def run_with_timeout(timeout, default, f, *args, **kwargs):
    if not timeout:
        return f(*args, **kwargs)
    try:
        timeout_timer = Timer(timeout, thread.interrupt_main)
        timeout_timer.start()
        result = f(*args, **kwargs)
        return result
    except KeyboardInterrupt:
        return default
    finally:
        timeout_timer.cancel()

不过,请注意,这使用一个中断来停止您提供的任何函数。对于所有函数来说,这可能不是一个好主意,而且它还阻止您在超时期间使用ctrl+c关闭程序(即,ctrl+c将作为超时处理)。你可以这样称呼它:

1
2
3
4
5
6
7
scan_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(some_criterium):
    line = run_with_timeout(timeout, None, scan_process.stdout.readline)
    if line is None:
        break
    else:
        some_criterium = do_something(line)

不过,可能有点过头了。我怀疑你的案子有一个更简单的选择,我不知道。


在python 3中,已经向子进程模块添加了一个超时选项。使用类似的结构

1
2
3
4
5
6
7
try:
    o, e = process.communicate(timeout=10)
except TimeoutExpired:
    process.kill()
    o, e = process.communicate()

analyze(o)

会是一个合适的解决方案。

由于输出预期包含一个新行字符,因此可以安全地假定它是文本(如可打印、可读),在这种情况下,强烈建议使用universal_newlines=True标志。

如果必须使用python2,请使用https://pypi.python.org/pypi/subprocess32/(backport)

对于纯python python 2解决方案,请查看使用带有超时的模块'subprocess'。


可移植的解决方案是,如果读取行花费的时间太长,则使用线程来终止子进程:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/usr/bin/env python3
from subprocess import Popen, PIPE, STDOUT

timeout = 10
with Popen(command, stdout=PIPE, stderr=STDOUT,
           universal_newlines=True) as process:  # text mode
    # kill process in timeout seconds unless the timer is restarted
    watchdog = WatchdogTimer(timeout, callback=process.kill, daemon=True)
    watchdog.start()
    for line in process.stdout:
        # don't invoke the watcthdog callback if do_something() takes too long
        with watchdog.blocked:
            if not do_something(line):  # some criterium is not satisfied
                process.kill()
                break
            watchdog.restart()  # restart timer just before reading the next line
    watchdog.cancel()

如果WatchdogTimer类与threading.Timer类类似,可以重新启动和/或阻止:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from threading import Event, Lock, Thread
from subprocess import Popen, PIPE, STDOUT
from time import monotonic  # use time.time or monotonic.monotonic on Python 2

class WatchdogTimer(Thread):
   """Run *callback* in *timeout* seconds unless the timer is restarted."""

    def __init__(self, timeout, callback, *args, timer=monotonic, **kwargs):
        super().__init__(**kwargs)
        self.timeout = timeout
        self.callback = callback
        self.args = args
        self.timer = timer
        self.cancelled = Event()
        self.blocked = Lock()

    def run(self):
        self.restart() # don't start timer until `.start()` is called
        # wait until timeout happens or the timer is canceled
        while not self.cancelled.wait(self.deadline - self.timer()):
            # don't test the timeout while something else holds the lock
            # allow the timer to be restarted while blocked
            with self.blocked:
                if self.deadline <= self.timer() and not self.cancelled.is_set():
                    return self.callback(*self.args)  # on timeout

    def restart(self):
       """Restart the watchdog timer."""
        self.deadline = self.timer() + self.timeout

    def cancel(self):
        self.cancelled.set()

尝试使用信号。警报:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#timeout.py
import signal,sys

def timeout(sig,frm):
  print"This is taking too long..."
  sys.exit(1)

signal.signal(signal.SIGALRM, timeout)
signal.alarm(10)
byte=0

while 'IT' not in open('/dev/urandom').read(2):
  byte+=2
print"I got IT in %s byte(s)!" % byte

几次跑步来展示它的效果:

1
2
3
4
$ python timeout.py
This is taking too long...
$ python timeout.py
I got IT in 4672 byte(s)!

有关更详细的示例,请参见pguides。


虽然您的(汤姆的)解决方案有效,但在C习惯用法中使用select()更紧凑。这相当于你的回答

1
2
3
4
5
6
7
from select import select
scan_process = subprocess.Popen(command,
                                stdout=subprocess.PIPE,
                                stderr=subprocess.STDOUT,
                                bufsize=1)  # line buffered
while some_criterium and not time_limit:
    poll_result = select([scan_process.stdout], [], [], time_limit)[0]

其余的都一样。

pydoc select.select

[注意:这是特定于Unix的,其他一些答案也是如此。]

[注2:根据操作请求编辑以添加行缓冲]

[注3:在所有情况下,行缓冲可能不可靠,导致readline()阻塞]