用python原子写入文件


atomic writing to file with Python

我使用python在一次操作中将文本块写入文件：

1
open(file, 'w').write(text)

如果脚本被中断，导致文件写入无法完成，我希望没有文件，而不是部分完成的文件。能做到吗？

相关讨论

将数据写入临时文件，成功写入数据后，将文件重命名为正确的目标文件，例如

1
2
3
4
5
6
7
8
9
f = open(tmpFile, 'w')
f.write(text)
# make sure that all data is on disk
# see http://stackoverflow.com/questions/7433057/is-rename-without-fsync-safe
f.flush()
os.fsync(f.fileno())
f.close()

os.rename(tmpFile, myFile)

根据文档http://docs.python.org/library/os.html os.rename

If successful, the renaming will be an atomic operation (this is a
POSIX requirement). On Windows, if dst
already exists, OSError will be raised
even if it is a file; there may be no
way to implement an atomic rename when
dst names an existing file

也

The operation may fail on some Unix flavors if src and dst are on different filesystems.

注：

  • 如果src和dest位置不在同一个文件系统上，则它可能不是原子操作。

  • 如果在电源故障、系统崩溃等情况下，性能/响应比数据完整性更重要，则可以跳过os.fsync步骤。

相关讨论

使用python tempfile实现原子写入的简单代码段。

1
2
with open_atomic('test.txt', 'w') as f:
    f.write("huzza")

或者读写同一个文件：

1
2
3
4
with open('test.txt', 'r') as src:
    with open_atomic('test.txt', 'w') as dst:
        for line in src:
            dst.write(line)

使用两个简单的上下文管理器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
import os
import tempfile as tmp
from contextlib import contextmanager

@contextmanager
def tempfile(suffix='', dir=None):
   """ Context for temporary file.

    Will find a free temporary filename upon entering
    and will try to delete the file on leaving, even in case of an exception.

    Parameters
    ----------
    suffix : string
        optional file suffix
    dir : string
        optional directory to save temporary file in
   """

    tf = tmp.NamedTemporaryFile(delete=False, suffix=suffix, dir=dir)
    tf.file.close()
    try:
        yield tf.name
    finally:
        try:
            os.remove(tf.name)
        except OSError as e:
            if e.errno == 2:
                pass
            else:
                raise

@contextmanager
def open_atomic(filepath, *args, **kwargs):
   """ Open temporary file object that atomically moves to destination upon
    exiting.

    Allows reading and writing to and from the same filename.

    The file will not be moved to destination in case of an exception.

    Parameters
    ----------
    filepath : string
        the file path to be opened
    fsync : bool
        whether to force write the file to disk
    *args : mixed
        Any valid arguments for :code:`open`
    **kwargs : mixed
        Any valid keyword arguments for :code:`open`
   """
    fsync = kwargs.get('fsync', False)

    with tempfile(dir=os.path.dirname(os.path.abspath(filepath))) as tmppath:
        with open(tmppath, *args, **kwargs) as file:
            try:
                yield file
            finally:
                if fsync:
                    file.flush()
                    os.fsync(file.fileno())
        os.rename(tmppath, filepath)
相关讨论

有一个简单的atomicfile助手：https://github.com/sashka/atomicfile

相关讨论

因为很容易弄乱细节，所以我建议使用一个小的库。图书馆的优势在于它处理了所有这些细节，并且正在被一个社区审查和改进。

其中一个库是untaker的python-atomicwrites，它甚至具有适当的Windows支持：

  • https://github.com/untitaker/python-atomicwrites

从自述文件：

1
2
3
4
5
6
7
from atomicwrites import atomic_write

with atomic_write('foo.txt', overwrite=True) as f:
    f.write('Hello world.')
    #"foo.txt" doesn't exist yet.

# Now it does.

我正在使用此代码自动替换/写入文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import os
from contextlib import contextmanager

@contextmanager
def atomic_write(filepath, binary=False, fsync=False):
   """ Writeable file object that atomically updates a file (using a temporary file).

    :param filepath: the file path to be opened
    :param binary: whether to open the file in a binary mode instead of textual
    :param fsync: whether to force write the file to disk
   """

    tmppath = filepath + '~'
    while os.path.isfile(tmppath):
        tmppath += '~'
    try:
        with open(tmppath, 'wb' if binary else 'w') as file:
            yield file
            if fsync:
                file.flush()
                os.fsync(file.fileno())
        os.rename(tmppath, filepath)
    finally:
        try:
            os.remove(tmppath)
        except (IOError, OSError):
            pass

用途：

1
2
3
with atomic_write('path/to/file') as f:
    f.write("allons-y!
")

这是根据这个食谱做的。

相关讨论

用于Windows循环文件夹和重命名文件的原子解决方案。测试，原子自动化，您可以增加概率，以最大限度地减少不具有相同文件名的事件的风险。字母符号组合的随机库对数字str(random.random.range(509999999999,2))使用random.choice方法。您可以根据需要更改数字范围。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import os import random

path ="C:\\Users\\ANTRAS\\Desktop\
UOTRAUKA\"

def renamefiles():
    files = os.listdir(path)
    i = 1
    for file in files:
        os.rename(os.path.join(path, file), os.path.join(path,
                  random.choice('ABCDEFGHIJKL') + str(i) + str(random.randrange(31,9999999,2)) + '.jpg'))
        i = i+1

for x in range(30):
    renamefiles()