关于性能：PostgreSQL如何比SQLite更快地执行写入？

How does PostgreSQL perform writes so much faster than SQLite?

我做了一个简单的整数更新性能测试。SQLite每秒只更新15次，PostgreSQL每秒更新1500次。

带有sqlite大小写的数字似乎是正常的。

sqlite站点中的常见问题解答解释了旋转磁盘的基本限制。

Actually, SQLite will easily do 50,000 or more INSERT statements per
second on an average desktop computer. But it will only do a few dozen
transactions per second. Transaction speed is limited by the
rotational speed of your disk drive. A transaction normally requires
two complete rotations of the disk platter, which on a 7200RPM disk
drive limits you to about 60 transactions per second. Transaction
speed is limited by disk drive speed because (by default) SQLite
actually waits until the data really is safely stored on the disk
surface before the transaction is complete. That way, if you suddenly
lose power or if your OS crashes, your data is still safe. For
details, read about atomic commit in SQLite..

By default, each INSERT statement is its own transaction. But if you
surround multiple INSERT statements with BEGIN...COMMIT then all the
inserts are grouped into a single transaction. The time needed to
commit the transaction is amortized over all the enclosed insert
statements and so the time per insert statement is greatly reduced.

Another option is to run PRAGMA synchronous=OFF. This command will
cause SQLite to not wait on data to reach the disk surface, which will
make write operations appear to be much faster. But if you lose power
in the middle of a transaction, your database file might go corrupt.

这个描述是真的吗？那么，postgresql怎么能比sqlite快那么多呢？(我在PostgreSQL中将fsync和synchronous_commit选项都设置为on)

更新：

以下是用Clojure编写的完整测试代码：

1
2
3
4
5
6
7

(defproject foo"0.1.0-SNAPSHOT"
:repositories {"sonatype-oss-public""https://oss.sonatype.org/content/groups/public/"}
:dependencies [[org.clojure/clojure"1.5.1"]
[org.clojure/java.jdbc"0.3.0-SNAPSHOT"]
[com.mchange/c3p0"0.9.2.1"]
[org.xerial/sqlite-jdbc"3.7.2"]
[postgresql"9.1-901.jdbc4"]])

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

(ns foo.core
(:require [clojure.java.jdbc :AS jdbc]
[clojure.java.jdbc.ddl :AS ddl])
(:import [com.mchange.v2.c3p0 ComboPooledDataSource]))

(def sqlite
(let [spec {:classname"org.sqlite.JDBC"
:subprotocol"sqlite"
:subname"test.db"}]
{:datasource (doto (ComboPooledDataSource.)
(.setDriverClass (:classname spec))
(.setJdbcUrl (str"jdbc:" (:subprotocol spec)":" (:subname spec)))
(.setMaxIdleTimeExcessConnections (* 30 60))
(.setMaxIdleTime (* 3 60 60)))}))

(def postgres
(let [spec {:classname"org.postgresql.Driver"
:subprotocol"postgresql"
:subname"//localhost:5432/testdb"
:USER"postgres"
:password"uiop"}]
{:datasource (doto (ComboPooledDataSource.)
(.setDriverClass (:classname spec))
(.setJdbcUrl (str"jdbc:" (:subprotocol spec)":" (:subname spec)))
(.setUser (:USER spec))
(.setPassword (:password spec))
(.setMaxIdleTimeExcessConnections (* 30 60))
(.setMaxIdleTime (* 3 60 60)))}))

(doseq [x [sqlite postgres]]
(jdbc/db-do-commands x
(ddl/create-TABLE :foo [:id :INT"PRIMARY KEY"] [:bar :INT])))

(doseq [x [sqlite postgres]]
(jdbc/INSERT! x :foo {:id 1 :bar 1}))

(defmacro bench
[expr n]
`(dotimes [_# 3]
(let [start# (. System (nanoTime))]
(dotimes [_# ~n]
~expr)
(let [end# (. System (nanoTime))
elapsed# (/ (double (- end# start#)) 1000000.0)
operation-per-sec# (long (/ (double ~n) (/ (double (- end# start#)) 1000000000)))]
(prn (str"Elapsed time:" elapsed#" ms (" (format"%,d" operation-per-sec#)" ops)"))))))

(bench (jdbc/query sqlite ["select * from foo"]) 20000)
(bench (jdbc/execute! sqlite ["update foo set bar=bar+1 where id=?" 1]) 100)

(bench (jdbc/query postgres ["select * from foo"]) 20000)
(bench (jdbc/execute! postgres ["update foo set bar=bar+1 where id=?" 1]) 5000)

输出为：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

; Running"select * from foo" 20000 times IN SQLite

"Elapsed time: 1802.426963 ms (11,096 ops)"
"Elapsed time: 1731.118831 ms (11,553 ops)"
"Elapsed time: 1749.842658 ms (11,429 ops)"

; Running"update foo set bar=bar+1 where id=1" 100 times IN SQLite

"Elapsed time: 6362.829057 ms (15 ops)"
"Elapsed time: 6405.25075 ms (15 ops)"
"Elapsed time: 6352.943553 ms (15 ops)"

; Running"select * from foo" 20000 times IN PostgreSQL

"Elapsed time: 2898.636079 ms (6,899 ops)"
"Elapsed time: 2824.77372 ms (7,080 ops)"
"Elapsed time: 2837.622659 ms (7,048 ops)"

; Running"update foo set bar=bar+1 where id=1" 5000 times IN PostgreSQL

"Elapsed time: 3213.120219 ms (1,556 ops)"
"Elapsed time: 3564.249492 ms (1,402 ops)"
"Elapsed time: 3280.128708 ms (1,524 ops)"

pg同步测试结果：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

C:\temp>"C:\Program Files\PostgreSQL\9.3\bin\pg_test_fsync"
5 seconds per test
O_DIRECT supported ON this platform FOR open_datasync AND open_sync.

Compare file sync methods USING one 8kB WRITE:
(IN wal_sync_method preference ORDER, EXCEPT fdatasync
IS Linux's default)
open_datasync 81199.920 ops/sec 12 usecs/op
fdatasync n/a
fsync 45.337 ops/sec 22057 usecs/op
fsync_writethrough 46.470 ops/sec 21519 usecs/op
open_sync n/a

Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync
is Linux's DEFAULT)
open_datasync 41093.981 ops/sec 24 usecs/op
fdatasync n/a
fsync 38.569 ops/sec 25927 usecs/op
fsync_writethrough 36.970 ops/sec 27049 usecs/op
open_sync n/a

Compare open_sync WITH different WRITE sizes:
(This IS designed TO compare the cost OF writing 16kB
IN different WRITE open_sync sizes.)
1 * 16kB open_sync WRITE n/a
2 * 8kB open_sync writes n/a
4 * 4kB open_sync writes n/a
8 * 2kB open_sync writes n/a
16 * 1kB open_sync writes n/a

Test IF fsync ON non-WRITE file descriptor IS honored:
(IF the times are SIMILAR, fsync() can sync DATA written
ON a different descriptor.)
WRITE, fsync, close 45.564 ops/sec 21947 usecs/op
WRITE, close, fsync 33.373 ops/sec 29964 usecs/op

Non-Sync'ed 8kB writes:
write 889.800 ops/sec 1124 usecs/op

相关讨论

它分为如何实现快照隔离。

SQLite使用文件锁定作为隔离事务的一种方法，只允许在完成所有读取后命中写操作。

相比之下，Postgres使用了一种更复杂的方法，称为多并发版本控制(MVCC)，它允许多个写操作与多个读操作并行进行。

http://www.sqliteconcepts.org/si_index.html

http://www.postgresql.org/docs/current/static/mvc-intro.html网站

网址：http://wiki.postgresql.org/wiki/mvc

相关讨论

你猜疑是对的。PostgreSQL使用您所指示的设置，不应该以每秒独立的顺序事务对旋转媒体执行任何接近1500次的更新。

可能IO堆栈中有关于如何实现同步的谎言或错误。这意味着您的数据在意外断电或操作系统故障后有严重损坏的风险。

从pg_test fsync的结果来看，情况确实如此。open-datasync是windows下的默认设置，它的速度似乎不太现实，因此必须是不安全的。当我在windows7机器上运行pg测试同步时，我也看到了同样的情况。

相关讨论

丹尼斯的答案有你需要的所有链接。我将寻求一个不太详细但可能更容易理解的答案。

sqlite不使用任何复杂的事务管理器，它没有隐藏高级多任务逻辑。它执行您告诉它执行的内容，完全按照这个顺序执行。换句话说：它完全按照你告诉它要做的去做。如果您尝试使用来自两个进程的同一个数据库，则会遇到问题。

另一方面，PostgreSQL是一个非常复杂的数据库：它有效地支持多个并发读写。把它想象成一个异步系统——你只安排要完成的工作，实际上你并不控制它的细节——Postgres是为你做的。

你的效率如何？在一个事务中加入几个-几十个-数百个更新/插入。对于一个简单的桌子，你会得到一个非常好的性能。

实际上，在旋转盘上的任何写入都是10毫秒的量级(典型的数字是8毫秒)。

这意味着，如果您在磁盘中写入相同的位置，则每秒写入的次数略多于100次，这对于数据库来说是非常奇怪的情况。请参阅ACM中的"你不了解磁盘的杰克"，通常一个磁盘可以在一次旋转中安排10次读或写。

http://queue.acm.org/detail.cfm？ID＝864058

因此，数据库可以每秒执行1000次写入，甚至更多。10年前，我见过应用程序在台式机上每秒执行1500个事务。

假设您使用的是普通硬盘(即没有SSD)，则每秒最多可以写入50-100次。似乎每秒15次的写入速度略低，但并非不可能。

因此，如果Postgres每秒进行1500次更新，那么它们要么被写入某个缓冲区/缓存，要么折叠成一个更新。在不了解更多关于实际测试的信息的情况下，很难说这是真正的原因，但是如果您要打开一个事务，那么更新一行1500次，然后提交，Postgres应该足够聪明，只执行一次对磁盘的"实际"写操作。

相关讨论