Continuing in Python's unittest when an assertion fails
编辑：切换到一个更好的例子，并澄清为什么这是一个真正的问题。
我想在Python中编写单元测试，在断言失败时继续执行，这样我就可以在单个测试中看到多个失败。例如：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | class Car(object): def __init__(self, make, model): self.make = make self.model = make # Copy and paste error: should be model. self.has_seats = True self.wheel_count = 3 # Typo: should be 4. class CarTest(unittest.TestCase): def test_init(self): make ="Ford" model ="Model T" car = Car(make=make, model=model) self.assertEqual(car.make, make) self.assertEqual(car.model, model) # Failure! self.assertTrue(car.has_seats) self.assertEqual(car.wheel_count, 4) # Failure! |
这里，测试的目的是确保Car的
如果我们假设这里最好不分解方法，那么我有一个新问题：我无法立即看到所有错误。当我修复
为了比较，Google的C ++单元测试框架区分了非致命的
The assertions come in pairs that test the same thing but have different effects on the current function. ASSERT_* versions generate fatal failures when they fail, and abort the current function. EXPECT_* versions generate nonfatal failures, which don't abort the current function. Usually EXPECT_* are preferred, as they allow more than one failures to be reported in a test. However, you should use ASSERT_* if it doesn't make sense to continue when the assertion in question fails.
有没有办法在Python的
顺便说一下，我很好奇有多少现实测试可能会从非致命断言中受益，所以我看了一些代码示例(编辑2014-08-19使用searchcode代替Google Code Search，RIP)。在第一页中随机选择的10个结果中，所有结果都包含在同一测试方法中进行多个独立断言的测试。所有人都将受益于非致命的断言。
使用非致命断言的另一种方法是捕获断言异常并将异常存储在列表中。然后声明该列表是空的，作为tearDown的一部分。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | import unittest class Car(object): def __init__(self, make, model): self.make = make self.model = make # Copy and paste error: should be model. self.has_seats = True self.wheel_count = 3 # Typo: should be 4. class CarTest(unittest.TestCase): def setUp(self): self.verificationErrors = [] def tearDown(self): self.assertEqual([], self.verificationErrors) def test_init(self): make ="Ford" model ="Model T" car = Car(make=make, model=model) try: self.assertEqual(car.make, make) except AssertionError, e: self.verificationErrors.append(str(e)) try: self.assertEqual(car.model, model) # Failure! except AssertionError, e: self.verificationErrors.append(str(e)) try: self.assertTrue(car.has_seats) except AssertionError, e: self.verificationErrors.append(str(e)) try: self.assertEqual(car.wheel_count, 4) # Failure! except AssertionError, e: self.verificationErrors.append(str(e)) if __name__ =="__main__": unittest.main() |
一个选项是立即将所有值断言为元组。
例如：
1 2 3 4 5 6 7 8 | class CarTest(unittest.TestCase): def test_init(self): make ="Ford" model ="Model T" car = Car(make=make, model=model) self.assertEqual( (car.make, car.model, car.has_seats, car.wheel_count), (make, model, True, 4)) |
此测试的输出将是：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | ====================================================================== FAIL: test_init (test.CarTest) ---------------------------------------------------------------------- Traceback (most recent call last): File"C:\temp\py_mult_assert\test.py", line 17, in test_init (make, model, True, 4)) AssertionError: Tuples differ: ('Ford', 'Ford', True, 3) != ('Ford', 'Model T', True, 4) First differing element 1: Ford Model T - ('Ford', 'Ford', True, 3) ? ^ - ^ + ('Ford', 'Model T', True, 4) ? ^ ++++ ^ |
这表明模型和车轮计数都不正确。
您可能想要做的是派生
在单个单元测试中具有多个断言被认为是反模式。单个单元测试预计只测试一件事。也许你的测试太多了。考虑将此测试分成多个测试。这样您就可以正确命名每个测试。
但有时候，可以同时检查多个内容。例如，当您断言同一对象的属性时。在这种情况下，您实际上断言该对象是否正确。一种方法是编写一个自定义帮助器方法，该方法知道如何在该对象上断言。您可以编写该方法，使其显示所有失败的属性，或者例如在断言失败时显示预期对象的完整状态和实际对象的完整状态。
每个断言都用一个单独的方法。
1 2 3 4 5 6 7 8 9 10 11 12 | class MathTest(unittest.TestCase): def test_addition1(self): self.assertEqual(1 + 0, 1) def test_addition2(self): self.assertEqual(1 + 1, 3) def test_addition3(self): self.assertEqual(1 + (-1), 0) def test_addition4(self): self.assertEqaul(-1 + (-1), -1) |
PyPI中有一个名为
例如，这段代码：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | import softest class ExampleTest(softest.TestCase): def test_example(self): # be sure to pass the assert method object, not a call to it self.soft_assert(self.assertEqual, 'Worf', 'wharf', 'Klingon is not ship receptacle') # self.soft_assert(self.assertEqual('Worf', 'wharf', 'Klingon is not ship receptacle')) # will not work as desired self.soft_assert(self.assertTrue, True) self.soft_assert(self.assertTrue, False) self.assert_all() if __name__ == '__main__': softest.main() |
...生成此控制台输出：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | ====================================================================== FAIL:"test_example" (ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File"C:\...\softest_test.py", line 14, in test_example self.assert_all() File"C:\...\softest\case.py", line 138, in assert_all self.fail(''.join(failure_output)) AssertionError: ++++ soft assert failure details follow below ++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The following 2 failures were found in"test_example" (ExampleTest): ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Failure 1 ("test_example" method) +--------------------------------------------------------------------+ Traceback (most recent call last): File"C:\...\softest_test.py", line 10, in test_example self.soft_assert(self.assertEqual, 'Worf', 'wharf', 'Klingon is not ship receptacle') File"C:\...\softest\case.py", line 84, in soft_assert assert_method(*arguments, **keywords) File"C:\...\Python\Python36-32\lib\unittest\case.py", line 829, in assertEqual assertion_func(first, second, msg=msg) File"C:\...\Python\Python36-32\lib\unittest\case.py", line 1203, in assertMultiLineEqual self.fail(self._formatMessage(msg, standardMsg)) File"C:\...\Python\Python36-32\lib\unittest\case.py", line 670, in fail raise self.failureException(msg) AssertionError: 'Worf' != 'wharf' - Worf + wharf : Klingon is not ship receptacle +--------------------------------------------------------------------+ Failure 2 ("test_example" method) +--------------------------------------------------------------------+ Traceback (most recent call last): File"C:\...\softest_test.py", line 12, in test_example self.soft_assert(self.assertTrue, False) File"C:\...\softest\case.py", line 84, in soft_assert assert_method(*arguments, **keywords) File"C:\...\Python\Python36-32\lib\unittest\case.py", line 682, in assertTrue raise self.failureException(msg) AssertionError: False is not true ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (failures=1) |
注意：我创建并维护
期望在gtest中非常有用。
这是gist中的python方式，代码：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | import sys import unittest class TestCase(unittest.TestCase): def run(self, result=None): if result is None: self.result = self.defaultTestResult() else: self.result = result return unittest.TestCase.run(self, result) def expect(self, val, msg=None): ''' Like TestCase.assert_, but doesn't halt the test. ''' try: self.assert_(val, msg) except: self.result.addFailure(self, sys.exc_info()) def expectEqual(self, first, second, msg=None): try: self.failUnlessEqual(first, second, msg) except: self.result.addFailure(self, sys.exc_info()) expect_equal = expectEqual assert_equal = unittest.TestCase.assertEqual assert_raises = unittest.TestCase.assertRaises test_main = unittest.main |
我喜欢@ Anthony-Batchelor的方法来捕获AssertionError异常。但是这种使用装饰器的方法略有不同，也是一种通过pass / fail报告测试用例的方法。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | #!/usr/bin/env python # -*- coding: utf-8 -*- import unittest class UTReporter(object): ''' The UT Report class keeps track of tests cases that have been executed. ''' def __init__(self): self.testcases = [] print"init called" def add_testcase(self, testcase): self.testcases.append(testcase) def display_report(self): for tc in self.testcases: msg ="=============================" +" " + \ "Name:" + tc['name'] +" " + \ "Description:" + str(tc['description']) +" " + \ "Status:" + tc['status'] +" " print msg reporter = UTReporter() def assert_capture(*args, **kwargs): ''' The Decorator defines the override behavior. unit test functions decorated with this decorator, will ignore the Unittest AssertionError. Instead they will log the test case to the UTReporter. ''' def assert_decorator(func): def inner(*args, **kwargs): tc = {} tc['name'] = func.__name__ tc['description'] = func.__doc__ try: func(*args, **kwargs) tc['status'] = 'pass' except AssertionError: tc['status'] = 'fail' reporter.add_testcase(tc) return inner return assert_decorator class DecorateUt(unittest.TestCase): @assert_capture() def test_basic(self): x = 5 self.assertEqual(x, 4) @assert_capture() def test_basic_2(self): x = 4 self.assertEqual(x, 4) def main(): #unittest.main() suite = unittest.TestLoader().loadTestsFromTestCase(DecorateUt) unittest.TextTestRunner(verbosity=2).run(suite) reporter.display_report() if __name__ == '__main__': main() |
控制台输出：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | (awsenv)$ ./decorators.py init called test_basic (__main__.DecorateUt) ... ok test_basic_2 (__main__.DecorateUt) ... ok ---------------------------------------------------------------------- Ran 2 tests in 0.000s OK ============================= Name: test_basic Description: None Status: fail ============================= Name: test_basic_2 Description: None Status: pass |
我意识到这个问题几年前就被问过了，但现在(至少)有两个Python软件包允许你这样做。
一个是最软的：https：//pypi.org/project/softest/
另一个是Python-Delayed-Assert：https：//github.com/pr4bh4sh/python-delayed-assert
我也没用过，但看起来和我很相似。
@Anthony Batchelor回答我遇到了问题，因为它迫使我在我的单元测试中使用
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | import unittest import traceback class AssertionErrorData(object): def __init__(self, stacktrace, message): super(AssertionErrorData, self).__init__() self.stacktrace = stacktrace self.message = message class MultipleAssertionFailures(unittest.TestCase): def __init__(self, *args, **kwargs): self.verificationErrors = [] super(MultipleAssertionFailures, self).__init__( *args, **kwargs ) def tearDown(self): super(MultipleAssertionFailures, self).tearDown() if self.verificationErrors: index = 0 errors = [] for error in self.verificationErrors: index += 1 errors.append("%s AssertionError %s: %s" % ( error.stacktrace, index, error.message ) ) self.fail( ' ' +" ".join( errors ) ) self.verificationErrors.clear() def assertEqual(self, goal, results, msg=None): try: super( MultipleAssertionFailures, self ).assertEqual( goal, results, msg ) except unittest.TestCase.failureException as error: goodtraces = self._goodStackTraces() self.verificationErrors.append( AssertionErrorData(" ".join( goodtraces[:-2] ), error ) ) def _goodStackTraces(self): """ Get only the relevant part of stacktrace. """ stop = False found = False goodtraces = [] # stacktrace = traceback.format_exc() # stacktrace = traceback.format_stack() stacktrace = traceback.extract_stack() # https://stackoverflow.com/questions/54499367/how-to-correctly-override-testcase for stack in stacktrace: filename = stack.filename if found and not stop and \ not filename.find( 'lib' ) < filename.find( 'unittest' ): stop = True if not found and filename.find( 'lib' ) < filename.find( 'unittest' ): found = True if stop and found: stackline = ' File"%s", line %s, in %s %s' % ( stack.filename, stack.lineno, stack.name, stack.line ) goodtraces.append( stackline ) return goodtraces # class DummyTestCase(unittest.TestCase): class DummyTestCase(MultipleAssertionFailures): def setUp(self): self.maxDiff = None super(DummyTestCase, self).setUp() def tearDown(self): super(DummyTestCase, self).tearDown() def test_function_name(self): self.assertEqual("var","bar" ) self.assertEqual("1937","511" ) if __name__ == '__main__': unittest.main() |
结果输出：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | F ====================================================================== FAIL: test_function_name (__main__.DummyTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File"D:\User\Downloads\test.py", line 77, in tearDown super(DummyTestCase, self).tearDown() File"D:\User\Downloads\test.py", line 29, in tearDown self.fail( ' ' +" ".join( errors ) ) AssertionError: File"D:\User\Downloads\test.py", line 80, in test_function_name self.assertEqual("var","bar" ) AssertionError 1: 'var' != 'bar' - var ? ^ + bar ? ^ : File"D:\User\Downloads\test.py", line 81, in test_function_name self.assertEqual("1937","511" ) AssertionError 2: '1937' != '511' - 1937 + 511 : |
正确的堆栈跟踪捕获的更多替代解决方案可以发布在如何正确覆盖TestCase.assertEqual()，产生正确的堆栈跟踪？
我不认为有一种方法可以用PyUnit做到这一点，并且不希望看到PyUnit以这种方式扩展。
我更倾向于每个测试函数坚持一个断言(或者更具体地说，每个测试断言一个概念)并且将
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | .FF. ====================================================================== FAIL: test_addition_with_two_negatives (__main__.MathTest) ---------------------------------------------------------------------- Traceback (most recent call last): File"test_addition.py", line 10, in test_addition_with_two_negatives self.assertEqual(-1 + (-1), -1) AssertionError: -2 != -1 ====================================================================== FAIL: test_addition_with_two_positives (__main__.MathTest) ---------------------------------------------------------------------- Traceback (most recent call last): File"test_addition.py", line 6, in test_addition_with_two_positives self.assertEqual(1 + 1, 3) # Failure! AssertionError: 2 != 3 ---------------------------------------------------------------------- Ran 4 tests in 0.000s FAILED (failures=2) |
如果您认为此方法不适合您，您可能会发现此答案很有帮助。
更新
看起来您正在使用更新的问题测试两个概念，我会将它们分成两个单元测试。第一个是在创建新对象时存储参数。这将有两个断言，一个用于
第二个概念更值得怀疑......您正在测试是否初始化了一些默认值。为什么？在实际使用它们时测试这些值会更有用(如果它们没有被使用，那么为什么它们在那里？)。
这两个测试都失败了，两者都应该。当我进行单元测试时，我对失败的兴趣远远超过我的成功，因为这是我需要集中注意力的地方。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | FF ====================================================================== FAIL: test_creation_defaults (__main__.CarTest) ---------------------------------------------------------------------- Traceback (most recent call last): File"test_car.py", line 25, in test_creation_defaults self.assertEqual(self.car.wheel_count, 4) # Failure! AssertionError: 3 != 4 ====================================================================== FAIL: test_creation_parameters (__main__.CarTest) ---------------------------------------------------------------------- Traceback (most recent call last): File"test_car.py", line 20, in test_creation_parameters self.assertEqual(self.car.model, self.model) # Failure! AssertionError: 'Ford' != 'Model T' ---------------------------------------------------------------------- Ran 2 tests in 0.000s FAILED (failures=2) |