关于C#：KIF：如何自动运行/压力测试iOS应用程序以查找罕见UI错误的原因？


KIF: How to auto-run/stress test an iOS app to find the cause of a rare UI bug?

注意：我在标题中添加了kif，只是为了搜索索引蛹，考虑到大多数答案都是为了讨论它。

我在寻找类似于Selenium for iOS的东西，基本上是一个测试自动化/单元测试框架，它可以多次运行特定的UI场景，直到崩溃为止，这将帮助我缩小发生非常罕见和随机的UI错误的原因。

(顺便说一下，我已经记录了数据源/表交互的每一行代码，并花了数小时分析了潜在的原因。)但没有得出结论……同样，这个错误很少发生)。

我研究了iOS中的一些单元测试框架，但它们似乎太多了。我不知道该选哪一个。另外，我提到硒是基于推测，因为我曾与过去在大型Web项目中使用过硒的QA人员合作过(我假设iOS中一定有类似的东西)。

既然我是一个为一个iOS项目工作的单人团队，我就必须戴上一顶质量保证帽来解决这个问题。

当UITableView中插入的实际行数与DataSource委托返回的行数不一致时，我会遇到一个典型的错误。这是错误消息：

1
2
3
4
5
6
7
8
9
*** Assertion failure in -[UITableView
 _endCellAnimationsWithContext:] Exception in insertRows: Invalid
 update: invalid number of rows in section 0.

The number of rows contained in an existing section after the update (2) must be equal to
 the number of rows contained in that section before the update (2),
 plus or minus the number of rows inserted or deleted from that section
 (1 inserted, 0 deleted) and plus or minus the number of rows moved
 into or out of that section (0 moved in, 0 moved out).

我点击一个UITableViewCell，进入另一个UITableView。有时它会起作用

enter image description here

有时(极少数情况下)不会(有上述错误)：

enter image description here

相关讨论

更新:..我在分隔符后面的底部添加了关于kif 2.0的示例代码。对于那些比我所面临的具体问题更感兴趣的人：

经过一些研究和试验……我已经将选项缩小到两个测试自动化库：弗兰克和KIF。我最终决定在借用黄瓜的小黄瓜语法来描述我的单元测试时使用KIF。

我选择EDOCX1(而不是Frank)的原因是KIF是基于100%的obj-c，而不是使用ruby，就像选择Frank一样。所以设置更简单，更适用于我狭窄的测试用例需求。也就是说，我承认如果我的应用程序更复杂(如使用多个服务器的intput等)，Frank会更有用。您可以看到这个优秀的演示的最后一个季度，了解更多关于KIF、Frank和其他自动化测试框架(包括苹果自己的UI自动化)的优缺点。

在使用kif之后，我发现了导致上述错误的bug，并且我可以100%地使用kif复制它！它之所以很少发生，是因为它只发生在我快速点击屏幕的时候。既然kif自动化了这些步骤……它以难以置信的速度完成了任务。这暴露了错误：)。

下面是我用于测试的代码的示例。这只是为了让您快速了解kif(和小黄瓜)可以为您做什么：

在一个文件中，我指定了要运行的方案：

1
2
3
4
5
6
7
- (void)initializeScenarios;
{
    [self addScenario:[KIFTestScenario scenarioToCompleteSignInAndLoadInbox]];
    [self addScenario:[KIFTestScenario scenarioToFillAttachmentsWithData]];
    [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucket]];
    [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucketSubView]];
}

每个场景都映射到步骤(为了进一步了解基于测试驱动程序开发的小黄瓜语法和行为驱动开发，我强烈建议阅读这本关于黄瓜的优秀书籍)：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
/* @given the application is at a fresh state
   @and   the user already has an imap email account with a valid username/pwd

   @then  the user can successfully log in
   @and   the inbox view will be loaded
   @and   the inbox will get loaded with the latest batch of emails in the user inbox
 */
+ (id)scenarioToCompleteSignInAndLoadInbox
{
    KIFTestScenario *scenario =
      [KIFTestScenario scenarioWithDescription:@"Test that a user
                                                 can successfully log in."];
    [scenario addStepsFromArray:[KIFTestStep stepsCompleteSignInAndLoadInbox]];

    return scenario;
}


/* @given that the user is already signed in
   @and   the user has already downloaded their folders

   @then  the user can click on the folders view
   @and   the user can click on the 'attachments' remote folder
   @and   the latest batch from the 'attachments' remote folder will download
 */
+ (id)scenarioToFillAttachmentsWithData {
    KIFTestScenario* scenario =
      [KIFTestScenario scenarioWithDescription:@"Test that we can view the
                                                 attachments folder and fill
                                                 it with data."];
    [scenario addStepsFromArray:[KIFTestStep stepsToFillAttachmentsWithData]];
    return scenario;

}

/* @given that the user is already signed in
   @and   the user has already downloaded their folders
   @and   the user has already downloaded attachments

   @then  the user can click on inbox menu button
   @and   the user can click on folder list menu button
   @and   the user can click on the file bucket icon (on the account list view)
   @and   the data for the file bucket is fetched from the dbase
   @and   the file bucket view displayes the attachments
 */
+ (id)scenarioToViewAndLoadFileBucket {
    KIFTestScenario *scenario =
       [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully
                                                  view and load
                                                  file bucket parent view"];
    [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketPage]];

    return scenario;
}

/* @given that the user is already signed in
   @and   the user has already downloaded their folders
   @and   the user has already downloaded attachments
   @and   the user has already opened file bucket view

   @then  the user can click on a random row in the file bucket view table
   @and   the subview will retrieve data from the dbase pertaining to that row
   @and   the subview will display the data in the uitableview
 */
+ (id)scenarioToViewAndLoadFileBucketSubView {
    KIFTestScenario *scenario =
       [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully
                                                  view and load filet
                                                  bucket sub view"];
    [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketSubPage]];
    return scenario;  
}

步骤是使用KIF的UI自动化方法定义的(这只是一个例子)：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// this step assumes there is an attachment folder that contains emails with attachments
+ (NSArray *)stepsToFillAttachmentsWithData {

    NSMutableArray* steps = [@[] mutableCopy];

    [steps addObject:
        [KIFTestStep stepToTapViewWithAccessibilityLabel:@"InboxMenuButton"]];

    NSIndexPath* indexPath =
        [NSIndexPath indexPathForRow:remoteAttachmentFolderNumber inSection:0];
    KIFTestStep* tapAttachmentRowStep =
        [KIFTestStep stepToTapRowInTableViewWithAccessibilityLabel:
                                     @"attachments" atIndexPath:indexPath];

    [steps addObject:[KIFTestStep stepToWaitForNotificationName:
         (NSString *)kBeganSyncingOlderEmails object:nil          
                          whileExecutingStep:tapAttachmentRowStep]];

    [steps addObject:tapAttachmentRowStep];

    [steps addObject:
        [KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"attachments"]];

    KIFTestStep *fillingInboxStep =
        [KIFTestStep stepToWaitForNotificationName:
                                 (NSString *)kOldMailBatchDelivered object:nil];

    [fillingInboxStep setTimeout:kSpecialTimeoutForLongTests];
    [steps addObject:fillingInboxStep];

    return steps;
}

KIF 2.0样本代码：kif 2.0使用Xcode5的全新测试导航器。这比Kif 1.0所做的要大得多。现在你的测试比过去更加有机和自然。(即实时运行..而不是创建未来运行的场景等)。你甚至可以用播放按钮等来测试每一个。你应该试试看。

以下是一些例子(再次使用gherkin语法)：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#import <KIF/KIF.h>
#import"KIFUITestActor+EXAdditions.h"
#import"KIFUITestActor+UserRegistration.h"

@interface LoginTests : KIFTestCase

@end
@implementation LoginTests

- (void)testReset {
    [tester flushDbase];
    [tester reset];
}

/* @given that the app is in a fresh clean state
 @and   that no one has ever registered with the server

 @then  the user can register their themselves with the server
 @and   immediately start with the rider's map
 @and   their location on the map shows
 */

- (void)testRegistration
{
    [tester flushDbase];
    [tester reset];
    [tester singleUserRegistration];
    [tester showUserCurrentLocationOnMap];
}

/* @given that the user has already registered with the server
   @and the user is not currently logged in

 @then  the user can login using their user name and password
 @and   immediately start with the rider's map
 @and   their location on the map shows
 */
- (void)testSuccessfulLogin
{
    [tester reset];
    [tester login];
    [tester showUserCurrentLocationOnMap];
}

/* @given that the user has already registered
   @and that the user is already logged in before app launch

 @then the user starts on the map view with the location visible
 @and the button prompts them to set pick up location
 */
- (void)testStartOfApplication {
    [tester showUserCurrentLocationOnMap];
    [tester showsPickUpButton];
}
@end

下面是类别文件中一些测试用例的实现：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
- (void)reset
{
    [self runBlock:^KIFTestStepResult(NSError **error) {
        BOOL successfulReset = YES;

        // Do the actual reset for your app. Set successfulReset = NO if it fails.
        AppDelegate* appDelegate = [[UIApplication sharedApplication] delegate];
        [appDelegate resetApp];

        KIFTestCondition(successfulReset, error, @"Failed to reset some part of the application.");

        return KIFTestStepResultSuccess;
    }];
}

- (void)flushDbase {
    [self runBlock:^KIFTestStepResult(NSError **error){
        NSURL *url = [NSURL URLWithString:@"http://randomdomain.com/flush_db"];
        NSURLRequest *request = [NSURLRequest requestWithURL:url];
        NSError *connectionError = nil;

        BOOL databaseFlushSucceeded = YES;

        NSURLResponse *response;
        NSData *resultData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&connectionError];
        if (!resultData) {
            databaseFlushSucceeded = NO;
            KIFTestCondition(databaseFlushSucceeded, error, @"failed to connect to server!");
        }

        if (connectionError) {
            databaseFlushSucceeded = NO;
            KIFTestCondition(databaseFlushSucceeded, error, [NSString stringWithFormat:@"connection failed. Error: %@", [connectionError localizedDescription]]);
        }

        return KIFTestStepResultSuccess;
    }];
}


- (void)navigateToLoginPage
{
    [self tapViewWithAccessibilityLabel:@"login email"];
}

- (void)returnToLoggedOutHomeScreen
{
    [self tapViewWithAccessibilityLabel:@"Logout"];
    [self tapViewWithAccessibilityLabel:@"Logout"]; // Dismiss alert.
}
相关讨论