Wednesday, July 23, 2014

Breaking Mac Apps

One of the reasons I took my current gig as a Software Development Engineer Test (SDET) was so I could learn more about my craft of software development. Too often as the developer under stress to get something out the door it can be tempting to do the minimum testing possible to get the feature out the door. That temptation is tempered by the fact that you know if you ship crap then you will have more headaches than spending the extra time now to get it right.

So, I wanted to concentrate on breaking a product too see what I learn by finding it's rough spots. This has been fun at times in the sense that I am actually learning ways to more effectively test my own software. I plan to use this knowledge to write better software myself.

I am currently testing a product on both Mac OS X and iOS. As a dev I have some things I learned from writing Mac apps that I have been able to put to use in my job as a tester.

One of those is to always test the First Run case. What is this? The First Run test involves running the app as if you were a new user installing and running the app for the first time.

How to do this on Mac?

  • Does the app use the Keychain to store anything? Find out and make sure to clear it before you run the test.
  • Does the app use the User Defaults to store anything? Probably, so go learn the command line tool defaults. Here is a short primer:
    • Open Terminal.app
    • Run this: defaults
      • Now you have the standard usage (screenshot shown above).
    • If you app's defaults are stored in the domain "com.mycompany.myapp" then type this to view what the app is storing: 
      • defaults read com.mycompany.myapp
    • Now before a First Run test run this to clear the stored setting: 
      • defaults delete com.mycompany.myapp
  • Does your app use any other storage? SQLLite for example? CoreData store?
    • Find out, find out where the files are stored and then delete them before you run a First Run test.
Run the test and see what blows up.

Too often we forget to retest this scenario as we work on features and the simple First Run case can get broken.

Test it often to weed out problems before new users find them.  I always run this on candidate builds for a new release to end users. I am also always surprised when this breaks since we seldom think the simple changes we are making in our app will break the First Run.

3 comments:

Grant McSheffrey said...

One thing to remember about iOS apps is that removing the app doesn't remove all settings related to your app. The device remembers things like privacy settings that the user previously responded to. There is an option to clear settings that you should use if you want to get the First Run experience.

Mark Thistle said...

Are you speaking of the simulator or some other option? I know of the option to clear settings in the simulator but not for an app on device. Where is the option to clear settings for an installed app?

From my understanding, to be persistent on device the settings would need to be stored in the Keychain or iCloud.

Are there others?

Grant McSheffrey said...

I'm talking about the system settings (ones that result from an iOS prompt inside your app like "AwesomeApp wants to access your contacts"). You can clear them from Settings | General | Reset | Reset Location & Privacy.