Upgrading a mobile tester’s weapons with advanced debugging

Upgrading a Mobile Tester’s Weapons with Advanced Debugging 

Abhijeet Vaikar, Deepak Thorecha

 

This paper was presented in #ATAGTR2017

Abstract

Almost all software teams are aggressively adopting Agile practices for software development thereby dissolving the collaboration barriers between developers and testers. The industry is demanding more technical testers who possess additional skills that enable them to independently test a product across different layers of a mobile product’s architecture. Testing each layer needs knowledge of its respective technology and tools. On the contrary, many manual mobile testers are still limiting themselves to performing purely black box testing of mobile websites and mobile native/hybrid apps.

Challenges that can be seen due to lack of open mindset towards tool based testing techniques:

  • Testing coverage and strategy is poor.
  • Sufficient details for defects are not provided.
  • Testing activity gets delayed due to a strong dependency on developers to guide non-technical testers on the technology and tools.

 

Keeping in mind the above challenges, it is imperative that manual mobile testers adopt an open, proactive mindset and upgrade themselves with the latest techniques and tools that would enable them to perform better testing independently.

This whitepaper outlines some of the tools and techniques for advanced debugging that can be added to the best practices for testing mobile web, native and hybrid apps on the Android and iOS platforms.

 

Background

Typical Mobile Web App Architecture

A mobile web application/site is accessed primarily via mobile browsers like Chrome, Firefox, and Safari. The core technologies that make up a mobile web application’s presentation layer are:

  • HTML – Core DOM structure that makes up a web page
  • CSS – Styling sheet that defines the layout and look-and-feel of elements on the web page.
  • JavaScript (For application logic, data binding, complex UI rendering and networking operations)
  • Local storage for offline access.

Typical Mobile Native/Hybrid App Architecture

A mobile native application is built with components that handle presentation, interaction, networking, logic, and data processing using the vendor’s native SDK. Each vendor has its own SDK. A hybrid application is a mix of native and web components that run eventually on a native run-time.

 

 

Conventional black-box approach towards testing mobile web apps and native/hybrid apps

Mobile testers that adopt the conventional black-box testing approach are exposed only to the applications alone for testing and are limited to the types of issues that they can find and the tests that they can execute. The closest that they can get to debugging web applications are by running them on desktop browsers in a simulated fashion. As far as native or hybrid applications are concerned, it is not possible to debug them unless testers adopt the development tools that provide debugging features.

Challenges due to traditional black-box testing approach

  • Limited testing coverage and strategy. Certain tests which could otherwise be covered and executed with the help of debugging tools cannot be covered without them. For example:
    • Figuring out how many requests are being sent to the server on click of a button and what is the accurate response time taken.
    • Testing a negative scenario on the app which can be triggered only when a certain parameter is sent in the request that gets sent on performing an action.
    • Testing behavior of the app that depends on a certain value that is stored in the local storage of the browser
  • Limited exposure to defect related information. Without the use of additional tools, testers are left with only the application’s visual behavior as the source of truth for any defects found by them. This leads to the tester not being able to provide sufficient information to the developer about defects found. This increases chances of developers finding it difficult to quickly reproduce defects on their environment.
  • Overall impact on project timelines. Project timeline can get impacted due to exclusion of debugging techniques in formally defined test strategy which in turn leads to poorer coverage of tests. This leads to bugs being discovered in the later stages of the project or on production exactly from the uncovered tests. Timelines can also get impacted due to developer-tester collaboration overhead that arises due to testers not providing all the necessary technical information for defects right when they were logged. Guiding testers on technical tasks also takes time which developers need to invest into if testers are non-technical.

 

Technical Testing Debt. We can define Technical Testing Debt as debt that arises due to performing quick informal testing without the efficient use of tools and techniques, initially in the project. This debt is ok as long as testers ensure that they eventually evolve and upgrade themselves with knowledge on the product technology and tools that help in better test coverage, defect management and collaboration with developers.

 

Example of tester developer collaboration that involves the mentioned challenges

 

Current Testing Trends

 

These testing trends are indicative of the need for traditional black-box mobile testers to up-skill themselves on the technologies and tools that enable them to:

  • Debug application issues & provide better defect artefacts to the developers.
  • Obtain better test coverage.
  • Develop a mind-set of proactively seeking out alternative & additional techniques of testing with the help of debugging tools.
  • Stay relevant in the industry.

 

Solutions

Purely black-box testers can make use of the following tools to support and upgrade their approach for testing mobile web applications and native applications:

  • Browser Developer Tools
  • Network Proxy Debugging Tools
  • Device Logs Monitoring Tools

Let us look at these tools from 2 factors perspective:

  1. Debug information they provide.
  2. Additional testability features they provide.

 

1. Browser Developer Tools

One of the widely adopted debugging strategies for web applications, in general, is using the browser’s developer tools. These developer tools are shipped along with the browser installations and can provide an array of features that can let developers/testers debug web applications. In the early days of mobile development, many developers and testers used to debug mobile websites on desktop browsers using the mobile simulation mode. This mode does not do complete justice to the testing as testing on a real device does. This practice is followed by many mobile testers even today.

Simulated testing of mobile web

We are really lucky to be in the age where the rise of a large number of web development platforms and technology stacks, has given birth to a feature-rich suite of robust developer tools by the major browser vendors to test mobile web remotely on devices directly. These developer tools can be connected to real devices and can provide meaningful information about your web application.

A generic process of connecting devices to developer tools

A generic process of connecting devices to developer tools

 

Comparison of browser developer tools with respect to mobile debugging

 

1.2 How to debug embedded web views in native Android and iOS apps?

Android: You need to configure the web views in your native Android app so that they can be debugged within developer tools. In order to configure, you need to call the static method setWebContentsDebuggingEnabled on the WebView class. Testers can get this done by asking the developers to include the below code in the application on the test builds:

if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
     WebView.setWebContentsDebuggingEnabled(true);
 }

iOS: Unfortunately, you cannot debug web views of an iOS app that is downloaded from the app store. To do so you will need to build and run the app from XCode with your device detected by and connected to XCode. Once your app is built and run on the device, you can then debug webviews inside it either by connecting to Chrome using iOS-webkit-debug-proxy OR to Firefox using Valence add-on OR to Safari browser on your desktop.

 

Debugging information browser developer tools provide:

  1. Complete DOM structure of the web page.
  2. CSS styles applied to the DOM
  3. Request and Response information to find network issues
  4. Page Resources (Html, CSS, JS, Images)
  5. Runtime performance of different events that take place on loading web page
  6. Memory profiling to find memory leak problems.
  7. JavaScript errors and warnings.
  8. Non-binary data sent over WebSockets (Non-HTTP Real-time communication)
  9. Cached content
  10. Cookie information
  11. Data stored in local storage, session storage, IndexedDB, Web SQL.

Additional testability features browser developer tools provide:

  1. Live-edit HTML elements (add, update, delete elements and their attributes), CSS values.
  2. Switch context to debug iframes embedded inside pages.
  3. Edit and Re-send request from Network Panel (Supported by Firefox Dev Tools)
  4. Enable/Disable Cache
  5. Edit/Delete cookies
  6. Edit/Delete local and session storage data

 

2. Network Proxy Debugging Tools

Unlike earlier days of mobile apps, most of the modern mobile native applications connect to the internet to fetch data. Application UI, internal logic, and background services thus depend a lot on the network calls made. It’s easier to debug network activity of web applications by connecting them to browser developer tools. Native applications, however, do not have consumer friendly developer tools to inspect network activity. Many issues can be uncovered by letting device’s network calls flow through a network proxy debugging tool. Tools that we can utilise are:

  • Charles Proxy Debugging Tool
  • Fiddler Web Debugging Proxy

 

Some scenario examples where network proxy debugging tools can come to rescue:

  1. An expectation in a test is to display specific data on the UI but the app shows a blank screen.
  2. An expectation in a test is to display specific data on the UI but data shown is different.
  3. An application should display a locally triggered notification on the device based on data received from the server but there is no notification shown.
  4. An expectation is that calls made to an API should be secured and no vulnerable data is exposed in the call.
  5. An expectation is that an email should be sent to the developer when application crashes, but email is not received.

 

General Setup and Operating Mechanism of Network Proxy Debugging Tools

 

 

2.1 Charles Proxy Debugging Tool

Charles is a network proxy tool that acts more like a middleware between the client (native apps) and the server (Internet) and logs every inbound/outbound network calls that are made by the device. One can then, monitor the request that goes through & response that comes back in. One can also edit the request/response data to assess the stability & error-handling capabilities of the application. Charles provides various options to play around with network calls, which makes it quite a convenient tool for testing mobile native apps. It is a cross-platform tool and requires a paid license. It also has a trial version that can be used for 30 days.

2.2 Fiddler Web Debugging Proxy

Fiddler is another popular network proxy debugging tool for Windows users. It is similar to Charles and is free to use. A feature of Fiddler that sets it apart from Charles is the ability to extend its core functionality using FiddlerScript. FiddlerScript is a scripting interface using which you can enhance UI of the tool, add new features and modify requests and responses in run time.

 

2.3 Distinct features provided by Network Proxy Debugging Tools useful from a tester’s perspective:

  1. Monitor Request/Response behaviours – As a tester, what matters is what and how the data, when fetched from the server, shows. In situations where response doesn’t seem to match, monitoring the request sent makes sense. Network proxy debugging tools provide fine grained information about each request and response that passes through it. This also includes non-HTTP connections like WebSocket and data protocols like Protocol Buffer.
  2. Rewrite Request/Response behaviour in run-time – This is where these tools prove extremely useful. There are scenarios where testers need to trigger a negative outcome, for example, show an error popup on the screen. If such scenarios are dependent on the app’s network calls then these tools allow the ability to modify the request data OR the response data in run-time to reproduce such cases.
  3. Simulating varying network speeds – Network connections tend to give poor performance, and this certainly can impact an application’s behaviour. Network proxy tools provide capability to throttle network speed to simulate poor network condition and test the application.
  4. Black-Listing Request/Response – Sometimes, the need of a test is to verify if the responding server is down or non-responsive. When an application makes a request to fetch data in the presence of a non-responsive domain, then the app can crash if it is not handled properly. Charles makes this testing possible with black-listing certain domains in the requests.

 

3. Device Logs Monitoring Tools

Both Android and iOS platforms provide a logging mechanism for native applications to push information (Debug messages, warnings, errors, fatal issues) to an output stream. Developers often enable this mechanism in the development builds of apps, and a tester can take advantage of this by monitoring this information while testing native apps. There are various tools that can help access logging outputs.

 

3.1 Android Debug Bridge (ADB)

Android Debug Bridge is a command line tool that allows communication with an Android device. This tool comes along with the Android SDK package, but can also be downloaded independently when the need is to just debug applications on your device as part of your testing routine.

You can obtain device logs by connecting the device to your desktop (that ADB runs on) with a USB cable and USB debugging enabled on the device. You can also connect to ADB over Wi-Fi with a few more additional steps. Once connected, you can issue commands in the following manner:

adb logcat

You can also, alternatively, push the output of this command to a file and view the contents of the file in a modern text editor.

adb logcat > logfile.txt

So when submitting an issue, for instance, a crashing app, a tester can submit this log as well to support more on the issue & help developer get a clear picture of what went wrong. Alternatively, there is an android application called aLogCat on Google Play Store which can be installed on the device to read device logs on the device itself.

 

3.2 XCode for iOS device logs

For iOS devices, there isn’t an exact equivalent of ADB. You can make use of XCode, the development IDE which allows a view of all the device logs. To view logs:

  • Download and install XCode on your MacOSX machine.
  • Plug in the device and open Xcode
  • Choose Window -> Devices from the menu bar
  • Under the DEVICES section in the left column, choose the device
  • To see the device console, click the up-triangle at the bottom left of the right-hand panel
  • Click the down arrow on the bottom right to save the console as a file
  • To see crash logs, select the View Device Logs button under the Device Information section on the right-hand panel
  • Find your app in the Process column and select the Crash log to see the contents.
  • To save a crash log, right-click the entry in the left column and choose “Export Log”

 

3.3 Apple Configurator 2 for iOS device logs

You can also access iOS device logs using Apple Configurator 2. To view logs:

  • Download and install Apple Configurator 2 on a Mac.
  • Attach the iOS device to the Mac and trust the computer.
  • Start Apple Configurator 2 tool.
  • Within the “All Devices” view double-click on the device attached to the Mac. An overview of the device is shown.
  • On the left-hand menu bar click on “Console”. The console of the device opens showing a live log.
  • Try reproducing the issue.
  • Press the “Save” button to save a copy of the log.

 

3.4 iOSConsole tool for iOS device logs

XCode is heavy-weight and complicated to deal with. iOSConsole is a small, lightweight app which allows viewing iOS device logs without any setup trouble. This tool also provides log filtering and export-to-file functionality.

 

Conclusion

Today mobile application ecosystem is split majorly into 3 different categories: Mobile web apps which run entirely on mobile browsers, mobile native apps, and mobile hybrid apps. Performing testing for each of these application categories requires knowledge on the debugging toolset as well as the underlying technology stack of the application to a certain extent. Debugging tools like Browser developer tools, network debugging proxies and device log monitors make it easier for mobile testers to find issues with finer details and help developers fix them quickly.

Any debugging tool alone cannot be a concrete solution to get debug information from an application. It is ideal to make use of different tools in tandem, depending upon the type of application that you are testing. If it’s a mobile web application, using the browser’s developer tools can be sufficient. If it’s a native application, you should use a network debugging proxy along with device log monitors to get the complete picture. If it’s a hybrid application, you should use browser developer tools to debug webviews, network debugging proxy for all the network calls and device log monitors to check for useful logging messages.

Agile teams are going for faster software releases which means developers need to fix bugs faster. Recent trends in mobile testing are going to demand testers to be more skilled technically. Thus, knowing how to use debugging tools is definitely going to help them achieve building a comprehensive testing strategy, perform better testing, achieve better collaboration with developers and stay relevant in the industry.

 

 

 

References

  1. https://saucelabs.com/blog/guest-post-functional-testing-in-2016-forecast
  2. https://az184419.vo.msecnd.net/sauce-labs/white-papers/web-and-mobile-testing-trends-2015.pdf
  3. https://www.slideshare.net/testingwhiz1/15-test-automation-trends-of-2016-infographic-testingwhiz
  4. http://qablog.practitest.com/stop-being-a-non-technical-tester/
  5. http://www.evoketechnologies.com/blog/software-testing-trends-predictions-2016/
  6. https://www.utest.com/articles/which-testing-skills-will-get-you-hired-in-2015
  7. https://www.smashingmagazine.com/2014/09/testing-mobile-emulators-simulators-remote-debugging/
  8. https://developers.google.com/web/tools/chrome-devtools/remote-debugging/
  9. https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Debugging_Firefox_for_Android_with_WebIDE
  10. https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Debugging_Firefox_for_Android_over_Wifi
  11. https://developers.google.com/web/tools/chrome-devtools/remote-debugging/webviews
  12. https://developer.mozilla.org/en-US/docs/Tools/Valence
  13. http://washamdev.com/2016/02/debug-a-website-in-ios-safari-on-windows/
  14. http://devcoma.blogspot.in/2014/03/how-to-investigate-ios-safari-crashes.html
  15. https://medium.com/@nikoloza/how-to-debug-remote-ios-device-using-chrome-devtools-f44d697003a7#.90pczegej
  16. https://webdesign.tutsplus.com/articles/quick-tip-using-web-inspector-to-debug-mobile-safari–webdesign-8787
  17. http://codewithchris.com/tutorial-using-charles-proxy-with-your-ios-development-and-http-debugging/
  18. https://visualstudiomagazine.com/articles/2014/07/01/http-debugging-tools.aspx
  19. http://docs.telerik.com/fiddler/Configure-Fiddler/Tasks/ConfigureForAndroid
  20. https://jaanus.com/debugging-http-on-an-android-phone-or-tablet-with-charles-proxy-for-fun-and-profit/
  21. https://www.charlesproxy.com/
  22. http://www.telerik.com/fiddler
  23. https://developer.android.com/studio/command-line/adb.html
  24. https://developer.android.com/studio/command-line/logcat.html
  25. http://www.makeuseof.com/tag/get-logcat-reporting-bugs-android/
  26. http://www.androidpolice.com/2017/01/05/google-makes-adb-fastboot-platform-tools-available-without-full-sdk-android-studio-download/
  27. https://www.utest.com/articles/grabbing-console-log-from-an-ios-device-using-os-x
  28. https://www.podfeet.com/blog/2016/07/ios-console/
  29. https://lemonjar.com/iosconsole/
  30. All monochrome icons are used from ‘The Noun Project’: https://thenounproject.com/
  31. Google chrome icon: http://www.freeiconspng.com/free-images/google-chrome-icon-3127
  32. Firefox icon: https://cdn1.iconfinder.com/data/icons/appicns/513/appicns_Firefox.png
  33. Safari icon: http://www.freeiconspng.com/free-images/safari-icon-14835
  34. Android icon: http://www.freeiconspng.com/free-images/android-icon-3062
  35. iOS icon: http://www.freeiconspng.com/free-images/ios-png-4091

Leave a Reply