Cloud Service Providers - A Comparison

The proliferation of mobile devices in the market has pushed the software industry to create and maintain apps for every market need. Mobile testing is one of the most complex aspects of running an app and the job is not getting any easier with newer OS and version releases every other day. For such a diverse landscape, cloud computing platforms offer a unified, dynamic and secure testing solution. A single cloud subscription platform can help testers connect thousands of mobile devices, with different device configurations, at one go. The increasing adoption of cloud testing methods has resulted in service providers offering diverse features, use cases and pricing models based solutions for the market. Here is a table that compares several functionalities of three of the most prominent cloud-based mobile testing platforms - AWS Device Farm, Sauce Labs, Xamarin Test Cloud. This can serve as a guide for any company to make an informed decision over their cloud investments.

Features SERVICE PROVIDERS
Usage
  • Automated testing of apps
  • Remote access of devices onto which you can load, run and interact with apps in real time
  • Automated testing of apps
  • Manual testing of apps
Automated testing of apps
OS Supported Android, iOS, Fire Os Android, iOS Android, iOS
Other key features Remote access of Android apps for interactive testing

Quality Report

  • Ensures app consistency across diffrent devices
  • Follows end-to-end stress testing
  • Provides app preview across different devices
Easy test uploading into Test cloud with Xamarin Studio IDE
TEST TYPES Android
  • Appium Java JUnit
  • Appium Java TestNG
  • Appium Python
  • Calabash
  • Instrumentation(JUnit, Espresso, Robotium, or any instrumentation-based tests)
  • UI Automator
  • Explorer
  • Appium-Independent of Language/Fromework
  • Espresso
  • Robotium
  • Xamarin UI Test
  • Calabash
  • Espresso
  • Appium-Java-Junit(For executing Appium with any other framework or Language have to consult with Cloud team)
iOS
  • Appium Java JUnit
  • Appium Java TestNG
  • Appium Python
  • Calabash
  • UI Automation
  • XCTest(including KIF)
  • XCTest UI
  • Appium-Independent of Language/Framework
  • Robotium
  • Xamarin UI Test
  • Calabash
  • XCUItest
  • Appium-Java-Junit(For executing Appium with any other framework or Language have to consult with cloud team)
Web Apps
  • Appium Java JUnit
  • Appium Java TestNG
  • Appium Python
Appium-Independent of Language/Framework Not Supported
App Support Native, Hybrid, Web Native, Hybrid, Web Xamarin, Native, Hybrid
Real Devices 350+ Devices 312 Devices 1320 Android Device, 1317 IOS Device
Emulator Support Yes Yes No
Manual Testing No Yes No
Device Capabilities Wi-Fi Connecton
No carrier - Cannot make phone calls or send SMS messages
Can take photos - May look dark & blurry
Reports
  • Report storage for 400 days
  • Components - Pass/Fail, crash reports, test & device logs, screenshots
  • Captures performance samples every second
  • Logcat captures for Android, device console logs for iOS
  • Regular archival of test results
  • Easy third party report generation and Log4j logs
  • Provision to log issues directly into JIRA or GitHub
  • Xamarin Test cloud report consists of following components: Screenshot of last step, Pass-Fail status, Memory usage, CPU usage, Time taken for test execution
  • Recorded video of the test execution is available for paid subscribers
Limits
  • Maximum app file size - 4GB
  • 60 minute limit on automated testing (extended upto 150 mins)
App will be deleted if idle for more than seven days.
  • Test runs period limits:
  • Paid subscription : 360 minutes
  • Trial account : 90 minutes
  • Xamarin University account : 30 minutes
  • Individual Xamarin : UI Tests may not exceed 30 minutes each
  • Individual Calabash steps may not exceed 10 minues each
  • Trial and University accounts are limited to 10 test runs per day
Tools & Plugins
  • Jenkins CI Plugin
  • Gradle Build System
  • Jenkins CI plugin
  • Direct Connectivity with Hockey App
  • Jenkins CI plugin
  • Maven Dependencies
Cost
  • Pay as You Go - $0.17/device minute
  • Unlimited Testing - Starts at $250/slot/month
  • Private Devices - Starts at $200/month
  • Individual - $149/month : 2 concurrent sessions, 1000 mins/month
  • Expert - $249/month : 4 concurrent sessions, 2000 mins/month
  • Professional - $399/month : 6 concurrent sessions, 3000 mins/month
  • Small Team - $529/month : 8 concurrent sessions, 4000 mins/month
  • Enterprise - To be contacted : 10 or more concurrent sessions, unlimited mins
  • New Bie- $99 per month : 1 concurrent device, 1 Device Hour per day
  • Small Startup-$379 per month : 3 device,5 Device hour per day
  • Growing business-$799 per month : 5 Devices, 10 Device Hours
  • Enterprise-To be contacted for unlimited usage
  • *12 mins test x 5 devices = 1 device hour