Device farms are a valuable alternative to testing that helps streamline processes for app and software development. Whether built in-house or from a third-party provider, they are efficient, cost-friendly, and highly scalable.
Picture this: you just finished developing your brand new mobile app. After many long days and nights, you’re finally ready to ship it and start to see the big bucks come in. Right? As we’re sure you already know, it doesn’t quite work that way. For any digital product to provide a universal experience, it must be fully functional anywhere and anytime. So, long before your app is ready to go live, you must make sure it works seamlessly across all devices, networks, operating systems, browsers, or any combination of these your users prefer. You may think this sounds like an obvious step, but keep in mind that there are currently about 3.8 billion smartphones in the world.
Now, let’s take a minute and consider the number of different devices, versions, browsers, and operating systems that those 3.8 billion smartphones use. Your app must be compatible with most–or all–of them under various circumstances such as a low battery, different versions and updates, poor network connection, and screen resolutions, to name a few. The only fail-safe way to ensure that your app works under all of these variables is to test it. Implementing proper testing methodologies is a non-negotiable part of any wholesome development process. It is the culmination of all those months and sleepless nights you spent bringing your product to life. Consequently, you don’t want to skimp on your testing strategy.
At Foonkie Monkey, we have been implementing different testing approaches for over ten years. We understand that developers– mainly startups–don’t have access to all the tools needed to leverage fast and automated testing. As a result, they may end up relying solely on manual testing, which works perfectly when coupled with other testing approaches but can be highly time-consuming, expensive, and unnecessarily complex when used on its own. To try and tackle this issue, we want to look at device farms as helpful and efficient tools for implementing solid testing processes without breaking the bank.
Let’s dig a little into what device farms are and why they may be a solid alternative to implement robust testing methodologies.
What is a Device Farm?
With modern technologies advancing at mind-blowing speeds, we developers are now tasked with adapting their products, processes, and practices to fit the entire scope of devices available today. Testing our products in today’s technological environment means we must monitor their performance under conditions that closely–almost perfectly– simulate those of actual devices. Consequently, modern QA engineers and analysts that want to deliver bullet-proof products need to test all mobile and software products on real devices. However, this approach isn’t as straightforward as it seems.
Since most applications need to be tested on different devices to evaluate their performance under different configurations, you would have to own all those devices in reality. We’re talking about not only purchasing several different devices but running automated regression tests on all of them and spending too much time installing builds and monitoring every device. So what’s the solution? Is there a tool that can help you test on multiple devices in a short time frame? Yes, it exists and is called a device farm.
Device farms are cloud-based testing environments that allow developers to remotely test the performance of their software and mobile products on a large number of real devices. They cover tests on all the different configurations, environments, conditions, and available combinations of browsers, operating systems, and devices. This way, device farms give developers a chance to identify issues regarding functionality, adaptability, performance, and design by seeing their products in action from the perspective of their end-users.
Also, these devices help you gauge your app’s performance in the presence of common hardware issues and limitations such as a low battery, different CPU usage levels, network strength, and even interruptions such as incoming calls and texts. Additionally, you can interact with the devices by using swipe gestures, fingerprints, or FaceID to see if your app’s responsive design is stable and fully functional.
Lastly, with a device farm, you don’t have to guess what problems may occur or improvise any aspect of your testing strategy. You can just run your code on an actual device, gauge its performance, and get very accurate results on your tests. This way, the amount of time and resources you can save by purchasing a device farm to leverage your testing procedures will positively impact your development processes.
Should You DIY or Purchase a Cloud-based Device Farm?
It’s important to note that, even if this article is focused on cloud-based device farms, you always have the option of piecing together an in-house device farm. DIY device farms are not that uncommon in the development industry but keep in mind that they are challenging to build and maintain.
First, you have the considerable upfront cost of purchasing multiple devices, each with its monthly carrier plan. You must buy carrier plans because one of the main aspects to consider in your testing procedures is data network variability and performance. Second, you must keep in mind that most testing devices have a fixed lifespan; they aren’t eternal, so you’ll need to replace them eventually. Third, there are the infrastructure and hardware costs, including control devices, Wi-Fi access points, repeaters, racks, and servers.
Additionally, in-house device farms are harder to scale, pose some increased data security risks, and cross-team collaboration is often more problematic due to physical devices being rooted to a fixed location. Finally, you have maintenance costs that can quickly add up when devices break or have any issue that, when not addressed, can impair the efficiency of your testing procedures. And, since new devices are released frequently, keeping your in-house device farm updated can take a toll on your pocket.
On the other hand, cloud-based device farms provide you with all the benefits of the in-house solution but with more minor monetary implications, security issues, hardware maintenance considerations, scalability problems, and overhead costs. In most cases, all you have to do is login into your account, select your device-browser-OS combination, and start performing actions.
However, if you feel like building a DIY device farm is suitable for your development company, make sure to consider all the implications stated above–as well as others that may not be included–to make an informed decision.
Advantages of Using a Cloud-based Device Farm
A 2020 study found that 65% of app users think less of a brand if it provides a poor mobile experience. That same study also found that 70% of users will abandon an app if it takes too long to load. These figures represent a pressing need for developers to ship products that work at top performance all the time; users are not very forgiving. Hence, you must implement a type of automated testing solution to ensure the quality of your apps.
You can choose from any of the available automated QA testing tools or implement manual testing, but know that device farms offer the comprehensiveness and efficiency you need to implement robust testing processes. All that being said, device farms are an excellent alternative for developers who want to minimize mistakes and deliver top-notch products as often as possible.
Here are some of the many advantages of using a device farm for your mobile testing needs:
- It is easy to use and helps increase your platform testing coverage by offering a wide breadth of devices.
- It can help QA engineers test the app’s operability from the perspective of the end-user.
- You can test on many devices without the need to physically purchase them, hire data plans, implement infrastructure, or manage hardware.
- It gives access to testing environments from any part of the world and any timezone, making it ideal for offshore development teams.
- Most device farms offer access to both modern and legacy versions of operating systems and multiple device models and manufacturers, even outdated ones.
- It offers integration with CI/CD pipelines.
- It’s a significantly less expensive alternative to having an in-house device lab.
- There are no overhead costs when purchasing a cloud-based device farm.
- It provides unparalleled scalability for tests.
- It gives you the option of using built-in test suites.
- Most vendors provide detailed reports and crash logs.
Drawbacks of Using a Cloud-based Device Farm
As you can see, device farms have a myriad of advantages that make them an attractive solution for automated testing. However, there are several drawbacks and critical components that cloud-based device farms lack.
These drawbacks include:
- Working with a cloud-based device farm means hiring a third-party service, which means you have to purchase a plan and depend on the vendor for updates and bug fixing.
- Sometimes vendors lag on updating devices, so you have to wait or work with what they provide,
- The more popular the vendor you choose, the longer you have to wait to access some devices.
- As with any cloud-based tool, security can be an issue when not addressed appropriately.
- Some vendors don’t provide test reporting.
Hopefully, by now, with the advantages and drawbacks clear, you feel more confident to decide what solution works better for you and if device farms are the way to go for your company. Whatever your approach, if you’re still considering device farms as your go-to solution for cross-device testing, here are some of the top vendors on the market.
Best Device Farm Vendors
AWS Device Farm
Amazon’s device farm is the market leader with nearly 400 different devices and 100 models available on Android, iOS, and Fire operating systems (phone and tablets). AWS device farm has a pay-as-you-go payment model where they charge $0.17/device per minute, but they also have a free trial of 1,000 minutes and an unlimited plan of $250 per month. Their most exciting features include delivering logs and performance data, integration with most CI/CD tools, real-world environment simulations, and the diversity of models and devices you can access. They also have a built-in test suite and offer customized testing with frameworks such as Appium and Calabash.
Albeit expensive, Amazon has one of the most comprehensive and complete device farms you will find on the market. It can be costly, but when you consider the broadness of its features, you can see there’s an actual value to using the AWS device farm.
Kobiton
Kobiton’s device farm is one of the most comprehensive and affordable testing solutions on the market. It grants access to over 350 different Android and iOS devices with a wide range of manufacturers and models. Kobiton’s device testing lab allows developers to perform manual and automated testing on real devices, on the public cloud, or on-premise. It also offers very low latency, a centralized administration console for more straightforward testing and device management, and full video logs to have accurate reports of bugs and errors at your disposal.
Kobiton offers several payment options: the first tier is for startups and costs $50/month. The next level is called Accelerate and costs $200/month, followed by the third tier, Scale, which costs $700/month. The final tier is Growth, which costs $1,400/month. All payment plans include unlimited users.
Kobiton is the device farm of our preference here at Foonkie Monkey. We find it extremely useful, fast, efficient, and it helps us leverage our testing strategies to deliver the top-notch products we are known for.
Sauce Labs
Sauce Labs device farm supports over 2,000 iOS and Android devices (phone and tablet) on private and public clouds in the US and Germany. They offer seamless integration with CI/CD tools and pipelines; they provide very effective debugging tools, test analytics, and parallel testing. Sauce Labs’ device farm also offers the possibility of using emulators and simulators to complement your testing strategy further. Additionally, they provide seamless integrations with frameworks such as Appium and Espresso.
They offer a free trial of 14 days and a yearly plan for real devices of $349/month on automated testing and $600/month on unlimited automated testing. They also offer customized billing plans depending on your needs and the features you want to use for your testing purposes.
Google’s Firebase Test Lab
Google’s Firebase Test Lab provides access to countless iOS and Android devices and seamlessly integrates with existing workflows. They also offer exceptionally detailed reports that include videos, logs, and screenshots. Firebase Test Lab has a groundbreaking tool called Robo, an intelligent crawler that navigates your app to validate bugs, issues, and crashes and gives you a detailed report of its findings. Additionally, it requires no code, and you can also use it for A/B testing. Lastly, it easily integrates with Google Cloud command line, Firebase console, and .
Google’s Firebase Test Lab is one of the most complete and affordable device farms on the market. They have three payment plans:
- Spark: Free but limited to 5 tests per day on real devices.
- Flame: Costs $25 per month.
- Blaze: Pay as you go ($5/hour per device).
BrowserStack
BrowserStack gives you access to over 2,000 real browsers and devices from various manufacturers and models for testing on both Android and iOS. It is a flexible and scalable device farm that gives you free Javascript unit tests and supports over 750 configurations. One of its most significant advantages is its simplicity of use where there is no setup required; you simply log in, and you’re ready. Additionally, it gives detailed crash reports, screenshots, and logs.
They charge a $199/month fee for automated testing, $49/ month for live testing, and $12.50/month for freelancers.
Device Farms: The Takeaway
When it comes to app testing, there is no in-size-fits-all approach. There are many efficient ways to test your product and which one you choose depends entirely on your project’s specifications and your development needs. However, as we tried to explain in this article, device farms are one of the most affordable and professional solutions that enable automated testing across a wide variety of devices. Using them alone or in tandem with other testing approaches can significantly impact the quality of the product you deliver.
Furthermore, it will undoubtedly make your testing processes faster, more accessible, more accurate, and in most cases, more affordable. On that same token, if you’re a startup owner, you can significantly benefit from the more cost-friendly device farm options on the market. They will give you access to all the benefits exposed above without breaking the bank, and you’ll deliver wholesome, robust products that will, without a doubt, build you a budding client base.
If you’re looking into device farms for your testing processes or looking for a development company with a wide range of testing services, don’t hesitate to contact our team! At Foonkie, we’re always here to help!