Robot vacuum beta tester guide: what to test and log
To beta test a robot vacuum fairly, give it the same repeatable jobs each week (a mapping run, an obstacle course, a measured debris test, a dock and station cycle), record the conditions every time and film every failure with the time stamped in. BetaSuite: Tester's Toolkit (betasuite.app) is an iPhone, iPad and Android app built for this: it burns the date and time into every photo and video and turns your sessions into a report for the manufacturer. The checklist below works with any tool, and there is a free 11-step protocol to import.
How people get into robot vacuum betas
Vacuum makers announce testing programmes where their owners already are: the brand's newsletter, its official community forum and its social channels. The application is mostly about your home, because a vacuum is tested against your floors and your clutter, not a lab's. Have these ready:
- Floor plan: how many rooms and levels, door thresholds, stairs and step-downs.
- Floors: hard floor, carpet pile and rugs, including fringes and very dark or glossy surfaces.
- Household: pets that shed, children, and how much usually ends up on the floor (cables, toys, socks).
- Set-up: Wi-Fi coverage, the smart-home system you use, and any robot vacuums you have owned, so they can compare.
Expect to share more than you would with a shop. The ECOVACS US New Product Testing Program statement (checked 4 October 2026), for example, says each round is announced with the details it needs, lists the personal data it may collect (including your home address and the layout of your home), and says the test unit may collect data about your home environment for development. Read that kind of statement before you apply, especially if the unit has a camera, and agree it with the people you live with. BetaSuite isn't connected with ECOVACS.
Getting picked is the start. Read the agreement for what you may photograph and post, and apply only if you can keep testing for the whole period. The rest of this guide is how to do that well.
What to test in a robot vacuum beta
Start with mapping and docking, because everything else depends on them. If the map is wrong or the vacuum can't find its dock, later results are hard to trust. Then work through the rest, and repeat anything that fails so you can say how often it happens.
| Area | What you are checking | Evidence worth capturing |
|---|---|---|
| Set-up and mapping | Time to the first map, how closely rooms, doors and furniture match reality, and whether the map survives a reboot, an update or a move to a new room. | A screenshot of the map after each run, and a photo of the real layout. |
| Obstacles and hazards | Cables, socks, shoes, pet bowls, toys, thin chair legs, and dark or glossy floors near a step. Does it avoid them, push them, tangle on them or drive off an edge? | A low-angle clip of each approach, filmed from floor level. |
| Rugs and thresholds | Rug edges, fringes, door tracks and changes of carpet. Stalling, flipping a rug edge, getting stuck. If it mops, does it lift the pad or avoid carpet? | A clip of it crossing the threshold or rug edge. |
| Cleaning performance | How much of a measured amount of debris it picks up in one pass. Use the repeatable test below. | Before and after photos from the same spot. |
| Edges and corners | Along walls, into corners and under furniture. Whether it fits under low furniture it should reach. | A photo of the wall edge and the corner after a pass. |
| Mopping (if fitted) | Even wetness, streaks, drips from the dock, and the water tank and flow behaviour. | A photo of the wet trail and a clip of the pad lifting. |
| Dock and station | Returns on its own, charges and resumes. If there is a station: empties and washes without spills, clogs or error codes. | A clip of the whole cycle and a photo of the station afterwards. |
| App and connectivity | Pairing, no-go zones, room selection, schedules, map edits, voice assistants, Wi-Fi dropouts and notifications. | A screen recording, or a dual recording of the app and the vacuum together. |
| Noise | Loudness at normal distance on each mode, and during station emptying. | Notes of the distance and mode. A phone sound meter is fine for comparing runs. |
| Battery and reliability | Area cleaned per charge, recharge and resume, and how many times you had to rescue it in a week. | A dated tally of every stuck event, error code and missed dock. |
Keep it safe. Test drop sensors only where a fall can't hurt anyone or break anything: keep people and breakables clear of stairs and steps while you test. Pick up loose cords and small items you don't want to be run over, and keep pets out of the room if the vacuum is behaving oddly.
A repeatable debris test
"It cleaned well" can't be compared from one week to the next. This test can, so you can tell whether a firmware update or a setting made things better or worse. It is a relative test, not a lab result.
- Choose a patch of hard floor and a patch of rug, each about one square metre. Clean both first.
- Weigh out the same amount of dry rice or oats every time (for example 30 g) and spread it evenly. Do the same near a wall and in a corner.
- Photograph from directly above, from the same spot each time.
- Run one pass in a fixed mode (same suction level, same mop setting).
- Photograph again. Sweep up what is left and weigh it.
- Work out the result: picked up % = (amount spread − amount left) ÷ amount spread × 100.
- Repeat after every firmware update or settings change, and log the result in the same row each time.
Conditions to record every time
A fault you can't reproduce is just a story. Log the same details every time so engineers can line your problem up against everyone else's.
| Record | Why it matters |
|---|---|
| Floor types and rooms | Hard floor, low or high pile carpet, rugs, room size and clutter. Performance varies a lot by surface. |
| Settings | Suction level, mop level, mode (vacuum, mop or both), no-go zones and schedule. |
| Maintenance state | Bin and filter empty or full, brush cleaned, mop pad fresh, and runs since the last service. |
| Lighting and surfaces | Very dark rooms, strong sun through a window or glossy black floors can upset some sensors. Note them. |
| Network | The Wi-Fi band, the distance to the router, and whether anything changed. |
| Software | Vacuum firmware and app version. Note them again after every update. |
| Household | Pets, shedding, toys left out. Anything that changes the mess it faces. |
| Date, time and battery | The date, time and battery percentage at the start of each run. |
BetaSuite adds the date and time to every photo and video for you, and can add GPS, live weather, the season and the time zone, each of which you can turn on or off in Settings.
A first-week routine
Adjust this to your brief. The point is to do the same checks in the same order, so a change in the result means something changed in the vacuum.
| When | Focus |
|---|---|
| Day 1 | Unbox, charge and pair. Note the firmware. Photograph each room. Watch the first mapping run. |
| Days 2–3 | The obstacle course, rugs and thresholds, and the debris test on hard floor and rug. |
| Days 4–5 | Dock and station cycles, app controls (no-go zones, room selection, schedule) and a noise check. |
| Days 6–7 | Let it clean on its own. Note every stuck event, rescue, error code and missed dock. |
| Every week | Repeat the debris test and the map screenshot. After any firmware update, repeat the set-up and mapping checks. |
A pass/fail checklist for a robot vacuum
Agree your thresholds with the manufacturer's brief first. If it doesn't give any, write your own before you start (for example, "needs rescuing no more than once a week") so you aren't judging after the fact.
- Pairs on the first attempt and reaches a ready state with no error.
- The first map matches the real layout (rooms and doors) and survives a restart.
- Avoids, or safely handles, cables, socks and shoes without tangling or dragging them.
- Crosses rug edges and thresholds without stalling or flipping the rug. If it mops, it lifts the pad or avoids carpet.
- Picks up at least your agreed share of the debris test, including along walls and in corners.
- Returns to the dock and recharges, then resumes if it says it will.
- If it has a station, the station empties or washes without spills, clogs or error codes.
- Obeys no-go zones, room selection and schedules on the next run.
- Stays connected to the app for the whole week.
- Needs rescuing no more often than your agreed limit.
- Is acceptably quiet on every mode while you are in the room.
- Installs a firmware update without losing the map or your settings.
A fail isn't a verdict on the vacuum. It's a finding. Write it up with the conditions and a clip.
Cameras and microphones in your home. If the vacuum has a camera or microphone, check what it records and where it sends it before you start. Tell the people you live with, and keep clips of other people out of your evidence. If the unit is under NDA, check what you may show.
How to capture evidence engineers can use
- Film from floor level. For obstacle and rug tests, prop the phone on its side on a small stand or a stack of books and keep it steady.
- Put the map and the vacuum in one clip. Dual recording in BetaSuite records your phone's screen and the rear camera into a single video, so the live map and the robot sit side by side on one timeline.
- Screenshot the map after every run. A map that quietly changes between runs is a classic regression, and it only shows up if you keep the history.
- Capture error codes word for word. Photograph or screenshot the message with the time visible, and record any voice alert.
- Keep before and after photos. For the debris test, shoot from the same spot every time.
- Burn the time into the picture. A stamp that is part of the image travels with the file. File metadata is often stripped when a photo is shared or compressed. BetaSuite stamps every photo and video, with optional GPS, weather, season and time zone. More on that in the timestamp camera app guide.
- Catch the sudden stuff. When the vacuum is working in the same room as you, Instant Capture in BetaSuite keeps a rolling buffer and saves the last 15–30 seconds when you tap. Without a tool like that, start recording at the start of each run you can watch.
- Mark it as it happens. Add a one-tap marker or a quick voice note with the time, then write it up properly later.
What to send the manufacturer
A good report is short at the top, specific in the middle and backed up at the bottom. See how to write a beta test report for the full structure.
| Section | What goes in it |
|---|---|
| At a glance | Model and serial number, firmware and app versions, test dates, floor types and layout, and any settings you changed. Three sentences on how it went. |
| Findings, worst first | Safety issues first, then anything that stops it cleaning, then annoyances. For each one: what happened, what you expected, how to repeat it, how often it happens and the conditions. |
| Evidence | Map screenshots before and after, clips of each failure, debris test photos and the error codes with the time visible. |
| What worked | A short list. Engineers also need to know what not to change. |
| Data files | A CSV or JSON of your findings if they ask, so they can load it into their own tools. BetaSuite reports can ship with both. |
Here is the format for one finding. The values are made up, to show the shape.
Title: Drags a charging cable, stalls and reports "wheel stuck" Severity: Medium (needs a rescue) Expected: Avoids the cable or flags it in the app Actual: Pulled the cable about 1 m, stalled, then showed a wheel error Frequency: 2 of 3 passes Conditions: Hard floor, living room, firmware 3.4.1, standard suction Evidence: Floor-level clip, stamped 18:42:10, plus a screenshot of the error
Want to see a finished report? Open the sample report (PDF). It has an At a Glance summary, findings sorted by severity and an image appendix.
Running it in BetaSuite
- Create a project for the vacuum, and note the model, serial number and firmware in your first session.
- Import the free protocol below.
- Start a session each time you test. Capture the photos and clips the steps ask for, and drop a marker whenever something happens.
- At the end of the week, generate a PDF or Word report with an At a Glance summary and findings by severity, plus JSON and CSV data files if you want them.
AI summaries are optional and opt-in. For units under NDA, Confidential Mode blocks cloud AI and cloud uploads, and warns you before you share or export anything. Everything is stored on your device, with no account.
Free BetaSuite protocol: robot vacuum home test
This is a ready-made, 11-step protocol for the first two weeks with a robot vacuum. It was built with the site's own Protocol Builder, so the link below is a real import link. It mixes pass/fail checks with choice, multiple-choice and rating questions, and asks for a photo or video where the evidence matters. For the general idea, see the test protocol template.
Robot vacuum home test · 11 steps · free to import and edit
- Set-up: unbox, charge and pair the vacuum with its app. It connects on the first attempt and reaches a ready state with no error.
- First map: the first full run produces a map of the floor. Walls and rooms match the real layout, and doors are in the right places. Screenshot: the map in the app.
- Obstacle avoidance: with a charging cable, a sock and a shoe on the floor, the vacuum avoids them or deals with them without tangling, dragging or pushing them out of reach.
- Rugs and thresholds: it crosses a rug edge and a door threshold without stalling, flipping the rug edge or getting stuck. If it mops, the pad lifts or it avoids carpet.
- Edges and corners: spread a measured scoop of dry rice or oats along a wall and into a corner, then run one pass. How much is left? Photo: the wall and corner.
- Dock and station: it returns to the dock on its own and recharges. If your model has a station, it empties the bin or washes the mop without spills, blockages or error codes. Photo: the dock.
- App controls: room selection, no-go zones and a schedule set in the app are all obeyed on the next run.
- Over a week, how many times did you find it stuck or need to rescue it?
- Noise: how acceptable was the noise while you were in the room? 1 = unbearable, 5 = barely noticeable.
- Which problems did you see? Tick all that apply.
- Overall: how happy would you be to rely on this vacuum every day? 1 = not at all, 5 = very.
Show the raw code
How to import it
- On your phone, tap Open this protocol. If BetaSuite is installed, it opens its import sheet with the code filled in. If the link opens in your browser instead, tap Copy code there (or here).
- In BetaSuite, open Test protocols, choose Import and paste the code or link.
- Edit any step to suit your unit, then run it. Your answers are saved to a session and roll up into the report. Protocols, sessions and reports are part of Advanced mode; if you're in Basic mode, Test protocols offers to switch for you.
Want to change the steps first? Open the link in the free Protocol Builder. It runs in your browser with no account, and you can generate a fresh code from your edits. Running a whole beta programme? See how to give every tester the same protocol and merge the results.
When BetaSuite is the wrong choice
BetaSuite covers everything from a single stamped photo to a full report for the manufacturer. It is the wrong tool only when:
- You need data from the vacuum itself. BetaSuite doesn't connect to the vacuum, so it can't read its logs, map data or error history. You note what the app shows, or record the screen.
- You need lab numbers. A phone can't measure suction power or noise to a standard. Use the debris test and a phone sound meter to compare your own runs, not to certify anything.
- You want unattended monitoring. BetaSuite records the moments you start or save. Use a separate fixed camera if you need to watch the floor while you're out.
- You want to test an app, not hardware. BetaSuite is not a TestFlight or Google Play testing tool. See what a hardware beta testing app needs to do.
Don't need a report? If the manufacturer takes findings through its own portal or form, or you only want stamped photos and video, switch BetaSuite to Basic mode (Settings > App mode). It keeps every capture tool, including dual recording and Instant Capture, and leaves out projects, sessions and reports, so it works as a plain timestamp camera. Attach the files wherever the manufacturer asks, and switch back to Advanced at any time without losing anything.
If you run the programme rather than test in it, see the programme page. For the longer case for the app, see why BetaSuite.
Frequently asked questions
- How do I become a robot vacuum beta tester?
- Watch where the brand talks to owners (its newsletter, official community forum and social channels) and apply when a testing programme is announced. Expect questions about your floor plan, floor types, rugs, pets and Wi-Fi, and some programmes ask for your address and the layout of your home. Read what data the test unit collects before you apply, especially if it has a camera, and apply only if you can test for the whole period.
- What should I test first in a robot vacuum beta?
- Mapping and docking. Everything else depends on the vacuum knowing where it is and finding its way home, so if either is unreliable, later results are hard to trust.
- How do I test robot vacuum obstacle avoidance fairly?
- Use the same objects, in the same places, in the same room, approached from the same direction, and film from floor level. Run at least three passes and record how many it handled (for example 2 of 3). One run proves very little.
- How can I compare cleaning between firmware versions?
- Use a repeatable debris test: the same measured amount of dry rice or oats on the same floor, one pass in a fixed mode, then weigh what is left. Repeat after each update and log the percentage picked up. It shows change over time, not a lab result.
- What if my robot vacuum has a camera?
- Check what it records and where the footage goes before you start, tell the people you live with, and keep other people out of your evidence. If the unit is under NDA, check what you are allowed to show.
- Can I use BetaSuite for a robot vacuum under NDA?
- Yes. Everything is stored on your device with no account. Confidential Mode blocks cloud AI and cloud uploads, and warns you before you share or export anything. Photos and videos also go to your camera roll by default, so if your photo library syncs to the cloud, turn Camera Roll off in Settings → Save Destinations before you turn Confidential Mode on. Check your NDA first for what you may photograph and what you may send, and by what route.
- Is BetaSuite a TestFlight alternative?
- No. BetaSuite: Tester's Toolkit is not an app-beta tool and has no connection to TestFlight or Google Play testing tracks. It is for people who beta test physical products in the field and need timestamped evidence and a report for the manufacturer.
- Is BetaSuite a subscription?
- No. It is a one-time purchase from the App Store or Google Play, with no subscription, no ads and no account. Each store sets the price.
Related guides
Start testing with timestamped evidence
BetaSuite: Tester's Toolkit burns the date and time into every photo and video, then turns your sessions, markers and findings into a PDF or Word report, on iPhone, iPad and Android. Only need a timestamp camera? Use Basic mode.
One-off purchase · No subscription · No ads · No tracking · Yours for life