Combined Rabbit Eye learning and device control mode

[leahcim]

Here I have a really huge task as a suggestion for the developers, but please prepare yourself for a headache if you really want to implement it… *lol

If it does become reality, then it would be the ultimate implementation of AI in combination with the LAM and the teach mode.

Now comes my idea…

3…

2…

1…

TaaA DaaaaaDadADaaaA! TaaaaDaaaaaDaaaAAA!

Just imagine the following:

For example, you switch on your PC, which is connected to r1 either via USB or Bluetooth, then you activate the Rabbit Eye, tell r1 what it should do on the PC and then simply put r1 in its case so that it can see the screen well.

r1 then takes a picture of the screen every few seconds or as quickly as possible or, if possible and feasible at all, simply starts a stream to be able to complete the task from the cloud (unfortunately this is unimaginable locally) and then simply carries out the actions that were previously requested on the PC to which it is connected.

Of course only if the necessary applications are available.

And now comes the thing: the skills required for this could come from the teach mode, for example.

What do you think of this idea?

Is it good or is it still too futuristic?

[BlueNexus9]

Hi,
could be useful for this suggestion as an expansion:

> USB-C as a connector/interface for other devices (for ‘teach-mode’, remote control)

@BlueNexus9

Hi rabbit-team,
Hi community,

mainly the USB-C interface is in use for charging our rabbit r1.

I think it could be a great improvement to have the USB-C interface as a connector for other devices like smartphones, computer etc…

As a connector to a computer you can use the rabbit r1 to process daily tasks or automatism.

By connecting with a computer maybe the possibility is given to create prompts like

‘Write an email in Outlook to xxx to agree the appointment’, etc…

So the chance is given to have an other fast and easy way to create automatism as an alternative to the planned ‘teach-mode’.

Also the possibility is given to control other devices with prompts, the rabbit keyboard, etc…

[Designer]

Are you talking about the R1 using the PC as a HID while looking what it is doing through its camera?

[PaulBacon]

So you want to be able to control the R1 from your computer, which then takes control of your computer to complete the given task(s)?

The ultimate goal seems to be to be able to execute a specific set of actions on applications on your computer on your behalf?

If that is the case, the (future) ambition for teach mode seemingly is to do this to some extend at some point, but:

• the applications that you want the R1 to use do not run on your local PC but on a virtual PC accessible to the R1
• you would teach a rabbit to go through a predefined set of actions on the applications you need to reach a certain goal. So the skills come from teach mode but would have to be taught first (by you or as a shared rabbit from another user).
• control of the R1 from a computer seems to deviate from the intention of the R1, which is for the R1 to control the computer, not vice versa.

So it sounds a bit like what you want to achieve with this should be covered by the the current ambition, I’ll leave it to others to say if that is (too) futuristic .

[leahcim]

@Designer

Are you talking about the R1 using the PC as a HID while looking what it is doing through its camera?

Yes, this is it!

[leahcim]

As I see it, a possible extension of the teach mode would be that the r1 could actually perform the necessary computing power completely locally, basically a sequence of clicks, movements and inputs based on screenshots. If a little AI comes into play, then this could even be transferred to other systems.