Just Speak Updated With More Hands-Free Functionality

    *Just Speak*: Controlling Android By Voice

1 JustSpeak: Controlling Android By Voice

JustSpeak is an Android Accessibility Service that enables voice control of your Android device. Once enabled, you can activate on-screen controls, launch installed applications, and trigger other commonly used Android actions using spoken commands.

1.1 Enabling Just Speak

Once installed, you can enable the Just Speak service under Settings->Accessibility on your Android device. Enabling the service is an action that only needs to be performed once as Just Speak will be automatically restarted when the phone is rebooted.

Just Speak is an Accessibility Service that uses Acessibility APIs on the platform to augment Android's user interface. Just Speak augments the Android user interface with voice-input control; other Accessibility Services like TalkBack provide spoken feedback. Note that Just Speak can be used either by itself, or in conjunction with other platform Accessibility Services such as TalkBack.

1.2 Initiating Voice Recognition

Once Just Speak is installed and enabled on your device, you can initiate voice commands by either performing an up-swipe from the Home button (if your device has soft keys) or by performing multiple taps on the Home button (if your device has hard keys). Note that this is the same gesture that activates Google Now on devices running the stock version of Android; When using Just Speak , you can get to Google Now by saying Launch Google Now.

Successful invocation of Just Speak starts voice recognition; this is indicated by playing an auditory icon (accompanied by vibration if available) and a visual overlay. Depending on the settings enabled in Just Speak, as well as other Accessibility Services, received voice input may be spoken and/or displayed after voice recognition has completed. As an example, TalkBack can work in conjunction with Just Speak to speak the recognized commands. Just Speak supports both local and global commands as described in later sections of this document.

1.3 Cancelling Voice Recognition

Voice recognition can be stopped by performing the same action used to initiate voice recognition (either performing an up-swipe from the Home button or by performing multiple clicks of the Home button). Just Speak is also programmed to stop listening if no voice input is received after a specific amount of time. Finally, an overlay, that covers the entire screen, is displayed whenever voice recognition is active; Clicking anywhere on this overlay will dismiss the overlay and terminate voice recognition.

1.4 Global Voice Commands

Just Speak supports a set of global commands that are available on any screen. These global commands include:

Command	Utterance	Synonym	Action
Open	Open <Installed Application>	Launch, Run	Launch Application
Recent	Recent Apps	Recent Applications	Recent Applications
Quick Settings	Quick Settings	Open	Quick Settings
Toggle WiFi	Switch WiFi (On/Off)		Toggle WiFi
Toggle Bluetooth	Switch Bluetooth (On/Off)		Toggle Bluetooth
Toggle Tethering	Switch Tethering (On/Off)		Toggle Tethering
Home	Go Home		Return to home screen
Back	Go Back		Return to previous screen
Notifications	Open Notifications		Open notifications shade
Easy Labels	Easy Labels		Display Easy Labels

Note that voice commands are flexible in that action words may be substituted with synonymous verbs (eg: “launch” in place of “open”). In addition, voice commands can be formulated as complete sentences, e.g., "Please open GMail".

In addition, Just Speak provides the following spoken aliases as a means of triggering commonly used applications:

Utterance	Action
Browser	Launch default Web browser
Web	Launch default Web Browser
OK Google Now	Launch Google Now
Search	Launch Voice Search
Voice Search	Launch Voice Search

1.5 Local Voice Commands

In addition to the global commands that are available all screens, Just Speak allows you to interact with on-screen controls in a variety of ways.

Command	Utterance	Synonym	Action
Activate	Click <control name>	Click, Tap	Activate control by its on-screen name
Scroll	Scroll Up/Down	Forward, Backward	Scroll  e.g., Lists.
Switch		Switch On/Off Toggle	Toggle Switches
Long Press	Long Press	Long Click, Long Tap	Long Press on on-screen controls
Check	Check	Check, Uncheck	Toggle CheckBox values

1.6 Labeling On-Screen Control

When using Just Speak, the text labels that appear next to on-screen controls determine the set of available local commands. For many controls, such as images, checkboxes, and switches, there may be no visible text to associate with the control. In these instances, Just Speak uses underlying Accessibility Metadata provided by the application developer to construct relevant labels; note that this metadata is also used by Accessibility Services such as TalkBack to meaningfully speak on-screen controls.

We leverage the visual overlay that indicates that Just Speak is active to visually display this additional metadata — this serves as a hint as to what you can say to activate the available controls. These overlay labels take one of two forms:

1. Controls receives a centered label in the simple case

where there are no actionable child controls that need additional labeling.

1. Where the control itself has actionable children,

Just Speak displays a labled frame around the actionable children.

1.7 Chaining Commands

Just Speak can be configured to take multiple voice commands at once and perform them sequentially. This chaining works with both local and global commands, performing the preceding action after the previous action has been executed. Commands are chained via simple connectives such as “and” and “then”. An example of this would be

“Click confirm and then go home”.

In this case, Just Speak would click the “Confirm” control (assuming it’s present) and upon completion of that task, return to the Home screen.

1.8 Easy Labels

Another configurable setting in Just Speak is the ability to replace the labels associated with on-screen controls with easy labels. These are phonetic labels that are designed to be unambiguous and are displayed in the overlay over their respective controls, easily identifying what to say to interact with a specific control. Note that phonetic labels can be temporarily activated or deactivated via the Just Speak global command Easy Labels.

1.9 Persistent Overlay

To aid users with limited dexterity, Just Speak also provides the option to make the overlay persistent, capturing all touch events received by the Android device. This effectively removes all traditional interactions a user can have with their device, replacing it with Just Speak functionality. The benefit of this is that the entire device essentially becomes a button, toggling between initiating and terminating voice recognition. In addition to this, overlay labeling is always present, allowing you to constantly be aware of what you can say to Just Speak.

1.10 Alternative Means Of Initiating Voice Recognition

We are continuing to experiment with alternative ways of initiating voice recognition. Toward this end, Just Speak enables you to configure an NFC tag that can then be tapped to initiate voice control; see “Just Speak -> Settings” for configuring an NFC tag.

Date:      2013-08-06 Tue

Author:    T.V Raman

Created: 2013-12-18 Wed 13:50

Emacs 24.3.50.1 (Org mode 8.2.3a)

Validate

JustSpeak: Control All Aspects Of Android By Voice

JustSpeak: Controlling Android By Voice

Table of Contents

• 1 JustSpeak: Controlling Android By Voice
  • 1.1 Starting JustSpeak
  • 1.2 Initiating Voice Commands
  • 1.3 Global Voice Commands
  • 1.4 Local Voice Commands
  • 1.5 Chaining Commands
  • 1.6 Alternative Means Of Triggering JustSpeak

1 JustSpeak: Controlling Android By Voice

JustSpeak is an Android Accessibility Service that enables voice control of your Android device. Once enabled, you can activate on-screen controls, launch installed applications, and trigger other commonly used Android actions using spoken commands.

1.1 Starting JustSpeak

Once installed, you can start JustSpeak under Settings->Accessibility on your Android device — you only need to do this once, i.e., JustSpeak will be automatically restarted when you reboot your phone.

JustSpeak is an Accessibility Service i.e., once started, it uses Acessibility APIs on the platform to augment Android's user interface. JustSpeak augments the Android user interface with voice-input control; other Accessibility Services like TalkBack provide spoken feedback. Note that JustSpeak can be used either by itself, or in conjunction with other platform Accessibility Services such as TalkBack.

1.2 Initiating Voice Commands

Once started, JustSpeak registers itself as an Assistant on your device; you can initiate voice commands by swiping up from the bottom of the screen. Note that this is the same gesture that activates Google Now on devices running the stock version of Android. When using JustSpeak, you can get to Google Now by saying Google. Successful invocation of voice-control produces an auditory icon, indicating that your device is ready to listen. The auditory tone, along with an optional vibration (if available) you hear is your cue to start speaking; JustSpeak displays a visual overlay on the right-edge of the screen to indicate that it is listening. the recognized utterance is displayed visually, — additionally, it is spoken out if TalkBack is active. JustSpeak supports two types of voice control commands as described below.

1.3 Global Voice Commands

JustSpeak supports a set of global commands — these are global in that they are available on any screen. Global commands include:

Command	Utterance	Action
Open	Open <Installed Application>	Launch Application
Recent	Recent	Recent Applications
Quick Settings	Quick Settings	Launch Quick Settings
Toggle WiFi	Switch WiFi (On/Off)	Toggle WiFi
Toggle Bluetooth	Switch Bluetooth (On/Off)	Toggle Bluetooth
Toggle Tethering	Switch Tethering (On/Off)	Toggle Tethering
Home	Go Home	Return to home screen
Back	Go Back	Return to previous screen
Notifications	Open Notifications	Pull down notifications shade

Note that voice commands are flexible in that you can use a number of synonyms for verbs such as open e.g., launch. In addition, voice control commands can be formulated as full sentences, e.g., "Please open GMail".

List Of Synonyms For Open:

• open
• go to
• show
• display

In addition, JustSpeak provides the following spoken aliases as a means of triggering commonly used applications:

Utterance	Action
Browser	Launch default Web browser
Web	Launch default Web Browser
OK Google Now	Launch Google Now
Search	Launch Voice Search
Voice Search	Launch Voice Search

1.4 Local Voice Commands

In addition to the global voice commands that are available on any screen, JustSpeak lets you activate on-screen controls in a variety of ways.

Command	Utterance	Action
Activate	Click <control name>	Activate control by its on-screen name
Toggle	Turn on/off <toggle>	Toggle on/off switch
Scroll	Scroll Up/Down	Scroll e.g., Lists.

List Of Synonym For Activate:

• touch
• click on
• press
• activate
• open
• push
• tap

1.5 Chaining Commands

Note that you can issue a sequence of commands via a single utterance by using simple connectives such as then or and when speaking; as an example, you can say:

Quick Settings then  WiFi

to open "Quick Settings" and then click the "WiFi" item that appears within quick settings. This is an example of chaining together a global command followed by a local command that becomes available after the global command has executed successfully. Similarly, you can say:

Open GMail, then compose

to launch GMail, and immediately start composing a new message.

1.6 Alternative Means Of Triggering JustSpeak

We are continuing to experiment with alternative ways of initiating voice control. Toward this end, JustSpeak enables you to configure an NFC tag that can then be tapped to initiate voice control; see JustSpeak->Settings for configuring an NFC tag. On Android 4.3 and above, you can also trigger JustSpeak by long-pressing both volume buttons at the same time.

Date: 2013-08-06 Tue

Author: T.V Raman

Org version 7.9.3f with Emacs version 24

Validate XHTML 1.0

Jelly Bean: Accessibility Gestures Explained

1 Jelly Bean: Accessibility Gestures Explained

This article details accessibility gestures in Jelly Bean and is a follow-up to Jelly Bean Accessibility Explained. It gives a conceptual overview of the 16 possible gestures and describes how they are used. The interaction behavior described here holds for all aspects of the Android user interface, including interactive Web pages within Chrome and the Android Web Browser.

1.1 Conceptual Overview Of The Gestures

Placing a finger on the screen speaks the item under the finger by first placing Accessibility Focus on that item. Moving the finger triggers touch exploration which moves Accessibility Focus.

To generate any of the Accessibility Gestures discussed below, one moves the finger much faster — how much faster is something we will tune over time, and if needed, make user customizable.

To remember the gestures, think of the four directions, Up, Down, Left and Right. In addition to these four basic navigational gestures, we defined an additional 12 gestures by picking pairwise combinations of these directional flicks, e.g., Left then Down — this gives a total of 16 possible gestures. In what follows, left then down means swipe left, and continue with a down flick. Note that in performing these additional gestures, speed matters the most. As an example, it is not essential that you make a perfect Capital L when performing the Down then Right gesture for instance; speed throughout the gesture, as well as ensuring that the finger moves some distance in each direction is key to avoid these being misinterpreted as basic navigational gestures.

1.2 Accessibility Focus And Accessibility Gestures

Accessibility Focus is moved using the four basic directional gestures. For now we have aliased Left with Up, and Down with Right; i.e., both Left and Up move to the previous item, whereas Down and Right move to the next item. Note that this is not the same as moving with a physical D-Pad or keyboard on Android; the Android platform moves System Focus in response to the D-Pad. Thus, moving with a D-Pad or trackball moves you through the various interactive controls on the screen; moving Accessibility Focus via the Accessibility Gestures moves you through everything on the screen.

1.3 Accessibility Gestures For Common Actions

In addition to the basic navigation describe above, we define the following gestures for common actions:

Navigation Granularity

You can increase or decrease navigation granularity by rapidly stroking Up then Down or Down then Up.

Scrolling Lists

You can scroll a list forward by rapidly stroking Right then Left; the reverse, i.e., Left then Right scrolls the list backward by a screenful.

1.4 User Configurable Gestures

Gestures Down then Left, Up then Left, Down then Right and Up then Right are user configurable; their default assignments are shown below.

Back

Gesture Down then Left is the same as pressing the Back button.

Home

Up then Left has the same effect as pressing the Home button.

Status Bar

Gesture Up then Right opens Status Notifications.

Recent

Down then Right has the same effect as pressing the Recent Applications button.

1.5 Summary

Gestures for manipulating and working with Accessibility Focus are an evolving part of the Android Accessibility; we will continue to refine these based on user experience. At this point, you are probably saying:

But wait, you said 16 gestures, but only told us the meanings of 12 of them

You are correct — we have left ourselves some gestures to use for future features.

Jelly Bean Accessibility Explained:Touch Exploration Augmented By Gestures

1 Jelly Bean Accessibility Explained:Touch Exploration Augmented By Gestures

We announced a number of accessibility enhancements in Android Jelly Bean — see our Google IO 2012 announcements and our Android Accessibility talk from I/O 2012. This article gives a user-centric overview of the Jelly Bean interaction model as enabled by touch exploration and navigational gestures. Note that as with every release, Android Accesssibility continues to evolve, and so as before, what we have today is by no means the final word.

1.1 High-Level Concepts

First, here's some shared vocabulary to ensure that we're all talking of the same thing when explaining Jelly Bean access:

Random Access

Enable user to reach any part of the on-screen UI with equal ease. We enabled this as of ICS with touch exploration.

Deterministic Access

Enable user to reliably land on a desired item on the screen. We enable this in Jelly Bean with linear navigation.

Accessibility Focus

The item that the user most recently interacted with — either via touch exploaration or linear navigation receives accessibility focus.

Activation

User can activate item having accessibility focus by double-tapping anywhere on the screen.

1.2 Sample Interaction Scenarios We Enable

The combination of random access via touch exploration, backed up by linear navigation starting from the point the user just explored enables users to:

• Touch explore an application to understand its screen layout,
• Use muscle memory to quickly touch parts of the display to access familiar application screens,
• Use linear navigation to reach the desired item when muscle memory is wrong by a small amount.

As an example, when using the Google Play Store, I can use muscle memory with touch exploration to find the Search button in the top action bar. Having found an application to install, I can once again use muscle memory to roughly touch in the vicinity of the Install button; If what I touch is not the Install button, I can typically find it with one or two linear navigation steps. Having found the Install button, I can double-tap anywhere on the screen.

The same use case in ICS where we lacked Accessibility Focus and linear navigation would have forced me to use touch exploration exclusively. In instances where muscle memory worked perfectly, this form of interaction was highly effective; but in our experience, it also tended to lead to breakdowns and consequent user frustration in instances where users almost found the control they were looking for but missed by a small amount.

Having introduced accessibility focus and linear navigation in Jelly Bean, we decided to eliminate the ICS requirement that the user tap on or near a control to activate it — we now enable users to activate the item with accessibility focus by tapping anywhere on the screen. To eliminate spurious taps --- especially on tablets, we made this a double-tap rather than a single tap. Note: based on user experience, we may optionally bring back single tap at some point in the future as an end-user customization.

1.3 Summary

Android Accessibility continues to move forward with Jelly Bean, and will continue to evolve rapidly along with the platform. Please use the Eyes-Free Google Group to provide constructive feedback on what works or doesn't work for you --- what is most effective is to objectively describe a given use case and your particular experience.

What's New In Google Accessibility From Google I/O 2012

Google IO 2012: What's New With Google Access

1 Google IO 2012: What's New From Google Access

We showcased a number of exciting advances in accessibility on Android and Chrome during IO 2012. With these advances, blind and low-vision users can leverage Google applications and services on Android and Chrome to collaborate effectively with their peers and to obtain on-the-go access. Advances include out-of-the-box access on Android (JellyBean), a new set of gestures that enable fluent interaction on touch-screen devices, Braille support on Android, an extension framework for ChromeVox, and a new, high-quality voice for use with Web applications on Chrome.

1.1 Enhanced Android Accessibility In JellyBean:

• Accessibility on Android can be enabled by long-pressing with two fingers (4 seconds) the setup screen to enable out-of-the-box access for blind users.
• Touch exploration has been enhanced with simple gestures that enable users navigate on-screen contents.
• JellyBean provides a set of Accessibility Actions that can be called from any AccessibilityService such as TalkBack; it also provides early support for Braille displays.
• Touch exploration and gesture navigation both set AccessibilityFocus —

double-tapping anywhere on the screen activates the item with AccessibilityFocus .

• TalkBack now has a sister service BrailleBack for providing Braille support on Android.

• Chrome on Android is now accessible and supports the latest in Web Access standards.

With these enhancements in Android access, blind users can use a combination of touch exploration and navigational gestures to access any part of the Android user interface.

As an example, I typically use the Android Play Store by touching the screen around the area where I expect a specific control; quick flicks of the finger then immediately get me to the item I want. With these touch gestures in place, I now use touch exploration to learn the layout of an application; with applications that I use often, I use a combination of muscle memory and gesture navigation for rapid task completion.

1.2 Chrome OS On Chrome Books And Chrome Box

Chrome OS comes with ChromeVox pre-configured — ChromeVox is our Web Accessibility solution for blind users. With the new high-quality voice that is being released on the Chrome Webstore, ChromeVox now provides smooth spoken feedback in Chrome on all desktop environments. Finally, we announced a flexible extension framework that enables Web developers leverage ChromeVox from within and outside of their own Web applications to provide rich, contextual spoken feedback.

1.3 Developer Tools For Ensuring Accessibility

To help developers better leverage Web Accessibility, we are releasing a new Accessibility Audit tool that enables Web developers detect and fix commonly occuring accessibility errors. This tool has been integrated into Chrome's Developer Tools and helps Web developers ensure accessibility while working within their normal workflow.

1.4 Accessibility Related Presentations At Google I/O 2012

Catch these on Youtube in the next week if you weren't able to attend I/O this week.

• Android Accessibility (T. V. Raman,Peter Lundblad, Alan Viverette and Charles Chen).
• Advanced Web Accessibility (Rachel Shearer, Dominic Mazzoni and Charles Chen).
• What's New In JellyBean: Android Team.
• JellyBean announcement in the Wednesday keynote.

Date: 2012-06-21 Thu

Author: T.V Raman

Org version 7.8.11 with Emacs version 24

Validate XHTML 1.0

Accessible GMail On Android — Eyes-Free Email On The Go

1 Accessible GMail On Android — Eyes-Free Email On The Go

I've been using Android as my primary smart phone since late 2008, and the level of email access I've had on Android in the past has always been a source of frustration. About a year ago, I first started accessing my email on Android with K9-Mail — that helped me bridge some of the accessibility gaps on the platform.

Over the last few months, our friends over in GMail Mobile have been adding accessibility support to the GMail client on Android. What is truly exciting is that this support is being added to existing releases of Android including Froyo (Android 2.2) and GingerBread (Android 2.3). This means that GMail on Android is now accessible on existing devices — get the update from Market and give it a spin.

1.1 Typical Usage Pattern

Here is my typical usage pattern when accessing my corporate email at Google. Note that the volume of email I receive on this account is extremely high, and includes many mailing lists that I typically do not read while on a mobile device. To limit how much email I download to the mobile device, and to ensure that I attend to the most pressing email messages while on the go I do the following:

• I have defined a GMail filter that assigns label to-mobile to messages I want to access when on the go.
• Typically, this includes email addressed directly to me, and other priority items.
• I launch GMail to open to this label.
• I quickly skim through the list of displayed messages to here the subject and a quick overview of the message.
• If I decide to read the complete message, I select that message via the trackball on my Nexus One to hear the message in its entirety.
• And finding an email thread I am looking for is just one click away — press the search button, and use up/down to navigate your search history.

See our help center documentation for additional details.

Author: T.V Raman <raman@google.com>

Date: 2011-08-10 Wed

HTML generated by org-mode 6.30c in emacs 23

Leveraging Android Access From Google IO 2011

You can watch our Google IO 2011 on Levarging Android Access APIs. The main take-aways from the talk:

• Android Access is easy --- the framework does most of the heavy-lifting.
• Implementing Android Access does not mean you take a performance hit.
• Accessibility is really about expanding the reach of your application.

Implementing accessibility within your application and thereby ensuring that it is usable in a wide variety of end-user scenarios will benefit your application --- both in terms of the number of users you gain, as well as how often your users use your application.