-
10.
- Summary of Conclusions of Test Results
- Triangle Pilot
- Web Pilot
- Main Auditory Graph Test
- Further Studies That Are Suggested By Test Results.
- Assumptions and Additional Auditory Graph Techniques
- Auditory Preference Test Pilot
- Overview
- Sample
- Data Collection
- Instrument Development
- Data Results
-
10.1.
The Triangle pilot was useful in gaining experience in question development and initial auditory graph display analysis. From this test, the auditory graphing techniques was modified in two significant ways. First, in order to accentuate the curvature, a derivative tick mark was added as a drum beat where the tempo of the drum represented the magnitude of the graph’s slope. Secondly, the data was mapped to a chromatic scale rather than the previous linear scaling. The tonal quality of the sound was also modified, but this was a result of utilizing a MIDI to implement the sound files rather than from research findings. The testing method was also modified after analysis of the Triangle pilot. The testing was changed from a guided interview method to accessing the test via web pages.
10.1.The web pilot was important to gauge student participation for a self guided test. Participation was not a problem as when the test was offered for token credit, more student subjects participated than were in attendance when the announcement of the test was mentioned in class. The web pilot demonstrated that the scripting method was adequate to display the auditory test and record subjects’ results. This pilot also displayed the inadequacies of the initial set of questions for a full comparative test. Thus, while the testing environment did not need to be modified, the questions used in the test were extended and reworked to provide a more complete comparison.
10.1.The Main Auditory Graph test demonstrated that student subjects with auditory graphs were able to perform at 80% the level of subjects given visual graphs. While this is a significant difference, it also demonstrates that subjects are able to use the graphs to answer many math and physics questions with very little self guided training.
This test also demonstrated that blind subjects around the world could not only access the test, but could effectively complete and answer the questions. In addition they were able to do so at a level that exceeded the student subjects, and was not significantly different from local graduate students taking the test with auditory graphs. Their response rate was also greater than 80% the level of graduate subjects with visual graphs.
10.-
10.2.
While the experiments performed were an adequate demonstration of an effective test with auditory graphs, there were many unanswered questions that arose. First, there were several assumptions as to the auditory representation. For example the piano tone was used for the data, a drum for the derivative, and the drum pitch were subjectively chosen by the author as a useful and convenient working model to begin with. There was no indication that these choices were necessarily the best ones to make. Therefore, future studies will want to make evaluations of these assumptions.
As was noted by one subject, the concept of 0 needs to be well represented with the auditory graph. In the previous studies, all graphs were produced for positive y axis values, thus eliminating the necessity to work with negative values. In order to have a more general and flexible graphing method, 0 and negative values need to be well represented. All graphs in these studies have been single valued, single data sets. Methods for construction of multiple data set comparisons need development, as well as the display of multi-valued functions.
Another important effect for further study is an analysis of training times on performance. The dramatic differences between the Web Pilot and the Main Auditory tests indicate that there is such an effect. Furthermore, there may be an effect that certain graphing techniques or indicators are only valid or useful under unique circumstances. Even if specific indicators are helpful, there is a question of whether or not they are desired by the user.
Since these studies concentrated on very simple graphs, it is still unknown how complex can graphs be for the y axis to pitch representation to still work. As graphed data becomes more complex, there may be a preference for audification (directly representing the data values as the wave pattern, and then using that pattern to drive a sound source) or other data sonification methods.
The auditory graphs in this study only allowed for limited control of the sound parameters. Playback rate, control on the investigation of data points by the user, the ability to listen to the sound forwards or backwards could all have an effect on graph comprehension.
These points demonstrate the need for future research. However, is it even possible to carry out such experimentation? To that extent, a method for comparing auditory graphing was developed and pilot tested as the Auditory Preference Test.
10.2.-
10.2.2.
In order to assess the effectiveness and desirability of various auditory graphing techniques, a test was developed that used a combination of pair-wise preference comparisons, graph identification questions, and Likert preference ratings. The preference questions were used to indicate which graphing styles subjects liked best, or thought were most useful. The graph identification questions were used to indicate which graphing styles has the highest rate of being answered correctly.
This test was created not only to find better elements for the auditory graph displays, but also to test and evaluate an alternative method of auditory graph production. This alternate technique utilizes Microsoft’s ActiveX controls to create "live" graphs that would have the potential for greater user control, customization, and flexibility than prerecorded graphs could attain.
The results of preference tests such as this one are able to directly guide software development incorporating auditory graphs. The Main Auditory Graph test demonstrated that basic auditory graphs could be effectively used for answering questions, but tests such as the Auditory Preference Pilot can be used discover what issues should be addressed for the best optimization of auditory graphs.
10.2.2.There were 13 subjects who participated in this study. As this was a web based test similar to the Web Pilot, one subject attempted the test from a remote location. Due to technical difficulties, the auditory graphs from the AudioPlot method were not active, thus the results for this subject were not included. There were twelve subjects that participated locally using the same computer, but at different times. These subjects were solicited primarily due to their proximity to the research location and included five advanced undergraduate physics students, two graduate science and math education students, and three employees of the toxicology department. These subjects were chosen because they had not been involved in previous auditory graph research which was an attempt to reduce bias due to familiarity with previous auditory graphing techniques.
10.2.2.Subjects were invited to an office with a desk computer with a web browser displaying the test’s introductory page. Due to the nature of the ActiveX components for creating some of the auditory graphs Microsoft’s Internet Explorer was used as the web browser. Subjects were told briefly what to expect from the test, such as that it used a web browser to display the test and it consisted of nine questions about auditory graphs. They were also shown the controls for adjusting the sound volume produced by a pair of speakers next to the computer. The investigator indicated that he would be in a neighboring room in case any technical difficulties arose, and left the subject to take the test.
Data collection was then similar to the method used in the Web Pilot and the Main Auditory Graph tests in that it utilized a web browser to display graphs and information, and used PERL script programs to record answers. The test consisted of an introductory page with the Informed Consent Document and a brief description of the test. Next, subjects were presented with a page to record their names from which a scripting program appended the information to a file and assigned a code number. Subjects were then presented with a series of pages containing one or more auditory or visual graphs, a multiple choice selection field, and a text entry box for comments on their graph choice. Another scripting program appended their code and text answers to a second data file and passed the code and multiple choice answer to the next page. After completing the last question, the scripting program appended the code number and the string of multiple choice answers to a third data file.
10.2.2.There were nine question pages: four consisted of pair-wise auditory graph comparisons, four involved matching an auditory graph to a visual graph (two questions were matching a visual graph to a choice of auditory graphs, and two were matching auditory graphs to a choice of visual graphs), and one page with five-point Likert ratings of 6 graph types. Each question page had a text field for subjects to provide comments and reasoning for their choices. The questions can be found in Appendix D.
The auditory graphs were produced by two methods. The first method was by playing prerecorded MIDI sound files using a piano instrument to represent the data values as was used in the previous Web Pilot and Main Auditory Graph tests. For this method, the data was mapped to a chromatic scale. The second method for generating the auditory graphs was with the AudioPlot ActiveX control from Oregon State University's Science Access Project. The AudioPlot (AP) control generated auditory graphs on the subjects’ computer from equations specified by the web page. It allowed for various graphing parameters which were also set within the web page code
It should be noted that another potential remote subject did not participate citing security concerns with ActiveX control modules. The choice of utilizing these controls to generate the auditory graphs on the web was based primarily on the transport of Visual Basic code written for an updated version of the TRIANGLE graphing calculator. This code was quickly modified to produce the AudioPlot control modules which could be incorporated into a web based testing environment.
Both the MIDI and AudioPlot methods would play the auditory graphs when the subject selected a "play" button on the page. The buttons were identical so the subject had no indication of a difference between the methods to produce the graphs. The AudioPlot graphs produced a smooth, continuously varying tone with optional clicks for the derivative information. The MIDI graphs consisted more of a staccato piano note with a courser resolution. The derivative information was displayed with a drum like tone. All graphs were accessed through the use of Internet Explorer on a desktop personal computer.
10.2.2.€Table 10.1 is a summary of the multiple choice results for each question. The questions and answer choices are abbreviated for reference. The full text for the questions can be found in Appendix D.
Table .1 Summary of Answer Choice per Question
|
Answer Choice as Percentage of Total |
|||||||||||||||
|
Question |
A |
B |
C |
D |
E |
F |
|||||||||
|
1. Gaussian curve, A = AP, B = MIDI, C = both, D = neither |
33% |
58% |
8% |
0% |
|||||||||||
|
2. Gaussian curve with derivative A = low +, high -; B = high +, low -, C = both good, D = neither good |
33% |
17% |
33% |
17% |
|||||||||||
|
3. xsin(x). A = no change at 0, B = instrument change at 0, C = both good, D = neither good |
50% |
42% |
0% |
8% |
|||||||||||
|
4. xsin(x). A = AP with deriv., B = MIDI with deriv. and pitch change a 0; C = both, D = neither |
33% |
50% |
0% |
17% |
|||||||||||
|
5. Match sin(x)exp(-x) to graph d, AP graphs: a=sin(x), b=cos(x), c=xsin(x), d=sin(x)exp(-x), e=cos(x)exp(-x), f=none |
17% |
0% |
17% |
58% |
0% |
8% |
|||||||||
|
6. Match cos(x)exp(-x) graph e, MIDI graph: a=sin(x), b=cos(x), c=xsin(x), d=sin(x)exp(-x), e=cos(x)exp(-x), f=none |
8% |
0% |
0% |
17% |
58% |
17% |
|||||||||
|
7. Match AP graph of cox(x) to picture a. a=cos(x), b=sin(x), c=xsin(x), d=cos(x)exp(-x), e=sin(x)exp(-x), f=none |
75% |
0% |
0% |
17% |
8% |
0% |
|||||||||
|
8. Match MIDI graph sound of sin(x) to picture b. a=cos(x), b=sin(x), c=xsin(x), d=cos(x)exp(-x), e=sin(x)exp(-x), f=none |
0% |
92% |
0% |
0% |
0% |
8% |
|||||||||
|
9. Likert preference of xsin(x) graph with different sound representations 1- 5, 1 is bad, 2 is poor, 3 is neutral, 4 is good, and 5 great. |
X avg |
std. dev |
|||||||||||||
a. MIDI |
3.75 |
0.87 |
|||||||||||||
b. AP |
3.75 |
1.14 |
|||||||||||||
c. MIDI, dx |
3.08 |
0.79 |
|||||||||||||
d. AP, dx |
3.42 |
1.08 |
|||||||||||||
e. MIDI, 0 |
3.83 |
1.34 |
|||||||||||||
f. MIDI, dx, 0 |
3.33 |
1.44 |
|||||||||||||
By equation 2.2, the error associated with the each of the Likert averages can be found. Using a 95% probability limit, the average of the standard deviations (s avg = 1.12 = 28%), and the sample size of twelve subjects,
, (10.1)
or about 16% since there was a 4 point range (5-1) in the rating scale.
10.2.2.While the testing results would provide more consistency for a larger number of subjects, the purpose of this pilot test is primarily to discover where any difficulties in the testing process and question statements may reside. However, there are still important conclusions that be drawn from these answers . Comparing the results above to the subjects’ written comments is very informative as to the reasons for the choices and greatly aids the analysis.
The first question compared MIDI and AudioPlot (AP) representations of a Gaussian curve. These graphs only used the y axis to pitch mapping. The results for question 1 imply that there was a preference (58 to 33%) for the MIDI graph over the AP graph, and that this was a significant difference since t = 3.09 > t critical = 2.20 (double tailed t-test, df = 11, a = 0.05, s » s = 0.28 from average standard deviations of results in question 9.). This is a somewhat surprising result as great effort went to produce a pleasing smooth sound. The commentary is very interesting as unexpected factors played a role in the choice. Subjects choosing the MIDI graph mentioned that it "seemed cleaner", and that the discontinuous sounds produced a more dramatic effect, making it easier to distinguish the maximum point on a graph. In contrast, at least one subject preferred the AP graph because it was continuous.
Several subjects commented that their choice was at least partially based on the frequency ranges of the graphs. "The greater difference between the maximum and minimum tones made the graph easier to visualize" and resulted for a MIDI preference choice for this subject, but another chose the AP graph because "I seem to make the connection better for the higher pitches." Thus, future testing will need to be careful that the that different graphing methods display the same range in frequencies.
These choices may reflect the difference in data mapping methods used by the two auditory plots. As has been noted in previous research by Stevens in Mansur [Man85], pitch has a logarithmic association with height. Thus there needs to be a larger difference between tones at higher values. The linear mapping used by the AP graphs thus has a perceptual flattening at higher pitches and may seem less distinct.
Question 2 investigated the pitch mapping preference for curvature. A very brief description of what the secondary, drum tone was supposed to represent was given at the top of the page. The first graph used a low drum tone to indicate positive curvature, and a high drum tone for negative curvature. The second graph had the reverse mapping. The graphs were again of a Gaussian curve. The results were that 33% (four subjects) preferred the first graph (A), while 17% (two subjects) chose the second (B). However, 33% didn't have a preference (C), and 17% didn't like either (D). This difference can not be considered significant (t = 1.97) however.
The comments provide the additional feedback that those choosing C or D, often did so because they found the graphs confusing, or had a difficult time distinguishing between the graphs. Also, two subjects who gave a preference for one graph type, gave the opposite graph a higher preference rating in question 9. This indicates both the necessity for better descriptions, and asking the same question for several different graph patterns for consistency.
Question 3 investigated the preference of including a change in the
graphs data sound when the y axis value was negative. This was brought
about from comments received during the Main Auditory Graph test (even
though all of the graphs of that study had positive values.) For this
representation, the data sound of the graph of 
changed from a piano tone, for positive values, to a
harpsichord tone for negative ones. There was only a slight,
non-significant, preference for the tone change. The reasons for not
preferring the change are very informative. Cited complaints were that the
tone change created "too many options for the ear to play with"
and "broke up the graph a little too much." Those who preferred
the tone change found it very helpful. One comment was: "I liked how
the pitch changed when the graph went below 0. I think it is important to
change the sound when some major distinction (like the zero line) is
involved." A more pleasing, less distracting tone change may greatly
improve the preference for the tone change.
Question 4 compared the graphs of for the AP and MIDI methods with derivative information. The AP
graph had a score of 33% and represented positive curvature with a high
pitch click, and negative curvature with a low pitch click. The MIDI graph
had a score of 50% with positive curvature represented by a low pitch
drum, and negative curvature by a high pitch drum. The MIDI graph also
incorporated a tone change for negative values which was not included in
the AP graph as it did not have a similar display option at the time. The
difference was just under that for significance at a
= 0.05 (t = 2.10 > t critical = 2.20 .) This
question would greatly benefit from a larger sample size.
Of the subjects choosing the AP graph (A) and providing comments, there is an indication that improvements were still desirable: "A would be better if the drum pitch had those high harmonics for positive values instead of the negative ones." and "A sound is good to me. … Sharp pitch is better to me, but this one also needs some different sound to express the ups and downs." Of the subjects commenting on the MIDI graph (B), they often cited that their choice was because "the distinct sounds in B were much more clear than in A," and because of the "negative change and you can pick up the slope/curvature better."
There was also a comment by one subject, who chose the neither (D) option that "both seemed rather arbitrary in relation to the graph, at least in the derivative department."
Comparing the results of questions 5 and 6, which were graph
identification questions, shows identical results between the those
choosing the correctly, and in the distribution of incorrect responses. In
question 5, subjects were asked to match a visually presented graph of
to one of 5 AP auditory graphs
(with derivative information.) In question 6, subjects were asked to match
a visually presented graph of 
to one of 5 MIDI auditory graphs (with derivative and 0 information.)
Thus, subjects seemed to be able to match a pictured graph to its auditory
representation equally well with both methods.
Comments about the AP graphs (question 5) indicated that some subjects found the choices indistinguishable. There were statements of "I started to choose E or D, but really I didn't like any of the choices" from a subject choosing None of the Above (F), and "But frankly, a-d sounded all the same" from a subject choosing the correct answer (D). One subject who chose incorrectly, noted the disparity between the choice and their reasoning: "I just like the sound of C the best, however, listening to the pitches, it almost seems like the two maximums reach the same pitch, but on the graph, the second one is lower."
Of the subjects commenting about their choices on the MIDI graphs, those who answered incorrectly indicated that "the drums in the background created confusion as to what was going on" but that there were many close choices as the comments of "A and E sounded nearly the same" from one who chose A, and "E seemed the closest, but the derivative portion seemed wrong" from one who chose F indicate. One subject stated gave an incorrect choice D but stated that "the tempo of the drum was most clear in describing the slope of the line, as was the change in sound describing the negative values of the curve."
For questions 7 and 8, which also involved graph identification. subjects were given an auditory graph and were asked to choose between several visual graphs or a "None of the Above" choice. Question 7 asked to match an AP graph of cos(x) and had a correct response rate of 75%, while question 8 matched a MIDI graph of sin(x) and had a correct response rate of 92%. The difference was just under the level for significance ( t = 2.10) and would also have benefited from a greater number of subjects.
In question 7, one subject who answered incorrectly mentioned a difficulty in identifying the starting sound. Question 8 would have had a 100% correct score, but the one subject who chose F mentioned that the graph "seemed to mostly fit B, but I don't think the derivative was correct." The response rate on question 8 may have been greater than that of question 7 due to it being a fairly similar graph as the prior question.
The last question asked subjects to rate different representations of
the graph of on a Likert scale
of 1 to 5, where 1 was bad, 2 was poor, 3 was neutral, 4 was good, and 5
was great. The results are given in Table 10.2, but are a bit vague due to
the high standard deviations and all rankings should be viewed as
essentially equivalent as the averages were all within the smallest
standard deviation. ANOVA analysis of the average Likert scores from
question 9 display of different graphing methods shows no significant
difference between the methods at the a
= 0.05 level. (F = 0.82 < Fcritical =
2.35) All the average scores were between 3 and 4 indicating that the
methods still could be greatly improved.
Table .2 Ranking of Preferred Graph Types
|
Rank |
Average Rating |
Graph Type |
|
1 |
3.83 |
MIDI with 0 |
|
2 (tie) |
3.75 |
MIDI plain, AP plain |
|
3 |
3.41 |
AP with derivative |
|
4 |
3.33 |
MIDI with 0 and derivative |
|
5 |
3.03 |
MIDI with derivative |
For question 9, several subjects also provided general comments on what they found helpful or annoying. These comments tended to focus on the drum beat (or clicks) indicating curvature, and the change in tone indicating negative values.
A few selected comments demonstrate the greatest strengths and some potential problems with these auditory graphs:
"They all represented the graph well, it just depended on if one was interested in slope and curvature."
"I like hearing positive and negative. I like having pauses between notes instead of one constant sound. I like really hearing the slope. I don't like the soft drums because it's hard to differentiate them from the sound of the computer loading." [The computer had a somewhat noisy fan.]
10.2.2.