This sounds really exciting. I'll be in Portland at the end of July/beginning of August for the Cognitive Science conference, and would love to stop by and check it out and meet you when I'm there. And my kids would love the museum too I'm sure. I can help implement any of the suggestions below, and I'm willing to help out to whatever extent I can and you want/need. I'm sure you have thought many of these issues through, but maybe not all of them:
* The tests you have approval for are a nice cross-section. If IRB approval is not onerous, there are some others you might want to consider, including ones that are not completely distributed yet but will be this summer. A few include: the stroop test, the SAtest (a dynamic visual tracking task with lizards and bugs as the targets), alternative versions of the memory span task that use a 3x3 picture grid to do responses (see mspan\instruc.pbl), and new versions of object comparison/mental rotation tasks that I have not yet distributed, the Corsi blocks test (visual-spatial WM), and a few versions of pattern recognition tests (delayed match-to-sample, matrix rotation, pattern comparison), and a face-search task where you see a target face and have to look for it in a field of distractors. There may be more, and I guess it would partly depend on the interests of you and your undergraduates. I suggest these because they are at least somewhat interesting and fun. In contrast to the various vigilance tasks which are extremely painful.
* I'm guessing that some/all of the tasks you use will need to be tweaked a little from the versions currently distributed. Especially with regard to the number of trials to scale it to a time period that a child wandering in off the street would tolerate (probably no more than about 5 minutes per task, but maybe that could be pushed for specific tasks). Also, some of the start-up sequences might be changed so that a particular version of the test is chosen automatically instead of selected by the experimenter (as in TOL). And feedback might be important as well. Not all the tasks offer immediate or delayed feedback about performance, and that could be important for an educational setting like the museum. Also, some tests might need to be tweaked to fit whatever screen sizes you are using. For some tasks, the trial structure could be made a little more resilient too, so, for example, if they don't solve TOL in something like 15 moves you stop and move on to the next problem. And maybe the particular data saved could change, to include timestamps and such. Finally, at the end of any study, you may want to give some short debriefing or specific feedback about performance. For some of the changes made, I will likely want to change the PEBL Test battery to closely match the tests you run.
* My biggest concern from a scientific perspective would be that parents would want to 'help out' their children. You may not really be able to prevent this, but you can at least anticipate it and find a way to record when it happened so you can throw those data out.
* I'm guessing you will have mouse input, and not touch screens? There is a mixture of keyboard and mouse entry tasks, and you should think about whether input modality could/should change. For a study like this, it would probably be nice to only give the subject a mouse to control the experiment (or just a touch screen). Right now, the tapping task, digit span, time wall, and mental rotation tasks are probably the only ones that need a keyboard. Each of these could probably be translated to mouse-only entry fairly easily, and only for the mental rotation task would this change the character of the task much (RT would be distorted by the need to make a mouse movement to the correct option).
* In a context like this, it is easiest to launch the studies from a batch file or something other than the standard PEBL launcher. In a study I ran this last weekend, I launched a sequence of tests from another PEBL script that collected some demographic/condition information and fed that information to each study so it was embedded in the data files. Something like this could be really useful, and you should check out the survey.pbl as a hint of what is possible. This would let you randomize the order of the tests as well. The launcher script could include some baseline demographics and instructions (with OMSI logo on top), and then launch one or more tests coded to be saved as a particular subject number.
* As for headphones, it partly depends on your budget. Large earcup headphones would be good, but you may not fit children and adults well, and those can be $40-$100+ each if you don't have them already. That might not be in your budget. Even cheap headphones you can get for under $10 could be effective if you piped low-level white noise through them (probably less distracting than video game music, but it should work for muffling environmental noise). This might prevent using auditory feedback or auditory instructions. I'm not sure that PEBL would support playing a long wave file in the background without hurting data collection, but you could have an MP3 with white noise looping on the computer in the background.
* Also re audio, you may also consider creating audio recordings of instructions, especially for children who are pre-literate or not good at reading. PEBL is fine for simple audio like this, and if you have headphones, that would probably work well. I would suggest Audacity as a nice free audio recorder if you don't have anything else. Similarly, having some additional audio feedback at different points would be useful and more engaging than simple text feedback. I can help edit the scripts so that either of these audio options could be used. Another route would be to have no audio (which can be done easily in the digit span task, or by replacing it with another version of memory span), which would eliminate the need for headphones other than as a sound buffer.
* The wisconsin card sort was originally tested by Berg on apes I believe, so it will probably work for children, but it might take them a long time. There are published norms of that task in books with titles like "handbook of neuropsychological assessment" which might provide a clue of whether children can do it. This is probably the most difficult of the tasks you have approval for. My 5-year-old loves to play the tower of london and solves it fine; at least the versions with unconstrained peg heights. The classic Shallice version might be too difficult for younger children.
* Of the tests you list, the 'hunt' task, and the 'mental rotation' task are probably the least well developed. Either of them may take additional work to make ready for testing. I'm also wondering which dexterity task you refer to. Is it the fitts law/aimed movement task? If it is the one where you move a noisy cursor to the bullseye, that is somewhat experimental, but the ptracker compensatory tracker is modeled after a pretty prominent task. Fitts is the only one that is really ready for use. The hunt task was basically a programming experiment on my part, and although it has some direct analogs in animal research and foraging, it was not designed to test any particular hypothesis, and I'm not sure if it is currently instrumented to record data. So, there is substantial room for changes/improvements in some of these tasks that might make them more interesting.
* A little thought should be given to how the data produced should be saved--should it all be saved to a single file which gets archived somewhere, or is every subject a new file? If you have a central server you want it archived to, you could run an scp or database command at the end of each test subject.
* I think the proposed sample would be great. Obviously, having 100 in each cell would be better, but that seems unlikely, and even the design you propose might be difficult to fill completely. 350 subjects x 5-10 tasks is a huge logistical undertaking as it is, and would be an amazing contribution to science and would be important for the PEBL project. The only real suggestion I'd make is to split up the 18-29 group into at least 2 groups--in lifespan tests of simple tasks, there is usually an inflection point on tasks like this somewhere between 18 and 24 which represents peak performance, so 18-24 and 25-29 would be reasonable categories, as would 18-20,21-24,25-29. Of course, this could be done post-hoc if you get enough subjects in that age range. The sample size would probably be a little small to use for diagnosis against a norm, but it should be sufficient to plot out basic increases/declines in mean with age, at least for many of the tasks.
* Are you planning on having people do multiple tests? If everyone will do all the tests, then you can identify correlations between performance on the tasks if you tie them together with a subject code; if not, and each person will do a few tests, there may be tests that fit together better, in the sense that it would be important to learn whether performance in them is correlated. If people would only do one task each, this is moot.
* I have fixed a few fairly serious bugs since the last release of PEBL, and I can give you a new PEBL executable archive to run from, which should reduce the chance of crashes.
Best of luck, and let me know if there is any way I can help out.
Shane T. Mueller, Ph.D.
Klein Associates Division
Applied Research Associates
1750 Commerce Center Boulevard
Fairborn, OH 45324
937 873 8166 ext 126
From: Brian Piper [psy391@...]
Sent: Friday, April 09, 2010 3:42 AM
Subject: [Pebl-norms] brainstorming
I wanted to throw some ideas out to the PEBL community for their feedback for a project in the development phase.
I have been running various computerized learning and memory tests on a community sample from the Oregon Museum of Science and Industry (OMSI) in Portland, Oregon, over the past couple years. The staff there has been extremely supportive.
This summer, about 5 undergraduate students will be joining the lab. We have IRB approval for 10 tests: tapping, dexterity, rotary pursuit, timewall, Berg's Card Sort, Digit Span, Tower of London, Trail-Making Test, mental rotation, and Hunt and have multiple computers to test folks simultaneously. Each student will pick 1-2 tests for their summer project.
Q1) How does this sample sound?
Age Male Female
6 10 10
7 10 10
8 10 10
9 10 10
10 10 10
11 10 10
12 10 10
13 10 10
14 10 10
15 10 10
16 10 10
17 10 10
18-29 10 10
30-39 10 10
40-49 10 10
50-59 10 10
60+ 10 10
70+ 5 5
Total 175 175 350
Of course, some tests might be too difficult for really young-kids (e.g. Berg's Card Sort) so there would have to be some piloting. Would that sample size be reasonable? Is anyone aware of any guidelines that clinical psychologists use when norming their tests?
The other detail to keep in mind is that this would be a community sample, largely middle class (OMSI has an admission fee of about $13 for adults). This being Portland, I'm anticipating about 95% Caucasian. Also, as this is a museum, we can ask simple questions (age, sex, handedness, do you have corrected to normal vision/hearing?) but that's about it. Anything more invasive (e.g. ADHD diagnosis, current medications) wouldn't be appropriate here.
Q2) The OMSI environment is like you'd probably imagine, there are occasional announcements on a loudspeaker, folks chatting excitedly, cell phones, etc. We can minimize visual distractions with physical dividers. Auditory distractions are less easily dealt with.
Any thoughts about headphones? I'm thinking the big, puffy kind that were popular in the 70's. These would be essential for digit span but what about other tasks? One possibility would be having music from a video game archive playing in the background (e.g. for Tower of London). The + would be the data would be less noisy. The - is that this could substantially alter the overall character of the test. Basically, I'm wrestling with the pros and cons of data collection in this environment.
Thanks for your suggestions and thoughts,
Brian J. Piper, Ph.D.
Department of Behavioral Neuroscience
Oregon Health & Science University