I was wondering if you know of a good way to calculate split-half reliability of PCA temporal/spatial factors derived from the EP Toolkit? E.g., Is there a way we can export trial-level data for each of our conditions (in this case gains vs. losses in a monetary reward task)?
Thank you!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
sure! I'll need more information to fully respond though. So first of all, was your PCA applied to the subject averages or to the individual trials? And for the split-half reliability, do you indeed need the reliability at the trial level? I ask, because computing the PCA at the subject level, but then trying to do trial-level things with the average-level results would be tricky.
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I believe the PCA was applied to the individual trials. And, what we are trying to calculate is the reliability for the amplitude measures from the different temporospatial factors. With mean amplitude-based ERPs, we just compute the ERP for even and odd trials separately and then calculate split-half reliability. We weren’t sure if this type of thing would be possible for PCA derived factors (without physically conducting the PCA separately on the even and odd trials, which might yield slightly different solutions). Is there a straightforward way to report the reliability (split-half or other measure) of the PCA-derived ERPs?
Thank you again- I appreciate your help.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well, first I need you to be much more precise about how this dataset was generated. The devil is in the details, as they say. Please confirm that it was applied to individual trials, and then if so, describe how this was conducted. You're not describing a conventional EP Toolkit analysis, so there are many possibilities here.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I apologize for the delayed reply. My advisor was out of town and I wanted to wait until he came back so that I cold confirm with him before responding. I was mistaken and the PCA was applied to subject averages (average of all trials for a given condition within each subject) rather than individual trials. So, it seems like we won’t be able to use these data for split-half reliability. Sorry, I should have known this earlier. I guess we’d need to do the PCA at the item level to be able to look at reliability data (split-half reliability or something like standardized measurement error), right? Is there some other way that people tend to report the reliability of obtained PCA solutions?
Thank you!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ah, that makes more sense! It depends on what kind of reliability you want. The EP Toolkit is already set up to provide SME and Dependibility measures for ERP measures. Truth to tell, I don't think either of these provide what you have in mind. They are both pretty specialized in their nature. PCA is layering another level of complexity, so I don't know of anyone who uses reliability measures with them, except for between session reliability (at the subject level). The EP Toolkit automatically provides that as well (in the form of an ICC measure). The whole topic is pretty complicated and not fully worked out. Even in Psychometrics, where these measures started, they tend to be problematic. For example, the most commonly used measure in psychometrics is Chronbach's Alpha, and yet the developer disavowed it some time back, but it keeps getting used. The EP Toolkit can also provide item averages. The first step in this is you need to specify what kind of reliability that you are looking to measure (trial vs. subject vs. session level). There are many kinds of reliability. And that's even before you get into the PCA aspects.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
I was wondering if you know of a good way to calculate split-half reliability of PCA temporal/spatial factors derived from the EP Toolkit? E.g., Is there a way we can export trial-level data for each of our conditions (in this case gains vs. losses in a monetary reward task)?
Thank you!
Hi,
sure! I'll need more information to fully respond though. So first of all, was your PCA applied to the subject averages or to the individual trials? And for the split-half reliability, do you indeed need the reliability at the trial level? I ask, because computing the PCA at the subject level, but then trying to do trial-level things with the average-level results would be tricky.
Joe
and which part of the statistics are you trying to do the split-half reliability on? The PCA solution itself or the ANOVAs?
Thank you for your quick response!
I believe the PCA was applied to the individual trials. And, what we are trying to calculate is the reliability for the amplitude measures from the different temporospatial factors. With mean amplitude-based ERPs, we just compute the ERP for even and odd trials separately and then calculate split-half reliability. We weren’t sure if this type of thing would be possible for PCA derived factors (without physically conducting the PCA separately on the even and odd trials, which might yield slightly different solutions). Is there a straightforward way to report the reliability (split-half or other measure) of the PCA-derived ERPs?
Thank you again- I appreciate your help.
Well, first I need you to be much more precise about how this dataset was generated. The devil is in the details, as they say. Please confirm that it was applied to individual trials, and then if so, describe how this was conducted. You're not describing a conventional EP Toolkit analysis, so there are many possibilities here.
I apologize for the delayed reply. My advisor was out of town and I wanted to wait until he came back so that I cold confirm with him before responding. I was mistaken and the PCA was applied to subject averages (average of all trials for a given condition within each subject) rather than individual trials. So, it seems like we won’t be able to use these data for split-half reliability. Sorry, I should have known this earlier. I guess we’d need to do the PCA at the item level to be able to look at reliability data (split-half reliability or something like standardized measurement error), right? Is there some other way that people tend to report the reliability of obtained PCA solutions?
Thank you!
Ah, that makes more sense! It depends on what kind of reliability you want. The EP Toolkit is already set up to provide SME and Dependibility measures for ERP measures. Truth to tell, I don't think either of these provide what you have in mind. They are both pretty specialized in their nature. PCA is layering another level of complexity, so I don't know of anyone who uses reliability measures with them, except for between session reliability (at the subject level). The EP Toolkit automatically provides that as well (in the form of an ICC measure). The whole topic is pretty complicated and not fully worked out. Even in Psychometrics, where these measures started, they tend to be problematic. For example, the most commonly used measure in psychometrics is Chronbach's Alpha, and yet the developer disavowed it some time back, but it keeps getting used. The EP Toolkit can also provide item averages. The first step in this is you need to specify what kind of reliability that you are looking to measure (trial vs. subject vs. session level). There are many kinds of reliability. And that's even before you get into the PCA aspects.