Task

Subjects (S1–S4) played a simple video game in which they controlled a spaceship with a small analogue joystick on a gamepad (Logitech® Rumblepad™ 2, Logitech Europe S.A., Morges, Switzerland) in the horizontal dimension (left-right; Figure 2a; Supplementary movie 1). The task was to evade blocks dropping from the top of the screen at a constant speed. The game was challenging enough so that the spaceship occasionally collided with a block (collision event, Figure 2b). At random moments, the spaceship moved in the direction opposite to the joystick movement for the duration of 500 ms (movement mismatch event, Figure 2c). Points were awarded for moving the spaceship, and subjects were instructed to gather as many points as possible. Subjects started the game with 20 “lives”. Each time the spaceship collided with a block, the number of “lives” was reduced by one. When the number of “lives” reached 0, the game, together with the recording session, ended. Recording sessions lasted between 5 and 24 minutes. We identified the neuronal responses to collision and mismatch events that were not mixed with neuronal responses to other events by defining a subgroup of “clean” outcome and “clean” mismatch events, consisting of events at least 2 seconds away from any other event of any kind. The total number of events recorded for each of the subjects is given in Table 1.

Figure 2

Task and error events.

Table 1

Number of recorded sessions and events for each of the subjects.

To earn more points, subjects had to try to stay in the game as long as possible. Therefore, collision events presented a clear disadvantage in reaching the goal of the game. Thus, collision events reflected outcome errors. During the movement mismatch event, there was a clear discrepancy between the intended movement and the movement performed by the spaceship. Thus, movement mismatch events reflected execution errors.

In our previous study

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Subjects and Recordings

Four subjects (3 male, 1 female) suffering from intractable pharmaco-resistant epilepsy voluntarily participated in the study after having given their informed consent. The study was approved by the Ethics Committee of the University Medical Center, Freiburg, Germany.

For pre-neurosurgical epilepsy diagnosis, the subjects were implanted with an 8×8 grid of subdural surface electrodes covering parts of the primary and pre-motor cortex (Figure 3). Additional subdural surface and deep brain electrodes were implanted for subjects S1, S2 and S3

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Figure 3

Locations of ECoG grid electrodes in relation to the cortex.

Recordings from all electrodes were digitized at 256 Hz sampling rate for S1 and S2 and at 1024 Hz sampling rate for S3 and S4. No analogue filters were used during the data acquisition. Power line frequency was 50 Hz. Data analysis presented here was performed after the experiment using the MATLAB software package (MATLAB version 7.4–7.11, Natick, Massachusetts: The MathWorks Inc., 2007–2011).

Measures of Detection Accuracy

Consider a process where a subject is actively observing a scene and, when a given stimulus appears, a neuronal response is elicited. Assume that neuronal activity is continuously recorded and a detection algorithm is continuously evaluating whether a stimulus appeared, given the neuronal activity. The efficiency of the detector can then be measured by comparing two point processes: the set of time points when the stimulus was presented and the set of time points when the detector detected the stimulus from the neuronal recordings.

Due to the internal processes in the brain and other sources of noise, even a perfect decoder will have a temporal noise in the detected times of events. On the other hand, detected events will still be useful, even if the times of detections are not perfectly aligned to the times of the events. For some applications, high temporal precision is not necessary. We describe this requirement on our detector as temporal tolerance.

If we tolerate the detected events within a time Δt from the real events, then any detection within this time window will be counted as a true positive detection. Every event window in which there are no detected events will be counted as false negative detection. For measuring the detection accuracy, we would also need to know the ability of the detector at predicting non-events. To obtain a fair estimate of such ability, the area between the event time windows has to be divided in windows of the same size, 2Δt. Every non-event time window in which there are no detected events will be counted as a true negative detection and every non-event time window in which there is a detected event will be counted as false positive detection.

Sensitivity and specificity of a detector

Accuracy of a detector can be described by measuring how well it performs in two different tasks: (i) detecting events when events are present and (ii) not detecting events when events are not present. One way to describe the first property is by measuring the sensitivity of the detector by calculating the true positive rate (TPR)

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[27] as a number of true positive detections (N_TP) divided by the total number of real events. Since the total number of real events is given by the sum of true positive detections and false negative detections (N_FN), the true positive rate is given by:

(1.1)

The second property can be described by measuring the specificity of the decoder by calculating the false positive rate (FPR)

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[27] as the ratio of false positive detections (N_FP) divided by the number of all detections. Since the total number of all detections is given by the sum of true positive detections and false positive detections (N_FP), the false positive rate is given by:

(1.2)

FPR definition used here should not be confused with the alternative definition of the false positive rate (FPR_ALT)

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[28]:

(1.3)

A disadvantage of measuring the detection accuracy by TPR and FPR is that one cannot directly compare two different detectors when both TPR and FPR of one detector are higher than the TPR and FPR of the other detector. Therefore, a metric incorporating both sensitivity and specificity of the detector is needed. One such metric is the mutual information.

Mutual information of a detector

One way to measure the performance of a detector is to calculate the mutual information

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[29] between a dataset containing times of real events and a dataset containing times of detected events. The mutual information is given by:

(1.4)

where X and Y are the sets of all possible states of the real and detected event datasets and x and y are specific states from those sets, p(x) and p(y) are the probabilities of specific states and p(x,y) is the joint probability that states x and y occur jointly. In our case, the set of real event states consists of “real event” (re) and “real non-event” (rne), while the set of detected events consists of “detected event” (de) and “detected non-event” (dne). Joint and marginal probabilities used to calculate the mutual information are given by:

(1.5)

Given a certain dataset of real event times and certain tolerance, the maximum value of the mutual information is obtained when detected event times perfectly match real event times. This value is identical to the entropy of real event times H(X):

(1.6)

To compare the mutual information over different tolerances, we calculated the normalized mutual information, C_YX

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[30]:

(1.7)

Calculating mutual information and entropy from a recorded dataset will give a good estimate of their true values, as long as the calculated probabilities are good estimates of the real probabilities. However, recorded datasets have a finite length, which will make the estimated probabilities fluctuate around their real values. Using the estimated probabilities to calculate the mutual information and entropy leads to a bias in the estimation

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[31]. To remove the bias of the mutual information, we used first and second order terms of the mutual information bias expansion derived in the study by Treves and Panzeri

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[31]:

(1.8)

(1.9)

(1.10)

where I_N(X,Y) is the mutual information estimated from a dataset of length N. Here, the values of joint and marginal distributions have also been estimated from the same dataset. To remove the bias of the entropy, we used first and second order terms of the entropy bias expansion derived in the study by Victor

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[32]:

(1.11)

(1.12)

(1.13)

Thus, the bias corrected value of the normalized mutual information was calculated as:

(1.14)

Experimental Data Analysis

Signal components

We analyzed the low and high frequency components of the recorded ECoG signals (Figure 4). The low frequency component was extracted by smoothing the preprocessed ECoG signals using a symmetric 2^nd order Savitzky-Golay filter

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[33],

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[34] with a time window of 250 ms (nominal 3 dB cut off frequency: 7.85 Hz for S1 and S2, 7.59 Hz for S3 and S4; estimated using table from

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[35]; for justification on using this filter see

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[16], section 1 of the supplementary material). We defined a window around each event of any kind, starting 3 seconds before the event and lasting until 3 seconds after it. The signals outside all of these windows were used as baseline activity. To enable a clear comparison to baseline, the average baseline activity was subtracted from the filtered recordings in each session for each electrode. The resulting signal was defined as the low frequency component of the signal (LFC).

Figure 4

Extraction of low and high frequency components (LFC and HFC) from the ECoG recordings.

To analyze the high frequency component of the signal, time-resolved Fourier transformation using a Hamming window (333 ms window width, shifted in steps of 31 ms) was applied to the preprocessed signals, and the amplitudes were used for further analysis. To account for the general decrease in amplitude with increasing frequency, the amplitudes of every frequency bin were normalized by dividing them by the average baseline amplitude of the same frequency bin in the respective session. We then extracted the average amplitude across a frequency band from 60 Hz to 128 Hz for S1 and S2 and from 60 Hz to 200 Hz for S3 and S4. Since recordings for subjects S1 and S2 were sampled at 256 Hz, spectral amplitudes could be calculated only for frequencies up to 128 Hz (Nyquist frequency). Therefore, the frequency band used to calculate high frequency component could only comprise frequencies up to 128 Hz for subjects S1 and S2. Amplitudes calculated from ECoG signals recorded at least 3 seconds away from any event were used as baseline. To enable a clear comparison to baseline, the average baseline activity was subtracted from the extracted amplitudes in each session for each electrode. The resulting signal was defined as the high frequency component of the signal (HFC).

When LFC and HFC were used together for detection, we normalized every electrode and signal component to zero mean and unit variance, to accommodate for their different scaling.

Detection algorithm

To detect error events from the neural activity, we trained a set of classifiers that captured the neuronal features which are specific to error events (Figure 5). To be sure that the training data did not include neuronal responses to non-error events which were erroneously identified as ERNRs, we used ERNRs elicited by “clean” events only. Given a signal component Φ, the peri-error feature vector was defined as:

(1.15)

where t_E is the time of an error event, el1, …, elm are the selected electrodes and t₁, …, t_n are the selected time points relative to the time of the error event. Therefore, the feature vector contains n•m features for one signal component. If more than one signal component was used, the feature vectors of the signal component were concatenated, yielding an l•n•m dimensional feature vector, where l is the number of signal components used (l=1 or 2 in this study). The time points t₁, …, t_n were always equidistant and defined by a set of parameters: (i) the time of the first feature in relation to the time of the error event, t₁, (ii) the number of time points, n and (iii) the temporal distance between the first and the last feature, t_n-t₁.

Figure 5

Classifier selection and performance evaluation of the error detection algorithm.

Each classifier was build using two classes of feature vectors, the error class containing peri-error feature vectors (E_class) and the baseline class containing feature vectors when no errors were present (B_class).

(1.16)

where δt is the time resolution of the signal component (LFC: 4 ms for S1 and S2, 1 ms for S3 and S4; 31 ms when HFC or both components were used). ”

These classes were used to build either a regularized linear discriminant analysis (rLDA) classifier or a regularized quadratic discriminant analysis (rQDA) classifier

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Destroy user interface control[36]. A QDA classifier is built by fitting a Gaussian distribution to each of the classes and gives the probability to belong to one of the classes for any arbitrary point in the feature space. Each Gaussian is represented by a class mean and a covariance matrix. LDA is a simplification of QDA, where fitted Gaussian distributions share a common covariance matrix. Regularization is implemented by modifying the covariance matrix of the fitted Gaussian distributions and has the purpose to improve the accuracy of the discriminant analyses on a new, independent set of data. We used the regularization of the form:

(1.17)

where C is the covariance matrix of the fitted Gaussian distribution, C_R is the regularized covariance matrix, γ is the regularization coefficient and I is the identity matrix of the same size as C. In the case of rQDA, the number of vectors in the baseline class is much higher than the number of vectors in the error class. Therefore, we regularized the covariance matrix of the Gaussian fitted to the error class only. The classifier was then used to calculate the probability of every feature vector F in the test dataset to belong to the error class.

(1.18)

where tF is the time point corresponding to the calculated probability and tend is the duration of the tested dataset. We then extracted all local maxima of the probabilities and assigned a ‘detected as non-error’ (dne) state to remaining points. A threshold value λ was then selected and we assigned a dne state to all time points for which the value of the maxima remained below the threshold. Finally, we assigned a dne state to all remaining maxima for which there was a higher maximum less than 1 second away. We assigned a ‘detected as error’ (de) state to all remaining maxima. Since the classifier should be able to detect error events for which ERNRs were only slightly mixed with neuronal responses to other events, we calculated the detection measures between the times of detected events and the times of all real events within the test dataset, instead of just using the “clean” events.

To properly validate the classifiers, we divided the recorded data into three similarly long parts by splitting each session into three parts, each containing one third of the “clean” events. First, we chose a set of parameter values consisting of: (i) a time of the first feature in relation to the error event, t₁, (ii) a number of features, n and (iii) a time distance between the first and the last feature, t_n-t₁, (iv) regularization coefficient, γ, and (v) probability threshold λ. An rLDA or rQDA classifier was then built using the first part of the dataset. Using the built classifier, we detected the events on the second part of the data and calculated the C_YX. Values of the parameters were then changed and the process was repeated, until all parameter values from the parameter grid were tested. We used the following grid of parameter values: (i) t₁: from −667 ms to 667 ms in steps of 56 ms; (ii) n: 1, 3, 4, 5 and 8 when using single electrodes and electrode quartets for detection and 1, 2 and 3 when using anatomical electrode subsets or all grid electrodes for detection; (iii) t_n-t₁100 ms, 125 ms, 250 ms, 500 ms, 750 ms and 1000 ms; (iv) γ: 0, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99 and 1; and (v) probability threshold: from 0.5 to 1 in steps of 0.017. The classifier that gave the maximum C_YX on the second part of the dataset was then used to detect the events on the third part of the dataset and TPR, FPR and C_YX were then calculated from this detection result. The same process was repeated, now using the third part of the dataset for testing the grid of parameter values and the second part of the dataset for classifier testing. The average values of the two sets of TPR, FPR and C_YX measures were then reported as the measured detection accuracy.

Different tolerance values were used to bin the experiment time into non-overlapping time bins, as described in section “Measures of detection accuracy”. The tolerance value directly determines the length of the dataset. Table 2 gives the dataset lengths for the tolerance values used.

Table 2

Total dataset length for different temporal tolerances.

Neuroanatomical Analysis

To determine whether the motor or the somatosensory cortex played a more distinctive role in generating ERNR, we classified electrodes as motor cortex electrodes, somatosensory cortex electrodes, and other electrodes (Figure 3) in the same way as done in the previous study by Milekovic et al.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Topographical Distribution of Informative Signals for Error Detection

To determine the topographical distribution of signals that were informative for the error detection, we performed detection using signals recorded from electrode quartets (Figure 5). For most of the subjects, several isolated, often spatially quite distant peaks of C_YX could be found over the cortical regions we recorded from. Locations of these peaks often differed for different error types and different signal components. Thus, we did not notice any topographical location that was systematically beneficial for detecting either outcome or execution errors.

Error Detection Using Signals from Motor or Somatosensory Areas

We compared the performance of error detection based on recordings from different anatomical subareas (motor, somatosensory and other areas) to the error detection performance based on recordings from all channels (Figure 9). For all 4 participants, the detection performance was highest when all electrodes were used; with subarea electrode sets reaching in some cases an equivalent performance.

Figure 9

Detection results when using signals from anatomical electrode subsets.

Next, we tested if recordings from motor or somatosensory areas provided enough information for high accuracy of error detection. To this end, we calculated the percentage of C_YX achieved when using signals from these areas compared to the C_YX values achieved when using all ECoG grid electrodes. These percentages were first averaged over all tolerance values and then over subjects. For S2, motor and somatosensory cortex was not well covered with electrodes as only 3 or 2 electrodes recorded from these areas (Table 4). We, therefore, excluded S2 from this analysis. Detection performance from motor cortex signals was 75±2% and 77±4% of the total detection performance for outcome and execution error respectively. Performance from somatosensory cortex signals reached 63±2% (outcome error) and 50±3% (execution error) of the total performance.

Table 4

Number of electrodes belonging to different anatomical subsets.

Detection from Smaller Electrode Sets

We investigated whether one can detect error events with smaller electrode subsets with accuracy similar to detection when all ECoG grid electrodes were used (Figure 10). When both frequency components were used for the tolerance of 366 ms, maximum C_YX from single electrodes was 60±6% for outcome error and 66±9% for execution error of the C_YX when all electrodes were used. For electrode quartets and both frequency components, maximum C_YX was 87±6% for outcome error and 78±10% for execution error of C_YX when all electrodes were used.

Figure 10

Detection results when using signals from electrode sets of different sizes.

Effect of MRNR Subtraction on the Normalized Mutual Information

We also investigated whether MRNR subtraction affected the detection of the error events (Figure 11). Over all subjects and tolerance values, the difference in C_YX when MRNR subtraction was and was not used was not significant when both LFC and HFC were used for detection, except for tolerances of 472 ms and 1 s for execution error for which the signals without MRNR subtraction gave higher C_YX values (C_YX difference for tolerance of 366 ms: outcome: −0.01±0.05; execution: 0.00±0.04). When only LFC was used for detection, using MRNR subtraction lead to a slight, but significant improvement, except for execution error for tolerances of 155 ms and 261 ms (C_YX difference for tolerance of 366 ms: outcome error: −0.14±0.04; execution error: −0.07±0.03). For the high frequency component, using MRNR subtraction did not change the C_YX values, except for outcome error for tolerances of 155 ms and 894 ms, for which the C_YX values were slightly significantly worse (C_YX difference for tolerance of 366 ms: outcome error: 0.03±0.05; execution error: 0.00±0.04).

Figure 11

Detection results when using signals with or without MRNR subtraction.

Characteristics of Signal Components Used for Detection

Here, we used both low (0–8 Hz) and high (60–128 Hz for S1 and S2 and 60–200 Hz for S3 and S4) frequency components of the neuronal signals to detect the error events. According to the study of Milekovic et al.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Detection of error events could still be possible, even in environments with more somatosensory stimuli and more different tasks at once. ERNRs used in this study were evoked on multiple, often quite distant, electrodes and these evoked responses exhibited quite different time courses. This makes the ERNR responses highly redundant and likely differentiable to other, non-error neuronal responses. Some of the earlier mentioned studies already used these signal properties to differentiate between neuronal responses to different non-error events

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Comparison to Previous Detection Studies

Several studies investigated the detection of epileptic seizures from neuronal recordings

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The afore mentioned studies used a wide variety of algorithms for detection: expectation maximization Gaussian mixture classifier

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The afore mentioned studies detecting movement states did not investigate the effect of temporal tolerance on the precision of the detector. Here, we showed that detection accuracy rises with the tolerance, until it saturates at around 300–500 ms. This implies that ERNRs used to detect error events are not perfectly timed to those events, and that the variability or ERNRs in response to error events is at the level of 300–500 ms. Reasons for this variability might be imprecision due to noise or other kinds of neuronal activity (e.g. ongoing activity) or the variability in the time subjects needed to recognize the error. Measurements of choice reaction time variability

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

Comparison to Previous ERNR Studies

Most of the earlier ERNR studies concentrated on activity of the anterior cingulate cortex (ACC) and its functional significance

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

ERNRs have also previously been found in motor cortex

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

The following popper user interface control may not be accessible. Tab to the next button to revert the control to an accessible version.

We would like to thank the subjects for participating in our study. We are grateful to the staff of the Freiburg University Hospital, Epilepsy Center for their help.