Abstract
Although the human hand has a complex structure with many individual degrees of freedom, joint movements are correlated. Studies involving simple tasks (grasping) or skilled tasks (typing or finger spelling) have shown that a small number of combined joint motions (i.e., synergies) can account for most of the variance in observed hand postures. However, those paradigms evoked a limited set of hand postures and as such the reported correlation patterns of joint motions may be task-specific. Here, we used an unconstrained haptic exploration task to evoke a set of hand postures that is representative of most naturalistic postures during object manipulation. Principal component analysis on this set revealed that the first seven principal components capture >90% of the observed variance in hand postures. Further, we identified nine eigenvectors (or synergies) that are remarkably similar across multiple subjects and across manipulations of different sets of objects within a subject. We then determined that these synergies are used broadly by showing that they account for the changes in hand postures during other tasks. These include hand motions such as reach and grasp of objects that vary in width, curvature and angle, and skilled motions such as precision pinch. Our results demonstrate that the synergies reported here generalize across tasks, and suggest that they represent basic building blocks underlying natural human hand motions.
Introduction
The human hand is remarkably complex: 27 bones are controlled by 39 muscles (Tubiana, 1981; Friden and Lieber, 1996). There are many degrees of freedom available for hand motion, yet joint movements are not independent. Anatomical factors, such as interdigit webbings, internal connections between various tendons (von Schroeder et al., 1990; Malerich et al., 1987; Linberg and Comstock, 1979; McGregor and Glover, 1988; el-Badawi et al., 1995) and multidigit insertions of extrinsic finger muscles (Tubiana and Valentin, 1964; Harris and Rutledge, 1972) result in mechanical coupling between various joints. Additionally, neuronal connections cause coupling. A single cortical motor neuron innervates multiple spinal motor neuron pools (Fetz and Cheney, 1980; Shinoda et al., 1981; Buys et al., 1986; McKiernan et al., 1998), which in turn drive multiple tendons (Schieber et al., 1997, 2001), and results in functional coupling between various muscles (Lang and Schieber, 2004). The sum of mechanical and neural coupling causes coordinated movements between various joints (Darling et al., 1994; Kilbreath and Gandevia, 1994; Kinoshita et al., 1995) referred to as kinematic synergies (Sherrington, 1947; Bernstein, 1967). Thus, although the hand has in excess of 20 degrees of freedom (Soechting and Flanders, 1997), the set of all naturalistic hand postures from everyday activities lies in a lower dimensional subspace.
Hand motion studies involving simple activities such as reach-to-grasp (Santello et al., 1998, 2002; Mason et al., 2001; Braido and Zhang, 2004), and skilled activities such as typing (Fish and Soechting, 1992; Soechting and Flanders, 1997) or finger-spelling (Jerde et al., 2003; Weiss and Flanders, 2004), reveal that a small dimensional space accounts for a large fraction of the postural variance observed during these tasks. Even individuated finger movements, which are the hallmark of any skilled activity, occur with synergistic couplings with adjacent digits (Schieber, 1995; Hager-Ross and Schieber, 2000). Synergies have also been observed in force patterns generated while grasping (Santello and Soechting, 2000) or during multidigit force production tasks (Zatsiorsky et al., 2000; Shinohara et al., 2004; Shim et al., 2005).
These studies involved very specific tasks invoking a limited range of hand postures or force patterns. Hence the synergies they report may be task specific (Rearick et al., 2003; Todorov and Ghahramani, 2004). The present study was designed to test whether a relatively small set of hand motions, or synergies, can be identified from unconstrained hand movements that occur during natural exploration tasks. First, we hypothesized that there is a basic set of synergistic hand motions that is task-independent. We tested this by evoking an unconstrained set of hand postures via a haptic exploration task. Specifically, postures were measured while blindfolded subjects explored fifty common objects. Second, we studied whether similar synergies are used when grasping objects with parametrically varying shapes under visual guidance. The aim was to test whether simpler paradigms, such as reach-to-grasp, sample the same space of postures used in object manipulation. Finally, we tested the generality of these synergies by using them to reconstruct postures evoked in skilled tasks, such as tip-to-tip pinches.
Materials and Methods
Experimental tasks
Eight subjects (seven males/one female, seven right handed/one left handed) participated in the experiments described below. The subjects gave informed consent to the procedures, which were approved by the Human Institutional Review Board of the Johns Hopkins University School of Medicine.
Haptic exploration task.
All subjects performed a haptic exploration task, in which they were blindfolded and asked to identify fifty common objects. Before each trial, the subject placed their dominant hand on a table in front of them and one of the objects (see Fig. 1b) was placed about 20 cm in front of their hand. On a go signal, the subject reached out, explored and identified the object. In addition, the subjects were asked to explore the surface curvature, edges and texture and in doing so rate the sharpest curvature, sharpest edge, and roughest surface encountered. These questions were posed to prolong the exploration time, because a simple object identification task is trivial and takes a very short time (Klatzky et al., 1985). Subjects were free to explore the objects as they wished. Sometimes they lifted the objects off the surface of the table, using different grasps ranging from a pinch grip between digits D1 (thumb) and D2 (index) to a power grasp that involves all five digits. At other times, subjects chose to explore with the object on the table and moved their hand around the object. Subjects also used their fingers to move the object around (in hand or on the table) to expose different surfaces for tactile exploration. Because subjects used unconstrained exploration strategies of objects that varied widely in their sizes and shapes, we believe that the set of postures evoked during this task is a representative sample of the naturalistic hand postures used in everyday object manipulation.
Experiment related information. a, Locations of the 23 markers on the hand. Four markers per digit were attached to the extreme ends of distal, middle and proximal phalanges of digits D2–D5 and distal, proximal, and metacarpal phalanges of D1. Three markers (markers 21, 22, and 23) were attached to the dorsum of the hand and were used to define a coordinate system attached to the dorsum of the hand. The x and y axes of this coordinate system are depicted in the figure. The z-axis points toward the palmar side. b, Image showing all the 50 objects used in haptic exploration task. The objects are listed in Materials and Methods, Principal component analysis. c, Schematics of shapes from the three sets used for the reach-to-grasp experiments. All shapes were 1 inch in thickness. R, Radius of curvature; W, width at the center.
Reach-to-grasp task.
Four of the eight subjects (P3–P6) performed an additional task involving reaching and grasping parametrically varying shapes. Unlike the previous task, subjects were not blindfolded and used vision to help guide their hands. Subjects were instructed to reach out and grasp various shapes between the distal pads of their digits (precision grasp involving all digits). Three sets of shapes were used (see Fig. 1c): (1) rectangular blocks of varying width (2, 5, and 9 cm), (2) curved surfaces such as circular plates of varying diameter (4, 10, and 16 cm) and convex or concave shapes with different central widths (5, 8, and 11 cm) and radii of curvatures (7.5 and 10 cm), and (3) oblique shapes with varying taper angle of the grasp surface using two different widths at the shorter surface (1 and 5 cm) and three different taper angles (15, 30, and 45°). Each object from the third set was presented twice, once with the shorter surface on the top, and then with the shorter surface at the bottom.
The objects were randomly presented one at a time while data were collected for 60 s. Subjects were instructed to reach out and grasp the object, hold the grasp for a couple of seconds, return to the starting position, rest for a couple of seconds and then repeat the procedure. Approximately eight reaches were performed within the collection time.
Precision pinch.
One subject (P8) performed an additional task that involved repeating four types of precision pinches (fingertip to thumb tip oppositions) between D1 and D2 through D5 successively. The subject placed his elbow on the table and held the digits of the hand in a comfortable extended position. On a go signal, the subject performed repeatedly the finger to thumb oppositions using D2 to D5 successively in order. Data was recorded for a total of 14 repeats of each of the four types of pinches.
Data acquisition
OPTOTRAK (Northern Digital, Waterloo, Ontario, Canada) sensors were used to record three-dimensional position data from 23 different points on the hand (Fig. 1a). Infrared emitting diodes were placed on the metacarpophalangeal (MCP), proximal interphalangeal (PIP), and distal interphalangeal (DIP) joints as well as tips of digits D2–D5. The position of the thumb (D1) was obtained using four markers placed at the MCP and IP joints of D1, tip and the base of the thumb. In addition three markers (markers 21, 22, and 23) were placed in a triangle on the dorsum of the hand. These markers were used to define a coordinate system attached to the dorsum of the hand. As a first step before any analysis, we corrected the marker coordinates for translation and rotation of the hand through space, by projecting the marker locations into this coordinate system (for details, see Appendix). Marker positions were sampled using 12 cameras and OPTOTRAK software at 100 Hz.
Data analysis
Principal component analysis.
If P“i = [P”x, P“y, P”z]T are the translation and rotation corrected coordinates of the ith marker at time t, the hand conformation or posture can be defined as a vector,V(t)= [P″1T,P″2T,…,P″23T]T, in a 69-dimensional space (23 markers × 3 coordinates for each). Vectors from every time step recorded while the subject explored the 50 objects (glasses, coffee mug, Coke can, pen, scissors, Scotch tape, computer mouse, key, screwdriver, stapler, teaspoon, pliers, blackboard eraser, file folder, Styrofoam cup, coffee jar, alligator clip, fork, comb, compact disc, lock, bulb, bowl, potato, tea cup, apple, paper clip, head phones, cell phone, 1.5 V cell, Gatorade bottle, frying pan, iron, milk bottle, remote control, bicycle helmet, phone, handset, glass jar, walkman, alarm clock, badminton shuttle, racquetball, tennis ball, camera, audiocassette, videocassette, ice tray, conical flask, computer speaker, and banana) were used to construct the covariance matrix:
Here, Vo(t) is the hand posture vector during the tth time step while the subject was exploring object o. V̄ is the subject's mean hand posture over the entire recorded data set from all objects. N(o) is the total number of time steps recorded while the subject was exploring the oth object. Time steps when any marker was occluded were skipped.
The covariance matrix represents the spread of the cloud of all conformations or postures as a hyperellipsoid around the mean posture in the 69-dimensional (69D) space. The directions and lengths of the axes of this hyperellipsoid can be obtained by an eigenvalue/eigenvector decomposition of the covariance matrix. For each subject, the eigenvectors were sorted according to the corresponding eigenvalues and the eigenvectors corresponding to the 10 highest eigenvalues were selected for further analyses. To study the contribution of a particular eigenvector Ei to the hand posture, we constructed hand postures Hi along the eigenvector Ei by adding to the mean conformation V̄, Ei scaled by a constant factor α, which acted to constrain the range of motion within biologically feasible limits. In other words, Hi = V̄ + αEi.
The 10 sets of hand postures corresponding to the 10 eigenvectors for each subject were visually compared across different subjects to identify similar trends. Those with similarities across multiple subjects were considered to belong to a common synergy. Nine synergies were similar across at least four subjects.
Similarity of synergies across subjects.
We used pairwise correlation coefficients between eigenvectors across subjects as indices of similarity of eigenvectors belonging to same or different synergies. The first measure, q1, for a synergy was a pairwise correlation coefficient between eigenvectors of various subjects that belonged to a given synergy. The second measure, q2, which measured the similarity of eigenvectors belonging to different synergies across subjects, was a pairwise correlation between an eigenvector of a subject belonging to the given synergy and all the eigenvectors, corresponding to the 10 highest eigenvalues, of other subjects that did not belong to the given synergy.
In a large multidimensional space, q2 is an underestimate on the similarity of vectors across subjects belonging to different synergies, because vectors that are random are more likely to be orthogonal in a higher versus lower dimensional space. As an extreme example, consider an eigenvector (v) of the first subject as a linear sum of two different eigenvectors (s1) and (s2) of a second subject as follows:
Although v is neither similar to s1 or s2, it is still orthogonal to the remaining 7 eigenvectors of the second subject, and hence there are 7 entries of zeroes in the distribution of q2 lowering the overall mean value. We therefore computed a third distribution q3 that was, for a pair of subjects and a given synergy, the maximum pairwise correlation between the synergy eigenvector of one subject, and all other eigenvectors of another subject that do not belong to the given synergy. Thus, q3 serves as a conservative estimate of similarity expected by chance.
Repeatability of the estimated synergy vectors.
Principal components (PCs) beyond the first three or four, account for a very small fraction of the observed variance (see Results). Thus, the shape of the variance cloud along these directions is susceptible to noise. To estimate the variations in PCs attributable to this undersampling of conformations, we divided the data into two sets corresponding to different sets of objects and then estimated PCs on each data set separately. Correlation coefficient between eigenvectors belonging to the same synergy but obtained separately from the two data sets was used as a measure of repeatability of the observed synergy (referred as q4).
Reach-to-grasp
Hand conformation during grasp.
The global coordinate system was aligned during the reach-to-grasp approximately along the x-axis, and the data were segmented into separate trials using hand (i.e., dorsum marker) velocity. Because the hand is not perfectly stationary during grasp, we identified the time steps before and after every grasp, where the velocity was 25% of the maximum value. The mean of these two time steps was a reliable indicator of the actual grasps.
Hand posture during maximum aperture.
Aperture size was defined as the root mean square of the distances between the D1 tip and the tips of every other digit. Marker coordinates around times when the aperture size maximized were used to represent hand postures during the approach to the given object.
Regression analysis.
Three sets of linear regressions were used to assess the effect of object width, curvature, and taper angle on the hand posture during approach and object contact. In the first regression, marker coordinates were regressed against a constant term and the width of the different objects from the first set. The second set of regressions captured the effects of changing the convexity or concavity of the grasped surface by regressing marker coordinates against a constant, the width of the object at the center, and the convexity or concavity of the grasp surface expressed as the inverse of the radius of curvature. We used positive (negative) values for convex (concave) surfaces. In the third set of regressions the coordinates of all markers were regressed against a constant term, width of the object at the center, and the taper angle. We used positive (negative) values for surfaces that tapered outward (inward) from top to bottom.
Six 69D vectors were constructed by using the 69 weights (23 markers × 3 coordinates) corresponding to object width from the first regression, surface curvature from the second regression and taper angle from the third regression respectively. These vectors point in the direction maximally affected by changing the object width, curvature and taper angle of the grasped surface during approach and actual grasp respectively. The degree of overlap between these vectors and the various synergies obtained by principal component analysis (PCA) was quantified by computing a pairwise correlation coefficient.
Precision pinch
We measured the fraction of variance in precision pinching that could be accounted for by the previously defined synergies. If V(t) is the translation and rotation corrected hand posture vector at time t, the total variance is σtotal2(t) = (V(t) − V̄)T (V(t) − V̄). Here, V̄ is the mean conformation obtained in the haptic exploration task. Variance captured by n synergy vectors (Ei to En) was computed as follows:
The ratio σn2(t)/σtotal2(t) denotes the fraction of variance captured by the n synergy vectors at time step t. Hand postures were then constructed from a subset of n synergy vectors as:
Synergies from interpolated data
Although using 12 cameras considerably minimized data loss caused by occlusion, we did lose data points, especially on occasions such as when the subject tried to explore the interiors of hollow objects. The results in this study are based only on time steps when all the markers were recorded, because we wanted to stay close to the true data. However, to investigate whether missing markers introduced any bias, we also computed synergies from a larger data set obtained by interpolating all of the missing markers except the markers on the tips of the fingers (because interpolation did not work reliably on the distal tip markers) and compared them with synergies obtained from the raw data via pairwise correlation coefficients. We interpolated the missing marker positions based on their distances from the other markers at any given time point. Within each trial, we computed the pairwise distances between all the 18 markers (23 markers minus five on the distal tip), at times when both the markers contributing to a given distance measure were captured. When either of the two markers was missing, the distance function was linearly interpolated. The markers positions were then estimated to minimize the sum of squares of the deviations from the interpolated distance measures. The deviations of each distance measure were normalized by the variance of the corresponding distance over the entire data set to weight each distance measure according to how likely it was expected to change.
Results
Haptic exploration
We identified a set of eigenvectors that described most of the variability of joint motion observed in the haptic exploration task (Fig. 2a). A cumulative plot of the normalized eigenvalues (Fig. 2b) shows that the first seven principal components account for >90% of the total variability.
Distribution of energies across principal components. a, Normalized eigenvalues (fraction of variance) along the first 15 principal components. b, Cumulative plot showing the fraction of variance account for by the first 15 principal components.
Structure of the principal components
The structure of principal components was remarkably similar across different subjects. However, the exact order of the principal components across subjects differed (Table 1). We identified nine vectors that were similar across multiple subjects. We refer to them as synergies and describe them below.
Occurrence of various synergies across subjects
Figure 3 and supplemental Figure 1 (available at www.jneurosci.org as supplemental material) show snapshots of hand postures along lines parallel to the nine synergy vectors and passing through the mean posture for two different subjects. The first rows in both figures illustrate hand postures along the first synergy. The salient aspect of this synergy is opening of the hand mostly via extension of MCP joints, but also because of PIP and DIP joints of digits D2, D3, D4, and D5. In addition, there is abduction (movement in the plane of the palm) and a slight flexion (movement in the plane perpendicular to the plane of the palm) of the thumb (D1) causing the thumb to “open” or move outward and closer to the plane of the palm. The second row of Figure 3 illustrates the second synergy for one subject. This synergy is characterized by digits D2 and D3 flexing inward toward the palm via a reduction in the MCP angles and digits D4 and D5 extending outward and away from the palm via an increase in the MCP angles at D4 and D5. In addition, there is a flexion of D1 causing D1 to move closer to the plane of the palm. The dominant feature of the third synergy was motion of a spread thumb back to a normal resting position via flexion and adduction of the thumb (Fig. 3, third row). This movement of the thumb is caused by flexion of the thumb, bringing the thumb closer to the plane of the palm, and a simultaneous abduction, causing the thumb to move laterally away from the other digits. The fourth synergy was characterized by a motion of the thumb toward the fingers (Figs. 3, fourth row). It differed from the third synergy, which is also dominated by thumb movement, in the direction of change of D1 abduction. In this case, the thumb adducted during a simultaneous flexion causing the thumb to move inward. The fifth synergy was characterized by flexing D2 and D5 by a decrease in MCP angles and extending D3 and D4 via an increase in the MCP angles (Fig. 3, fifth row). This caused digits D2 and D5 to move inward as digits D3 and D4 moved outward. In addition there is an increase in the PIP angles of D3 and D4 and a small increase in the DIP angle, causing the middle and distal phalanges of these two digits to extend from a curled position as the digits move outward. The sixth synergy is characterized by changes at the DIP and PIP joints of digits D2–5 (Fig. 3, sixth row), causing these digits to extend out from a curled position. There is also a slight flexing of the thumb causing it to move closer to the palm, and increased abduction causing it to move laterally outward. The seventh synergy involves a lateral movement of digits D2, D3, D4, and D5 with respect to the palm (Fig. 3, seventh row). There is an increased abduction of the thumb, causing the thumb to move outward (i.e., opposite to the motion of digits D2–D5). This motion is akin to what would be used to roll a ball between fingers and thumb. Also, there is a flexion of the thumb causing the thumb to move closer to the plane of the palm. The eighth synergy is characterized by flexing the hand via a decrease in the MCP angles of D2–D5, and extending the fingers via an increase in the PIP and DIP angles of the same digits (Fig. 3, eighth row). This causes the proximal phalanges of digits D2–D5 to flex in while the middle and distal phalanges flex out. The ninth synergy was marked by changes in the abduction angles; spread fingers were pulled together and straightened (Fig. 3, last row). Both examples show a decrease in the three abduction (ABD) angles D23 ABD, D34 ABD and D45 ABD. In addition, there is an increase in the PIP and DIP angles causing the middle and distal phalanges to extend outward. Note that although we have described the changes in hand postures from left to right, the directionality is immaterial (indicated by the bidirectional arrow at the bottom of Fig. 3), as the same synergies can account for changes in the opposite direction. For example, synergy S1 which captured opening of the hand can also account for closing of the hand.
Structure of nine synergies: Examples illustrating the structures of the nine synergies. Each row illustrates five hand postures along a particular synergy. Synergies S1 and S2 belong to subject P1, while Synergies S2–9 belong to subject P4. The circles indicate the marker positions on digits D1 (black), D2 (blue), D3 (green), D4 (red), and D5 (yellow).
Dynamic contribution of synergies during exploratory movements
The nine synergies described above, define a 9D coordinate system for each subject with its origin at the center of the cloud of all postures evoked by the particular subject. Any posture can be considered as a unique point in this 9D space. Similarly, any exploratory movement can be considered as a trace or curve meandering inside this 9D space. Just like a static posture (unique point) can be expressed as a linear combination of projections along the synergies, any curve or time series of hand motion segment can be expressed as a linear combination of corresponding time series of projections or movement patterns along each synergy. Figure 4 illustrates this point by plotting a 3 s segment of haptic exploration for a subject (P4), together with corresponding time series along the dominant synergies (S1, S2, and S4) that contribute during this time segment. The movement pattern along each synergy was obtained by projecting the deviation of the hand postures from the mean posture at successive time points onto the synergy vectors. The nine synergies together account for >90% of the variance throughout this segment (black curve). The first half of this segment is dominated by the S2 (red curve), whereas S1 dominates during the second half (blue curve). S4 briefly dominates for a short time interval near the center of the segment (green curve).
Contribution of various synergies during a 3 s segment of haptic exploration. The plot shows the fractions of variance accounted for by the three dominant synergies, namely S1 (blue), S2 (red), and S4 (green). The fractions captured by the remaining synergies is indicated by dotted lines. Black line indicates the total fraction captured by all the nine synergies. The first three rows of hand postures are conformations reconstructed using the projections along S1 (first row), S2 (second row), and S4 (third row) at five time points T1–T5. The fourth row shows the actual hand postures during the five time points. The black boxes correspond to time points when the contribution of that particular synergy changes dramatically.
The hand models illustrate the role of these three synergies (top three rows) in the actual changes in postures (last row) at five time points (T1 to T5). The contribution of S1 rises dramatically from T3 to T5, which indicates a motion along that synergy. This motion, comprising a flexion at the MCP joints of D2–D4, is reflected in hand models depicting projections along the first synergy (first row, hand models enclosed inside the box) and capture the corresponding flexion in the actual hand postures (fourth row, last three postures). Similarly, the contribution of S2 falls dramatically between T1 and T3, which indicates an extension of D2 and D3 and a simultaneous flexion at D4 and D5. This is reflected in the posture projections along the second synergy (second row, first three postures) and capture the corresponding motion component in the actual hand postures (last row, first three hand models). As the subject's hand changed postures through the five time steps (fourth row), the thumb initially flexed toward the other digits from T1 to T3 and then extended out from T3 to T5. This is captured by the projection along the fourth synergy as illustrated by the hand postures in the third row.
Thus any hand movement can be thought of as a combination of postural changes along contributing synergies. The change in the fraction of variance captured by a particular synergy during a certain time interval is related to the postural changes occurring along that synergy. Thus, the synergies can be thought of as building blocks that contribute in varying proportions to construct any arbitrary hand movement patterns.
Similarity of synergies across subjects
Figure 5 compares the similarity of eigenvectors belonging to the same synergy as well as of eigenvectors across synergies. Across all synergies, the distribution of q1 (dark gray bars), which measures the similarity of all vectors across subjects grouped under a particular synergy, is significantly higher than the distribution of q2 (white bars), which measures the similarity of vectors across subjects belonging to different synergies. The distribution of q1 is also higher than the distribution of q3 (light gray bars), which is a conservative measure of similarity between eigenvectors across subject that belong to different synergies (see Materials and Methods), indicating that the vectors categorized into the same synergy are more similar than what would be expected by chance because of random grouping of eigenvectors across subjects. The differences between the two distributions are lesser for S7 and S9; however, the differences are still significant (p < 0.01).
Comparison of eigenvectors across subjects within the same synergy class and between different synergy classes. For each, the synergy is plotted; error bars denote distributions of four kinds of measures (see Materials and Methods). Black bars are distribution of correlation coefficients q4 between eigenvectors belonging to the same subject and synergy but obtained separately from two sets of data. Dark gray bars are distributions of correlation coefficients q1 across subjects between eigenvectors belonging to the same synergy. Light gray bars are distributions of maximum of pairwise correlations q3 between eigenvectors across subjects and across synergies. White bars are distributions of pairwise correlations q2 between eigenvectors across subjects and across synergies.
Some synergies (S5, S7, S8, and S9) tend to have lower average correlations between similar synergy vectors across subjects. The fourth quantity (q4), which estimates the expected variations in eigenvectors caused by undersampling of conformations in the directions of principal components with low eigenvalues (see Materials and Methods), is represented by black bars in Figure 5. It can be seen that the distribution of q1 and q4 are comparable. Thus, the decrease in correlation between higher-order synergy vectors is similar to the decrease in correlation caused by noisy undersampling of the conformation space along eigenvectors with low eigenvalues.
Variations in precise synergy structure across subjects
Although there are striking similarities in synergy vectors across subjects, there are subtle differences as well. Such differences can be observed, for example, in Figure 3 and supplemental Figure 1 (available at www.jneurosci.org as supplemental material). In the third synergy, the abduction of the thumb is accompanied by a decrease in the MCP angle of D2, causing D2 to move inward, in subject P2 (supplemental Fig. 1, available at www.jneurosci.org as supplemental material), whereas it increases in subject P4 (Fig. 3). Such differences are in general larger in the synergies with lower eigenvalues. This indicates that although there are common underlying patterns of motions, there are variations between subjects. There is also a difference in the rank order of the synergies in each subject, implying that every subject has a unique order of preference for these synergy vectors. Some synergies are absent in some subjects (Table 1), indicating that they evoked too few postures along this synergy. An extreme example of individual differences is the emergence of a completely novel principal component that is unique to one subject. Subject P1 evoked peculiar postures while exploring objects. At times, especially for the smaller objects, the subject tended to keep D5 extended, while at other times D5 was flexed along with the other digits. Because of this large range of motion along D5 only, this subject had a unique vector as the third principal component that captured the motion of D5.
Postures evoked in grasping parametrically varying shapes
In this section we identify the space of postures evoked in visually guided reach-to-grasp task and relate these postures to the synergy vectors described above. Using regression, we computed vectors that represent maximal postural changes evoked by various parameters of the grasped object (see Materials and Methods). The correlations of various synergies with these vectors corresponding to object width (black), curvature (white), and taper angle (gray) are shown in Figure 6. Regression vectors during grasp (Fig. 6a) and approach (Fig. 6b) have similar correlations with the synergies indicating that the hand conforms to the shape of the object to be grasped during the approach (Jeannerod, 1986; Santello and Soechting, 1998). The postural variation induced by changing the width of the grasped objects is largely explained by S1. The second vector (surface curvature) has large correlations with S1 and S5. Because at the two extremes of the S5, digits D3 and D4 are inward or outward with respect to digits D2 and D5, postures at these ends would be useful for grasping convex versus concave objects. The third vector (taper angle) has a high correlation coefficient with S3 indicating that the most salient change in the grasp posture, induced by changing the taper angle, is abduction of the thumb.
Parametrically sampling the 9D space of synergies using simpler paradigms such as reach-to-grasp. Correlation coefficient between the vectors representing the direction of postural change induced by changing the width of target object (black), taper angle of grasping surface (gray) and curvature of the grasping surface (white) with the nine synergy vectors. a, Vectors are obtained from posture data during grasp. b, Vectors are obtained from the hand postures during maximum aperture before grasp.
Postures evoked in precision pinch
Here we evaluate whether the synergies described are used during skilled precision pinch tasks. Figure 7, a (black curve) and the last columns in b, c, and e, shows that nine synergies of subject P8 capture >95% of the variance for precision pinches involving D2, D3, and D5 at each time step. However, they perform poorly at explaining the pinch involving D4, as correlation drops below 60% (Fig. 7a,d). As the hand posture changes from an open hand to a particular pinch, there is a significant decrease in the fraction of variance captured by the first synergy (Fig. 7a, blue line) and an increase in the fractions captured by S2 (red line), S5 (green line), and S6 (yellow line) during D1–D2, D1–D3, and D1–D5 precision pinches. Figure 7, b and e, shows the distribution of fractions of variance accounted by various synergies during the 18 repeats of each type of pinch. S7–S9 play no role during the D1–D2 (Fig. 7b) or D1–D5 (Fig. 7e) pinches. The largest fraction is captured by the S1 for the D1–D2 (Fig. 7b) and D1–D4 (Fig. 7d) pinches, S5 for the D1–D3 pinch (Fig. 7c), and S2 for the D1–D5 pinch (Fig. 7e). The variance is more distributed over the various synergies in the case of the D1–D3 pinch (Fig. 7c), as all the synergies except for the S2 account for some fraction of the variance. Again, across the 18 repeats, the 9 synergies capture >95% of the variance during D1–D2, D1–D3, and D1–D5 pinches, whereas they capture ∼60% of the variance during the D1–D4 precision pinch.
Role of various synergies during a pinch. a, Time profile of fractions of variance accounted for by the first synergy (blue), second synergy (red), fifth synergy (green), sixth synergy (yellow), and all the nine synergies (black) while the subject performed the pinch grip task. b–e, Fraction of variance accounted for by each synergy and all synergies (last column) averaged across the 16 repeats of the D1–D2 pinch (b), D1–D3 pinch (c), D1–D4 pinch (d), and D1–D5 pinch (e).
Figure 8 illustrates that hand posture during precision pinches can be reconstructed using a combination of the synergies described above. When the D1–D2 pinch is constructed from just S1 of the same subject (Fig. 8a, second column), all the digits are aligned together and there is a significant difference between this reconstructed posture and the original posture (Fig. 8a, first column). When S6 is included in the reconstruction, the middle and distal phalanges of digits D2–D4 move outward (third column), so that the four digits are no longer aligned as in the postures in the second column. When S2 is also included in the reconstruction it moves digit D2 further inward toward D1 and digit D5 outward (fourth column). Finally, when all the six contributing synergies (S1–S6) are used for reconstruction, D2 moves further in toward D1 and the other digits extend outward (fifth column) and the reconstructed posture looks similar to the original posture (compare with first column). Note that in the original posture the tips of D1–D2 do not make contact because the markers were on the back of the finger.
Reconstruction of postures during precision pinches. a, Contribution of various synergies in the construction of D1–D2 pinch. The two rows show the front (row 1) and side (row 2) views of the rendered posture. The first column is a posture of the D1–D2 precision pinch of subject P8 averaged across 16 repeats. The second column shows two views of the posture reconstructed from just the first synergy. The third column shows two views of the posture reconstructed from the first and sixth synergies. The fourth column shows two views of the posture reconstructed from the first, sixth and second synergies. The last column shows two views of the posture reconstruction from all six contributing synergies (S1–S6). b–d, Comparison of the actual postures evoked during D1–D3 pinch (b), D1–D4 pinch (c), and D1–D5 pinch (d) with the postures reconstructed from contributing synergies. Top row shows the actual postures, whereas the bottom row shows the reconstructed postures. The left column shows the front view and the right column shows the side view for each posture.
Figure 8b–d compares the postures evoked during D1–D3 (Fig. 8b, top row), D1–D4 (c, top row), and D1–D5 (d, top row) precision pinches with the corresponding postures constructed from contributing synergies (bottom rows). The D1–D3 pinch was constructed from all synergies except S2, D1–D4 was constructed from five synergies (S1, S2, S4, D5, and S9), whereas the D1–D5 pinch was constructed from six synergies (S1–S6). The salient differences between the constructed pinches and the original pinches are at the DIP joints. Because none of the nine synergies described in this study exclusively involve the DIP joints, differences between DIP joint angles cannot be accounted for by these synergies. The reconstructed postures for the D1–D3 pinch (Fig. 8b, bottom row) and D1–D5 pinch (Fig. 8d, bottom row) are similar to the corresponding actual postures (Fig. 8b,d, top rows). However, the reconstruction fails in the case of the D1–D4 pinch (Fig. 8c). Although all of the contributing synergies are included in the reconstruction, it extends D2 outward, but the D3 and D5 remain curled inward along with digit D4 and the reconstructed hand posture (Fig. 8c, bottom row) is a poor replication of the original posture (Fig. 8c, top row).
Data occlusion
Because we used an optical system, occlusion caused loss of data points that can introduce bias. The results described above are based on times when all the marker positions were captured by the OPTOTRAK which was ∼38% of the total exploration time used by the subjects (Table 2). Synergies obtained from interpolated data have very high correlations with the corresponding synergies obtained from the raw unoccluded data set (Fig. 9). Thus, occlusion did not introduce any systematic bias in the data set.
Sizes of data sets used
Similarity of synergies obtained from interpolated data. Average correlation coefficients between synergies obtained from raw data with the corresponding synergies obtained from interpolated data are shown.
Discussion
We estimated synergistic patterns of hand motion using PCA on a set of postures evoked in an unconstrained haptic exploration task. We identified nine eigenvectors (i.e., synergies) that were similar across subjects and across manipulations of different objects and accounted for >90% of the variance. These synergies are also used during a simpler reach-to-grasp movement and skilled precision pinching. We find that these synergies account well for all tasks except for the D1–D4 pinch task.
True estimates of underlying task-independent synergies can only be obtained by including the set of all possible hand postures. We have tried to get as close to this set as possible via an unconstrained task. The estimated synergies shown here may be close to the true underlying task-independent synergies, given their similarity across subjects and across manipulations of different objects, and because they account well for skilled tasks involving individuated finger movements, such as precision pinches. At the same time, they failed to account for one movement, the D1–D4 pinch, implying that this set of synergies deviates from the underlying task-independent set, by at least one vector.
Data occlusion is unlikely to have biased our synergies, because we obtained similar vectors when all the markers except the markers on the distal tips were interpolated. Another possible source of bias could be under-representation of precision grips in the data, because only 4/50 objects were small. However, subjects typically used precision grips to explore small features of large objects such as edges, handles, etc. The fact that the synergies account well for the D1–D2 and D1–D3 precision pinches and fail at D1–D4 pinch, which is not typically used to explore small objects or features, further illustrates that precision grips relevant to haptic manipulations were captured.
Anatomical correlates
These synergies have direct correlates with the underlying muscle anatomy. S1 likely involves extrinsic finger extensors (extensor digitorum communis) and flexors (flexor digitorum superficialis and profundus) that extend and flex D2–D5, respectively. These muscles also dominate the first few muscle synergies in a finger-spelling task (Weiss and Flanders, 2004), the hand opening aspect of a reach-to-grasp task (Holdefer and Miller, 2002) and power grasp of a cylinder (Brochier et al., 2004). Other synergies also have a close relation to specific muscle functions. For example, S2 and S5 likely rely on extensor indicis proprius (extends D2) and extensor/flexor digiti minimi (extends/flexes D5) via a different pattern of coupling for each synergy. S3 (thumb spreads sideways) may use abductor pollicis longus, brevis, and adductor pollicis muscles to move the thumb.
Variations in the skeletal structures, musculature and connective tissue connections between subjects (von Schroeder et al., 1990) could underlie the subtle differences observed in the synergies across subjects. Alternately, the differences could also be the result of how the motor system has developed in each subject through learning and experience.
Previously defined hand synergies
A number of synergies reported previously are qualitatively similar to some of our synergies. For example, the dominant PC observed across all studies is a simultaneous flexion-extension across all digits together with adduction-abduction between digits. This is similar to S1, albeit the abduction–adduction observed is lower than previous reports. This is because a separate synergy (S9) accounts for the abduction–adduction between digits. The second synergy reported by Santello and Soechting (2000) involved forces generated in phase at digits D2 and D5 and opposite to the forces generated at digits D3 and D4. This is qualitatively similar to S5 (D2, D5 flex; D3, D4 extend). Furthermore, the second synergy reported by Santello et al. (1998), which involved extension at MCP and flexion at the PIP joints (S8) and flexion and abduction of the thumb (S4). The major PCs observed in the finger-spelling task do not directly correlate with our synergies. However, linear combination of these PCs obtained to match the first two muscles synergies during the same task (Weiss and Flanders, 2004) are very similar to our first two synergies. Thus, the synergies that we describe in this study include the hand conformation space described in previous studies.
Synergies and exploratory procedures
Because synergies combine in various ways to construct any continuous segment of hand motion, they can be thought of as building blocks of exploratory procedures (EPs) (Lederman and Klatzky, 1987; Roland and Mortensen, 1987) and some speculations can be drawn on the roles of synergies while executing a particular EP. S1 (i.e., hand opening) is likely to dominate EP's such as static encompassing or enclosure, which involve power grasps or molding of object. Similarly S7 is likely to dominate lateral motion or roll. Although, we did not record wrist or arm position with respect to the body, it is likely that S7 may be correlated with a lateral motion of the wrist.
Theoretical implications
There are at least two opinions regarding the emergence and role of synergies in motor control: (1) they provide a simplified substrate by reducing the dimensionality providing a more tractable control problem that still allows for great flexibility of hand motions. (2) Synergies could be a manifestation of an optimal control strategy that simply projects the noise along dimensions redundant to the current task (Todorov and Jordan, 2002). This view predicts that a given task dichotomizes the available 20D space into a subspace relevant to the task and a redundant subspace or uncontrolled manifold (Scholz and Schoner, 1999) that accumulates noise causing the observed synergies to be task dependent (Todorov and Ghahramani, 2004). Our study suggests that the two views need not be exclusive. The similarity of our synergies across subjects and across manipulations of different objects suggests that there might be an underlying hard-coupling of joints into a task-independent subspace. This is also consistent with anatomical and physiological studies (see Introduction) demonstrating mechanical and neural couplings across multiple joints and muscles. However, specific tasks occupy separate subspaces within the 9D space defined by our synergies. This is evident because (1) postures during the four pinches have different patterns of projections on the nine synergies, (2) variations induced by changing various parameters of the object in a reach-to-grasp task have different patterns of projections on the nine synergies, and (3) the smaller subspaces observed in previous studies are similar to subspaces formed by a subset of the nine vectors. Thus, a task specific optimal control strategy could still dichotomize the 9D space into a task relevant subspace and a redundant subspace used to accumulate noise.
Implications for studies of proprioception
A major challenge in studying the cortical representation of proprioception is the difficulty in manipulating hand postures in a parametric way. In the reach-to-grasp task, we obtained vectors that denote maximal postural changes induced during approach and while grasping objects with parametrically varying shapes. Changes in each parameter of the object introduce postural changes along a unique direction in the 9D space defined by the nine synergies. Thus, by systematically altering the target object along the three chosen parameters, hand postures during approach and grasp can be parametrically changed within a 3D space embedded in the 9D space. Similar principles can be used to identify, by modulating different parameters, other vectors that sample the remainder of the 9D space.
Functional implications
Because of 20 degrees of freedom, a transformation between two unique postures is precise only if specified in a 20D space. However, because a smaller subspace accounts for most of the variance, the 20D space can be split into two subspaces. First, defined by few task-independent synergies that accounts for the gross manipulations between two unique postures, and the second, that captures the rest of the 20D space and accounts for the minor adjustments in various joint angles. This simplifies the problem of understanding hand manipulation and control (Santello et al., 1998), because now hand manipulation can be treated as a two-step process, each involving a unique set of constraints and control conditions. On the sensory side, it reduces the dimensionality of the proprioceptive space during object discrimination/identification. This implies that neurons in somatosensory areas implicated in object manipulation and discrimination such as second somatosensory cortex (SII) (Murray and Mishkin, 1984; Binkofski et al., 1999; Fitzgerald et al., 2004; Hinkley et al., 2006) may show invariant cutaneous responses (Fitzgerald et al., 2006; Thakur et al., 2006) that are tuned within a space defined by these synergies. Such a reduction can also provide optimal compression of sensory information before it is relayed to the motor cortical areas along the dense connections between SII and motor areas (Krubitzer and Kaas, 1990).
Neurophysiological studies in ventral premotor cortex (PMv) have revealed a hand movements region, where neurons discharge during specific goal-directed movements such as grasping, holding and tearing (Rizzolatti et al., 1981, 1988; Kurata and Tanji, 1986; Hepp-Reymond et al., 1994). These neurons could potentially be coding for synergies. Given that PMv activation precedes primary motor cortex (M1) activation (Umilta et al., 2007), information regarding the activation patterns of various synergies required in a particular task carried by these neurons could ultimately be translated by M1 to recruit appropriate motor outputs to the hand.
Appendix
Correcting for arm translation and wrist motion
The global position coordinates of all markers in 3D space, were transformed into a coordinate system attached to the dorsum of the hand. The origin of this coordinate system was defined at the centroid of the triangle formed by the three markers (21, 22, and 23) on the dorsum. x-axis of the coordinate system was defined along the apex of the triangle (toward the MCP joints), whereas the y-axis was defined parallel to the base of the triangle. The z-axis was defined orthogonal to the plane of the palm. The transformation consisted of two steps. First, the centroid of the triangle at every time step, which gives the current coordinates of the origin in absolute space, was subtracted from the raw position coordinates of all markers at that time step, to correct for translations of the hand through space. Specifically, if Pi = [pix, piy, piz]T, were the absolute coordinates of the ith marker at a given time step, the coordinates of the origin at this time point are as follows:
The translation-corrected coordinates of the markers are P′i = Pi − O. Next the marker position data were corrected for any rotation of the hand in space. At every time step, we determined unit vectors along the three coordinate axes and projected the translation-corrected coordinates onto these vectors to obtain the translation and rotation corrected coordinates. Because the x-axis was defined to be along marker 21, unit vector along the x-axis can be obtained by normalizing the translation corrected coordinates of marker 21:
By definition, marker 23 lies in the x–y plane. The translation corrected coordinates of marker 23 (P′23) was projected onto the unit vector along the x-axis. This projection was subtracted from P′23 to obtain the component of P′23 along the y-axis. This component vector was divided by its norm to obtain a unit vector along the y-axis. Specifically,
The unit vector along z-axis was computed using a similar principle. The translation corrected coordinates of marker 4 (distal tip of thumb) was projected onto the unit vectors along x and y axes. When these projections are subtracted from P′4, it yields the component of P′4 that is orthogonal to the x–y plane and along the z-axis. Normalizing this component vector gives the unit vector along z-axis:
Given unit vectors along x, y and z axes, the translation and rotation corrected coordinates were computed as follows:
Thus, P″ represents the coordinates of various joints of the hand in reference to the dorsum of the hand, with the palm fixed in space. All of the analyses reported in this study were performed on P″.
Footnotes
-
This work was supported by National Institutes of Health Grants NS34086 and HO040289. We thank B. Nash and B. Quinlan for help in machining the objects.
- Correspondence should be addressed to Dr. Steven S. Hsiao, Zanvyl Krieger Mind/Brain Institute, Johns Hopkins University, 338 Krieger Hall, 3400 North Charles Street, Baltimore, MD 21218. Steven.Hsiao{at}jhu.edu.
