Besides Sven's answer, below is an answer from Jim to your question.
When A and B are indefinite, then the theory describing the possible
solutions of the eigenvalue problem gets complicated. In particular,
if A and B (nearly) share a null-space, then you have what is
called a "singular pencil", which means that the eigenvalues and
eigenvectors are not uniquely defined (or extremely ill-conditioned).
Is this the case with your A and B? One way to get error bounds
from DSYGV that would indicate there is such a problem is
described in the LAPACK manual at
http://www.netlib.org/lapack/lug/node98.html
If you do think you have this problem, and you still want an
eigenvalue decomposition of the more general kind (known
as the Kronecker Canonical Form), then there are two
possibilities.
First, you can try deflating the common nullspace,
by replacing A and B with Q'*A*Q and Q'*B*Q where
Q is the orthogonal eigenvector matrix of A and so
Lambda_A = Q*A*Q' is the diagonal matrix of eigenvalues
of A. Choose the order of eigenvectors in Q so that the k zero
(or tiny) eigenvalues of A appear at the bottom right of Lambda_A.
If the last k rows and columns of Q'*B*Q are also tiny enough,
you can interpret the pair (A,B) as having a k-dimensional
common null-space (with undefined eigenvalues), and the
remaining eigenvalues can be found from the matrices in
the leading n-k rows and columns.
Second, a more sophisticated code that computes the general
case of the Kronecker Canonical Form is called GUPTRI
(for "Generalized UPper TRIangular Form"). We have software
available for GUPTRI, though not in LAPACK; see
http://www.cs.umu.se/~guptri/
Finally, we are developing an "iterative refinement" technique for
problems such as yours, and would be interested in trying out
your matrix, if you are willing to share it, and it is not too big.
Please contact Jim Demmel (
demmel@cs.berkeley.edu) about
this if you are interested.
Jim