Oracle Weblogic Server: Understanding and Investigating SSL Issues

What is SSL and how does it work?

SSL is short for Secure Sockets Layer. The SSL protocol was developed by Netscape and is supported by all popular web browsers such as Internet Explorer, Mozilla Firefox, Google Chrome, and Opera. For SSL to work, a SSL certificate issued by a Certificate Authority must be installed on the web server. SSL can then be used to encrypt the data transmitted (secure SSL transactions) between a browser and web server (and vice versa). Browsers indicate a SSL-secured session by changing the HTTP to HTTPS and displaying a small padlock. Web site visitors can click on the padlock to view the SSL certificate.

What is PKC and how does it work?

Public Key Cryptography (PKC) is a method for securely exchanging messages, based on assigning two complimentary keys (one public, one private) to the individuals involved in a transaction. Public Key Cryptography is based on the science of encryption, the mathematical scrambling and unscrambling of messages.
Public key cryptography addresses several of the shortcomings of symmetric key cryptography. In public key cryptography, an individual or organization has two complimentary keys, one called a public key, and one called a private key. Any information encrypted using the private key can only be decrypted using the public key. Conversely, any information encrypted using the public key can only be decrypted using the private key. For example:

1. Bob has two complimentary keys
2. What one key encrypts on the other key can decrypt
3. Bob keeps one key private (Private Key)
4. Bob makes one key available to the public (Public Key)
5. If Alice needs to send Bob a message
6. Bob sends Alice a copy of his public key
7. Alice encrypts a message with Bob's public key
8. Bob decrypts the message with his private key

In Public Key Cryptography, if Alice wants to send a secret message to Bob, she must obtain a copy of his public key. Before doing so, however, she needs to make sure that the public key really belongs to Bob.
Certificates (also called Digital IDs) address this problem. A certificate is an electronic document that binds a public key to a particular individual or organization. Certificates are issued by a trusted third party, called a Certification Authority (CA). Before issuing a certificate, a good CA will go though a series of authentication procedures to make sure that Bob is who he claims to be, and that the public key in the certificate really belongs to Bob.

What is a Certificate?

Technically, SSL Certificates, also known as digital certificates, bind an identity to a pair of electronic keys that can be used to encrypt and sign digital information. An SSL Certificate makes it possible to verify someone's claim that they have the right to use a given key, helping to prevent people from using funny keys to impersonate other users. Used in conjunction with encryption, SSL Certificates provide a complete security solution, assuring the identity of one or all parties involved in a transaction.
An SSL Certificate is issued by a trusted third party called a Certification Authority (CA). A CA acts somewhat like a passport office. CAs must take steps to establish the identity of the people or organizations to whom they issue IDs. Once the CA establishes an organization's identity, it issues a certificate that contains the organization's public key and signs it with the CA's private Key .
By using a SSL Certificate, you are enabling your site to conduct authenticated, encrypted on-line commerce. Users visiting your site will be able to submit credit card numbers or other personal information to your site, with assurance that they are really doing business with you (and not an impostor) and that the information that they are sending to you can not be intercepted or decrypted by anyone other than the intended recipient. Your SSL Certificate will contain the following information:

• Your organization's common name (e.g., www.oracle.com)
• Additional identifying information (e.g., IP and physical address)
• Your public key
• Expiration date of the public key
• Name of the CA that issued the ID (i.e., VeriSign)
• A unique serial number
• VeriSign's digital signature

What is a Certificate Authority?

A Certificate Authority is a trusted third party similar to a passport office, or a Certified Public Accountant. Certificate Authorities are responsible for issuing, revoking, renewing, and providing directories of digital certificates. Certificate Authorities must follow rigorous procedures for authenticating the individuals and organizations to whom they issue certificates. All digital certificates are "signed" with the Certificate Authority's private key to ensure authenticity. The Certificate Authority's Public Key is widely distributed.

What is a SSL handshake?

The SSL handshake is the term given to the process of the browser and web server setting up a SSL session. The SSL handshake involves the browser receiving the SSL certificate and then sending "challenge" data to the web server in order to cryptographically prove whether the web server holds the SSL key associated with the SSL certificate. If the cryptographic challenge is successful then the SSL handshake has completed and the web server will hold a SSL session with the web browser. During a SSL session the data transmitted between the web server and web browser will be encrypted.
Before you can understand how to troubleshoot a handshake it is important to understand it. When you debug the handshake using debug options, each handshake phase needs to be analyzed so you can determine the problem. To understand what is going on in a handshake, refer to http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame2.doc_5.1/ss7aumst18.htm


The client sends a ClientHello message to which the server must respond with a ServerHello message, or else a fatal error will occur and the connection will fail. The client hello and server hello are used to establish security enhancement capabilities between client and server. The client hello and server hello establish the following attributes: protocol version, session ID, cipher suite, and compression method.
Following the hello messages, the server will send its Certificate, if it is to be authenticated.
If the server is authenticated, it may request for client authentication a certificate (CertificateRequest) from the client, if that is appropriate to the cipher suite selected (Optional 2 way).
Now the server will send the ServerHelloDone message, indicating that the hello-message phase of the handshake is complete. The server will then wait for a client response.
If the server has sent a CertificateRequest message, the client must send either the Certificate message or a no certificate alert (optional 2 way). The ClientKeyExchange message is now sent, and the content of that message will depend on the public key algorithm selected between the ClientHello and the ServerHello. If the client has sent a certificate with signing ability, a digitally-signed CertificateVerify message is sent to explicitly verify the certificate.
At this point, a ChangeCipherSpec message is sent by the client, and the client copies the pending Cipher Spec into the current Cipher Spec. The client then immediately sends the Finished message under the new algorithms, keys, and secrets. In response, the server will send its own ChangeCipherSpec message, transfer the pending to the current Cipher Spec, and send its Finished message under the new Cipher Spec. At this point, the handshake is complete and the client and server may begin to exchange application layer data. (See flow chart below.)

Certificate Formats

The primary certificate types are:

• PEM
• DER
• PKCS#12

PEM

Can contain all of private keys (RSA and DSA), public keys (RSA and DSA) and (x509) certificates. It stores data Base64 encoded DER format, surrounded by ASCII headers, so is suitable for text mode transfers between systems.

-----BEGIN CERTIFICATE-----
MIICJjCCAdCgAwIBAgIBITANBgkqhkiG9w0BAQQFADCBqTELMAkGA1UEBhMCVVMx
EzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28xFTAT
BgNVBAoTDEJFQSBXZWJMb2dpYzERMA8GA1UECxMIU2VjdXJpdHkxIzAhBgNVBAMT
GkRlbW8gQ2VydGlmaWNhdGUgQXV0aG9yaXR5MR4wHAYJKoZIhvcNAQkBFg9zdXBw
b3J0QGJlYS5jb20wHhcNMDAwNTMwMjEzODAxWhcNMDQwNTEzMjEzODAxWjCBjDEL
MAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBG
cmFuY2lzY28xFTATBgNVBAoTDEJFQSBXZWJMb2dpYzEZMBcGA1UEAxMQd2VibG9n
aWMuYmVhLmNvbTEeMBwGCSqGSIb3DQEJARYPc3VwcG9ydEBiZWEuY29tMFwwDQYJ
KoZIhvcNAQEBBQADSwAwSAJBALdsXEHqKHgs6zj0hU5sXMAUHzoT8kgWXmNkKHXH
79qbPh6EfdlriW9G/AbRF/pKrCQu7hhllAxREbqTuSlf2EMCAwEAATANBgkqhkiG
9w0BAQQFAANBACgmqflL5m5LNeJGpWx9aIoABCiuDcpw1fFyegsqGX7CBhffcruS
1p8h5vkHVbMu1frD1UgGnPlOO/K7Ig/KrsU=
-----END CERTIFICATE-----

DER

Distinguished Encoding Rules (DER) can contain all of private keys, public keys and certificates. It is the default format for most browsers, and is stored according to the ASN1 DER format. It is headerless -- PEM is text header wrapped DER.

PKCS#12

Public Key Cryptography Standards #12 (PKCS#12) can contain all private keys, public keys, and certificates. It stores in a binary format, and is also known as PFX files.

Generating Demo Certificates

To setup certificate configurations, ensure that you have a set of the following certificates:

• PRIVATE key
• PUBLIC key
• CERTIFICATION AUTHORITY (CA)

Step 1 - Create demo private keys

There are three ways to create a private key: You can use:

• keytool
• Certificate Servlet
• openssl

keytool (from your jdk)

To generate the private key:

Usage; keytool -genkey      [-v] [-alias <alias>] [-keyalg <keyalg>]
             [-keysize <taille_cle>] [-sigalg <sigalg>]
             [-dname <nomd>] [-validity <joursval>]
             [-keypass <mot_passe_cle>] [-keystore <keystore>]
             [-storepass <mot_passe_store>] [-storetype <type_store>]
             [-provider <classe_fournisseur>] ...

keytool -genkey -keyalg RSA -alias mykey -keystore mykeystore.jks
Enter keystore password:  password
What is your first and last name?
  [Unknown]:  Colette Gotfried
What is the name of your organizational unit?
  [Unknown]:  Customer Service
What is the name of your organization?
  [Unknown]:  BEA
What is the name of your City or Locality?
  [Unknown]:  Liberty Corner
What is the name of your State or Province?
  [Unknown]:  New Jersey
What is the two-letter country code for this unit?
  [Unknown]:  NJ
Is CN=Colette, OU=Customer Service, O=BEA, L=Liberty Corner, ST=NJ, C=NJ correct?
  [no]:  yes

Enter key password for <mykey>
        (RETURN if same as keystore password):

As a result you obtain a file: mykeystore.jks, containing a private key, and a self-signed public key. For more details on the use of keytool, refer to Keytool Manual.

Certificate Servlet from WebLogic (deprecated in 7.0)

The application is the file certificate.war when deployed. It is called as follows:

http://hostname:port/certificate

Step 2 - Sign the public key by a trusted CA

The next step is to have the public key signed by a know CA. This is done by retrieving the CSR (Cert Signature Request) and sending it to one of the Certificate Authorities.

keytool -certreq -keystore demokeystore
Enter keystore password :  password

-----BEGIN NEW CERTIFICATE REQUEST-----
MIIBrTCCARYCAQAwbTELMAkGA1UEBhMCRlIxEDAOBgNVBAgTB0Zsb3JpZGUxDjAMBgNVBAcTBVBh8
cmlzMQwwCgYDVQQKEwNCRUExEzARBgNVBAsTCkJFQVN5c3RlbXMxGTAXBgNVBAMTEFNlYmFzdGll
bk1hbGJvaXMwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMMfPz2S+6tXayJ2qmQ0yF6MFXxQ
XcmL2n9QqnE1k+hmIRfvD6dAYSOc8wZorbjZppEqHMK4ZVRKQuqvrRdyasR7DJO6ezQSGG0kyB+J
qLDLMkwT8ig0Z6eiIdQkvrbnCTi9hPWI76rdgwxF8UIeJbh7k0NctG1CISZdLGwRxeOJAgMBAAGg
ADANBgkqhkiG9w0BAQQFAAOBgQB5gK/tsCt3S6FTA+kfYpXMyplmbI7sDd4I0JOlciqe7ssw+va1
wA5UEVt8HqRXQMczEcZRwrf0QbxjQXJUMz4e6i4OuQvp/XMK+EOWHTcLwYgu708A81GyisXUd3/D
iecFRcH4TBCIHHbdqtFtVH0BXKsLUuiAxabRyI0507XfXg==
-----END NEW CERTIFICATE REQUEST-----

You will need to copy and paste all this (including -----BEGIN NEW CERTIFICATE REQUEST----- and -----END NEW CERTIFICATE REQUEST-----) to the Certification Authority.
The following example uses the Verisign CA:

• Go to the Verisign site
• Click on SSL Trial ID
• Fill in all your details and copy and paste your generated CSR
• Verisign will then send you an email with the PEM content of the public key. Save it under public.pem
• A link to the Verisign CA root certificate is given -- save it under a new file such as CA.pem

Store the CA root in your keystore. If you have an intermediate CA you will have to put it in there after the CA root etc.:

keytool -import -alias verisignCA -file CA.pem -keystore mykeystore -trustcacerts
keytool -import -alias verisignIntermediateCA -file IntermediateCA.pem -keystore mykeystore -trustcacerts

Import the public key into your keystore. It will go on the same alias as the private key:

keytool -import -alias mykey -file public.pem -keystore mykeystore -trustcacerts

Converting Certificate Formats

OpenSSL is an open software tool that provides similar functionality as keytool. Download OpenSSL from http://www.openssl.org/contrib/ssl.ca-0.1.tar.gz

To PKCS#12 (Mozilla, IE etc.) from PEM

openssl pkcs12 -export -in pem-certificate-and-key-file -out pkcs-12-certificate-and-key-file

openssl pkcs12 -export -in pem-certificate-file -inkey pem-key-file -out pkcs-12-certificate-and-key-file

From PKCS#12 to PEM

http://support.globalsign.net/en/serversign/pkcs12.cfm

• Extract the private key from the pkcs12 file
• Open a command line from the openssl/bin directory
• Run the following command to extract the private key from the pkcs12 file:
  
  openssl pkcs12 -in keyexport.pfx -nocerts -nodes -out keyexport.prv
• Enter the password used to create the pkcs12 (.pfx) file
• Extract the public key from the pkcs12 file
• Run the following command to extract the public key from the pkcs12 file:
  
  openssl pkcs12 -in keyexport.pfx -nokeys -out keyexport.pub
• Enter the password used to create the pkcs12 (.pfx) file
• Enter PEM pass phrase and a confirmation
• From PEM/DER to DER/PEM - RSA Keys
  
  openssl rsa -inform PEM|DER -outform DER|PEM -in pem-file|der-file -out der-file|pem-file
  
  
  A PEM key looks like:
  
  -----BEGIN CERTIFICATE-----
  ...ascii stuff....
  -----END CERTIFICATE-----
  
  
  NOTE: A DER key is not readable in ascii.

Look into a Certificate

Private keys:
     openssl -rsa -inform DER -in demokey.der -text

Public keys, Certifications Authorities:
     openssl -x509 -inform PEM -in democert.pem -text

Keystore:
keytool -list -keystore mykeystore.jks -v
Enter keystore password: password
Keystore type: jks
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: mykey
Creation date: Mar 30, 2013
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=Colette, OU=Customer Service, O=BEA, L=Liberty Corner, ST=NJ, C=NJ
Issuer: CN=Colette, OU=Customer Service, O=BEA, L=Liberty Corner, ST=NJ, C=NJ
Serial number: 4069afc4
Valid from: Tue Mar 30 12:35:00 EST 2013 until: Mon Jun 28 13:35:00 EDT 2013
Certificate fingerprints:
         MD5: 1D:D7:8F:82:19:6D:A6:BF:C9:B1:DA:E2:EB:B3:C1:93
         SHA1:0A:83:65:79:B2:59:35:33:24:56:AF:84:E6:A3:D0:E9:12:69:C8:67

*******************************************
*******************************************

Other commands:

Extract the private key from the pkcs12 file:

openssl pkcs12 -in keyexport.pfx -nocerts -nodes -out keyexport.prv

Extract the public key from the pkcs12 file:

openssl pkcs12 -in keyexport.pfx -nokeys -out keyexport.pub

For more information, refer to http://www.techonline.com/community/ed_resource/feature_article/14364

Configure WLS to use your keystore (one way SSL only)

From the Admin console, go to your server page, and in the Keystore&SSL tab choose:

Custom Identity and Custom Trust
Custom Identity
Custom Identity Key Store File Name:  mykeystore
Custom Identity Key Store Type: jks
Custom Identity Key Store Pass Phrase: password
Confirm Custom Identity Key Store Pass Phrase: password

Custom Trust
Custom Trust Key Store File Name: mykeystore
Custom Trust Key Store Type: jks
Custom Trust Key Store Pass Phrase: password
Confirm Custom Trust Key Store Pass Phrase: password

Private Key Alias: mykey
Passphrase: password
Confirm Passphrase: password

Ensure that SSL Listen Port Enabled is selected, then restart your server.

<19 dec. 2013 10 h 39 CET> <Debug> <TLS> <000000> <SSLManager.getServerCertificate()>
<19 dec. 2013 10 h 39 CET> <Notice> <Security> <BEA-090171> <Loading the identity certificate stored under the alias mykey from the jks keystore file D:\_Wk\supportpattern\mydomain\demokeystore.>
<19 dec. 2013 10 h 39 CET> <Notice> <WebLogicServer> <BEA-000298> <Certificate expires in 14 days: [
[
  Version: V3
  Subject: CN=CertServer, OU=BEASystems, O=BEA, L=Paris, ST=Florida, C=FR
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  com.sun.net.ssl.internal.ssl.JSA_RSAPublicKey@1bf
  Validity: [From: Fri Dec 19 01:00:00 CET 2013,
               To: Sat Jan 03 00:59:59 CET 2014]
  Issuer: OU=For VeriSign authorized testing only. No assurances (C)VS1997, OU=www.verisign.com/repository/TestCPS Incorp. By Ref. Liab. LTD., O="VeriSign, Inc"

  SerialNumber: [    0ed7bf9a 778fd148 175bac0b e1d3627d]

Certificate Extensions: 5
[1]: ObjectId: 2.5.29.31 Criticality=false
Extension unknown: DER encoded OCTET string =
0000: 04 3B 30 39 30 37 A0 35   A0 33 86 31 68 74 74 70  .;0907.5.3.1http
0010: 3A 2F 2F 63 72 6C 2E 76   65 72 69 73 69 67 6E 2E  ://crl.verisign.
0020: 63 6F 6D 2F 53 65 63 75   72 65 53 65 72 76 65 72  com/SecureServer
0030: 54 65 73 74 69 6E 67 43   41 2E 63 72 6C           TestingCA.crl

[2]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
[1.3.6.1.5.5.7.3.1, 1.3.6.1.5.5.7.3.2]]

[3]: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.16.840.1.113733.1.7.21]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier:
0000: 16 2A 68 74 74 70 3A 2F   2F 77 77 77 2E 76 65 72  .*http://www.ver
0010: 69 73 69 67 6E 2E 63 6F   6D 2F 72 65 70 6F 73 69  isign.com/reposi
0020: 74 6F 72 79 2F 54 65 73   74 43 50 53              tory/TestCPS

]]  ]
]

[4]: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

[5]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 08 3A F5 EC EE 10 AD 9C   3C D7 94 5A 84 9C 34 F2  .:......<..Z..4.
0010: 61 70 30 45 AF 99 03 79   AF 47 D9 A0 62 20 A6 D3  ap0E...y.G..b ..
0020: C1 21 98 59 A3 3D 6D 8F   E9 58 71 CE 87 FE AB 8A  .!.Y.=m..Xq.....
0030: 99 D8 F5 71 DE 44 55 2E   BB EB 86 15 C0 31 BF 25  ...q.DU......1.%

]>
<19 dec. 2013 10 h 39 CET> <Info> <WebLogicServer> <BEA-000307> <Exportable key maximum lifespan set to 500 uses.>
<19 dec. 2013 10 h 39 CET> <Info> <WebLogicServer> <BEA-000308> <Using full strength (domestic) SSL.>
<19 dec. 2013 10 h 39 CET> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file D:\_Wk\supportpattern\mydomain\demokeystore.>
<19 dec. 2013 10 h 39 CET> <Debug> <TLS> <000000> <Trusted CA: [
[
  Version: V1
  Subject: OU=For VeriSign authorized testing only. No assurances (C)VS1997, OU=www.verisign.com/repository/TestCPS Incorp. By Ref. Liab. LTD., O="VeriSign, Inc"
  Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4

  Key:  com.sun.net.ssl.internal.ssl.JSA_RSAPublicKey@166
  Validity: [From: Sun Jun 07 02:00:00 CEST 2008,
               To: Wed Jun 07 01:59:59 CEST 2006]
  Issuer: OU=For VeriSign authorized testing only. No assurances (C)VS1997, OU=www.verisign.com/repository/TestCPS Incorp. By Ref. Liab. LTD., O="VeriSign, Inc"

  SerialNumber: [    52a9f424 da674c9d af4f5378 52abef6e]

]
  Algorithm: [MD5withRSA]
  Signature:
0000: A5 A7 47 F2 8F 37 10 A0   96 94 CF E6 7C DB A3 E4  ..G..7..........
0010: 02 22 49 AC 08 F8 D3 08   C9 EF 9B B2 9C C0 32 60  ."I...........2`
0020: B9 A1 30 92 88 B5 80 14   98 F5 B8 89 A7 DA 0A F9  ..0.............
0030: CB F5 62 7D CA B9 53 3E   62 9B 5C 59 72 DF C7 12  ..b...S>b.\Yr...

]>
<19 dec. 2013 10 h 39 CET> <Debug> <TLS> <000000> <SSLManager: loaded 1 trusted CAs from D:\_Wk\supportpattern\mydomain\demokeystore>
<19 dec. 2013 10 h 39 CET> <Info> <WebLogicServer> <BEA-000307> <Exportable key maximum lifespan set to 500 uses.>
<19 dec. 2013 10 h 39 CET> <Info> <WebLogicServer> <BEA-000300> <Certificate contents: 2 certificate(s):
  fingerprint = 68dd50d604d078d8da79c2b93a6d9886, not before = Fri Dec 19 01:00:00 CET 2013, not after = Sat Jan 03 00:59:59 CET 2023, holder = C=FR SP=Florida L=Paris O=BEA OU=BEASystems CN=CertServer , issuer = O=VeriSign, Inc OU=For Veri Sign authorized testing only. No assurances (C)VS1997 , key =  modulus length=12 9 exponent length=3
  fingerprint = 40065311fdb33e880a6f7dd14e229187, not before = Sun Jun 07 02:00:
00 CEST 2013, not after = Wed Jun 07 01:59:59 CEST 2023, holder = O=VeriSign, Inc OU=For VeriSign authorized testing only. No assurances (C)VS1997 , issuer = O=VeriSign, Inc OU=For VeriSign authorized testing only. No assurances (C)VS1997 , key =  modulus length=65 exponent length=3>

<19 dec. 2013 10 h 39 CET> <Notice> <WebLogicServer> <BEA-000355> <Thread "SSLListenThread.Default" listening on port 7002, ip address *.*>

You are done. WebLogic is now configured successfully to do one-way SSL (no client authentication).

Problem Troubleshooting

A problem can originate in the SSL configuration of one of the parties, in the certificates used for SSL communication, or in the SSL software being used. The first step is to identify the exact SSL failure experienced by the parties.

1. Know the failure: Enable the SSL Debug Flags to track SSL issues

To diagnose an SSL issue, add the following to the java command line:

-Dweblogic.security.SSL.verbose=true
-Dssl.debug=true
-Dweblogic.StdoutDebugEnabled=true

The flag -Dweblogic.security.SSL.debugEaten=true can also be added to the server. It is recommended that this be added at the very last minute as it may be too wordy to be helpful.
Debug flags can also be enabled in the Admin Console: see http://docs.oracle.com/cd/E23943_01/apirefs.1111/e13952/taskhelp/task_runtimes/DefineDebugSettings.html for instructions. For our purposes, enable the weblogic.security.ssl tree of debug flags.
This should be done for the server and the client, if both are using the WebLogic SSL package, otherwise, only for the WebLogic party of the handshake.

2. What does a correct handshake look like?

NOTE: For more information refer to What is a SSL Handshake?

First SSL Handshake

If you grep an SSL log, a good handshake should show the following:

Server Side

type mylogserver | grep "HANDSHAKEMESSAGE"

<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientHelloV2>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientKeyExchange>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientKeyExchange RSA>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Finished>

type mylogserver | grep "CHANGE_CIPHER_SPEC"

<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <write CHANGE_CIPHER_SPEC offset = 0 length = 1>

Client Side

type mylogclient | grep "HANDSHAKEMESSAGE"

<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ServerHello>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Certificate>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ServerHelloDone>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Finished>

SSL resuming a session

This is what occurs when a handshake resumes a session on a second request:

Server Side

type mylogserver  | grep "HANDSHAKEMESSAGE"

<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientHelloV2>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientKeyExchange>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientKeyExchange RSA>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Finished>
<REQUEST ONE PROCESSED>
<NOW PROCESSING REQUEST TWO>
<18 dec. 2013 13 h 27 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientHello>
<18 dec. 2013 13 h 27 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Finished>

type mylogserver  | grep "CHANGE_CIPHER_SPEC"

<18 dec. 2013 13 h 27 CET> <Debug> <TLS> <000000> <write CHANGE_CIPHER_SPEC offset = 0 length = 1>
<18 dec. 2013 13 h 27 CET> <Debug> <TLS> <000000> <126795 received CHANGE_CIPHER_SPEC>

Client Side

type mylogclient | grep "HANDSHAKEMESSAGE"

<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ServerHello>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Certificate>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ServerHelloDone>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Finished>
<REQUEST ONE PROCESSED>
<18 dec. 2013 13 h 26 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ServerHello>
<18 dec. 2013 13 h 27 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Finished>

Handshake doing client authentication (2-way SSL)

This is what occurs when a handshake does client authentication (2-way SSL):

Server Side

type mylogserver  | grep "HANDSHAKEMESSAGE"

<19 dec. 2013 11 h 07 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientHello>
<19 dec. 2013 11 h 07 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Certificate>
<19 dec. 2013 11 h 07 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientKeyExchange>
<19 dec. 2013 11 h 07 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ClientKeyExchange RSA>
<19 dec. 2013 11 h 07 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: CertificateVerify>
<19 dec. 2013 11 h 07 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Finished>

Client Side

type mylogclient | grep "HANDSHAKEMESSAGE"

<23 dec. 2013 15 h 59 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ServerHello>
<23 dec. 2013 15 h 59 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Certificate>
<23 dec. 2013 15 h 59 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: CertificateRequest>
<23 dec. 2013 15 h 59 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ServerHelloDone>
<23 dec. 2013 15 h 59 CET> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Finished>

type mylogclient | grep "CHANGE_CIPHER_SPEC"

<23 dec. 2013 15 h 59 CET> <Debug> <TLS> <000000> <write CHANGE_CIPHER_SPEC offs

3. Analyze logs - determine the failure

By looking into your logs, you should be able to locate the first SSL exception being thrown by the server, or the client. All possible failures are:
close_notify
unexpected_message
bad_record_mac
decryption_failed
record_overflow
decompression_failure
handshake_failure
bad_certificate
unsupported_certificate
certificate_revoked
certificate_expired
certificate_unknown
illegal_parameter
unknown_ca
access_denied
decode_error
decrypt_error
export_restriction
protocol_version
insufficient_security
internal_error
user_canceled
no_renegotiation
The most common failures and troubleshooting methods are:

• General Certificate
• Failed hostname verification check
• CERT_CHAIN_UNTRUSTED
• BAD_CERTIFICATE (not signed properly causing SSL handshake failure)
• CLOSE_NOTIFY
• HANDSHAKE_FAILURE

General Certificate

Before you start looking or reproducing a problem you need to check all the certificates.

openssl x509 -in ca.pem -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: md5WithRSAEncryption
        Issuer: C=US, ST=California, L=San Francisco, O=BEA WebLogic, OU=Security, CN=Demo Certificate Authority/Email=support@bea.com
        Validity
            Not Before: May 30 21:37:44 2010 GMT
            Not After : May 14 21:37:44 2020 GMT
        Subject: C=US, ST=California, L=San Francisco, O=BEA WebLogic, OU=Security, CN=Demo Certificate Authority/Email=support@bea.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (512 bit)
                Modulus (512 bit):
                    00:dd:51:28:0f:64:36:96:7e:0f:ca:29:54:35:4c:
                    8f:6b:dc:90:c5:2e:98:a8:99:3b:c7:05:a5:00:76:
                    79:00:08:6a:ed:d9:28:b1:90:c2:35:96:18:61:36:
                    62:86:93:b1:19:c8:0b:c2:a6:3a:81:86:42:37:ba:
                    70:96:4f:a1:fd
                Exponent: 65537 (0x10001)
    Signature Algorithm: md5WithRSAEncryption
        00:5b:0a:65:9f:5d:73:59:da:e6:51:e9:3b:c1:0b:f3:91:0f:
        0c:f4:72:08:9f:65:4d:1c:37:6c:f3:04:a8:8b:72:06:e1:00:
        5e:1f:30:a1:18:06:37:98:fd:2a:29:c3:a1:6b:26:14:20:4e:
        e4:c1:72:a8:de:69:e0:03:cb:e3
-----BEGIN CERTIFICATE-----
MIICQzCCAe2gAwIBAgIBADANBgkqhkiG9w0BAQQFADCBqTELMAkGA1UEBhMCVVMx
EzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28xFTAT
BgNVBAoTDEJFQSBXZWJMb2dpYzERMA8GA1UECxMIU2VjdXJpdHkxIzAhBgNVBAMT
GkRlbW8gQ2VydGlmaWNhdGUgQXV0aG9yaXR5MR4wHAYJKoZIhvcNAQkBFg9zdXBw
b3J0QGJlYS5jb20wHhcNMDAwNTMwMjEzNzQ0WhcNMDQwNTE0MjEzNzQ0WjCBqTEL
MAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBG
cmFuY2lzY28xFTATBgNVBAoTDEJFQSBXZWJMb2dpYzERMA8GA1UECxMIU2VjdXJp
dHkxIzAhBgNVBAMTGkRlbW8gQ2VydGlmaWNhdGUgQXV0aG9yaXR5MR4wHAYJKoZI
hvcNAQkBFg9zdXBwb3J0QGJlYS5jb20wXDANBgkqhkiG9w0BAQEFAANLADBIAkEA
3VEoD2Q2ln4PyilUNUyPa9yQxS6YqJk7xwWlAHZ5AAhq7dkosZDCNZYYYTZihpOx
GcgLwqY6gYZCN7pwlk+h/QIDAQABMA0GCSqGSIb3DQEBBAUAA0EAAFsKZZ9dc1na
5lHpO8EL85EPDPRyCJ9lTRw3bPMEqItyBuEAXh8woRgGN5j9KinDoWsmFCBO5MFy
qN5p4APL4w==
-----END CERTIFICATE-----

Solution

Validate all the certificates and ensure that the expiration dates are correct.

Failed hostname verification check

Client Side

[Security:090504]Certificate chain received from localhost - 127.0.0.1 failed hostname verification check. Certificate contained CertServer but check expected localhost javax.net.ssl.SSLKeyException: [Security:090504]Certificate chain received from localhost - 127.0.0.1 failed hostname verification check. Certificate contained CertServer but check expected localhost
        at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
        at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertSent(Unknown Source)
        at com.certicom.tls.record.handshake.HandshakeHandler.fireAlert(Unknown Source)..

Solution

Use the switch -Dweblogic.security.SSL.ignoreHostnameVerification=true. This means that in the certificate, the CN for the server certificate is CertServer and it was not used in the URL being connected to. If t3://CertServer:7001 was used, the HostnameVerification switch would not be needed.
Another solution could be to set

-Dweblogic.security.SSL.hostnameVerifier=examples.security.sslclient.
    NulledHostnameVerifier

NulledHostnameVerifier is published as an example part of WLS.

CERT_CHAIN_UNTRUSTED

Client Side

<23 dec. 2013 15 h 26 CET> <Warning> <Security> <BEA-090542> <Certificate chain received from localhost - 127.0.0.1 was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.>

<23 dec. 2013 15 h 31 CET> <Debug> <TLS> <000000> <Validation error = 16>
<23 dec. 2013 15 h 31 CET> <Debug> <TLS> <000000> <Certificate chain is untrusted>
<23 dec. 2013 15 h 31 CET> <Debug> <TLS> <000000> <SSLTrustValidator returns: 16>
<23 dec. 2013 15 h 31 CET> <Debug> <TLS> <000000> <Trust status (16):   CERT_CHAIN_UNTRUSTED>

Solution

The chain that the server is sending is not trusted by the client. Add

-Dweblogic.security.TrustKeyStore=CustomTrust
-Dweblogic.security.CustomTrustKeyStoreFileName=<yourkeystore>

with all the trusted CAs that the server sends (CAs of the public key).
OR

-Dweblogic.security.SSL.trustedCAKeyStore=<your keystore>

The logs should also show the trusted certificates that the client loads at startup (or the server, if the server is acting as a client for SSL), before any SSL handshake is attempted. It should look like the following:

<Mar 22, 2013 11:54:43 AM EST> <Debug> <TLS> <000000> <SSLManager, getting trusted CAs from default key store: /web/bea/weblogic700/server/lib/cacerts>
<Mar 22, 2013 11:54:43 AM EST> <Debug> <TLS> <000000> <Trusted CA: Serial number: 69042098805081595651034369680212310004
Issuer:C=US, ST=MyState, L=MyTown, O=MyOrganization, OU=FOR TESTING ONLY, CN=CACERT
Subject:C=US, ST=MyState, L=MyTown, O=MyOrganization, OU=FOR TESTING ONLY, CN=CACERT
Not Valid Before:Thu Mar 21 15:12:27 EST 2002
Not Valid After:Tue Mar 22 15:12:27 EST 2022
Signature Algorithm:MD5withRSA
>
<Mar 22, 2013 11:54:43 AM EST> <Debug> <TLS> <000000> <Trusted CA: Serial number: 46914133237969612308202465797198785159
Issuer:C=US, ST=MyState, L=MyTown, O=MyOrganization, OU=FOR TESTING ONLY, CN=CertGenCAB
Subject:C=US, ST=MyState, L=MyTown, O=MyOrganization, OU=FOR TESTING ONLY, CN=CertGenCAB
Not Valid Before:Thu Oct 24 11:54:45 EDT 2002
Not Valid After:Tue Oct 25 11:54:45 EDT 2022
Signature Algorithm:MD5withRSA
>
<Mar 22, 2013 11:54:43 AM EST> <Debug> <TLS> <000000> <Trusted CA: Serial number: 11374952449
Issuer:C=US, O=VeriSign, Inc., OU=Class 4 Public Primary Certification Authority
Subject:C=US, O=VeriSign, Inc., OU=Class 4 Public Primary Certification Authority
Not Valid Before:Sun Jan 28 19:00:00 EST 2013
Not Valid After:Fri Dec 31 18:59:59 EST 2014
Signature Algorithm:MD2withRSA

When the server sends its certificate during the SSL handshake, the log should look like the following:

<Mar 22, 2013 11:54:43 AM EST> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: ServerHello>
<Mar 22, 2013 11:54:43 AM EST> <Debug> <TLS> <000000> <HANDSHAKEMESSAGE: Certificate>
<Mar 22, 2013 11:54:43 AM EST> <Debug> <TLS> <000000> <Performing hostname validation checks: secure.authorize.net>
<Mar 22, 2013 11:54:44 AM EST> <Debug> <TLS> <000000> <validationCallback: validateErr = 0>
<Mar 22, 2013 11:54:44 AM EST> <Debug> <TLS> <000000> <  cert[0] = Serial number: 61240024771365919750260051707246880350
Issuer:O=VeriSign Trust Network, OU=VeriSign, Inc., OU=VeriSign International Server CA - Class 3, OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
Subject:C=US, ST=Washington, L=Bellevue, O=InfoSpace, OU=Authorize.Net, OU=Terms of use at www.verisign.com/RPA (c)01, CN=secure.authorize.net
Not Valid Before:Mon Apr 21 20:00:00 EDT 2013
Not Valid After:Thu Apr 21 19:59:59 EDT 2018
Signature Algorithm:MD5withRSA
>
<Mar 22, 2013 11:54:44 AM EST> <Debug> <TLS> <000000> <  cert[1] = Serial number: 49573667635714834907930444256359116452
Issuer:C=US, O=VeriSign, Inc., OU=Class 3 Public Primary Certification Authority
Subject:O=VeriSign Trust Network, OU=VeriSign, Inc., OU=VeriSign International Server CA - Class 3, OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
Not Valid Before:Wed Apr 16 20:00:00 EDT 2007
Not Valid After:Mon Oct 24 19:59:59 EDT 2017
Signature Algorithm:SHAwithRSA
>
<Mar 22, 2013 11:54:44 AM EST> <Debug> <TLS> <000000> <  cert[2] = Serial number: 149843929435818692848040365716851702463
Issuer:C=US, O=VeriSign, Inc., OU=Class 3 Public Primary Certification Authority
Subject:C=US, O=VeriSign, Inc., OU=Class 3 Public Primary Certification Authority
Not Valid Before:Sun Jan 28 19:00:00 EST 2008
Not Valid After:Tue Aug 01 19:59:59 EDT 2028
Signature Algorithm:MD2withRSA

Compare both lists and try to determine which certificate is missing and why. Some possible reasons are:

• The certificate is trusted but it has expired.
• Your client does not load the cacerts certificates
• Your server uses a self-signed certificate, which your client must be configured to trust

BAD_CERTIFICATE (not signed properly causing SSL handshake failure)

Client Side

<23 dec. 2013 15 h 36 CET> <Debug> <TLS> <000000> <Alert received from peer, notifying peer we received it: com.certicom.tls.record.alert.Alert@1b34126>
<23 dec. 2013 15 h 36 CET> <Warning> <Security> <BEA-090482> <BAD_CERTIFICATE alert was received from localhost - 127.0.0.1. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.>
<23 dec. 2013 15 h 36 CET> <Debug> <TLS> <000000> <close(): 6386542>
[Security:090482]BAD_CERTIFICATE alert was received from localhost - 127.0.0.1.Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.
javax.net.ssl.SSLKeyException:
[Security:090482]BAD_CERTIFICATE alert was received from localhost - 127.0.0.1. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.

Server Side

<23 dec. 2013 15 h 38 CET> <Warning> <Security> <BEA-090478> <Certificate chain received from 127.0.0.1 - 127.0.0.1 was not signed properly causing SSL handshake failure.>

Solution

It could be that the CA certificates sent to the server are being shown in the wrong order for the chain. Review the CAs and see if the order such as Public, Intermediate, Root CA is respected.

CLOSE_NOTIFY

The alert for CLOSE_NOTIFY is normal and expected to occur; the tracing is not treating it specially. An ALERT severity 1 type means that the connection was closed. However, check that all the Handshake messages before that show completion of the handshake. The following is an example of expected CLOSE_NOTIFY ALERT:

<Jun 18, 2002 7:15:27 AM EDT> <Debug> <TLS> <000000> <NEW ALERT: com.certicom.tls.record.alert.Alert@2241f8 Severity: 1 Type: 0
java.lang.Exception:
Stack trace
at weblogic.security.utils.SSLSetup.debug(SSLSetup.java:290)
at com.certicom.tls.record.alert.Alert.<init>(Unknown Source)
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.closeWriteHandler(Unknown Source)
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.close(Unknown Source)
at javax.net.ssl.impl.SSLSocketImpl.close(Unknown Source)
at weblogic.socket.NTSocketMuxer.cleanup(NTSocketMuxer.java:500)
at weblogic.rjvm.t3.T3JVMConnection.close(T3JVMConnection.java:692)
at weblogic.rjvm.ConnectionManager.removeConnection(ConnectionManager.java:982)
at weblogic.rjvm.ConnectionManager.shutdown(ConnectionManager.java:573)
at weblogic.rjvm.ConnectionManagerServer.shutdown(ConnectionManagerServer.java:527)
at weblogic.rjvm.RJVMImpl.peerGone(RJVMImpl.java:937)
at weblogic.rjvm.RJVMImpl.gotExceptionReceiving(RJVMImpl.java:615)
at weblogic.rjvm.ConnectionManager.gotExceptionReceiving(ConnectionManager.java:826)
at weblogic.rjvm.t3.T3JVMConnection.hasException(T3JVMConnection.java:634)
at weblogic.socket.SSLFilter.hasException(SSLFilter.java:302)
at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:537)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:23)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:141)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:122

HANDSHAKE_FAILURE

This may happen when the WebLogic server connects a remote Microsoft server.

Server Side

####<18.feb.2013 kl 16.09 CET> <Debug> <TLS> <zkb1b74v> <raitnow-server> <ExecuteThread: '14' for queue: 'weblogic.kernel.Default'> <system> <> <000000> <26572483 received ALERT>
####<18.feb.2013 kl 16.09 CET> <Debug> <TLS> <zkb1b74v> <raitnow-server> <ExecuteThread: '14' for queue: 'weblogic.kernel.Default'> <system> <> <000000> <NEW ALERT: com.certicom.tls.record.alert.Alert@75ca3e Severity: 2 Type: 40
java.lang.Throwable: Stack trace
at weblogic.security.utils.SSLSetup.debug(SSLSetup.java:265)
at com.certicom.tls.record.alert.Alert.<init>(Unknown Source)
at com.certicom.tls.record.alert.AlertHandler.handleAlertMessages(Unknown Source)
at com.certicom.tls.record.ReadHandler.interpretContent(Unknown Source)
at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
at com.certicom.tls.record.ReadHandler.readUntilHandshakeComplete(Unknown Source)
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.completeHandshake(Unknown Source)
at com.certicom.tls.record.WriteHandler.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:127)
at java.io.FilterOutputStream.flush(FilterOutputStream.java:123)
at weblogic.net.http.HttpURLConnection.writeRequests(HttpURLConnection.java:99)
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:296)
at java.net.URLConnection.getContent(URLConnection.java:582)
at no.nordea.safe.raitnow.ejb.impl.HTTPSRAConnection.getConnectionReader(HTTPSRAConnection.java:73)
at no.nordea.safe.raitnow.ejb.impl.HTTPRARequester.retrieveResponse(HTTPRARequester.java:211)
at no.nordea.safe.raitnow.ejb.impl.HTTPRARequester.sendRequest(HTTPRARequester.java:138)
at no.nordea.safe.raitnow.ejb.impl.CastorRADataTransformer.order(CastorRADataTransformer.java:199)
at no.nordea.safe.raitnow.ejb.EndUserRegistrationAuthorityBean.order(EndUserRegistrationAuthorityBean.java:98)
at no.nordea.safe.raitnow.ejb.EndUserRegistrationAuthorityBean_mzdjm5_EOImpl.order (EndUserRegistrationAuthorityBean_mzdjm5_EOImpl.java:532)
at no.nordea.safe.raitnow.ejb.EndUserRegistrationAuthorityBean_mzdjm5_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:477)
at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:420)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:415)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
>
####<18.feb.2013 kl 16.09 CET> <Debug> <TLS> <zkb1b74v> <raitnow-server> <ExecuteThread: '14' for queue: 'weblogic.kernel.Default'> <system> <> <000000> <Alert received from peer, notifying peer we received it: com.certicom.tls.record.alert.Alert@75ca3e>
####<18.feb.2013 kl 16.09 CET> <Warning> <Security> <zkb1b74v> <raitnow-server> <ExecuteThread: '14' for queue: 'weblogic.kernel.Default'> <system> <> <BEA-090497> <HANDSHAKE_FAILURE alert received from 193.214.20.203 - 193.214.20.203. Check both sides of the SSL configuration for mismatches in supported ciphers, supported protocol versions, trusted CAs, and hostname verification settings.>

Solution

This happened before the client got the ServerHello message. The Microsoft Server was refusing the handshake because the cipher suites given to the remote server were not 128 bits - the remote server wasn't allowing anything lower. If the client license is changed on the WebLogic Server side, it will then work.