openssl to negotiate SSL encryption for STARTTLS

I'm using openssl to connect to an SMTP server normally (without encryption), send a STARTTLS command, negotiate the SSL encryption, and then interact with the encrypted session.

This is the command I'm using (through telnet):

openssl s_client -starttls smtp -crlf -connect 1.2.3.4:25

How can I ensure that TLS handshake was successful?

This is the sequence of commands used so far:

<< 220 example.com ESMTP ready
>> EHLO localhost
<< 250-smtp.mail.yahoo.com
<< 250-PIPELINING
<< 250-AUTH PLAIN LOGIN CRAM-MD5
<< 250 STARTTLS
>> STARTTLS
<< 220 2.0.0 Start TLS
>> openssl s_client -starttls smtp -crlf -connect 127.0.0.1:587

You seem to be confusing a shell (where you type commands such as telnet or openssl) and the socket's protocol.

Using telnet to connect to a port for a protocol like SMTP is a quick hack that allows you to type in directly what you'd normally have to program if implementing a client for that protocol. It can work a little for text-based protocols but it has limitations. In particular, you'll have a hard-time typing an TLS handshake this way: firstly you probably won't be able to find the right keys on your keyboard for some of the bytes you need to send; secondly, you certainly won't be able to read what the server sends you. In short, this approach doesn't make any sense.

openssl s_client -starttls smtp -crlf -connect 127.0.0.1:587 already does what you're trying to do with telnet: it opens the connection to that server, sends the EHLO SMTP command, sends the STARTTLS SMTP command and then starts the handshake. The OpenSSL command itself is not part of the SMTP protocol at all and mustn't be sent on the SMTP socket. What you'll get when running this command should be similar to having your telnet session with the handshake already performed, since you should be able to use its standard input/ouput in the same way you would be able telnet.

This being said, both telnet and openssl s_client to send SMTP commands are debugging techniques at best.