$Revision: 1.8 $
$Date: 2003/01/06 04:38:31 $
An unfortunate side-effect of working in IT is the number of tickets or email from users complaining of network, system, or application problems; more unfortunate is that most users' only diagnostic tool is their web browser or another GUI-rich high-level application that hides problems with so-called "friendly error messages." Also surprising is the number of "IT Professionals" who are no better than those same users I just described.
Surprising to some, a very versatile tool is provided by default on most operating systems: telnet. There are instances of operating systems where either telnet or the provided shell are so crippled, that the default environment is not sufficient. There are instances where a daemon process (FTP PORT) or a UDP client are required and telnet is not sufficient regardless of the OS. Enter netcat. netcat is a general tool to read/write across TCP/UDP network connections. Windows OS users will probably want a (Cygwin | U/WIN | MKS) shell; even Windows 2000 IPC has apparent buffer limitations when piping large output between processes.
I will describe some basic usage of common Internet protocols and how we can use netcat and/or telnet to drive end-to-end communication. Following these simple tutorials, I will discuss other topics in perimeter penetration testing and network capacity planning/testing.
Depending on your particular application, you will use any number of HTTP methods, but I will only discuss the more common ones: HEAD, GET, and POST.
Common mistakes are to issue lower-case or mixed-case commands; aside from the URI, they should be all upper-case. The command line should always be the first line as well, followed by the HTTP headers. The headers themselves are typically mixed-case with their first letter capitalized. Custom headers are typically prefixed with "X-" and all lines in the complete HTTP header are delimited by a <CRLF> carriage-return line-feed combination (like MS-DOS), also represented as \r\n, 0x0D0A, ^M^J, etc. And finally, there is a 2 <CRLF> combination to delimit the header from the data payload.
To determine if your HTTP service is simply running or to determine if it returns the proper headers, the HEAD command is sufficient. For some, HTTP/1.0 may be sufficient, but if you utilize HTTP host headers (aka virtual servers), you will require HTTP/1.1 and the "Host: " HTTP header. Also note the keepalive timeout (and connection pooling) when using HTTP/1.1.
$ cat http1.0.txt HEAD / HTTP/1.0 $ $ cat http1.1.txt HEAD /RCS/index.html,v HTTP/1.1 Host: www.example.com $ $ cat http1.0.txt | telnet www.example.com 80 HTTP/1.1 200 OK Date: Mon, 30 Dec 2002 02:22:15 GMT Server: Apache/1.3.26 (Unix) mod_perl/1.27 Last-Modified: Wed, 27 Feb 2002 09:28:34 GMT ETag: "3bd02-0-3c7ca6c2" Accept-Ranges: bytes Content-Length: 0 Connection: close Content-Type: text/html X-Pad: avoid browser bug Connection closed by foreign host. $ $ cat http1.1.txt | telnet www.example.com 80 HTTP/1.1 403 Forbidden Date: Mon, 30 Dec 2002 02:26:10 GMT Server: Apache/1.3.26 (Unix) mod_perl/1.27 Content-Type: text/html Connection closed by foreign host.
The HTTP/1.0 request is returned as a HTTP/1.1 response. This is normal, the 1.0 request forces a "Connection: close," which we expect. The ETag is important for server farms, where the default entity tag is "Last Modified Date"-"Size"-"Inode Number"; the "Inode Number" will obviously vary from machine to machine, so this appears to be a server misconfiguration.
The HTTP/1.1 response is a 403, which tells us that the requested file exists, but is forbidden to us due to file/directory permissions or ACL settings.
Whereas HEADs are typically used to determine availability, GET and POST requests are typically used to determine the correct functionality of your application.
$ cat http_get.txt GET /search?q=http+rfc HTTP/1.1 Host: www.google.com $ $ cat http_get.txt | telnet www.google.com 80 HTTP/1.1 200 OK Server: GWS/2.0 Date: Mon, 30 Dec 2002 02:35:07 GMT Transfer-Encoding: chunked Content-Type: text/html Cache-control: private Set-Cookie: ID=1269ea8c29cd902f:TM=1041215707:LM=1041215707:S=DiuLPcEx58qJyg1A; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com 9f6 <html><head><meta HTTP-EQUIV="content-type... [ output cut ]
When crafting GET requests, be sure to properly encode the URL. Here's a "safe" URL encoding subroutine in perl:
sub URLEncode
{
my($url)=@_;
my(@characters)=split(/(\%[0-9a-fA-F]{2})/,$url);
foreach(@characters)
{
if ( /\%[0-9a-fA-F]{2}/ ) # Escaped character set ...
{
# IF it is in the range of 0x00-0x20 or 0x7f-0xff
# or it is one of "<", ">", """, "#", "%",
# ";", "/", "?", ":", "@", "=" or "&"
# THEN preserve its encoding
#"
unless ( /(20|7f|[0189a-fA-F][0-9a-fA-F])/i
|| /2[2356fF]|3[a-fA-F]|40/i )
{
s/\%([2-7][0-9a-fA-F])/sprintf "%c",hex($1)/e;
}
}
else # Other stuff
{
# 0x00-0x20, 0x7f-0xff, <, >, and " ... "
s/([\000-\040\177-\377\074\076\042])
/sprintf "%%%02x",unpack("C",$1)/egx;
}
}
return join("",@characters);
}
Notice the hex just before the server returns the content; this is due to the "Transfer-Encoding: chunked" header. The "9f6" is the number of bytes in the HTTP data payload. This response is similar to the required POST request header "Content-Length: ".
$ cat http_post.txt POST /echo.pl HTTP/1.0 Content-Type: text/plain Content-Length: 5 hello $ $ cat http_post.txt | telnet example.com 80 HTTP/1.1 200 OK Content-Type: text/xml; charset=utf-8 Content-Length: 87 <?xml version="1.0" encoding="utf-8"?> <Response>Hello!</Response> Connection closed by foreign host.
Responses to a POST command are not cacheable, unless the response includes appropriate Cache-Control or Expires header fields.
To scan your network for rogue proxy servers, the following can be scripted to enumerate the desired net blocks.
$ cat http_proxy.txt GET http://www.yahoo.com HTTP/1.0 $ $ cat http_proxy.txt | telnet www.example.com 80 HTTP/1.1 404 Not Found Date: Mon, 30 Dec 2002 19:12:24 GMT Server: Apache/1.3.26 (Unix) mod_perl/1.27 Connection: close Content-Type: text/html
Resources that require Basic HTTP Authentication are simply encoded as base64 with this one-liner. MD5 isn't quite so simple, but given the challenge phrase, is still trivial.
$ perl -MMIME::Base64 -le "print encode_base64(shift)" "username:password"
dXNlcm5hbWU6cGFzc3dvcmQ=
$ echo "username:realm:password:challenge:HEAD:/secure" | md5sum
7ff531e2618d58a5587f81035997c9b5
$
$ cat http_basic.txt
HEAD /secure HTTP/1.0
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
$ cat http_digest.txt
HEAD /secure HTTP/1.0
Authorization: Digest username="username",
realm="realm",
nonce="challenge",
uri="/secure",
response="7ff531e2618d58a5587f81035997c9b5"
$
$ cat http_basic.txt | telnet www.example.com 80
HTTP/1.1 200 OK
Date: Mon, 30 Dec 2002 19:12:24 GMT
Server: Apache/1.3.26 (Unix) mod_perl/1.27
Content-Type: text/html
The techniques described in section Section 2.1 apply just the same here. The only difference is the underlying session layer. Tunneling HTTP commands through SSL can be done easily with the openssl s_client.
$ cat https_head.txt HEAD / HTTP/1.1 Host: www.example.com $ $ cat https_head.txt | openssl s_client -connect www.example.com:443
Depending on corporate and technical policies, different e-commerce sites will have varying support for SSL and TLS protocols, cipher type, and key bit depth. Web browsers typically do not provide a sufficiently fine-grained user-interface to test varying combinations of those three components. With openssl however, it is trivial.
Microsoft Internet Explorer v5.00 and v5.01 have, for example, serious bugs when using SSL. When your QA department refuses to certify your SSL infrastructure because MSIE 5.01 (default with Windows98) fails, here are some tests to prove them wrong.
$ cat https_head.txt \ | openssl s_client -ssl2 -cipher "RC2:EXPORT40" -connect example.com:443 $ cat https_head.txt \ | openssl s_client -ssl2 -cipher "RC2:EXPORT56" -connect example.com:443 $ cat https_head.txt \ | openssl s_client -ssl2 -cipher "RC4:EXPORT40" -connect example.com:443 $ cat https_head.txt \ | openssl s_client -ssl2 -cipher "RC4:EXPORT56" -connect example.com:443 $ ...
Other combinations would include SSLv2 vs SSLv3 (TLSv1 not supported in MSIE 5.01) protocols, MD5 vs SHA hash algorithms, IDEA vs DES vs RC2 vs RC4 symmetric key algorithms, RSA (DSA/DSS asymmetric key algorithm not supported in MSIE 5.01), and 40 vs 56 vs 128 bit key length. A quick script would be very handy here to cover all the permutations. Using HTTP/1.1 keepalives to pool HTTP requests is highly recommended here, since the SSL handshake with each HTTP connection establishment is highly CPU intensive, in addition to the numerous TCP 3-ways.
If your PRNG is poor, the -rand [filename] argument may be required to add sufficient entropy for the public key handshake.
FTP clients are simple to use and understand; they support authentication, remote directory navigation, and ascii/binary file transfer. The protocol itself however, is somewhat non-intuitive.
Common mistakes are to confuse typical FTP client commands with the actual FTP protocol commands. "dir" and "ls" are typically wrappers for the actual commands LIST and NLST. "get" and "put" are typically wrappers for the actual commands RETR and STOR. Also non-intuitive is the use of either server initiated (PORT) or client initiated (PASV) data transfers over a separate network pipe.
FTP requires two communication channels, a control and data channel and there are two data transfer modes: PORT and PASV. In PORT mode, the data channel is established from the server, whereas in PASV (passive) mode, the data channel is established from the client.
PORT mode is typically the default, but PASV mode is conceptually simpler and is becoming more prevalent due to strict firewalls in most corporate environments and many residential broadband NSPs. The bold text below are commands that we issue. TTY1 in this example is our control session and TTY2 is our data session.
tty1$ telnet ftp.example.com 21 tty2$ Connected to ftp.example.com Escape character is '^]'. 220 ftp.example.com FTP server ready. USER ftp 331 Guest login ok, send your complete e-mail address as password. PASS myname@ 230 Guest login ok, access restrictions apply. TYPE A 200 Type set to A. PASV 227 Entering Passive Mode (192,168,0,100,105,163) NLST tty2$ telnet 192.168.0.100 27043 Connected to ftp.example.com. Escape character is '^]'. 150 Opening ASCII mode data connection for file list. README 226 Transfer complete. Connection closed by foreign host. TYPE I tty2$ 200 Type set to I. PASV 227 Entering Passive Mode (192,168,0,100,126,43) RETR /README tty2$ telnet 192.168.0.100 32299 Connected to ftp.example.com. Escape character is '^]'. 150 Opening BINARY mode data connection for /README (27 bytes). Welcome to ftp.example.com! 226 Transfer complete. Connection closed by foreign host. QUIT tty2$ 221 Goodbye. Connection closed by foreign host. tty1$
The USER command typically accepts both "anonymous" and "ftp" for guest accounts and the PASS command typically requires only an "@" for a "complete e-mail address." The TYPE command sets your data transfer mode to ASCII (A) or BINARY (I).
The Passive (PASV) command returns a comma delimited set of integers. The first 4 fields are your dotted quad IP address of the server accepting the requested data transfer. The 2nd two fields are used to calculate the destination port of the data transfer. In our NLST example, ((105 * 256) + 163) returns 27043, so in our second TTY, we telnet to port 27043 of the server IP returned in the PASV output. A PASV command is required for each individual data request.
PORT mode transfers are exactly the opposite process from PASV mode transfers. The client sends the server a PORT request, encoding the client's local destination port to the remote server. This requires we setup a daemon obviously to listen on that port, which is easily done with netcat. In this particular case, tty2's IP is 192.168.0.101.
tty1$ telnet ftp.example.com 21 tty2$ nc -l -p1025 Connected to ftp.example.com Escape character is '^]'. 220 ftp.example.com FTP server ready. USER ftp 331 Guest login ok, send your complete e-mail address as password. PASS myname@ 230 Guest login ok, access restrictions apply. PORT 192,168,0,101,4,1 200 PORT command successful. NLST 150 Opening ASCII mode data connection for file list. README 226 Transfer complete. tty2$ QUIT 221 Goodbye. Connection closed by foreign host. tty1$
Problems with FTP are typically introduced by changes to firewall/packet filter rule sets or operating system tcp/ip stack tuning of dynamic client port restrictions. Analyzing the PORT and PASV responses complement the use of netstat and network sniffers in troubleshooting FTP problems.
Corporate policies might dictate strict SPAM/UCE controls, including mail relay limitations. We can also test anti-virus filters for correct functionality quite easily with netcat or telnet.
$ cat smtp_relay.txt HELO myhost MAIL FROM: Sender Name <user@example.org> RCPT TO: Recipient Name <user@example.net> DATA From: Sender Name <user@example.org> To: Recipient Name <user@example.net> Subject: test . QUIT $ $ cat smtp_relay.txt | telnet smtp.example.net 25 Connected to smtp.example.org. Escape character is '^]'. 220 smtp.example.net 250 smtp.example.net 250 Ok 250 Ok 250 Ok: queued as 4AF0413778 221 Bye Connection closed by foreign host. $ $ cat smtp_relay.txt | telnet smtp.example.org 25 Connected to smtp.example.org. Escape character is '^]'. 220 smtp.example.org 250 smtp.example.org 250 Ok 554 <user@example.net>: Recipient address rejected: Relay access denied Connection closed by foreign host.
SMTP generally doesn't prevent source email spoofing, but relaying recipient email for one domain through another domain's SMTP server is typically frowned upon.
SPAM filters typically parse the SMTP headers and pattern-match against a signature database of known offenders; when the number of patterns matched increases a specified threshold, the email is typically flagged as SPAM or UCE. As shown above, the DATA command allows the client to append to the email header. The actual email "message" is delimited from the header by two <CRLF>s (carriage return, line feed). Custom SMTP headers are typically prefixed with "X-".
Another SPAM technique is to use the VRFY (verify mailbox) or RCPT (recipient) commands to determine if the destination mailbox exists.
$ cat http_vrfy1.txt HELO myhost VRFY user1 VRFY user2 VRFY user3 $ $ cat http_vrfy2.txt HELO myhost MAIL FROM: <user@example.org> RCPT TO: <user1@example.net> RCPT TO: <user2@example.net> RCPT TO: <user3@example.net> $ $ cat http_vrfy1.txt | telnet smtp.example.net 25 Connected to smtp.example.net. Escape character is '^]'. 220 smtp.example.net 250 smtp.example.net 502 VRFY command is disabled 502 VRFY command is disabled 502 VRFY command is disabled $ $ cat http_vrfy2.txt | telnet smtp.example.net 25 Connected to smtp.example.net. Escape character is '^]'. 220 smtp.example.net 250 smtp.example.net 550 <user1@example.net>: User unknown 250 Ok 550 <user3@example.net>: User unknown
VRFY is typically disabled by default in modern SMTP servers, but it is still trivial to verify destination mailboxes with several RCPT commands.
Testing anti-virus filters is more difficult, but a common filter is to test the file name extensions in all MIME attachments. The following attaches a 0-byte vbscript file.
$ cat http_virus.txt
HELO myhost
MAIL FROM: Sender Name <user@example.org>
RCPT TO: Recipient Name <user@example.net>
DATA
From: Sender Name <user@example.org>
To: Recipient Name <user@example.net>
Subject: you just won!!
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----_=_NextPart_000_01C2B03F.ED539020"
------_=_NextPart_000_01C2B03F.ED539020
Content-Type: text/plain;
charset="iso-8859-1"
you just won!!
------_=_NextPart_000_01C2B03F.ED539020
Content-Type: text/plain;
name="prize.vbs"
Content-Disposition: attachment;
filename="prize.vbs"
------_=_NextPart_000_01C2B03F.ED539020--
.
QUIT
$
$ cat http_virus.txt | telnet smtp.example.net 25
$ cat ssh.txt SSH-2.0-client $ $ cat ssh.txt | nc www.example.com 80 SSH-1.99-OpenSSH_3.4p1 p3=diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1ssh-rsa,ssh-dssf aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijnd ael-cbc@lysator.liu.sefaes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes1 92-cbc,aes256-cbc,rijndael-cbc@lysator.liu.seUhmac-md5,hmac-sha1,hmac-ripemd160, hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96Uhmac-md5,hmac-sha1,hmac-ripe md160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 none,zlib none,zlib
What do we learn from this exercise? We learn which versions of SSH are supported. This information can be helpful if SSHv1 is disabled by the server, but your client defaults to v1.
The only remaining information that appears in (mostly) plain text are the negotiatible authentication parameters. If the server implements a strong policy, you can determine the minimum strength and protocol/ ciphers necessary without resorting to trial and error.