Skip to content

Tag: utility

Handy cURL shell script for http troubleshooting

The great cURL tool

Many people know about the famous cURL tool. For those don’t know yet, here is the introduction from it’s own man page.

curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.

curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more. As you will see below, the number of features will make your head spin!

Many developers, system admins, tech support people and users use it on a day to day basis. The typical way of using it is to view HTTP connection detail such as request and response headers, very handy.

Wait, are you really using it in a great way?

But many people never notice a powerful option cURL provided, the “-w” option. Here are some key information regarding this option from its man page.

-w, --write-out Make curl display information on stdout after a completed transfer. The format is a string that may contain plain text mixed with any number of variables. The format can be specified as a literal "string", or you can have curl read the format from a file with "@filename" and to tell curl to read the format from stdin you write "@-".

Some really useful variable “-w” option supports:

size_download The total amount of bytes that were downloaded.
size_request The total amount of bytes that were sent in the HTTP request.
size_upload The total amount of bytes that were uploaded.
speed_download The average download speed that curl measured for the complete download. Bytes per second.
speed_upload The average upload speed that curl measured for the complete upload. Bytes per second.
time_appconnect The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
time_connect The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
time_namelookup The time, in seconds, it took from the start until the name resolving was completed.
time_pretransfer The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
time_redirect The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
time_starttransfer The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
time_total The total time, in seconds, that the full operation lasted.

Try this!

Let’s put these together in a shell script, say


set -e

              Downloaded (byte)  :  %{size_download}
            Request sent (byte)  :  %{size_request}
                Uploaded (byte)  :  %{size_upload}

       Download speed (bytes/s)  :  %{speed_download}
         Upload speed (bytes/s)  :  %{speed_upload}

            DNS lookup time (s)  :  %{time_namelookup}
  Connection establish time (s)  :  %{time_connect}
           SSL connect time (s)  :  %{time_appconnect}
          Pre-transfer time (s)  :  %{time_pretransfer}
              Redirect time (s)  :  %{time_redirect}
        Start-transfer time (s)  :  %{time_starttransfer}

                 Total time (s)  :  %{time_total}


exec curl -w "$curl_format" -o /dev/null -s "[email protected]"

Then we can call this script in such way:


              Downloaded (byte)  :  251340
            Request sent (byte)  :  121
                Uploaded (byte)  :  0

       Download speed (bytes/s)  :  2483400.000
         Upload speed (bytes/s)  :  0.000

            DNS lookup time (s)  :  0.000111
  Connection establish time (s)  :  0.000559
           SSL connect time (s)  :  0.000000
          Pre-transfer time (s)  :  0.000623
              Redirect time (s)  :  0.000000
        Start-transfer time (s)  :  0.023913

                 Total time (s)  :  0.101208

We have a pretty view of how may data are transferred and how much time spent, isn’t it nice?

Next time you need to diagnostic some HTTP issue, besides of the regular curl command you used to run, don’t forget to give this one a try. I use it a lot, hope you will find it helpful as well.

http connection timing, http connection troubleshooting, http connection diagnostic, curl advanced tips
Leave a Comment