system.verbs.builtins.tcp.httpClient

on httpClient (method="GET", server="", port=nil, path="", proxy=user.webBrowser.proxy.domain, proxyPort=user.webBrowser.proxy.port, proxyUserName=user.webBrowser.proxy.userName, proxyPassword=user.webBrowser.proxy.password, data="", datatype="", username="", password="", adrHdrTable=nil, cookiesOn=false, debug=false, timeOutTicks=60*30, flMessages=true, ctFollowRedirects=0, flJustHeaders=false, flAcceptOpml=false, flUseProxy=nil, adrRedirectInfo=nil)
  Changes
    11/10/11; 4:12:08 PM by DW
      Allow data to be an address, if so we don't make a copy of it.
    Before November 2011
      1/22/06; 7:33:35 PM by DW
        The code that checked to see if we're running on the "samemachine" wasn't checking if server was 127.0.0.1, and was only checking the port on user.inetd.config.http, when there may be more listeners running on different ports, as there are in the OPML Editor. It would only shortcut of it was localhost and if it was running on one of the daemons.
      2/7/03; 3:49:12 PM by JES
        When testing for proxy-exempt servers, don't inclue the port number if specified in the server parameter.
      10/20/02; 2:32:23 AM by JES
        Respect user.webBrowser.proxy.exceptions.
      10/16/02; 6:55:56 PM by JES
        Removed debugging code.
      10/14/02; 11:10:16 AM by JES
        If server is 127.0.0.1 or "localhost", set flUseProxy to false.
      10/13/02; 6:23:28 PM by DW
        New optional param -- adrRedirectInfo. If it's non-nil (the default) and we're redirecting, we fill the table with info about the redirect. The RSS aggregator needs this information, and it may prove useful in other places. See xml.rss.readService for an example of its usage.
      12/19/01; 9:58:10 PM by JES
        Added a call to webBrowser.init, which initializes prefs at user.webBrowser.proxy. Added optional parameter, flUseProxy, which defaults to nil. If nil, then the value at user.webBrowser.proxy.enabled is used. Pass the value of flUseProxy when following redirects.
      8/14/01; 1:49:11 PM by JES
        When recursing to following redirects, pass in the flJustHeaders and flAcceptOpml parameters.
      8/9/01; 6:10:17 PM by JES
        If the stream is unexpectedly closed by the remote host, call tcp.abortStream, to avoid consuming the available connections.
      8/8/01; 3:21:27 PM by JES
        If no Accept header has been passed in, add an Accept header with the value: text/x-opml, */*, so that we get the response in opml instead of html.
      3/1/01; 1:57:33 PM by PBS
        Trim white space around the content-length header before coercing it to a number. If a server adds white space here, the coercion would fail, resulting in a connection counter leak. (History: this was discovered in Radio when reading the Red Herring channel.)
      1/10/01; 10:36:30 AM by DW
        Add an optional parameter, flJustHeaders, if true, we don't read the body, we just return the header portion of the response. This is useful if you just want to know how big a download is, and want to defer downloading big things until the middle of the night when it's quiet on the Internet.
      1/7/01; 9:44:18 PM by DW
        In Radio, set User-Agent to Radio UserLand.
        Changed the header so it doesn't use backslashes. The outliner now supports wrapping headlines.
      9/22/00; 11:27:50 AM by PBS
        Respect the user's offline settings.
      11/5/99; 3:00:39 AM by AR
        If the server doesn't send a Content-Length header, read until connection closes.
      10/21/99; 3:57:29 PM by AR
        Adapted to use new TCP verbs in 6.1.
      6/14/99; 5:23:37 AM by AR
        If flMessages is true, we now clean up the message area before we return.
      6/13/99; 9:20:04 AM by AR
        Both the read and write loop now yield time to other Frontier threads and to the system.
        The Host header is now generated correctly if the request goes thru a proxy server.
        Fixed a few inconsistencies in the cookie handling code. Thinks should now work correctly when connecting via a proxy server.
        Added an optional parameter, ctFollowRedirects, determining the number of times that HTTP redirects will be followed. In order to preserve previous behaviour, the default value is 0.
      3/4/99; 11:54:30 AM by PBS
        If adrHdrTable^ contains a "Host" sub-item, don't add a Host header item.
      3/2/99; 10:43:18 AM by DW
        if adrHdrTable^ contains a "User-Agent" sub-item, don't add a User-Agent header item
      10/25/98; 6:33:59 AM by DW
        allow the use of a colon in the server name to specify the port
          example: www.scripting.com:80
        if present, we use that port, not the one specified in the call
          the assumption is that most callers either specify 80 or don't specify the port at all
          for crawlers, for example, they definitely want to use the port specified in the server string
          some commands, esp in the webBrowser class, get the url from another piece of software
      9/28/98; 4:42:12 AM by DW
        if we're sending to the same machine, set the stream ID to 0
        user.webserver.responders.CGI.methods.any is copying it, hope no CGIs are using it!
      9/17/98; 7:11:56 AM by DW
        msgs now say that they're from tcp.httpClient.
        changed Connection to xxx timed out message to...
          Can't process the request because the connection to xxx timed out.
        allow server to be "localhost", if so, we totally optimize the functionality
      2/13/98; 8:57:10 AM by DW
        Initial code assembled from various sample scripts by Dave Winer.
        The goal is to have a single documented way to talk to HTTP servers.
        It's got a long parameter list, but requires no configuration, doesn't run off a user.xxx table.
        The target user of this script is someone who needs to understand how the code works.
        It will be distributed with Frontier on both platforms as soon as we agree that this is complete and works.
        When you call it, use parameter labels, don't depend on the order of these params.
        For examples, see tcp.examples.
  local (adrdata)
  if tcp.isOffline () //PBS 09/22/00: check offline status
    scriptError ("Can't send HTTP request because TCP is offline.")
  bundle //set adrdata, 11/10/11 by DW
    if sizeof (data) > 0
      if typeof (data) == addresstype
        adrdata = data
      else
        adrdata = @data
    else
      adrdata = @data
  webBrowser.init ()
  try //set flUseProxy
    if (flUseProxy == nil) and (typeOf (flUseProxy) == unknownType)
      flUseProxy = user.webBrowser.proxy.enabled
    if (server == "127.0.0.1" or string.lower (server) == "localhost") //JES 10/14/02: don't call the loopback address through the proxy
      flUseProxy = false
    if flUseProxy //check the exceptions table -- if we find a match then set flUseProxy to false
      on wildcardIpMatch (s, oneip)
        local (i)
        if s == oneip //optimization
          return (true)
        for i = 1 to 4
          try
            local (part = string.nthField (oneip, ".", i))
            local (pattern = string.nthField (s, ".", i))
            if pattern == "*"
              continue
            if pattern contains "-"
              local (min = number (string.nthField (pattern, "-", 1)) )
              local (max = number (string.nthField (pattern, "-", 2)) )
              local (n = number (part))
              if n >= min and n <= max
                continue
              else
                return (false)
            if pattern != part
              return (false)
          else
            return (false)
        return (true)
      local (flIpAddress = false, hostToCheck = string.lower (string.nthField (server, ":", 1)) )
      try //see if flIpAddress should be true
        local (serverWithoutDots = string.replaceAll (server, ".", ""))
        double (serverWithoutDots) //will error if non-numeric characters are present
        flIpAddress = true
      else //check domains, and then try a DNS lookup
        local (adr)
        local (lowerserver = string.lower (server))
        for adr in @user.webBrowser.proxy.exceptions
          if string.wildcardMatch (hostToCheck, string.lower (nameOf (adr^)) )
            if adr^
              flUseProxy = false
              break
        if flUseProxy //try a reverse lookup so we can check against IP address
          try
            hostToCheck = tcp.dns.getDottedId (server)
            flIpAddress = true
      if flUseProxy //check to see if this specific IP is excepted
        if defined (user.webBrowser.proxy.exceptions.[hostToCheck])
          if user.webBrowser.proxy.exceptions.[hostToCheck]
            flUseProxy = false
      if flUseProxy //check for IP-address exceptions which contain wildcard characters and ranges
        local (i)
        for i = 1 to sizeOf (user.webBrowser.proxy.exceptions)
          if user.webBrowser.proxy.exceptions[i]
            local (oneexception = nameOf (user.webBrowser.proxy.exceptions[i]))
            if wildcardIpMatch (oneexception, hostToCheck)
              flUseProxy = false
              break
  on displayMsg (s)
    msg ("tcp.httpClient: " + s)
  local (myAddress = tcp.myDottedID (), flSameMachine = false)
  bundle //look for the port specified in the server string
    if server contains ":"
      try {port = number (string.nthField (server, ":", 2))} //it might not be a number
      server = string.nthField (server, ":", 1)
    if port == nil //no port was specified, either in the call or in the server string
      port = 80
  bundle //see if it's the same machine
    local (lowerserver = string.lower (server))
    if (lowerserver == "localhost") or (lowerserver == "127.0.0.1")
      flSameMachine = true
    else
      try {flSameMachine = tcp.equalNames (server, myAddress)} //DW 3/9/98
    bundle //check that we're running on the requested port, 1/22/06 by DW
      if flsamemachine //may turn false
        local (adrd, flfound = false)
        for adrd in @user.inetd.config
          if port == adrd^.port
            if inetd.isDaemonRunning (adrd)
              flfound = true
              break
        if not flfound
          flsamemachine = false
      bundle //old code
        PBS 9/18/98: check that the server on the requested port is Frontier.
        if port != user.inetd.config.http.port
          flSameMachine = false //it's the same machine, but it's not Frontier as a server
        else
          if not (inetd.isDaemonRunning (@user.inetd.config.http))
            flSameMachine = false //Frontier isn't running as a webserver, there must be another server on this machine
  if not (path beginsWith "/")
    path = "/" + path
  local (httpCommand = "", adrCookiesTable = @user.webBrowser.cookies)
  bundle //build the HTTP request in "httpCommand"
    on add (s)
      httpCommand = httpCommand + s + "\r\n"
    local (uri = path)
    if flUseProxy and (sizeOf (proxy) > 0) //the request is going thru a proxy server: set uri accordingly
      uri = "http://" + server
      if port != 80
        uri = uri + ":" + port
      uri = uri + path
    add (method + " " + uri + " HTTP/1.0")
    if (adrHdrTable == nil) or (not defined (adrHdrTable^.["User-Agent"])) //3/2/99; 10:43:10 AM by DW
      if system.environment.isPike
        add ("User-Agent: Radio UserLand/" + Frontier.version () + " (" + sys.os () + ")")
      else
        add ("User-Agent: Frontier/" + Frontier.version () + " (" + sys.os () + ")")
    local (host = "Host: " + server)
    if port != 80
      host = host + ":" + port
    if (adrHdrTable == nil) or (not defined (adrHdrTable^.["Host"])) //3/4/99; 11:54:30 AM by PBS
      add (host)
    if flAcceptOpml //8/8/01; 3:19:50 PM by JES
      if (adrHdrTable == nil)
        add ("Accept: text/x-opml, */*")
      else //headers were passed in
        if defined (adrHdrTable^.Accept)
          local (acceptHeader = adrHdrTable^.Accept)
          if not (acceptHeader contains "text/x-opml")
            if sizeOf (string.trimWhiteSpace (acceptHeader)) > 0
              acceptHeader = "text/x-opml, " + acceptHeader
            else
              acceptHeader = "text/x-opml"
          if not (acceptHeader contains "*/*")
            acceptHeader = acceptHeader + ", */*"
          add ("Accept: " + acceptHeader)
        else
          add ("Accept: text/x-opml, */*")
    if sizeOf (adrdata^) > 0
      add ("Content-Type: " + datatype)
      add ("Content-length: " + sizeOf (adrdata^))
    if (sizeOf (username) > 0) or (sizeOf (password) > 0)
      add ("Authorization: Basic " + base64.encode (username + ":" + password, 0))
    if flUseProxy and (sizeOf (proxy) > 0) //the request is going thru a proxy server
      if (sizeOf (proxyUsername) > 0) or (sizeOf (proxyPassword) > 0) //the proxy requires authentication
        add ("Proxy-Authorization: Basic " + base64.encode (proxyUsername + ":" + proxyPassword, 0))
    if adrHdrTable != nil //added 3.27.98 by ASG to include header items
      local (i)
      for i = 1 to sizeOf (adrHdrTable^)
        local (headerName = nameOf (adrHdrTable^[i]))
        if flAcceptOpml //8/8/01; 3:19:50 PM by JES: Only add the Acept header once
          if string.lower (headerName) == "accept"
            continue
        add (headerName + ": " + adrHdrTable^[i])
    if cookiesOn and defined (adrCookiesTable^) //add cookies to the request header: ASG 4/9/98
      local (i, j, k, cookiePath = "/", cookiestring)
      if path != "/"
        cookiePath = string.popSuffix (path, "/")
      for i = 1 to sizeOf (adrCookiesTable^) //loop through cookies table
        local (adrdomain = @adrCookiesTable^ [i])
        if host endsWith nameOf (adrdomain^)
          for j = 1 to sizeOf (adrdomain^)
            local (adrcookie = @adrdomain^ [j])
            if cookiePath contains nameOf (adrcookie^)
              for k = 1 to sizeOf (adrcookie^)
                if adrcookie^ [k].expires > clock.now () //add this cookie
                cookiestring = cookiestring + nameOf (adrcookie^ [k]) + "=" + adrcookie^ [k].value + "; "
                else
                delete (@adrcookie^ [k]) //this cookie has expired so delete it
      if cookiestring != nil
        cookiestring = string.delete (cookiestring, sizeof (cookiestring) - 1, 2) //delete trailing "; "
        add ("Cookie: " + cookiestring) //add the cookies to the header
    add ("") //blank line
    if sizeOf (adrdata^) > 0
      httpCommand = httpCommand + adrdata^
  if debug //save a copy of the HTTP request for debugging
    wp.newTextObject (httpCommand, @scratchpad.httpCommand)
  local (httpResult = "")
  if flSameMachine //shortcut the call, turn our worst case into our best case: 9/17/98; 7:21:13 AM by DW
    local (params)
    new (tabletype, @params)
    params.client = myAddress
    params.port = port
    params.stream = 0 //9/28/98; 4:42:45 AM by DW
    params.request = httpCommand
    httpResult = webserver.server (@params, httpCommand)
  else
    local (serverAddress = server, serverPort = port)
    if flUseProxy and (sizeOf (proxy) > 0) //the request is going thru a proxy server: switch server and port
      serverAddress = proxy
      serverPort = proxyPort
    if flMessages //PBS 5/5/98
      displayMsg ("Connecting to " + serverAddress + ".")
    local (stream = tcp.openStream (serverAddress, serverPort))
    local (chunksize = 5 * 1024)
    if flMessages
      displayMsg ("Sending request to " + serverAddress + ".")
    try
      tcp.writeStringToStream (stream, httpCommand, chunkSize, timeOutTicks/60) //write the command to the server
    else
      tcp.abortStream (stream)
      scriptError (tryError)
    if flMessages
      displayMsg ("Waiting for data from " + serverAddress + ".")
    local (pat = "\r\n\r\n")
    try
      tcp.readStreamUntil (stream, pat, timeOutTicks/60, @httpResult)
    else
      tcp.abortStream (stream)
      scriptError (tryerror)
    local (headerTable)
    webserver.util.parseHeaders (httpResult, @headerTable)
    if flJustHeaders //the caller only wants the headers, not the body 1/10/01 by DW
      tcp.closeStream (stream)
    else
      if (string.upper (method) != "HEAD") and defined (headerTable.["Content-Length"]) //we know there's content left to be read
        local (lenHeader = string.patternMatch (pat, httpResult) - 1)
        local (lenContent = number (string.trimWhiteSpace (headerTable.["Content-Length"]))) //PBS 03/01/01: trim white space before coercing
        local (lenTotal = lenHeader + string.length (pat) + lenContent)
        if flMessages
          displayMsg ("Receiving response body (" + string.kBytes (lenContent) + ") from " + serverAddress + ".")
        try
          tcp.readStreamBytes (stream, lenTotal, timeOutTicks/60, @httpResult)
        else
          tcp.abortStream (stream)
          scriptError (tryerror)
      else //just read until connection closes
        if flMessages
          displayMsg ("Receiving response from " + serverAddress + ".")
        try
          tcp.readStreamUntilClosed (stream, timeOutTicks/60, @httpResult)
        else
          tcp.abortStream (stream)
          scriptError (tryerror)
      tcp.closeStream (stream)
      if (httpResult != "") and flMessages
        displayMsg ("Finished receiving " + string.kBytes (string.length (httpResult)) + " from " + serverAddress + ".")
  if (httpResult != "") and cookiesOn // find, parse, and store any cookies: ASG 4/9/98
    local (ixEndHeader, responseHeader, i, j)
    ixEndHeader = string.patternMatch ("\r\n\r\n", httpResult) //try to find the end of the response header
    if ixEndHeader > 0 //there is a valid header, look for cookies
      responseHeader = string.mid (httpResult, 1, ixEndHeader - 1) //get the header
      if string.lower (responseHeader) contains "set-cookie:" //the response header contains at least one cookie
        if not defined (adrCookiesTable^)
          new (tableType, adrCookiesTable)
        for i = 1 to string.countFields (responseHeader, "\r") //find all the cookies
          local (onecookie = string.nthField (responseHeader, "\r", i)) //get the cookie
          onecookie = string.popLeading (onecookie, '\n')
          if string.lower (onecookie) beginsWith "set-cookie:"
            local (flsecure = false) //not secure by default
            local (domain = server) //default domain to this server
            local (cookiePath = "/")
            local (expires = clock.now () + (24 * 60 * 60)) //default expires at "end of session"
            local (cookieName, cookieValue)
            if path != "/"
              cookiePath = string.popSuffix (path, "/")
            onecookie = string.popLeading (string.mid (onecookie, 12, infinity), ' ')
            for j = 1 to string.countFields (onecookie, ';')
              local (cookiepart = string.nthField (onecookie, ';', j))
              cookiepart = string.popLeading (cookiepart, ' ')
              case string.lower (string.nthField (cookiepart, '=', 1)) //take care of standard parts
                "domain"
                domain = string.nthField (cookiepart, '=', 2)
                "path"
                cookiePath = string.nthField (cookiepart, '=', 2)
                "expires"
                on cookieDateToSystemDate (cookieDate)
                try //some cookie dates can be converted to dates directly
                local (systemDate = date (cookieDate))
                if systemDate < date ("Monday, 6-Feb-2040")
                return (systemDate)
                try //it may be a date in the 21st century
                local (lastpart = string.nthField (cookieDate, "-", 3))
                local (ixstart = string.patternMatch (lastpart, cookieDate))
                return (date (string.insert ("20", cookieDate, ixstart)))
                return (false)
                expires = cookieDateToSystemDate (string.nthField (cookiepart, '=', 2))
                if expires == false //invalid cookie, ignore it
                break
                "secure"
                flsecure = true
              else //this is the actual cookie name/value
                cookieName = string.nthField (cookiepart, '=', 1)
                cookieValue = string.nthField (cookiepart, '=', 2)
            if clock.now () < expires //store non-expired cookies only
              if not defined (adrCookiesTable^.[domain]) //have we cookies already for this domain?
                new (tableType, @adrCookiesTable^.[domain])
              if not defined (adrCookiesTable^.[domain].[cookiePath])
                new (tableType, @adrCookiesTable^.[domain].[cookiePath])
              if not defined (adrCookiesTable^.[domain].[cookiePath].[cookieName])
                new (tableType, @adrCookiesTable^.[domain].[cookiePath].[cookieName])
              adrCookiesTable^.[domain].[cookiePath].[cookieName].value = cookieValue
              adrCookiesTable^.[domain].[cookiePath].[cookieName].expires = expires
              adrCookiesTable^.[domain].[cookiePath].[cookieName].secure = flsecure
  if debug //save a copy of the server's response for debugging
    wp.newTextObject (httpResult, @scratchpad.httpResult)
  if (ctFollowRedirects > 0) //check whether we are being redirected: AR 6/13/1999
    local (statusCode = string.nthField (httpResult, ' ', 2))
    if ({"301", "302", "303"} contains statusCode) //response codes for Moved Permanently, Moved Temporarily, See Other
      local (flRedirect = false, redirectServer, redirectPath)
      local (t); new (tableType, @t)
      webserver.util.parseHeaders (httpResult, @t)
      if defined (t.location) //the response has a Location: header field
        on isAbsoluteURL (s)
          if not (s contains ":")
            return (false)
          local (scheme = string.lower (string.nthField (s, ":", 1)))
          if scheme == ""
            return (false)
          local (ix)
          for ix = 1 to sizeof (scheme)
            if not string.isAlpha (scheme[ix])
              return (false)
          return (true)
        on isAbsolutePath (s)
          return (s beginsWith "/")
        case true
          isAbsoluteURL (t.location)
            if string.lower (t.location) beginsWith "http://" //it's an absolute HTTP URL
              local (s = string.delete (t.location, 1, string.length ("http://")))
              redirectServer = string.nthField (s, "/", 1)
              redirectPath = string.delete (s, 1, string.length (redirectServer) + 1)
              flRedirect = true
          isAbsolutePath (t.location)
            redirectServer = server
            redirectPath = t.location
            flRedirect = true
        else //assume it is a relative path
          redirectServer = server
          redirectPath = string.popSuffix (path, "/") + "/" + t.location
          flRedirect = true
      if flRedirect
        if flMessages
          displayMsg ("Redirected to " + t.Location + ".")
        if statusCode == "303" //use the GET method to retrieve the specified entity
          method = "GET" //see section 10.3.4 of RFC2068
        if adrRedirectInfo != nil //10/13/02 by DW, accumulate info about the redirect
          local (adrsubtable = @adrRedirectInfo^.[string.padWithZeros (sizeof (adrRedirectInfo^) + 1, 5)])
          new (tableType, adrsubtable)
          adrsubtable^.code = statusCode
          adrsubtable^.server = redirectServer
          adrsubtable^.port = port
          adrsubtable^.path = redirectPath
        httpResult = tcp.httpClient (method, redirectServer, port, redirectPath, proxy, proxyPort, proxyUserName, proxyPassword, data, datatype, username, password, adrHdrTable, cookiesOn, debug, timeOutTicks, flMessages, --ctFollowRedirects, flJustHeaders, flAcceptOpml, flUseProxy, adrRedirectInfo)
  if flMessages //clean up the message area: AR 6/14/1999
    msg ("")
  return (httpResult)
bundle //test code
  tcp.httpclient ("POST", server:"localhost", path:"/uppercase", port:1408, datatype:"text", data:"oh the buzzing", ctFollowRedirects:5, debug:false)