[frontierkernel-commits] odbs/frontierRoot/system/verbs/builtins/tcp httpClient,1.2,1.3

[creedon <icr...@us...>]

Update of /cvsroot/frontierkernel/odbs/frontierRoot/system/verbs/builtins/tcp
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv28176

Modified Files:
	httpClient 
Log Message:
when deleting expired cookies, delete in reverse order

Index: httpClient
===================================================================
RCS file: /cvsroot/frontierkernel/odbs/frontierRoot/system/verbs/builtins/tcp/httpClient,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** httpClient	7 Sep 2005 17:05:55 -0000	1.2
--- httpClient	6 Nov 2005 01:28:51 -0000	1.3
***************
*** 1,3 ****
  FrontierVcsFile:1:scpt:system.verbs.builtins.tcp.httpClient
  
! on httpClient (method="GET", server="", port=nil, path="", proxy=user.webBrowser.proxy.domain, proxyPort=user.webBrowser.proxy.port, proxyUserName=user.webBrowser.proxy.userName, proxyPassword=user.webBrowser.proxy.password, data="", datatype="", username="", password="", adrHdrTable=nil, cookiesOn=false, debug=false, timeOutTicks=60*30, flMessages=true, ctFollowRedirects=0, flJustHeaders=false, flAcceptOpml=false, flUseProxy=nil, adrRedirectInfo=nil) {
	ÇChanges
		Ç5/3/04; 12:14:50 AM by TAC
			Çcookies weren't working for a path that looks like "/mypath"
		Ç2/7/03; 3:49:12 PM by JES
			ÇWhen testing for proxy-exempt servers, don't inclue the port number if specified in the server parameter.
		Ç2002
			Ç10/20/02; 2:32:23 AM by JES
				ÇRespect user.webBrowser.proxy.exceptions.
			Ç10/16/02; 6:55:56 PM by JES
				ÇRemoved debugging code.
			Ç10/14/02; 11:10:16 AM by JES
				ÇIf server is 127.0.0.1 or "localhost", set flUseProxy to false.
			Ç10/13/02; 6:23:28 PM by DW
				ÇNew optional param -- adrRedirectInfo. If it's non-nil (the default) and we're redirecting, we fill the table with info about the redirect. The RSS aggregator needs this information, and it may prove useful in other places. See xml.rss.readService for an example of its usage.
		Ç2001
			Ç12/19/01; 9:58:10 PM by JES
				ÇAdded a call to webBrowser.init, which initializes prefs at user.webBrowser.proxy. Added optional parameter, flUseProxy, which defaults to nil. If nil, then the value at user.webBrowser.proxy.enabled is used. Pass the value of flUseProxy when following redirects.
			Ç8/14/01; 1:49:11 PM by JES
				ÇWhen recursing to following redirects, pass in the flJustHeaders and flAcceptOpml parameters.
			Ç8/9/01; 6:10:17 PM by JES
				ÇIf the stream is unexpectedly closed by the remote host, call tcp.abortStream, to avoid consuming the available connections.
			Ç8/8/01; 3:21:27 PM by JES
				ÇIf no Accept header has been passed in, add an Accept header with the value: text/x-opml, */*, so that we get the response in opml instead of html.
			Ç3/1/01; 1:57:33 PM by PBS
				ÇTrim white space around the content-length header before coercing it to a number. If a server adds white space here, the coercion would fail, resulting in a connection counter leak. (History: this was discovered in Radio when reading the Red Herring channel.)
			Ç1/10/01; 10:36:30 AM by DW
				ÇAdd an optional parameter, flJustHeaders, if true, we don't read the body, we just return the header portion of the response. This is useful if you just want to know how big a download is, and want to defer downloading big things until the middle of the night when it's quiet on the Internet.
			Ç1/7/01; 9:44:18 PM by DW
				ÇIn Radio, set User-Agent to Radio UserLand.
				ÇChanged the header so it doesn't use backslashes. The outliner now supports wrapping headlines.
		Ç9/22/00; 11:27:50 AM by PBS
			ÇRespect the user's offline settings.
		Ç1999
			Ç11/5/99; 3:00:39 AM by AR
				ÇIf the server doesn't send a Content-Length header, read until connection closes.
			Ç10/21/99; 3:57:29 PM by AR
				ÇAdapted to use new TCP verbs in 6.1.
			Ç6/14/99; 5:23:37 AM by AR
				ÇIf flMessages is true, we now clean up the message area before we return.
			Ç6/13/99; 9:20:04 AM by AR
				ÇBoth the read and write loop now yield time to other Frontier threads and to the system.
				ÇThe Host header is now generated correctly if the request goes thru a proxy server.
				ÇFixed a few inconsistencies in the cookie handling code. Thinks should now work correctly when connecting via a proxy server.
				ÇAdded an optional parameter, ctFollowRedirects, determining the number of times that HTTP redirects will be followed. In order to preserve previous behaviour, the default value is 0.
			Ç3/4/99; 11:54:30 AM by PBS
				ÇIf adrHdrTable^ contains a "Host" sub-item, don't add a Host header item.
			Ç3/2/99; 10:43:18 AM by DW
				Çif adrHdrTable^ contains a "User-Agent" sub-item, don't add a User-Agent header item
		Ç1998
			Ç10/25/98; 6:33:59 AM by DW
				Çallow the use of a colon in the server name to specify the port
					Çexample: www.scripting.com:80
				Çif present, we use that port, not the one specified in the call
					Çthe assumption is that most callers either specify 80 or don't specify the port at all
					Çfor crawlers, for example, they definitely want to use the port specified in the server string
					Çsome commands, esp in the webBrowser class, get the url from another piece of software
			Ç9/28/98; 4:42:12 AM by DW
				Çif we're sending to the same machine, set the stream ID to 0
				Çuser.webserver.responders.CGI.methods.any is copying it, hope no CGIs are using it!
			Ç9/17/98; 7:11:56 AM by DW
				Çmsgs now say that they're from tcp.httpClient.
				Çchanged Connection to xxx timed out message to...
					ÇCan't process the request because the connection to xxx timed out.
				Çallow server to be "localhost", if so, we totally optimize the functionality
			Ç2/13/98; 8:57:10 AM by DW
				ÇInitial code assembled from various sample scripts by Dave Winer.
				ÇThe goal is to have a single documented way to talk to HTTP servers.
				ÇIt's got a long parameter list, but requires no configuration, doesn't run off a user.xxx table.
				ÇThe target user of this script is someone who needs to understand how the code works.
				ÇIt will be distributed with Frontier on both platforms as soon as we agree that this is complete and works.
				ÇWhen you call it, use parameter labels, don't depend on the order of these params.
				ÇFor examples, see tcp.examples.
	
	if tcp.isOffline () { //PBS 09/22/00: check offline status
		scriptError ("Can't send HTTP request because TCP is offline.")};
	
	webBrowser.init ();
	try { //set flUseProxy
		if (flUseProxy == nil) and (typeOf (flUseProxy) == unknownType) {
			flUseProxy = user.webBrowser.proxy.enabled};
		if server == "127.0.0.1" or string.lower (server) == "localhost" { //JES 10/14/02: don't call the loopback address through the proxy
			flUseProxy = false};
		if flUseProxy { //check the exceptions table -- if we find a match then set flUseProxy to false
			on wildcardIpMatch (s, oneip) {
				local (i);
				if s == oneip { //optimization
					return (true)};
				for i = 1 to 4 {
					try {
						local (part = string.nthField (oneip, ".", i));
						local (pattern = string.nthField (s, ".", i));
						if pattern == "*" {
							continue};
						if pattern contains "-" {
							local (min = number (string.nthField (pattern, "-", 1)) );
							local (max = number (string.nthField (pattern, "-", 2)) );
							local (n = number (part));
							if n >= min and n <= max {
								continue}
							else {
								return (false)}};
						if pattern != part {
							return (false)}}
					else {
						return (false)}};
				return (true)};
			local (flIpAddress = false, hostToCheck = string.lower (string.nthField (server, ":", 1)) );
			try { //see if flIpAddress should be true
				local (serverWithoutDots = string.replaceAll (server, ".", ""));
				double (serverWithoutDots); //will error if non-numeric characters are present
				flIpAddress = true}
			else { //check domains, and then try a DNS lookup
				local (adr);
				local (lowerserver = string.lower (server));
				for adr in @user.webBrowser.proxy.exceptions {
					if string.wildcardMatch (hostToCheck, string.lower (nameOf (adr^)) ) {
						if adr^ {
							flUseProxy = false;
							break}}};
				if flUseProxy { //try a reverse lookup so we can check against IP address
					try {
						hostToCheck = tcp.dns.getDottedId (server);
						flIpAddress = true}}};
			if flUseProxy { //check to see if this specific IP is excepted
				if defined (user.webBrowser.proxy.exceptions.[hostToCheck]) {
					if user.webBrowser.proxy.exceptions.[hostToCheck] {
						flUseProxy = false}}};
			if flUseProxy { //check for IP-address exceptions which contain wildcard characters and ranges
				local (i);
				for i = 1 to sizeOf (user.webBrowser.proxy.exceptions) {
					if user.webBrowser.proxy.exceptions[i] {
						local (oneexception = nameOf (user.webBrowser.proxy.exceptions[i]));
						if wildcardIpMatch (oneexception, hostToCheck) {
							flUseProxy = false;
							break}}}}}};
	
	on displayMsg (s) {
		msg ("tcp.httpClient: " + s)};
	
	local (myAddress = tcp.myDottedID (), flSameMachine = false);
	bundle { //look for the port specified in the server string
		if server contains ":" {
			try {port = number (string.nthField (server, ":", 2))}; //it might not be a number
			server = string.nthField (server, ":", 1)};
		if port == nil { //no port was specified, either in the call or in the server string
			port = 80}};
	bundle { //see if it's the same machine
		if string.lower (server) == "localhost" {
			flSameMachine = true}
		else {
			try {flSameMachine = tcp.equalNames (server, myAddress)}}; //DW 3/9/98
		
		ÇPBS 9/18/98: check that the server on the requested port is Frontier.
		if port != user.inetd.config.http.port {
			flSameMachine = false} //it's the same machine, but it's not Frontier as a server
		else {
			if not (inetd.isDaemonRunning (@user.inetd.config.http)) {
				flSameMachine = false}}}; //Frontier isn't running as a webserver, there must be another server on this machine
	
	if not (path beginsWith "/") {
		path = "/" + path};
	
	local (httpCommand = "", adrCookiesTable = @user.webBrowser.cookies);
	bundle { //build the HTTP request in "httpCommand"
		on add (s) {
			httpCommand = httpCommand + s + "\r\n"};
		local (uri = path);
		if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server: set uri accordingly
			uri = "http://" + server;
			if port != 80 {
				uri = uri + ":" + port};
			uri = uri + path};
		add (method + " " + uri + " HTTP/1.0");
		if (adrHdrTable == nil) or (not defined (adrHdrTable^.["User-Agent"])) { //3/2/99; 10:43:10 AM by DW
			if system.environment.isPike {
				add ("User-Agent: Radio UserLand/" + Frontier.version () + " (" + sys.os () + ")")}
			else {
				add ("User-Agent: Frontier/" + Frontier.version () + " (" + sys.os () + ")")}};
		local (host = "Host: " + server);
		if port != 80 {
			host = host + ":" + port};
		if (adrHdrTable == nil) or (not defined (adrHdrTable^.["Host"])) { //3/4/99; 11:54:30 AM by PBS
			add (host)};
		if flAcceptOpml { //8/8/01; 3:19:50 PM by JES
			if (adrHdrTable == nil) {
				add ("Accept: text/x-opml, */*")}
			else { //headers were passed in
				if defined (adrHdrTable^.Accept) {
					local (acceptHeader = adrHdrTable^.Accept);
					if not (acceptHeader contains "text/x-opml") {
						if sizeOf (string.trimWhiteSpace (acceptHeader)) > 0 {
							acceptHeader = "text/x-opml, " + acceptHeader}
						else {
							acceptHeader = "text/x-opml"}};
					if not (acceptHeader contains "*/*") {
						acceptHeader = acceptHeader + ", */*"};
					add ("Accept: " + acceptHeader)}
				else {
					add ("Accept: text/x-opml, */*")}}};
		if sizeOf (data) > 0 {
			add ("Content-Type: " + datatype);
			add ("Content-length: " + sizeOf (data))};
		if (sizeOf (username) > 0) or (sizeOf (password) > 0) {
			add ("Authorization: Basic " + base64.encode (username + ":" + password, 0))};
		if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server
			if (sizeOf (proxyUsername) > 0) or (sizeOf (proxyPassword) > 0) { //the proxy requires authentication
				add ("Proxy-Authorization: Basic " + base64.encode (proxyUsername + ":" + proxyPassword, 0))}};
		if adrHdrTable != nil { //added 3.27.98 by ASG to include header items
			local (i);
			for i = 1 to sizeOf (adrHdrTable^) {
				local (headerName = nameOf (adrHdrTable^[i]));
				if flAcceptOpml { //8/8/01; 3:19:50 PM by JES: Only add the Acept header once
					if string.lower (headerName) == "accept" {
						continue}};
				add (headerName + ": " + adrHdrTable^[i])}};
		
		if cookiesOn and defined (adrCookiesTable^) { //add cookies to the request header: ASG 4/9/98
			local (i, j, k, cookiePath = "/", cookiestring);
			if path != "/" {
				cookiePath = string.popSuffix (path, "/");
				if cookiePath == "" { // 5/3/04; 12:14:50 AM by TAC - cookies weren't working for a path that looks like "/mypath"
					cookiePath = "/"}};
			for i = 1 to sizeOf (adrCookiesTable^) { //loop through cookies table
				local (adrdomain = @adrCookiesTable^ [i]);
				if host endsWith nameOf (adrdomain^) {
					for j = 1 to sizeOf (adrdomain^) {
						local (adrcookie = @adrdomain^ [j]);
						if cookiePath contains nameOf (adrcookie^) {
							for k = 1 to sizeOf (adrcookie^) {
								if adrcookie^ [k].expires > clock.now () { //add this cookie
									cookiestring = cookiestring + nameOf (adrcookie^ [k]) + "=" + adrcookie^ [k].value + "; "}
								else {
									delete (@adrcookie^ [k])}}}}}}; //this cookie has expired so delete it
			if cookiestring != nil {
				cookiestring = string.delete (cookiestring, sizeof (cookiestring) - 1, 2); //delete trailing "; "
				add ("Cookie: " + cookiestring)}}; //add the cookies to the header
		
		add (""); //blank line
		if sizeOf (data) > 0 {
			httpCommand = httpCommand + data}};
	
	if debug { //save a copy of the HTTP request for debugging
		wp.newTextObject (httpCommand, @scratchpad.httpCommand)};
	
	local (httpResult = "");
	if flSameMachine { //shortcut the call, turn our worst case into our best case: 9/17/98; 7:21:13 AM by DW
		local (params);
		new (tabletype, @params);
		params.client = myAddress;
		params.port = port;
		params.stream = 0; //9/28/98; 4:42:45 AM by DW
		params.request = httpCommand;
		httpResult = webserver.server (@params, httpCommand)}
	else {
		local (serverAddress = server, serverPort = port);
		if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server: switch server and port
			serverAddress = proxy;
			serverPort = proxyPort};
		
		if flMessages { //PBS 5/5/98
			displayMsg ("Connecting to " + serverAddress + ".")};
		
		local (stream = tcp.openStream (serverAddress, serverPort));
		local (chunksize = 5 * 1024);
		
		if flMessages {
			displayMsg ("Sending request to " + serverAddress + ".")};
		
		try {
			tcp.writeStringToStream (stream, httpCommand, chunkSize, timeOutTicks/60)} //write the command to the server
		else {
			tcp.abortStream (stream);
			scriptError (tryError)};
		
		if flMessages {
			displayMsg ("Waiting for data from " + serverAddress + ".")};
		
		local (pat = "\r\n\r\n");
		
		try {
			tcp.readStreamUntil (stream, pat, timeOutTicks/60, @httpResult)}
		else {
			tcp.abortStream (stream);
			scriptError (tryerror)};
		
		local (headerTable);
		webserver.util.parseHeaders (httpResult, @headerTable);
		
		if flJustHeaders { //the caller only wants the headers, not the body 1/10/01 by DW
			tcp.closeStream (stream)}
		else {
			if (string.upper (method) != "HEAD") and defined (headerTable.["Content-Length"]) { //we know there's content left to be read
				local (lenHeader = string.patternMatch (pat, httpResult) - 1);
				local (lenContent = number (string.trimWhiteSpace (headerTable.["Content-Length"]))); //PBS 03/01/01: trim white space before coercing
				local (lenTotal = lenHeader + string.length (pat) + lenContent);
				
				if flMessages {
					displayMsg ("Receiving response body  (" + string.kBytes (lenContent) + ") from " + serverAddress + ".")};
				try {
					tcp.readStreamBytes (stream, lenTotal, timeOutTicks/60, @httpResult)}
				else {
					tcp.abortStream (stream);
					scriptError (tryerror)}}
			else { //just read until connection closes
				if flMessages {
					displayMsg ("Receiving response from " + serverAddress + ".")};
				try {
					tcp.readStreamUntilClosed (stream, timeOutTicks/60, @httpResult)}
				else {
					tcp.abortStream (stream);
					scriptError (tryerror)}};
			
			tcp.closeStream (stream);
			if (httpResult != "") and flMessages {
				displayMsg ("Finished receiving " + string.kBytes (string.length (httpResult)) + " from " + serverAddress + ".")}}};
	
	if (httpResult != "") and cookiesOn { // find, parse, and store any cookies: ASG 4/9/98
		local (ixEndHeader, responseHeader, i, j);
		ixEndHeader = string.patternMatch ("\r\n\r\n", httpResult); //try to find the end of the response header
		if ixEndHeader > 0 { //there is a valid header, look for cookies
			responseHeader = string.mid (httpResult, 1, ixEndHeader - 1); //get the header
			if string.lower (responseHeader) contains "set-cookie:" { //the response header contains at least one cookie
				if not defined (adrCookiesTable^) {
					new (tableType, adrCookiesTable)};
				for i = 1 to string.countFields (responseHeader, "\r") { //find all the cookies
					local (onecookie = string.nthField (responseHeader, "\r", i)); //get the cookie
					onecookie = string.popLeading (onecookie, '\n');
					if string.lower (onecookie) beginsWith "set-cookie:" {
						local (flsecure = false); //not secure by default
						local (domain = server); //default domain to this server
						local (cookiePath = "/");
						local (expires = clock.now () + (24 * 60 * 60)); //default expires at "end of session"
						local (cookieName, cookieValue);
						if path != "/" {
							cookiePath = string.popSuffix (path, "/")};
						onecookie = string.popLeading (string.mid (onecookie, 12, infinity), ' ');
						for j = 1 to string.countFields (onecookie, ';') {
							local (cookiepart = string.nthField (onecookie, ';', j));
							cookiepart = string.popLeading (cookiepart, ' ');
							case string.lower (string.nthField (cookiepart, '=', 1)) { //take care of standard parts
								"domain" {
									domain = string.nthField (cookiepart, '=', 2)};
								"path" {
									cookiePath = string.nthField (cookiepart, '=', 2)};
								"expires" {
									on cookieDateToSystemDate (cookieDate) {
										try { //some cookie dates can be converted to dates directly
											local (systemDate = date (cookieDate));
											if systemDate < date ("Monday, 6-Feb-2040") {
												return (systemDate)}};
										try { //it may be a date in the 21st century
											local (lastpart = string.nthField (cookieDate, "-", 3));
											local (ixstart = string.patternMatch (lastpart, cookieDate));
											return (date (string.insert ("20", cookieDate, ixstart)))};
										return (false)};
									expires = cookieDateToSystemDate (string.nthField (cookiepart, '=', 2));
									if expires == false { //invalid cookie, ignore it
										break}};
								"secure" {
									flsecure = true}}
							else { //this is the actual cookie name/value
								cookieName = string.nthField (cookiepart, '=', 1);
								cookieValue = string.nthField (cookiepart, '=', 2)}};
						if clock.now () < expires { //store non-expired cookies only
							if not defined (adrCookiesTable^.[domain]) { //have we cookies already for this domain?
								new (tableType, @adrCookiesTable^.[domain])};
							if not defined (adrCookiesTable^.[domain].[cookiePath]) {
								new (tableType, @adrCookiesTable^.[domain].[cookiePath])};
							if not defined (adrCookiesTable^.[domain].[cookiePath].[cookieName]) {
								new (tableType, @adrCookiesTable^.[domain].[cookiePath].[cookieName])};
							adrCookiesTable^.[domain].[cookiePath].[cookieName].value = cookieValue;
							adrCookiesTable^.[domain].[cookiePath].[cookieName].expires = expires;
							adrCookiesTable^.[domain].[cookiePath].[cookieName].secure = flsecure}}}}}};
	
	if debug { //save a copy of the server's response for debugging
		wp.newTextObject (httpResult, @scratchpad.httpResult)};
	
	if (ctFollowRedirects > 0) { //check whether we are being redirected: AR 6/13/1999
		local (statusCode = string.nthField (httpResult, ' ', 2));
		if ({"301", "302", "303"} contains statusCode) { //response codes for Moved Permanently, Moved Temporarily, See Other
			local (flRedirect = false, redirectServer, redirectPath);
			local (t); new (tableType, @t);
			webserver.util.parseHeaders (httpResult, @t);
			if defined (t.location) { //the response has a Location: header field
				on isAbsoluteURL (s) {
					if not (s contains ":") {
						return (false)};
					local (scheme = string.lower (string.nthField (s, ":", 1)));
					if scheme == "" {
						return (false)};
					local (ix);
					for ix = 1 to sizeof (scheme) {
						if not string.isAlpha (scheme[ix]) {
							return (false)}};
					return (true)};
				on isAbsolutePath (s) {
					return (s beginsWith "/")};
				case true {
					isAbsoluteURL (t.location) {
						if string.lower (t.location) beginsWith "http://" { //it's an absolute HTTP URL
							local (s = string.delete (t.location, 1, string.length ("http://")));
							redirectServer = string.nthField (s, "/", 1);
							redirectPath = string.delete (s, 1, string.length (redirectServer) + 1);
							flRedirect = true}};
					isAbsolutePath (t.location) {
						redirectServer = server;
						redirectPath = t.location;
						flRedirect = true}}
				else { //assume it is a relative path
					redirectServer = server;
					redirectPath = string.popSuffix (path, "/") + "/" + t.location;
					flRedirect = true}};
			if flRedirect {
				if flMessages {
					displayMsg ("Redirected to " + t.Location + ".")};
				if statusCode == "303" { //use the GET method to retrieve the specified entity
					method = "GET"}; //see section 10.3.4 of RFC2068
				
				if adrRedirectInfo != nil { //10/13/02 by DW, accumulate info about the redirect
					local (adrsubtable = @adrRedirectInfo^.[string.padWithZeros (sizeof (adrRedirectInfo^) + 1, 5)]);
					new (tableType, adrsubtable);
					adrsubtable^.code = statusCode;
					adrsubtable^.server = redirectServer;
					adrsubtable^.port = port;
					adrsubtable^.path = redirectPath};
				
				httpResult = tcp.httpClient (method, redirectServer, port, redirectPath, proxy, proxyPort, proxyUserName, proxyPassword, data, datatype, username, password, adrHdrTable, cookiesOn, debug, timeOutTicks, flMessages, --ctFollowRedirects, flJustHeaders, flAcceptOpml, flUseProxy, adrRedirectInfo)}}};
	
	if flMessages { //clean up the message area: AR 6/14/1999
		msg ("")};
	
	return (httpResult)}
\ No newline at end of file
--- 1,3 ----
  FrontierVcsFile:1:scpt:system.verbs.builtins.tcp.httpClient
  
! on httpClient (method="GET", server="", port=nil, path="", proxy=user.webBrowser.proxy.domain, proxyPort=user.webBrowser.proxy.port, proxyUserName=user.webBrowser.proxy.userName, proxyPassword=user.webBrowser.proxy.password, data="", datatype="", username="", password="", adrHdrTable=nil, cookiesOn=false, debug=false, timeOutTicks=60*30, flMessages=true, ctFollowRedirects=0, flJustHeaders=false, flAcceptOpml=false, flUseProxy=nil, adrRedirectInfo=nil) {
	ÇChanges
		Ç11/2/05; 9:37:38 AM by TAC
			Çwhen deleting expired cookies, delete in reverse order
		Ç5/3/04; 12:14:50 AM by TAC
			Çcookies weren't working for a path that looks like "/mypath"
		Ç2/7/03; 3:49:12 PM by JES
			ÇWhen testing for proxy-exempt servers, don't inclue the port number if specified in the server parameter.
		Ç2002
			Ç10/20/02; 2:32:23 AM by JES
				ÇRespect user.webBrowser.proxy.exceptions.
			Ç10/16/02; 6:55:56 PM by JES
				ÇRemoved debugging code.
			Ç10/14/02; 11:10:16 AM by JES
				ÇIf server is 127.0.0.1 or "localhost", set flUseProxy to false.
			Ç10/13/02; 6:23:28 PM by DW
				ÇNew optional param -- adrRedirectInfo. If it's non-nil (the default) and we're redirecting, we fill the table with info about the redirect. The RSS aggregator needs this information, and it may prove useful in other places. See xml.rss.readService for an example of its usage.
		Ç2001
			Ç12/19/01; 9:58:10 PM by JES
				ÇAdded a call to webBrowser.init, which initializes prefs at user.webBrowser.proxy. Added optional parameter, flUseProxy, which defaults to nil. If nil, then the value at user.webBrowser.proxy.enabled is used. Pass the value of flUseProxy when following redirects.
			Ç8/14/01; 1:49:11 PM by JES
				ÇWhen recursing to following redirects, pass in the flJustHeaders and flAcceptOpml parameters.
			Ç8/9/01; 6:10:17 PM by JES
				ÇIf the stream is unexpectedly closed by the remote host, call tcp.abortStream, to avoid consuming the available connections.
			Ç8/8/01; 3:21:27 PM by JES
				ÇIf no Accept header has been passed in, add an Accept header with the value: text/x-opml, */*, so that we get the response in opml instead of html.
			Ç3/1/01; 1:57:33 PM by PBS
				ÇTrim white space around the content-length header before coercing it to a number. If a server adds white space here, the coercion would fail, resulting in a connection counter leak. (History: this was discovered in Radio when reading the Red Herring channel.)
			Ç1/10/01; 10:36:30 AM by DW
				ÇAdd an optional parameter, flJustHeaders, if true, we don't read the body, we just return the header portion of the response. This is useful if you just want to know how big a download is, and want to defer downloading big things until the middle of the night when it's quiet on the Internet.
			Ç1/7/01; 9:44:18 PM by DW
				ÇIn Radio, set User-Agent to Radio UserLand.
				ÇChanged the header so it doesn't use backslashes. The outliner now supports wrapping headlines.
		Ç9/22/00; 11:27:50 AM by PBS
			ÇRespect the user's offline settings.
		Ç1999
			Ç11/5/99; 3:00:39 AM by AR
				ÇIf the server doesn't send a Content-Length header, read until connection closes.
			Ç10/21/99; 3:57:29 PM by AR
				ÇAdapted to use new TCP verbs in 6.1.
			Ç6/14/99; 5:23:37 AM by AR
				ÇIf flMessages is true, we now clean up the message area before we return.
			Ç6/13/99; 9:20:04 AM by AR
				ÇBoth the read and write loop now yield time to other Frontier threads and to the system.
				ÇThe Host header is now generated correctly if the request goes thru a proxy server.
				ÇFixed a few inconsistencies in the cookie handling code. Thinks should now work correctly when connecting via a proxy server.
				ÇAdded an optional parameter, ctFollowRedirects, determining the number of times that HTTP redirects will be followed. In order to preserve previous behaviour, the default value is 0.
			Ç3/4/99; 11:54:30 AM by PBS
				ÇIf adrHdrTable^ contains a "Host" sub-item, don't add a Host header item.
			Ç3/2/99; 10:43:18 AM by DW
				Çif adrHdrTable^ contains a "User-Agent" sub-item, don't add a User-Agent header item
		Ç1998
			Ç10/25/98; 6:33:59 AM by DW
				Çallow the use of a colon in the server name to specify the port
					Çexample: www.scripting.com:80
				Çif present, we use that port, not the one specified in the call
					Çthe assumption is that most callers either specify 80 or don't specify the port at all
					Çfor crawlers, for example, they definitely want to use the port specified in the server string
					Çsome commands, esp in the webBrowser class, get the url from another piece of software
			Ç9/28/98; 4:42:12 AM by DW
				Çif we're sending to the same machine, set the stream ID to 0
				Çuser.webserver.responders.CGI.methods.any is copying it, hope no CGIs are using it!
			Ç9/17/98; 7:11:56 AM by DW
				Çmsgs now say that they're from tcp.httpClient.
				Çchanged Connection to xxx timed out message to...
					ÇCan't process the request because the connection to xxx timed out.
				Çallow server to be "localhost", if so, we totally optimize the functionality
			Ç2/13/98; 8:57:10 AM by DW
				ÇInitial code assembled from various sample scripts by Dave Winer.
				ÇThe goal is to have a single documented way to talk to HTTP servers.
				ÇIt's got a long parameter list, but requires no configuration, doesn't run off a user.xxx table.
				ÇThe target user of this script is someone who needs to understand how the code works.
				ÇIt will be distributed with Frontier on both platforms as soon as we agree that this is complete and works.
				ÇWhen you call it, use parameter labels, don't depend on the order of these params.
				ÇFor examples, see tcp.examples.
	
	if tcp.isOffline () { //PBS 09/22/00: check offline status
		scriptError ("Can't send HTTP request because TCP is offline.")};
	
	webBrowser.init ();
	try { //set flUseProxy
		if (flUseProxy == nil) and (typeOf (flUseProxy) == unknownType) {
			flUseProxy = user.webBrowser.proxy.enabled};
		if server == "127.0.0.1" or string.lower (server) == "localhost" { //JES 10/14/02: don't call the loopback address through the proxy
			flUseProxy = false};
		if flUseProxy { //check the exceptions table -- if we find a match then set flUseProxy to false
			on wildcardIpMatch (s, oneip) {
				local (i);
				if s == oneip { //optimization
					return (true)};
				for i = 1 to 4 {
					try {
						local (part = string.nthField (oneip, ".", i));
						local (pattern = string.nthField (s, ".", i));
						if pattern == "*" {
							continue};
						if pattern contains "-" {
							local (min = number (string.nthField (pattern, "-", 1)) );
							local (max = number (string.nthField (pattern, "-", 2)) );
							local (n = number (part));
							if n >= min and n <= max {
								continue}
							else {
								return (false)}};
						if pattern != part {
							return (false)}}
					else {
						return (false)}};
				return (true)};
			local (flIpAddress = false, hostToCheck = string.lower (string.nthField (server, ":", 1)) );
			try { //see if flIpAddress should be true
				local (serverWithoutDots = string.replaceAll (server, ".", ""));
				double (serverWithoutDots); //will error if non-numeric characters are present
				flIpAddress = true}
			else { //check domains, and then try a DNS lookup
				local (adr);
				local (lowerserver = string.lower (server));
				for adr in @user.webBrowser.proxy.exceptions {
					if string.wildcardMatch (hostToCheck, string.lower (nameOf (adr^)) ) {
						if adr^ {
							flUseProxy = false;
							break}}};
				if flUseProxy { //try a reverse lookup so we can check against IP address
					try {
						hostToCheck = tcp.dns.getDottedId (server);
						flIpAddress = true}}};
			if flUseProxy { //check to see if this specific IP is excepted
				if defined (user.webBrowser.proxy.exceptions.[hostToCheck]) {
					if user.webBrowser.proxy.exceptions.[hostToCheck] {
						flUseProxy = false}}};
			if flUseProxy { //check for IP-address exceptions which contain wildcard characters and ranges
				local (i);
				for i = 1 to sizeOf (user.webBrowser.proxy.exceptions) {
					if user.webBrowser.proxy.exceptions[i] {
						local (oneexception = nameOf (user.webBrowser.proxy.exceptions[i]));
						if wildcardIpMatch (oneexception, hostToCheck) {
							flUseProxy = false;
							break}}}}}};
	
	on displayMsg (s) {
		msg ("tcp.httpClient: " + s)};
	
	local (myAddress = tcp.myDottedID (), flSameMachine = false);
	bundle { //look for the port specified in the server string
		if server contains ":" {
			try {port = number (string.nthField (server, ":", 2))}; //it might not be a number
			server = string.nthField (server, ":", 1)};
		if port == nil { //no port was specified, either in the call or in the server string
			port = 80}};
	bundle { //see if it's the same machine
		if string.lower (server) == "localhost" {
			flSameMachine = true}
		else {
			try {flSameMachine = tcp.equalNames (server, myAddress)}}; //DW 3/9/98
		
		ÇPBS 9/18/98: check that the server on the requested port is Frontier.
		if port != user.inetd.config.http.port {
			flSameMachine = false} //it's the same machine, but it's not Frontier as a server
		else {
			if not (inetd.isDaemonRunning (@user.inetd.config.http)) {
				flSameMachine = false}}}; //Frontier isn't running as a webserver, there must be another server on this machine
	
	if not (path beginsWith "/") {
		path = "/" + path};
	
	local (httpCommand = "", adrCookiesTable = @user.webBrowser.cookies);
	bundle { //build the HTTP request in "httpCommand"
		on add (s) {
			httpCommand = httpCommand + s + "\r\n"};
		local (uri = path);
		if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server: set uri accordingly
			uri = "http://" + server;
			if port != 80 {
				uri = uri + ":" + port};
			uri = uri + path};
		add (method + " " + uri + " HTTP/1.0");
		if (adrHdrTable == nil) or (not defined (adrHdrTable^.["User-Agent"])) { //3/2/99; 10:43:10 AM by DW
			if system.environment.isPike {
				add ("User-Agent: Radio UserLand/" + Frontier.version () + " (" + sys.os () + ")")}
			else {
				add ("User-Agent: Frontier/" + Frontier.version () + " (" + sys.os () + ")")}};
		local (host = "Host: " + server);
		if port != 80 {
			host = host + ":" + port};
		if (adrHdrTable == nil) or (not defined (adrHdrTable^.["Host"])) { //3/4/99; 11:54:30 AM by PBS
			add (host)};
		if flAcceptOpml { //8/8/01; 3:19:50 PM by JES
			if (adrHdrTable == nil) {
				add ("Accept: text/x-opml, */*")}
			else { //headers were passed in
				if defined (adrHdrTable^.Accept) {
					local (acceptHeader = adrHdrTable^.Accept);
					if not (acceptHeader contains "text/x-opml") {
						if sizeOf (string.trimWhiteSpace (acceptHeader)) > 0 {
							acceptHeader = "text/x-opml, " + acceptHeader}
						else {
							acceptHeader = "text/x-opml"}};
					if not (acceptHeader contains "*/*") {
						acceptHeader = acceptHeader + ", */*"};
					add ("Accept: " + acceptHeader)}
				else {
					add ("Accept: text/x-opml, */*")}}};
		if sizeOf (data) > 0 {
			add ("Content-Type: " + datatype);
			add ("Content-length: " + sizeOf (data))};
		if (sizeOf (username) > 0) or (sizeOf (password) > 0) {
			add ("Authorization: Basic " + base64.encode (username + ":" + password, 0))};
		if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server
			if (sizeOf (proxyUsername) > 0) or (sizeOf (proxyPassword) > 0) { //the proxy requires authentication
				add ("Proxy-Authorization: Basic " + base64.encode (proxyUsername + ":" + proxyPassword, 0))}};
		if adrHdrTable != nil { //added 3.27.98 by ASG to include header items
			local (i);
			for i = 1 to sizeOf (adrHdrTable^) {
				local (headerName = nameOf (adrHdrTable^[i]));
				if flAcceptOpml { //8/8/01; 3:19:50 PM by JES: Only add the Acept header once
					if string.lower (headerName) == "accept" {
						continue}};
				add (headerName + ": " + adrHdrTable^[i])}};
		
		if cookiesOn and defined (adrCookiesTable^) { //add cookies to the request header: ASG 4/9/98
			local (i, j, k, cookiePath = "/", cookiestring);
			if path != "/" {
				cookiePath = string.popSuffix (path, "/");
				if cookiePath == "" { // 5/3/04; 12:14:50 AM by TAC - cookies weren't working for a path that looks like "/mypath"
					cookiePath = "/"}};
			for i = 1 to sizeOf (adrCookiesTable^) { //loop through cookies table
				local (adrdomain = @adrCookiesTable^ [i]);
				if host endsWith nameOf (adrdomain^) {
					for j = 1 to sizeOf (adrdomain^) {
						local (adrcookie = @adrdomain^ [j]);
						if cookiePath contains nameOf (adrcookie^) {
							for k = sizeOf (adrcookie^) downTo 1 { // 11/2/05; 9:37:38 AM by TAC
								if adrcookie^ [k].expires > clock.now () { //add this cookie
									cookiestring = cookiestring + nameOf (adrcookie^ [k]) + "=" + adrcookie^ [k].value + "; "}
								else {
									delete (@adrcookie^ [k])}}}}}}; //this cookie has expired so delete it
			if cookiestring != nil {
				cookiestring = string.delete (cookiestring, sizeof (cookiestring) - 1, 2); //delete trailing "; "
				add ("Cookie: " + cookiestring)}}; //add the cookies to the header
		
		add (""); //blank line
		if sizeOf (data) > 0 {
			httpCommand = httpCommand + data}};
	
	if debug { //save a copy of the HTTP request for debugging
		wp.newTextObject (httpCommand, @scratchpad.httpCommand)};
	
	local (httpResult = "");
	if flSameMachine { //shortcut the call, turn our worst case into our best case: 9/17/98; 7:21:13 AM by DW
		local (params);
		new (tabletype, @params);
		params.client = myAddress;
		params.port = port;
		params.stream = 0; //9/28/98; 4:42:45 AM by DW
		params.request = httpCommand;
		httpResult = webserver.server (@params, httpCommand)}
	else {
		local (serverAddress = server, serverPort = port);
		if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server: switch server and port
			serverAddress = proxy;
			serverPort = proxyPort};
		
		if flMessages { //PBS 5/5/98
			displayMsg ("Connecting to " + serverAddress + ".")};
		
		local (stream = tcp.openStream (serverAddress, serverPort));
		local (chunksize = 5 * 1024);
		
		if flMessages {
			displayMsg ("Sending request to " + serverAddress + ".")};
		
		try {
			tcp.writeStringToStream (stream, httpCommand, chunkSize, timeOutTicks/60)} //write the command to the server
		else {
			tcp.abortStream (stream);
			scriptError (tryError)};
		
		if flMessages {
			displayMsg ("Waiting for data from " + serverAddress + ".")};
		
		local (pat = "\r\n\r\n");
		
		try {
			tcp.readStreamUntil (stream, pat, timeOutTicks/60, @httpResult)}
		else {
			tcp.abortStream (stream);
			scriptError (tryerror)};
		
		local (headerTable);
		webserver.util.parseHeaders (httpResult, @headerTable);
		
		if flJustHeaders { //the caller only wants the headers, not the body 1/10/01 by DW
			tcp.closeStream (stream)}
		else {
			if (string.upper (method) != "HEAD") and defined (headerTable.["Content-Length"]) { //we know there's content left to be read
				local (lenHeader = string.patternMatch (pat, httpResult) - 1);
				local (lenContent = number (string.trimWhiteSpace (headerTable.["Content-Length"]))); //PBS 03/01/01: trim white space before coercing
				local (lenTotal = lenHeader + string.length (pat) + lenContent);
				
				if flMessages {
					displayMsg ("Receiving response body  (" + string.kBytes (lenContent) + ") from " + serverAddress + ".")};
				try {
					tcp.readStreamBytes (stream, lenTotal, timeOutTicks/60, @httpResult)}
				else {
					tcp.abortStream (stream);
					scriptError (tryerror)}}
			else { //just read until connection closes
				if flMessages {
					displayMsg ("Receiving response from " + serverAddress + ".")};
				try {
					tcp.readStreamUntilClosed (stream, timeOutTicks/60, @httpResult)}
				else {
					tcp.abortStream (stream);
					scriptError (tryerror)}};
			
			tcp.closeStream (stream);
			if (httpResult != "") and flMessages {
				displayMsg ("Finished receiving " + string.kBytes (string.length (httpResult)) + " from " + serverAddress + ".")}}};
	
	if (httpResult != "") and cookiesOn { // find, parse, and store any cookies: ASG 4/9/98
		local (ixEndHeader, responseHeader, i, j);
		ixEndHeader = string.patternMatch ("\r\n\r\n", httpResult); //try to find the end of the response header
		if ixEndHeader > 0 { //there is a valid header, look for cookies
			responseHeader = string.mid (httpResult, 1, ixEndHeader - 1); //get the header
			if string.lower (responseHeader) contains "set-cookie:" { //the response header contains at least one cookie
				if not defined (adrCookiesTable^) {
					new (tableType, adrCookiesTable)};
				for i = 1 to string.countFields (responseHeader, "\r") { //find all the cookies
					local (onecookie = string.nthField (responseHeader, "\r", i)); //get the cookie
					onecookie = string.popLeading (onecookie, '\n');
					if string.lower (onecookie) beginsWith "set-cookie:" {
						local (flsecure = false); //not secure by default
						local (domain = server); //default domain to this server
						local (cookiePath = "/");
						local (expires = clock.now () + (24 * 60 * 60)); //default expires at "end of session"
						local (cookieName, cookieValue);
						if path != "/" {
							cookiePath = string.popSuffix (path, "/")};
						onecookie = string.popLeading (string.mid (onecookie, 12, infinity), ' ');
						for j = 1 to string.countFields (onecookie, ';') {
							local (cookiepart = string.nthField (onecookie, ';', j));
							cookiepart = string.popLeading (cookiepart, ' ');
							case string.lower (string.nthField (cookiepart, '=', 1)) { //take care of standard parts
								"domain" {
									domain = string.nthField (cookiepart, '=', 2)};
								"path" {
									cookiePath = string.nthField (cookiepart, '=', 2)};
								"expires" {
									on cookieDateToSystemDate (cookieDate) {
										try { //some cookie dates can be converted to dates directly
											local (systemDate = date (cookieDate));
											if systemDate < date ("Monday, 6-Feb-2040") {
												return (systemDate)}};
										try { //it may be a date in the 21st century
											local (lastpart = string.nthField (cookieDate, "-", 3));
											local (ixstart = string.patternMatch (lastpart, cookieDate));
											return (date (string.insert ("20", cookieDate, ixstart)))};
										return (false)};
									expires = cookieDateToSystemDate (string.nthField (cookiepart, '=', 2));
									if expires == false { //invalid cookie, ignore it
										break}};
								"secure" {
									flsecure = true}}
							else { //this is the actual cookie name/value
								cookieName = string.nthField (cookiepart, '=', 1);
								cookieValue = string.nthField (cookiepart, '=', 2)}};
						if clock.now () < expires { //store non-expired cookies only
							if not defined (adrCookiesTable^.[domain]) { //have we cookies already for this domain?
								new (tableType, @adrCookiesTable^.[domain])};
							if not defined (adrCookiesTable^.[domain].[cookiePath]) {
								new (tableType, @adrCookiesTable^.[domain].[cookiePath])};
							if not defined (adrCookiesTable^.[domain].[cookiePath].[cookieName]) {
								new (tableType, @adrCookiesTable^.[domain].[cookiePath].[cookieName])};
							adrCookiesTable^.[domain].[cookiePath].[cookieName].value = cookieValue;
							adrCookiesTable^.[domain].[cookiePath].[cookieName].expires = expires;
							adrCookiesTable^.[domain].[cookiePath].[cookieName].secure = flsecure}}}}}};
	
	if debug { //save a copy of the server's response for debugging
		wp.newTextObject (httpResult, @scratchpad.httpResult)};
	
	if (ctFollowRedirects > 0) { //check whether we are being redirected: AR 6/13/1999
		local (statusCode = string.nthField (httpResult, ' ', 2));
		if ({"301", "302", "303"} contains statusCode) { //response codes for Moved Permanently, Moved Temporarily, See Other
			local (flRedirect = false, redirectServer, redirectPath);
			local (t); new (tableType, @t);
			webserver.util.parseHeaders (httpResult, @t);
			if defined (t.location) { //the response has a Location: header field
				on isAbsoluteURL (s) {
					if not (s contains ":") {
						return (false)};
					local (scheme = string.lower (string.nthField (s, ":", 1)));
					if scheme == "" {
						return (false)};
					local (ix);
					for ix = 1 to sizeof (scheme) {
						if not string.isAlpha (scheme[ix]) {
							return (false)}};
					return (true)};
				on isAbsolutePath (s) {
					return (s beginsWith "/")};
				case true {
					isAbsoluteURL (t.location) {
						if string.lower (t.location) beginsWith "http://" { //it's an absolute HTTP URL
							local (s = string.delete (t.location, 1, string.length ("http://")));
							redirectServer = string.nthField (s, "/", 1);
							redirectPath = string.delete (s, 1, string.length (redirectServer) + 1);
							flRedirect = true}};
					isAbsolutePath (t.location) {
						redirectServer = server;
						redirectPath = t.location;
						flRedirect = true}}
				else { //assume it is a relative path
					redirectServer = server;
					redirectPath = string.popSuffix (path, "/") + "/" + t.location;
					flRedirect = true}};
			if flRedirect {
				if flMessages {
					displayMsg ("Redirected to " + t.Location + ".")};
				if statusCode == "303" { //use the GET method to retrieve the specified entity
					method = "GET"}; //see section 10.3.4 of RFC2068
				
				if adrRedirectInfo != nil { //10/13/02 by DW, accumulate info about the redirect
					local (adrsubtable = @adrRedirectInfo^.[string.padWithZeros (sizeof (adrRedirectInfo^) + 1, 5)]);
					new (tableType, adrsubtable);
					adrsubtable^.code = statusCode;
					adrsubtable^.server = redirectServer;
					adrsubtable^.port = port;
					adrsubtable^.path = redirectPath};
				
				httpResult = tcp.httpClient (method, redirectServer, port, redirectPath, proxy, proxyPort, proxyUserName, proxyPassword, data, datatype, username, password, adrHdrTable, cookiesOn, debug, timeOutTicks, flMessages, --ctFollowRedirects, flJustHeaders, flAcceptOpml, flUseProxy, adrRedirectInfo)}}};
	
	if flMessages { //clean up the message area: AR 6/14/1999
		msg ("")};
	
	return (httpResult)}
\ No newline at end of file