Readme for analog 3.3

This Readme describes analog3.3. For the latest version of analog, see the analog home page. For examples of the output see

• our local statistics.
• statistics for my pages.

Analog is freeware, but its use is covered by a licence. You must agree to the terms of the licence before using the program.

This is a version of the Readme in one page. If you're reading it on line, you might prefer the version on several smaller pages. There is an index at the end of this document.

Now you can go to

• Starting to use analog
• Customising analog
• What the results mean
• Errors and warnings
• Frequently asked questions (FAQ)
• Mailing lists
• Helper applications
• Acknowledgements
• What's new in this version?
• Map
• Index

If you log in to your ISP's machine from your home machine, you have two options. If you have the right permissions, you can run analog on your ISP's machine. Otherwise, you can download (e.g., ftp) the logfiles from their machine to yours, and then run analog on your machine.

Once you've downloaded the right version of analog for your computer from the analog home page (or a mirror site), you need to know how to set it up and run it. This is very easy, but the instructions are slightly different depending which platform you're using.

• Mac users
• Windows users
• OS/2
• Everyone else (Unix, VMS, Acorn, Windows 3.1 etc.)

If you can't manage to set up analog after reading the instructions, send a message to the analog-help mailing list.

LOGFILE logfilename    # to set where your logfile lives

There's a list of basic commands later in the Readme. Also there are a few to get you started in the configuration file already, but there are lots of others available. You can read about all the commands in the section on customising analog.

One note: on other platforms, there is another way to give options, via command line arguments. You'll see these mentioned in this Readme from time to time, but the Mac doesn't have a command line, so ignore these.

If you want to compile your own version of analog (it's written in C), or just to read the source code, it's available from the analog home page. (It's the same source code for all versions).

When you've downloaded analog, and either you or your browser has unzipped it, you will find in the analog directory a configuration file called analog.cfg and the analog executable itself, as well as the Readme, the Licence (which you must read and agree to before using analog) and a couple of other files.

(Some unzip programs are broken, and do not create folders when they should. If you don't have a folder called lang, create one and put all the files called *.lng into it.)

There are two ways of running analog. You can either run it from Windows (by single-clicking or double-clicking on its icon, depending on your setup), or you can run it from the DOS command prompt (under Start-Programs). If you run it from Windows, it will create a DOS window to run in. When it's finished, it will produce an output file called Report.html. The first time you run it, this may all happen almost instantly.

LOGFILE logfilename    # to set where your logfile lives

There's a list of basic commands later in the Readme. Also there are a few to get you started in the configuration file already, but there are lots of others available. You can read about all the commands in the section on customising analog.

In some ways, it's easier to run analog from the DOS command prompt, because you get to see any error or warning messages more easily. Also, if you run analog from the command prompt, there is another way to give options, via command line arguments, given on the command line after the program name. These are just shortcuts for configuration file commands.

If you want to compile your own version of analog (it's written in C), or just to read the source code, it's available from the analog home page. (It's the same source code for all versions).

LOGFILE logfilename    # to set where your logfile lives

There's a list of basic commands later in the Readme. Also there are a few to get you started in the configuration file already, but there are lots of others available. You can read about all the commands in the section on customising analog.

There is one other way to give options to analog, via command line arguments, given on the command line after the program name. These are just shortcuts for configuration file commands.

If you want to compile your own version of analog (it's written in C), or just to read the source code, it's available from the analog home page. (It's the same source code for all versions). There are instructions about compiling on another page.

First, you will want to look at the file analhead.h. These are all user-settable options, but most of them you can override later. You will need to check the first few options in the file.

When you have done that, you need to compile the program. How to do that depends on which system you're using.

make

If you haven't got gcc, you will need to change the compiler - try acc or cc instead. If it still doesn't compile, try DEFS=-DNODNS to ignore the DNS lookup code.

There is a known problem with HP-UX 10 and some versions of gcc. If it complains about an error in the <sys/stat.h> library, you need to upgrade to gcc version 2.7.2.3 or later, or use HP's cc compiler. HP's compiler is not an ANSI C compiler by default, so you need to specify -Ae in the CFLAGS to tell the compiler to use ANSI C.

SunOS 4's cc and gcc don't have the necessary header files for ANSI C. If you have the ANSI C compiler acc, use that. Otherwise use the DEFS given in the Makefile.

SunOS 5 users need to change the LIBS= line in the Makefile. Also, this OS sometimes seems to have a broken strcmp() function. If you get an "illegal instruction" error when running analog, compile it with the -DNEED_STRCMP in the DEFS= line.

Compiling under VMS. Type

MMS

Compiling under Acorn RiscOS. The Makefile is called Make.Risc, and you will have to rename it to Makefile before running make. Also you have to make directories called C, H and O, and move the sources files into the appropriate directories: e.g., alias.c must be renamed C.alias. And you will find that there are some filenames in the header file analhead.h that you want to change to fit into the RiscOS directory structure.

Compiling under OS/2. Although there is a precompiled version of analog for OS/2, if you want to compile your own you will need the EMX package. You should edit the Makefile to have OS=OS2 and LIBS=-lsocket. Then after running Make, you need to run the command

EMXBIND -b ANALOG

analog

You can configure analog by putting commands in the configuration file, which is called analog.cfg by default. Two commands you will need straight away are

LOGFILE logfilename      # to set where your logfile lives
OUTFILE outputfile.html  # to send the output to a file instead of the screen

There's a list of basic commands later in the Readme. Also there are a few to get you started in the configuration file already, but there are lots of others available. You can read about all the commands in the section on customising analog.

There is one other way to give options to analog, via command line arguments, given on the command line after the program name. These are just shortcuts for configuration file commands.

• basic commands

The following section is a technical (i.e., dull but important) section on the

• syntax of configuration commands.

• Choosing a logfile
• Specifying a log format
• Aliases
• Inclusions and exclusions
• Configuring the output
• Time reports
• Other reports
• Hierarchical reports
• The domains file
• Computer-readable output
• Cache files
• DNS lookups
• Coping with low memory
• Debugging
• Form interface and CGI program

Analog reads logfiles produced by your web server, and produces an output file based on the data in them. So you need to know how to specify which logfile to read, and which file to send the output to. The relevant commands look like

LOGFILE my_logfile
OUTFILE output.html

LOGFILE new1.log,old*.log
LOGFILE new2.log

HOSTNAME "Spam Widgets Inc."
HOSTURL http://www.spam-widgets.com/

If you have broken images in the output instead of graphs, you need to say in which directory on your server the images are stored. You do this by a command like

IMAGEDIR /analog/images/

MONTHLY ON    # one line for each month
WEEKLY ON     # one line for each week
FULLDAILY ON  # one line for each day
DAILY ON      # one line for each day of the week
HOURLY ON     # one line for each hour of the day
GENERAL ON    # the General Summary at the top
REQUEST ON    # which files were requested
FAILURE ON    # which files were not found
DIRECTORY ON  # directory report
HOST ON       # which computers requested files
DOMAIN ON     # which countries they were in
REFERRER ON   # where people followed links from
FAILREF ON    # where people followed broken links from
BROWSER ON    # which browsers people were using
FILETYPE ON   # types of file requested
SIZE ON       # sizes of files requested

REQINCLUDE pages

LANGUAGE FRENCH

As I said, these are only a few of the commands available. To find out about all the commands, you'll have to read the remaining sections of the Readme, starting with a short section on the syntax of configuration commands.

CONFIGFILE other.cfg

In the Mac version, you can start up a program with a particular configuration file by dragging it onto the analog icon. The configuration file must start with a #. The default configuration file is still read first.

You can also specify any configuration command on the command line even if it doesn't have a command line abbreviation, by use of the +C command. For example, +C"UNCOMPRESS *.gz" will include that command.

DAILY      OFF   # We don't want a daily summary
FULLDAILY  "ON"  # We want a full daily report instead 
HOSTNAME (Spam Widgets Inc.)  # Spaces, so quotes or brackets needed

analog -settings [other options]

analog -settings > file

LOGFILE logfilename

You can have several LOGFILE commands. You can include wildcards in the logfile name (but not necessarily in the directory name: this is system-dependent), and you can use a list of logfiles separated by commas (without spaces). So the following commands would tell analog to read logfile1, c:\logs\logfile2, and all files ending in .log:

LOGFILE logfile1,*.log
LOGFILE c:\logs\logfile2

LOGFILE "Hard Drive:Internet Applications:Analog:Logs:*"

LOGFILE none

If your logfile is not in one of the standard formats, you will probably still be OK, because it is possible to tell analog about other formats using a LOGFORMAT command. This is explained in the next section. But most users don't ever need to know about this because they have logfiles in a standard format. So the best thing to do is just to try analysing your logfile and see if analog will understand it. If it does, you don't need to worry about LOGFORMATs.

If analog can't understand your logfile, it will warn you that it can't detect the format, or possibly that it found a lot of corrupt lines. There are basically four reasons why this might happen:

1. Since analog tries to deduce the format based on the first line of the logfile, it could just be that the first line is corrupt. In this case, you could tell analog the format, or you could just fix the first line.
2. For the same reason, if the format changes midway through the log, analog will count the remaining lines as corrupt. In this case, you will find that your report contains a partial analysis but with a large number of corrupt lines too. You will need to give analog two LOGFORMAT commands to tell it about the two different formats.
3. Some log formats are not very well designed and analog can't analyse them reliably. In this case it will give up rather than risk doing a bad job, usually with a helpful message. So if you believe that your logfile is in a standard format, but analog still can't analyse it, you should read the notes on all the built-in formats at the end of this page where some common problems with those formats are described.
4. Finally, some logfiles really aren't in one of the standard formats. In this case you will need to read the next section and learn how to tell analog about your format.

LOGFILE log1,log2 http://www.%v.mydomain.com

If %v is included in the argument and the logfile line doesn't have a virtual host, that line will be marked as corrupt. If VHOSTLOWMEM 3 is specified, the %v's will not be translated and will just appear as %v in the output.

UNCOMPRESS *.gz,*.Z  /usr/bin/gzcat

UNCOMPRESS *.gz "c:\Program Files\gzip\gzip -cd"

UNCOMPRESS *.LOG-GZ;*  "gunzip -c"

If analog determines when it starts to uncompress a logfile that that file isn't wanted for the analysis, two undesirable things can happen. Either the program might pause until the logfile is fully uncompressed, or there might be a "broken pipe" error reported. This is system dependent, and out of analog's control.

Logfile formats

The common logfile format is written by most servers. Its lines look like

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0" 200 1243

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ "HTTP/1.0" 200 1243

[25/Dec/1998:17:45:35] http://www.site.com/ -> /~sret1/

[25/Dec/1998:17:45:35] Mozilla/2.0 (X11; I; HP-UX A.09.05)

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0" 200 1243
"http://www.site.com/" "Mozilla/2.0 (X11; I; HP-UX A.09.05)"

LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\""

192.64.25.41, -, 25/12/98, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
30, 303, 1243, 200, 0, GET, /~sret1/, -,

There are also various third-party extensions to the Microsoft format to include, for example, the browser and referrer. But they all do it in different ways, so analog can't automatically diagnose them, and again, you need to write a LOGFORMAT command for them.

12/25/98 17:45:35  jay.bird.com  host1  Server  fred  GET  /~sret1/
http://www.site.com/    Mozilla/2.0 (X11; I; HP-UX A.09.05)  200  1243  2178

If analog finds that the header line is corrupt, it will usually tell you what was wrong with it. Here are two common problems. First, the header line musn't contain the same item twice, even under two different names. If it does, you will have to use a LOGFORMAT command to tell analog which one you want to ignore. Secondly, you're not allowed the time without the date or vice versa -- in particular, having the date just at the top of the logfile is not sufficient; you must have it on each line. Microsoft servers produce extended logs with the date only at the top. But if the date changes during the logfile, the server doesn't then write a new date line. For this reason analog can't analyse such logfiles safely. There are some programs on the helper applications page to put the date on each line. If you already have such a logfile you might want to use one of these programs, but they have to assume that the date doesn't change during the logfile, so it would be safer to tell your server to log in a better format in future.

The extended log is described at http://www.w3.org/TR/WD-logfile.html. Its header line looks like

#Fields: date time cs-uri

The WebSTAR file has a header line like

!!LOG_FORMAT DATE TIME RESULT URL BYTES_SENT HOSTNAME

Finally, the Netscape header line looks like

format=%Ses->client.ip% [%SYSDATE%] "%Req->reqpb.clf-request%"
%Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%

The basic command to specify a log format looks like

LOGFORMAT format

LOGFILE log0
LOGFORMAT format1
LOGFILE log1
LOGFORMAT format2
LOGFILE log2
LOGFILE log3

APACHELOGFORMAT (%h %l %u %t \"%r\" %s %b)

There are format words for all the built-in formats analog knows about. You might need one of these words if your logfile is in a standard format, but analog can't detect which format it's in for some reason; for example, maybe the first line is corrupt; or maybe analog can't tell whether you're using North American or international dates. So for example

LOGFORMAT COMMON

If your logfile is not in one of the recognised formats, you can tell analog about your format using a log format string. You only ever need this if your logfile has lines which are not in one of the standard formats. And even if it isn't in a standard format, if you're using the Apache web server, you will find APACHELOGFORMAT (above) easier.

The format string consists of a template for the logfile line, with the various fields and special characters replaced by codes as follows. Please note that these codes are case sensitive -- for example, %b is completely different from %B!

%S

host (computer making the request)

%r

file requested

%B

browser

%A

browser with +'s instead of spaces

%f

referrer (URL referring to the file)

%u

user (tip: a cookie can usefully be defined as %u too)

%v

virtual host (also called virtual domain)

%d

day of the month

%m

month in digits

%M

month, three letter English abbreviation

%y

year, last two digits

%Y

year, four digits

%h

hour of the day

%n

minute of the hour

%a

a for am or p for pm, if %h is 12-hour clock (so to match "am" you need %am)

%U

"Unix time" (seconds since beginning of 1970, GMT)

%b

number of bytes transferred

%c

HTTP status code

%C

Special code, specific to particular servers

%q

query string (part of filename after ?, if recorded in a separate field)

%j

junk: ignore this field (field can be empty too)

%w

white space: spaces or tabs

%W

optional white space

%%

% sign

\n

new line

\t

tab stop

\\

single backslash

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0" 200 1243

LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b)

Fri 25/12/98 5:45pm, /~sret1/, jay.bird.com, 200, 1243, http://www.site.com, Mozilla/2.0 (X11; I; HP-UX A.09.05)

LOGFORMAT (%j %d/%m/%y %h:%n%am, %r, %S, %c, %b, %f, %B)

LOGFORMAT COMMON
LOGFORMAT COMBINED
LOGFILE log1
LOGFORMAT (%j %d/%m/%y %h:%n%am, %r, %S, %c, %b, %f, %B)
LOGFILE log2
LOGFILE log3

If you specify several formats, analog tries to match each line to the first format first, then if that fails the next, and so on, so the order of the formats is important. Usually you want to specify the most common one first, to minimise the time spent trying to match lines to inappropriate formats.

So let's go back to the first example:

LOGFILE log0
LOGFORMAT format1
LOGFILE log1
LOGFORMAT format2
LOGFILE log2
LOGFILE log3

The times you need to use the DEFAULTLOGFORMAT instead of the LOGFORMAT are if you want to change the format of logfiles which aren't given in a LOGFILE command -- for example, ones specified on the command line, or dragged onto the program icon on a Mac, or compiled in. It is also useful to use the DEFAULTLOGFORMAT if your logfiles are always in the same format, so that you don't have to worry about putting in enough LOGFORMATs in the right places.

The "Unix time", %U, is always recorded in GMT. So you will probably need to use a LOGTIMEOFFSET command to convert to your local timezone. Also, it's just the integer part of the time, so if you have decimals you will have to use %U.%j .

The log formats which analog can handle are those which are known as instantaneously decipherable: in practice, this means that the character which terminates a string can never occur in the string. So for example, in common format, which looks like

LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b)

Sometimes you need to read one of the fields in a logfile, but not analyse it. For example, if you have a separate common log and referrer log, the referrer log might look like

http://guide-p.infoseek.com/Titles -> /~sret1/analog/

LOGFORMAT (%f -> %*r)

A tip: sometimes it is more efficient to specify two or more adjacent fields to ignore with a single %j, as long as the whole group ends with a recognisable character. So common format is more efficiently specified as

LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b)

Another tip: %j can also be used to ignore whole lines, rather than just fields analog doesn't use. For example, the extended log format ignores lines beginning with # by using

LOGFORMAT #%j

LOGFORMAT (%*S, %*u, %m/%d/%y, %h:%n:%j, %j)

Common format, LOGFORMAT COMMON

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0" 200 1243
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r%wHTTP%j" %c %b)
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r" %c %b)

Microsoft common format, LOGFORMAT MS-COMMON

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ "HTTP/1.0" 200 1243
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r%w"HTTP%j" %c %b)
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r" %c %b)

Combined log, LOGFORMAT COMBINED

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0" 200
     1243 "http://www.site.com/" "Mozilla/2.0 (X11; I; HP-UX A.09.05)"
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r%wHTTP%j" %c %b "%f" "%B")
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r" %c %b "%f" "%B")

Referrer log, LOGFORMAT REFERRER

[25/Dec/1998:17:45:35] http://www.site.com/ -> /~sret1/
or http://www.site.com/ -> /~sret1/
LOGFORMAT ([%d/%M/%Y:%h:%n:%j] %f -> %*r)
LOGFORMAT (%f -> %*r)

Browser log, LOGFORMAT BROWSER

[25/Dec/1998:17:45:35] Mozilla/2.0 (X11; I; HP-UX A.09.05)
LOGFORMAT ([%d/%M/%Y:%h:%n:%j] %B)

Microsoft log, North American dates, LOGFORMAT MICROSOFT-NA

192.64.25.41, -, 12/25/98, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
     30, 303, 1243, 200, 0, GET, /~sret1/, -,
LOGFORMAT (%S, %u, %m/%d/%y, %h:%n:%j, W3SVC%j, %j, %v, %j, %j, %b, %c, %j, %j, %r, %q,)
LOGFORMAT (%*S, %*u, %m/%d/%y, %h:%n:%j, %j)

Microsoft log, international dates, LOGFORMAT MICROSOFT-INT

192.64.25.41, -, 25/12/98, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
     30, 303, 1243, 200, 0, GET, /~sret1/, -,
LOGFORMAT (%S, %u, %d/%m/%y, %h:%n:%j, W3SVC%j, %j, %v, %j, %j, %b, %c, %j, %j, %r, %q,)
LOGFORMAT (%*S, %*u, %d/%m/%y, %h:%n:%j, %j)

WebSite log, North American dates, LOGFORMAT WEBSITE-NA

12/25/98 17:45:35  jay.bird.com  host1  Server  fred  GET  /~sret1/
     http://www.site.com/    Mozilla/2.0 (X11; I; HP-UX A.09.05)  200  1243  2178
LOGFORMAT (%m/%d/%y %h:%n:%j\t%S\t%v\t%j\t%u\t%j\t%r\t%f\t%j\t%B\t%c\t%b\t%j)

WebSite log, international dates, LOGFORMAT WEBSITE-INT

25/12/98 17:45:35  jay.bird.com  host1  Server  fred  GET  /~sret1/
     http://www.site.com/    Mozilla/2.0 (X11; I; HP-UX A.09.05)  200  1243  2178
LOGFORMAT (%d/%m/%y %h:%n:%j\t%S\t%v\t%j\t%u\t%j\t%r\t%f\t%j\t%B\t%c\t%b\t%j)

CASE INSENSITIVE
CASE SENSITIVE

Next it applies built-in aliases to each item. For example, it knows that %7E in a filename or referrer is equivalent to ~ and translates it accordingly. It also strips off the directory suffix from any filenames which have it. This suffix is normally index.html, but you can specify another one instead with a command such as

DIRSUFFIX default.htm

FILEALIAS /football.html /soccer.html
HOSTALIAS lion lion.statslab.cam.ac.uk

The alias commands for the other items are called BROWALIAS, REFALIAS, USERALIAS and VHOSTALIAS. Only one alias is ever applied to any item. So after

FILEALIAS /football.html /soccer.html
FILEALIAS /soccer.html /brazil.html

You can also use wildcards (? and *) in alias commands. The left hand side can contain at most one *, unless the right hand side contains no *'s. If the right hand side contains a * too, then the part of the name represented by the * on the left hand side will be substituted at the position of the * on the right hand side. So, for example,

FILEALIAS /football/* /soccer/

FILEALIAS /football/* /soccer/*

TYPEOUTPUTALIAS .txt ".txt (Plain text files)"

There can be some confusion between some ALIAS and OUTPUTALIAS commands. For example, what is the difference between HOSTALIAS and HOSTOUTPUTALIAS? In fact, there are several differences, resulting from the different times at which the aliases are processed. The HOSTALIAS applies to the host items, but the HOSTOUTPUTALIAS only applies to the lines in the host report. This means that the HOSTALIAS also affects the other reports which use the hosts, such as the domain report, whereas the HOSTOUTPUTALIAS only affects the host report. Also the HOSTOUTPUTALIAS applies separately to each line of the host report. This means that if two separate hosts translate to the same thing in a HOSTALIAS command, they will become one host ever after. But if one were to use the same HOSTOUTPUTALIAS commands, there would be two hosts, which would just happen to have the same name in one report.

In summary, HOSTALIAS would normally be used if a single host had two different names, so might otherwise appear to be two hosts, whereas HOSTOUTPUTALIAS would normally be used to annotate or clarify the host report.

The full list of output aliases is REQOUTPUTALIAS, REDIROUTPUTALIAS, FAILOUTPUTALIAS, TYPEOUTPUTALIAS, DIROUTPUTALIAS, HOSTOUTPUTALIAS, DOMOUTPUTALIAS, REFOUTPUTALIAS, REFSITEOUTPUTALIAS, REDIRREFOUTPUTALIAS, FAILREFOUTPUTALIAS, BROWOUTPUTALIAS, FULLBROWOUTPUTALIAS, VHOSTOUTPUTALIAS, USEROUTPUTALIAS and FAILUSEROUTPUTALIAS.

There is one known bug with OUTPUTALIAS. The report is sorted before the OUTPUTALIAS is applied. This means that if the SORTBY for the report is set to ALPHABETICAL, then the report will not be sorted correctly.

HOSTEXCLUDE mycomputer.myisp.com

The rule for determining whether an item is included or excluded is as follows. All the INCLUDE and EXCLUDE commands for that item are considered one by one in order, and the item is included or excluded according to the last command it matched. Items which don't match any of the INCLUDE or EXCLUDE commands are included if the first command was an exclusion, and excluded if the first command was an inclusion. For example, the configuration

FILEINCLUDE /~sret1/*
FILEEXCLUDE /~sret1/backgammon/*,/~sret1/analog/*
FILEINCLUDE /~sret1/backgammon/*.gif

FILEEXCLUDE /~sret1/*/img/*

The full list of these commands is HOSTINCLUDE and HOSTEXCLUDE; FILEINCLUDE and FILEXCLUDE; BROWINCLUDE and BROWEXCLUDE; REFINCLUDE and REFEXCLUDE; USERINCLUDE and USEREXCLUDE; and VHOSTINCLUDE and VHOSTEXCLUDE.

Sometimes a line doesn't contain a particular sort of item, either because there is no field reserved for it on the line, or because the browser didn't send it for that request. You can include or exclude these lines by making a special blank entry in the INCLUDE or EXCLUDE command. For example,

USERINCLUDE jim
USERINCLUDE ""

If you get confused with all the inclusions and exclusions, remember that you can always run analog -settings to see what the options you have specified represent.

FROM 990701
TO   000630

FROM -01-00+01   # from tomorrow last year
TO -00-0131  # to the end of last month (OK even if last month
             # didn't have 31 days)
FROM -00-00-112
TO   -00-00-01  # statistics for the last 16 weeks
FROM -00-00-00:-06+01  # statistics for the last 6 hours

REFREPEXCLUDE http://your.site.com/*

The full list of these commands is REQINCLUDE and REQEXCLUDE; REDIRINCLUDE and REDIREXCLUDE; FAILINCLUDE and FAILEXCLUDE; TYPEINCLUDE and TYPEEXCLUDE; DIRINCLUDE and DIREXCLUDE; HOSTREPINCLUDE and HOSTREPEXCLUDE; DOMINCLUDE and DOMEXCLUDE; REFREPINCLUDE and REFREPEXCLUDE; REFSITEINCLUDE and REFSITEEXCLUDE; REDIRREFINCLUDE and REDIRREFEXCLUDE; FAILREFINCLUDE and FAILREFEXCLUDE; BROWSUMINCLUDE and BROWSUMEXCLUDE; FULLBROWINCLUDE and FULLBROWEXCLUDE; VHOSTREPINCLUDE and VHOSTREPEXCLUDE; USERREPINCLUDE and USERREPEXCLUDE; and FAILUSERINCLUDE and FAILUSEREXCLUDE. The inclusion or exclusion applies to the unaliased name, if you are doing any output aliases.

REQINCLUDE pages

Analog determines which files should count as pages (and thus which requests count as page requests) using another INCLUDE/EXCLUDE pair, called PAGEINCLUDE and PAGEEXCLUDE. By default, *.html, *.htm and directories (*/) count as pages. But you change the list by commands like

PAGEINCLUDE *.ps,*.ps.gz
PAGEEXCLUDE sret1.html

Finally, there are commands called ARGSINCLUDE and ARGSEXCLUDE, and REFARGSINCLUDE and REFARGSEXCLUDE. Sometimes a URL contains arguments after a question mark. For example, the URL

/cgi-bin/script.pl?x=1&y=2

Analog can either read or ignore the arguments. If the command ARGSEXCLUDE /cgi-bin/script.pl were given, analog would ignore the arguments to that file, and so treat the above URL as being the same as /cgi-bin/script.pl. On the other hand, if ARGSINCLUDE /cgi-bin/script.pl were specified, analog would read the arguments, and treat the above URL as a different file from /cgi-bin/script.pl (or from /cgi-bin/script.pl?y=2&x=1), although a grand total for /cgi-bin/script.pl would still be listed in the Request Report.

REFARGSINCLUDE and REFARGSEXCLUDE are the same for referrers. By default, all arguments are included. The check for whether the arguments should be included happens before the filename is aliased: this means that you can't use pages in this command, because we don't know whether a file is a page until after it's been aliased.

If you want to see the arguments in a report you may also have to set the appropriate ARGSFLOOR command.

There are 27 different reports which analog can produce, if your logfiles contain the necessary information. Each one has a short name, and a code letter or number, as follows:

x  GENERAL      General Summary
m  MONTHLY      Monthly Report
W  WEEKLY       Weekly Report
D  FULLDAILY    Daily Report
d  DAILY        Daily Summary
H  FULLHOURLY   Hourly Report
h  HOURLY       Hourly Summary
4  QUARTER      Quarter-Hour Report
5  FIVE         Five-Minute Report
S  HOST         Host Report
o  DOMAIN       Domain Report
r  REQUEST      Request Report
i  DIRECTORY    Directory Report
t  FILETYPE     File Type Report
z  SIZE         File Size Report
E  REDIR        Redirection Report
I  FAILURE      Failure Report
f  REFERRER     Referrer Report
s  REFSITE      Referring Site Report
k  REDIRREF     Redirected Referrer Report
K  FAILREF      Failed Referrer Report
B  FULLBROWSER  Browser Report
b  BROWSER      Browser Summary
v  VHOST        Virtual Host Report
u  USER         User Report
J  FAILUSER     Failed User Report
c  STATUS       Status Code Report

FIVE OFF
REFSITE ON

You can turn the "Go To" lines in the report off with the command

GOTOS OFF

The figures in parentheses in the General Summary are for the last seven days: either the seven days before the TO time, or if no TO time is given, the seven days before the time of the program start. The figures for the last seven days are normally included if some, but not all, of the requests fall in those seven days; but you can turn them off by means of the command

LASTSEVEN OFF

You can change the order of the reports by means of the REPORTORDER command. You should list the code letters for all possible reports in the order you want them, like this:

REPORTORDER xcmdDhH45WriSoEItzsfKkuJvbB

OUTFILE stats.htm

OUTFILE /usr/bin/httpd/htdocs/stats.html  # Unix
OUTFILE "Hard Disk:Server Apps:WebSTAR:Analog:Report.html" # Mac

Sometimes it's convenient to include the date in the name of the OUTFILE. You can do this by including the following codes in the filename.

%D  date of month
%m  month name
%M  month number
%y  two-digit year
%Y  four-digit year
%H  hour
%n  minute
%w  day of week

OUTFILE stats%y%M.html

OUTPUT ASCII

Next, you can change the language of the output. There are two ways to do this. The usual way is to use the LANGUAGE command. For example, the command

LANGUAGE FRENCH

The other way is to use the LANGFILE command. This is useful if you want to download a new language from the analog home page, or if you want to translate one yourself, or even if you want to change some words or phrases or the way the dates and times are formatted in the output. The LANGFILE command tells analog in which file to find the various words and phrases for a new language. For example, the command

LANGFILE lang/guarani.lng

Some languages also have domains files available. These are normally selected automatically by the LANGUAGE command. But you can tell analog to use a different domains file with the DOMAINSFILE command. Also, some languages have translations of the form interface.

If you want to translate another language, I would be delighted! You'd be wise to contact me first to make sure that no-one else is already translating the same language. The English language file contains some brief instructions for translating new languages.

You have to be careful using this command. Because of daylight savings time in operation in different parts of the world at different times, analog cannot attempt to convert between different timezones. So it's your responsibility to set the right offset for different times of year. For example, if you were in Chicago, but your server was recording time in GMT, you would need to specify two different time offsets, one of minus five hours for summer and one of minus six hours for winter. You would need to split your logfiles in the right places and then run commands like

LOGTIMEOFFSET -300
LOGFILE summer*.log
LOGTIMEOFFSET -360
LOGFILE winter*.log

There is also a related command called TIMEOFFSET. This tells analog how much to offset the time of the computer on which it is running (rather than the computer running the server), to get your local time.

IMAGEDIR img/   # within the same directory as the output
IMAGEDIR /img/  # off the root directory of your server

There are three commands which affect the top line of the output. First, the LOGO command allows you to replace the analog logo with another image (for example, your organisation's logo). You can say

LOGO picture.gif  # for this file
LOGO /images/picture2.gif  # a different file
LOGO none         # for no logo

Then there are commands HOSTNAME and HOSTURL which affect the name and link at the end of the title line. For example, I might specify

HOSTNAME "Stephen Turner"
HOSTURL  http://www.statslab.cam.ac.uk/~sret1/

HOSTNAME "M\üller & S\öhne"

There are commands called HEADERFILE and FOOTERFILE. These let you specify files to be inserted near the top and bottom of your output. You can specify

HEADERFILE none

There are three related commands called SEPCHAR, REPSEPCHAR and DECPOINT. These specify single characters to be used as the thousands separator in numbers, the thousands separator within the columns in the reports, and the decimal point. For example, a French user might choose

SEPCHAR " "
REPSEPCHAR none
DECPOINT ,

There is a command called RAWBYTES. Specify RAWBYTES ON if you want the exact number of bytes to be listed in reports, or RAWBYTES OFF if you want the number of kilobytes or Megabytes as appropriate to be listed instead.

Finally there is a command called PAGEWIDTH which specifies the width of the page. The output is not guaranteed to fit in this width, but analog will take notice of it when choosing the width of the time graphs, and when sorting the host report alphabetically; and if the output format is ASCII, when drawing horizontal rules and printing some bits of text. I recommend about PAGEWIDTH 65 for HTML output, and PAGEWIDTH 75 for ASCII output.

Each time report can contain columns listing the requests, requests for pages, and bytes transferred at that time, using the following code letters.

R

Number of requests

r

Percentage of the requests

P

Number of page requests

p

Percentage of the page requests

B

Number of bytes transferred

b

Percentage of the bytes

HOURCOLS Pb

FULLDAYGRAPH P

FULLDAYGRAPH b

MONTHBACK ON  # Monthly Report backwards
WEEKBACK OFF  # Weekly Report forwards

QUARTERROWS 96  # only the last day's worth
MONTHROWS 0 # 0 means no restriction: show all time

MARKCHAR =

There is a parameter called MINGRAPHWIDTH which sets the minimum nominal size of the graphs. For example, if you set

MINGRAPHWIDTH 10

There is one more command which affects the time reports. You can specify which day should be counted as the first day of the week. This affects the layout of the Daily Report, Daily Summary and Weekly Report. For example, our local student newspaper publishes a new edition on the web every Friday, so they like to specify WEEKBEGINSON FRIDAY for their reports.

In the next section, we'll look at commands relating to the non-time reports.

First, these reports have COLS commands, just like the time reports. (See the section on Time reports for how to use these commands.) In the non-time reports, one additional column is possible, namely D for date of last access. So, for example,

REQCOLS RD

HOSTSORTBY ALPHABETICAL

There is one known bug concerned with SORTBY ALPHABETICAL. The report is sorted before any OUTPUTALIAS is applied. This means that if an OUTPUTALIAS has been specified for the report, then the report will not be sorted correctly.

DOMFLOOR 1000r       # all domains with at least 1000 requests
DOMFLOOR 1000p       # at least 1000 requests for pages
DOMFLOOR 1000000b    # at least 1,000,000 bytes transferred
DOMFLOOR 1Mb         # at least 1 megabyte
DOMFLOOR 0.5%r       # 0.5% of the requests (ditto %p and %b)
DOMFLOOR 0.5:r       # 0.5% of the maximum number of requests
                     # for any domain (ditto :p and :b)
DOMFLOOR 970701d     # last access since 1st July 1997
DOMFLOOR -00-01-00d  # last access in last month (see
                     # documentation on FROM and TO commands)
DOMFLOOR -100r       # domains with top 100 number of requests
                     # (ditto -100p, -100b, -100d)

r

REQUESTS

p

PAGES

b

BYTES

d

DATE

a

ALPHABETICAL

x

RANDOM

+fp

means turn the referrer report on and sort it by page requests, but says nothing about the floor;

+f100r

means list all referrers with at least 100 requests, but says nothing about the sort method;

+fb10000

means list all referrers with at least 10,000 bytes, sorted by bytes;

+fa-000101d

means list all referrers with accesses this year, sorted alphabetically.

There's one other command which affects the links in the Request Report. The command BASEURL prepends an additional string to the URLs in the target of the link. For example, after the command

BASEURL http://www.statslab.cam.ac.uk

In the next section, we'll look at commands for generating hierarchical reports, which are closely related to the commands in this section.

First, you need to be able to control what gets listed in the reports. For this you need to use the SUB family of commands. So, for example, the command SUBDIR /~sret1/* would ensure that the Directory Report would not only contain an entry for the sum of my files, but also one for each of my subdirectories, something like this:

29,111: /~sret1/
10,234:   /~sret1/analog/
 5,179:   /~sret1/backgammon/
11,908: /~steve/

If you specify a SUB command, all the intermediate levels are included automatically. So, for example, after

SUBDOMAIN statslab.cam.ac.uk

Here are examples of the other three SUB commands:

SUBTYPE *.gz      # in the File Type Report
SUBBROW Mozilla/*  # in the Browser Summary
REFDIR http://search.yahoo.com/*   # Referring Site Report

The SUBDOMAIN report (but none of the others) can included a second argument describing the subdomain. For example

SUBDOMAIN cam.ac.uk 'University of Cambridge'

SUBDOMAIN 131.111 'University of Cambridge'

One other use for the SUBDIR command is if you have used the second argument to the LOGFILE command. Suppose you have translated files like /index.html into http://www.mycompany.com/index.html. Then the command

SUBDIR http://www.mycompany.com/*

An sub-item is listed in a hierarchical report only if it is above the sub-FLOOR, and it is included with a SUB command, and its immediate parent is listed. For example, specifying

SUBDIR /*/*/
SUBDIRFLOOR -3r
SUBDIRSORTBY REQUESTS

The report INCLUDE and EXCLUDE commands for a hierarchical report only apply to the top level of the report: you can use the SUB commands for the lower levels.

The three file reports (Request Report, Redirection Report and Failure Report) and the three referrer reports (Referrer Report, Redirected Referrer Report and Failed Referrer Report) are not fully hierarchical, but they do list search arguments together under the file to which they refer (provided that the arguments have been read in: see the ARGSINCLUDE command). So they have similar sub-FLOOR and sub-SORTBY commands, namely REQARGSFLOOR, REDIRARGSFLOOR, FAILARGSFLOOR, REFARGSFLOOR, REDIRREFARGSFLOOR and FAILREFARGSFLOOR; and REQARGSSORTBY, REDIRARGSSORTBY, FAILARGSSORTBY, REFARGSSORTBY, REDIRREFARGSSORTBY and FAILREFARGSSORTBY.

That concludes the description of all the output configuration commands. Now we move on to some other individual topics, starting with the domains file.

DOMAINSFILE lang/mydomains.tab

If you haven't got a domains file, you can download one from http://www.statslab.cam.ac.uk/~sret1/analog/ukdom.tab. It should contain each domain code followed by its location on a new line, thus:

ad   Andorra
ae   United Arab Emirates
[...]

If you want HTML special characters in the domains file, you have to precede them with a backslash, like this:

am   Arm\énie

Only domains which occur in the domains file will get their own line in the Domain Report: the rest are probably spurious, and will be accumulated together as "unknown domains". If you have debugging turned on, you can see which domains were unknown.

OUTPUT COMPUTER

Each line in the output is separated into fields by means of a special string. You can specify this string by means of the COMPSEP command; for example

COMPSEP ,

Each line in the preformatted output begins with a letter indicating which report the line is part of. (The code letters for the reports are listed in the section on Configuring the Output.) After that, there follows a field indicating the remaining columns in the report (using the letters RrPpBbD as usual). Then there are the numerical data and then the name of the item. Times actually take up several fields: year, month, date, hour & minute, or as many of those as are necessary to identify the time.

The first line of most reports has f instead of the normal column letters, followed by the floor for the report, in the form it would be written for a FLOOR command, followed by the SORTBY using the code letters

r

REQUESTS

p

PAGES

b

BYTES

d

DATE

a

ALPHABETICAL

x

RANDOM

The general summary is a bit different. After an initial x, there is a two-character code saying what the line contains. The possible codes are

HN

HOSTNAME

HU

HOSTURL

PS

Program start time

FR

Time of first request

LR

Time of last request

E7

Time last 7 days ends

SR

Total successful requests

S7

Total successful requests in last 7 days

PR

Total successful requests for pages

P7

Total successful requests for pages in last 7 days

FL

Total failed requests

F7

Total failed requests in last 7 days

RR

Total redirected requests

R7

Total redirected requests in last 7 days

NC

Logfile lines without status code

C7

Lines without status code in last 7 days

NF

Number of distinct files requested

N7

Number of distinct files requested in last 7 days

NH

Number of distinct hosts served

H7

Number of distinct hosts served in last 7 days

CL

Number of corrupt lines in the logfile

UL

Number of unwanted lines in the logfile

BT

Total number of bytes transferred

B7

Total number of bytes transferred in last 7 days

For most people, the cache file will not be needed: compressing the logfile using a standard compression utility such as gzip will be sufficient. Compressing a logfile is very efficient owing to the large number of repeated strings: I find about 12 times compression in practice. That in itself may solve your filespace problems, without needing to throw away any information.

The cache file is not the best format for post-processing the data or feeding it into a spreadsheet. For that you should use the computer readable output style.

If you are going to use the cache file feature, it is very important that you understand what is and what is not recorded. It is not possible to reconstruct everything of interest in the logfile from the cache file. The cache file does contain information about the total number of requests for each host and each file, but not about, for example, which files were read by which hosts. (To do so would take up as much disk space as the compressed logfile.) So you cannot later look at only one file and see which hosts read that file. Similarly, you cannot later restrict the files or hosts by date, using FROM and TO commands.

In summary, you should do all the inclusions and exclusions you want when you create the cache file. If you want different sets of inclusions and exclusions, you should create several cache files from the same logfile. You cannot later apply extra inclusions and exclusions accurately.

One other minor point: the pattern of failed requests and redirected requests over time is not recorded in the cache file. So although the total number will still be correct, the number in the last 7 days can be under-reported subsequently.

CACHEOUTFILE none

You can read in a previously-made cache file with the CACHEFILE command, or with the +U command line option. As with the LOGFILE command, you can use commas and wild cards to read in several cache files, and read compressed cache files using the UNCOMPRESS mechanism. Note that if you don't want to read a logfile as well as the cache file, you will have to explicitly set the LOGFILE to none.

When analog reads in a cache file, it will respect inclusions and exclusions as far as it can, but it does not apply any more aliases to the items. (This is to avoid double-aliasing.) So you must do any aliases you want at the time you create the cache file. Similarly, it does not obey the LOGTIMEOFFSET variable, to avoid double-offsetting, so any offset you want must be applied at cache-creation time too.

Sometimes you don't want to record all the types of item in the cache file. You might want to forget about which hosts had accessed your web site, for example, and only remember how many times each file was requested. You can choose not to include one type of item in the cache file by setting its LOWMEM to 3; for example, specify

HOSTLOWMEM 3

1. Archive the old logfile, and restart the server with a fresh logfile. (See your server documentation for how to do this.)
2. Make both a cache file and an ordinary report from the old logfile.
3. Make a report from the cache file and compare it against the report from the logfile to check it works.

I prefer to make a separate cache file from each logfile, in case something goes wrong with one of them, rather than a single cache file combining several logfiles, or a single cache file combining an old cache file and a logfile.

Unfortunately DNS lookups are typically very slow, because your computer has to ask across the network to find out the names of the hosts. For this reason, analog saves the addresses it has looked up in a file, so that you don't have to look them up again next time. (Even so, you may find the DNS lookups too slow to be usable.) The file is specified by a command like

DNSFILE dnsfile.txt

There are four possible levels of DNS activity. If you specify DNS NONE, no numerical addresses will be resolved. If you specify DNS READ, then analog will read the DNS file for old lookups, but no new lookups will take place. This mode is suitable if you are running analog while not connected to the internet. The third level is DNS WRITE. This reads the old file, looks up new addresses, and adds them to the file. (The first time you use DNS WRITE, you will get a missing-file warning as it tries to read the old file, but it will exist the next time.) The final level is DNS LOOKUP. This reads the old file and looks up new addresses, but doesn't add the new addresses to the file, so that they will not be remembered for next time. This is not normally a level that the user wants to specify, but analog will switch to this the behaviour if DNS WRITE fails for some reason.

If two copies of analog were allowed to write to the DNS file at the same time, the file could become corrupted. So when analog is running in DNS WRITE mode, it creates a lock file which tells other copies of analog to back off to DNS LOOKUP. You can change the location of that file with the command

DNSLOCKFILE filename

Analog never deletes anything from the DNS file: this means that the DNS file will grow, and can become quite large. You should delete the top of it every so often.

There are two parameters which say how long to trust old lookups for. If you set

DNSGOODHOURS 672

Finally, there is a debugging command, DEBUG +D to show all the DNS lookups that analog is making.

timestamp IP_address name

Recall what happens to an item when it has been read in. First it is aliased. Secondly, it is checked to see whether it is included or excluded. Then finally, if all the items are wanted, one request is added to its score.

Normally the name of the item is saved before the aliasing takes place. This avoids analog having to do the aliasing again next time the same item is encountered. But this can take up more memory than necessary. So there is a family of LOWMEM commands provided, which tell analog to record the name at a later stage, or even not at all. If you use these commands, analog will have to do a bit more work than normal, but it will use less memory. On most sites, the hosts take up most of the memory, so I'll use the HOSTLOWMEM command as an example.

The command

HOSTLOWMEM 0

HOSTLOWMEM 1

HOSTLOWMEM 2

HOSTLOWMEM 3

First, remember the option we mentioned before, to list the current settings of all of analog's variables. To get this, just put -settings on the command line, or PRINTVARS ON in one of your configuration files, along with your other commands. Then analog will produce the list of settings instead of running in the normal way.

DEBUG ON

C

list all corrupt logfile lines

D

information about DNS lookups

F

information about file opening and closing

S

summary information about each logfile when it's closed

U

list unknown domains

DEBUG FS

DEBUG +CD

DEBUG -CD

The WARNINGS command acts similarly. As well as WARNINGS ON and WARNINGS OFF, there are warnings in the following categories.

C

invalid configuration specified

D

dubious configuration specified

E

ERRFILE command used (see below)

F

files missing

L

apparent problems in logfiles

M

possibly problems in logfiles

R

turning off empty reports

You can also use command line abbreviations for these commands. The DEBUG command is represented by +V (for ON), -V (for OFF), +VFS (to select options FS), +V+FS (to add those options), and +V-FS (to remove them). Similarly the WARNINGS command can be given by +q, -q, +q<options>, +q+<options> or +q-<options>.

PROGRESSFREQ 20000  # say

ERRFILE newfile

ERRFILE stderr

The form interface is suitable for ordinary users to use, but it needs to be set up by a system administrator or other expert. In order to set it up, you have to be running a web server. You need to know what CGI programs are, where they live on your server, and how to set up their permissions properly. And you need to know how to write HTML forms. I shall assume this level of background knowledge for the rest of this section. Also, I shall assume that analog has already been set up and is running properly on its own.

Warning: CGI programs can contain security loopholes which allow an unscrupulous user to harm your system. (If you don't know about this, you shouldn't be running CGI programs at all. Read and understand the World Wide Web Security FAQ first.) I have tried to make this form interface safe, but I cannot guarantee it, and I take no responsibility if anything goes wrong. You use it at your own risk. (See the licence.) Furthermore, you should be aware that unless you take special measures like password protection, setting up the form interface implies making analog executable, and your logfiles analysable, by anyone on the internet.

The form interface consists of two parts: a form to choose the options, and a cgi program to interpret them and pass them to the analog program. You don't in fact need the form at all: if you want to create a link to the cgi program, with the arguments passed in the URL in the usual way, then that's fine.

On Unix, to compile the cgi program, you must first edit the top of anlgform.c to indicate where analog lives on your system. Then type make form, which should compile this source into a program called anlgform.cgi.

On Windows, the cgi program is compiled already, and called anlgform.exe. It assumes that analog is at \analog\analog.exe (on the same drive as anlgform.exe), so you must put a copy of analog there.

There is a form called anlgform.html distributed with the program. You can also find forms in languages other than English in the lang directory. Before you use the form, you must uncomment and edit the action at the top to indicate where anlgform.cgi (or anlgform.exe on Windows) lives on your server. There are other instructions at the top of the form which you must read and do before the form will work.

The form which is distributed with the program should only be regarded as an example form. Almost every configuration command has a counterpart in the CGI program, and so you can add to the form options to do almost anything you want. (The main exceptions are aliases, which are too complicated, and HEADERFILE, FOOTERFILE and LOGFORMAT, which would allow people to view any file on your system.) I shall give the full list in a minute. But as an example, if you have a form element with name="lo", then its argument will be passed to a LOGFILE command, so you can allow your users to select a logfile in this way.

(It is theoretically possible, though I think very unlikely, that another file on your system could conform sufficiently closely to one of the predefined log formats that analog could be persuaded to analyse it and so reveal some of its contents. If you're worried about this, or even if you want to force only one particular logfile to be analysed from the form, you can put a LOGFILE command in your mandatory configuration file. This will override earlier LOGFILE commands, and thus make analog ignore any logfiles specified on the form. This trick is only possible on Windows if you compile your own version of analog.)

You can specify other configuration files to be included. When analog is called by the CGI program, it first processes the default configuration file as usual. Then it processes any configuration file specified by an argument with name cg. Then it processes all the other arguments which the CGI program specifies. After that, it processes any configuration file specified by an argument with name cm. Finally, it processes the mandatory configuration file as usual. You may therefore want two copies of analog, one for form use and one for non-form use, with different configuration files compiled in. Note that the commands in the default and mandatory configuration files will contribute to the configuration: some of them may even override options specified on the form.

If the option qv=1 is sent to the CGI program, then analog is not run, but a list of the configuration commands which would have been sent to analog is printed instead. This is useful for checking that the CGI program is working properly. It can also be used for using the form to produce a configuration file.

Troubleshooting

1. Do other CGI programs work on your server?
2. Look in the server's error log for clues.
3. Are all relevant files (analog itself, logfiles, configuration files etc.) readable by your web server?
4. Include qv=1 in the arguments as explained above. If this works, then at least the CGI program is working.
5. Try running the cgi program from the shell. Set the environment variable QUERY_STRING to equal "xq=1", and run anlgform.cgi directly.
6. If some form options don't seem to take effect, then check whether they are being overridden by a command in a configuration file.
7. If you get a long wait, then no data returned, the server is probably timing out the request before analog has finished. The remedy is to increase the timeout interval.
8. There is one DNS issue which confuses people. The form always sets either DNS READ (if a DNS file is set on the form) or DNS NONE (otherwise). This may override what you have set somewhere else. You can override this behaviour in the mandatory configuration file, but you are likely to run into timeout problems if you do.

Time reports

        lc   uc    value
ON/OFF   q    p    1 for on, 0 for off
GRAPH    g    h
ROWS     r    s
COLS     c    d

Other reports

                lc   uc    value
ON/OFF           q    p    1 for on, 0 for off
FLOOR            f    g    Excluding floor method
Floor method     h    i    r, p, b or d
SORTBY           s    t    0 for requests, 1 for pages, 2 for bytes,
                           3 for date, 4 for alphabetical, 5 for random
SUB              j         (Where applicable)
SUBFLOOR         w    x    As above
Subfloor method  y    z    As above
SUBSORTBY        u    v    As above
COLS             c    d
Report INCLUDE   l    m
Report EXCLUDE   n    o

Items

Browser       b
Referrer      f
File          r
Host          s
User          u
Virtual host  v

LOWMEM    k
INCLUDE   x
EXCLUDE   z

Miscellaneous

Command        Code    Value/Notes

ALLBACK        ab      1 for on, 0 for off
BASEURL        ba
CASE           ca      1 for sensitive, 0 for insensitive
CONFIGFILE     cg/cm   See above
COMPSEP        cp
               cr      Charset of language file
DNSGOODHOURS   da
DNSBADHOURS    db
DECPOINT       de
DOMFILE        df
DIRSUFFIX      di
DNSFILE        dn      Also sets DNS READ; o/wise DNS is set to NONE
FROM           fr
MINGRAPHWIDTH  gw
HOSTNAME       hn
HOSTURL        hu
IMAGEDIR       ie
LANGUAGE       la      Name of language: LANGFILE overrides
CACHEFILE      lc
LANGFILE       lf      Overrides LANGUAGE
LOGO           lg
LOGFILE        lo
LASTSEVEN      ls
LOGTIMEOFFSET  lt      For all logfiles
REFLINKINCLUDE lw
LINKINCLUDE    lx
REFLINKEXCLUDE ly
LINKEXCLUDE    lz
MARKCHAR       ma
OUTPUT         ot      0 for HTML, 1 for ASCII, 2 for COMPUTER
PAGEWIDTH      pw
PAGEINCLUDE    px
PAGEEXCLUDE    pz
               qv      Output configuration file, rather than run analog
RAWBYTES       rb
REPORTORDER    re
SEPCHAR        sa
REPSEPCHAR     sb
TIMEOFFSET     tm
TO             to
WARNINGS       wa      1 for on, 0 for off. If unspecified, get WARNINGS FL.
WEEKBEGINSON   wb      0 for Sunday, 1 for Monday, ..., 6 for Saturday
GOTOS          xp
GENERAL        xq
REFARGSINCLUDE yw
ARGSINCLUDE    yx
REFARGSEXCLUDE yy
ARGSEXCLUDE    yz

• How the web works. This section discusses what happens when somebody connects to your web site, and what you can and can't find out about them. If you think that you can get statistics on how many people have visited your web site (or want to know why you can't), then this section is for you.
• Analog's definitions. This section gives precise details on what is included in each report, what analog counts as a successful request, and so on.

I should say that these ideas are not new to me. In particular, I can recommend four excellent articles about this subject: Interpreting WWW Statistics by Doug Linder; Making Sense of Web Usage Statistics by Dana Noonan; Getting Real about Usage Statistics by Tim Stehle; and, the most negative of all, Why Web Usage Statistics are (Worse Than) Meaningless by Jeff Goldberg.

So, what do you know about it? First, I make one request for your front page. You know the date and time of the request and which page I asked for (of course), and the internet address of my computer (my host). I also usually tell you which page referred me to your site, and the make and model of my browser. I do not tell you my user name or my e-mail address.

Next, I look at the page (or rather my browser does) to see if it's got any graphics on it. If so, and if I've got image loading turned on in my browser, I make a separate connection to retrieve each of these graphics. I never log into your site: I just make a sequence of requests, one for each new file I want to download. The referring page for each of these graphics is your front page. Maybe there are 10 graphics on your front page. Then so far I've made 11 requests to your server.

After that, I go and visit some of your other pages, making a new request for each page and graphic that I want. Finally, I follow a link out of your site. You never know about that at all. I just connect to the next site without telling you.

The other sort of cache is on a larger scale. I'm in the UK. Because the link across the Atlantic is sometimes very congested, we've set up a national cache. (Many individual ISP's also do the same thing.) I can set my browser to get your pages from the national cache instead of directly from you. If anyone else in the country has used the cache to look at your pages recently, the cache will have saved them, and will give them out to me without ever telling you about it. So hundreds of people could read your pages, even though you'd only sent it out once. Also, if the page I wanted wasn't already stored in the cache, the cache would ask for it from you on my behalf. This would mean that the request appeared to come from the cache, rather than from me. If several people did this, you would think that only one host was accessing the cache, rather than lots of different ones.

You can also know what people told you their browsers were, and what the referring pages were. You should be aware, though, that many browsers lie deliberately about what sort of browser they are, or even let users configure the browser name. Also, a few browsers send incorrect referrers, telling you the last page that the user was on even if they weren't referred by that page.

1. You can't tell the identity of your readers. Unless you explicitly require users to provide a password, you don't know who connected or what their e-mail addresses are.
2. You can't tell how many visitors you've had. You can guess by looking at the number of distinct hosts that have requested things from you. But this is not always a good estimate for three reasons. First, if users get your pages from a local cache server, you will never know about it. Secondly, sometimes many users appear to connect from the same host: either users from the same company or ISP, or users using the same cache server. Finally, sometimes one user appears to connect from many different hosts. AOL now allocates users a different hostname for every request. So if your home page has 10 graphics on, and an AOL user visits it, most programs will count that as 11 different visitors!
3. You can't tell how many visits you've had. Many programs, under pressure from advertisers' organisations, define a "visit" (or "session") as a sequence of requests from the same host until there is a half-hour gap. This is an unsound method for several reasons. First, it assumes that each host corresponds to a separate person and vice versa. This is simply not true in the real world, as discussed in the last paragraph. Secondly, it assumes that there is never a half-hour gap in a genuine visit. This is also untrue. I quite often follow a link out of a site, then step back in my browser and continue with the first site from where I left off. Should it really matter whether I do this 29 or 31 minutes later? Finally, to make the computation tractable, such programs also need to assume that your logfile is in chronological order: it isn't always, and analog will produce the same results however you jumble the lines up.
4. Cookies don't solve these problems. Some sites try to count their visitors by using cookies. But this can only work if you refuse to let people read your pages who can't or won't take a cookie. And you still have to assume that your visitors will use the same cookie for their next request.
5. You can't follow a person's path through your site. Even if you assume that each person corresponds one-to-one to a host, you don't know their path through your site. It's very common for people to go back to pages they've downloaded before. You never know about these subsequent visits to that page, because their browser has cached them. So you can't track their path through your site accurately.
6. You often can't tell where they entered your site, or where they found out about you from. If they are using a cache server, they will often be able to retrieve your home page from their cache, but not all of the subsequent pages they want to read. Then the first page you know about them requesting will be one in the middle of their true visit.
7. You can't tell how they left your site, or where they went next. They never tell you about their connection to another site, so there's no way for you to know about it.
8. You can't tell how long people spent reading each page. The same comments apply as in the previous paragraph. You can't tell which pages they are reading between successive requests for pages. They might be reading some pages they downloaded earlier. They might have followed a link out of your site, and they might or might not return later. They might have interrupted their reading for a quick game of Minesweeper. You just don't know.

I've presented a somewhat negative view on this page, emphasising what you can't find out. Web statistics are still informative: it's just important not to slip from "this page has received 30,000 requests" to "30,000 people have read this page." In some sense these problems are not really new to the web -- they are present just as much in print media too. For example, you only know how many magazines you've sold, not how many people have read them. In print media we have learnt to live with these issues, using the data which are available, and it would be better if we did on the web too, rather than making up spurious numbers.

The host is the computer which has asked you for a file. The file might be a page (i.e., an HTML document) or it might be something else, such as an image. The total requests counts all the files which have been requested, including pages, graphics, etc. (Some people call this the number of hits, but that word is used in different ways by different people, so I avoid it). The requests for pages obviously only counts pages. The referrer for a request is the place that the user (or his computer) heard about your file from. If he followed a link to reach a page, it will be the previous page. In the case of a graphic on a page, the referrer will be the page containing the graphic.

First, successful requests are those with HTTP status codes in the 200's (where the document was returned) or with code 304 (where the document was requested but was not needed because it had not been recently modified and the user could use a cached copy). Sometimes the logfile line doesn't contain a status code. These lines are also assumed by analog to be successes.

Redirected requests are those with other codes in the 300's, indicating that the user was directed to a different file instead. The most common cause of these requests is that the user has incorrectly requested a directory name without the trailing slash. The server replies with a redirection ("you probably mean the following") and the user then makes a second connection to get the correct document (although usually the browser does it automatically without the user's intervention or knowledge). The other common cause of redirected requests is their use as "click-thru" advertising banners.

Failed requests are those with codes in the 400's (error in request) or 500's (server error). They come about for a variety of reasons, but the most common are when the requested file is not found or is read-protected.

Finally, requests returning informational status code are those with status codes in the 100's. These are very rare at the moment.

There are a few other types of logfile lines listed in the General Summary. Lines without status code refers to those logfile lines without a status code, and the successful requests in the General Summary only counts the ones with a status code: except if the line contains the name of the file requested, and the filename is being counted (not starred in the LOGFORMAT), then it's listed in the successes. Corrupt logfile lines are those which analog didn't manage to parse. And unwanted logfile entries are ones which we have specifically excluded. Successful requests for pages refers to those lines on which the file requested was given and was defined as a page by the PAGEINCLUDE command.

The "not listed" line at the bottom of each of the non-time reports includes both those items which were explicitly excluded at the output stage with an OUTPUTEXCLUDE command, and those which were not listed because they were below the floor for the report.

The figures in parentheses in the General Summary are for the last seven days: either the seven days before the TO time, or if no TO time is given, the seven days before the time of the program start. (It would be nicer to use the seven days before the last time in the logfile, but we don't know when this is until we've read the whole logfile, and by then it's too late.) The figures for the last seven days are not included if all, or none, of the requests fall in the last seven days.

In the Domain Report, "domain not given" means that the hostname did not contain a dot. "Unknown domain" means that it did contain a dot, but that the domain name was not in the domains file.

First, you should understand the difference between a crash, an error, a warning, and a debugging message. First, a crash is when analog exits prematurely, without producing the whole output file. The system might give a message, but analog will not give one of its own messages. Analog should never crash. If it does crash, please tell me about it.

An error is something which stops analog finishing its job. Whenever an error is detected, analog gives a message starting something like analog: Fatal error: and will then tell you what type of thing went wrong before quitting.

A warning is a problem which is not fatal to analog: it will keep on with its processing. These vary from the possibly serious, such as files which could not be found, to purely informational. They produce a message starting analog: Warning. You can turn warnings off using the WARNINGS command.

Finally, a debugging message gives information on the state of the program. They just begin with a single code letter followed by a colon. You don't get any debugging messages unless you've asked for them.

Now I shall describe all the possible errors and warnings in detail.

Errors

Ran out of memory: cannot continue

Analog ran out of memory. Try increasing the memory available to the process, if your operating system will allow it, or using the LOWMEM commands.

Cannot ignore mandatory configuration file

See the section in the Readme on the mandatory configuration file.

Can't find language file
Language file too short
Language file contains excessively long lines

Analog can't run without a well-formed language file. See the documentation on language files.

Attempted to read more than 50 configuration files

The most likely explanation for this is that you have accidentally created a loop using the CONFIGFILE command.

Incorrect default given in analhead.h
Default given in analhead.h too short

If you've compiled your own version, and you've specified an incorrect configuration in the file analhead.h, analog gives up to allow you to fix it.

Failed to open output file for writing

Analog couldn't create, or couldn't write to, the output file you specified.

Cache output file already exists: won't overwrite

Analog won't overwrite an old cache file. You must move or delete it yourself first.

OUTFILE and CACHEOUTFILE are the same

This can't be what you wanted, because one would overwrite the other.

OUTPUT NONE and CACHEOUTFILE none selected

You requested no output.

Warnings

Category C

Category D

LOGFORMAT with no subsequent logfile

You have specified a LOGFORMAT command, but no subsequent logfile to which it could be applied. Most likely you put the LOGFORMAT after the LOGFILE command. You must put the LOGFORMAT before the LOGFILE command or use DEFAULTLOGFORMAT instead. See the section on Choosing a logfile for more details.

Offset not a multiple of 30
Offset more than 25 hours

The time offsets are meant to be for correcting between differences in time zones. These differences are usually multiples of 30 minutes between -25 and +25 hours. Maybe you specified the offset in hours instead of minutes by mistake, or something like that.

SORTBY doesn't match FLOOR
SORTBY (or FLOOR) isn't included in COLS

Within one report, it's helpful to your readers to have the sort method and the floor compatible, and both included in the COLS. (See the section on Non-time reports).

Time reports have not all got same value of BACK

It's usually helpful to have all the time reports running in the same direction.

Report contains no COLS

You've got an empty COLS list for one report, so you'll just get a list of names, not any information about them.

LOWMEM 3 prevents that item being cached

You're making a cache file, but one item is not being recorded because of a LOWMEM command, and will therefore not be saved in the cache file.

OUTFILE and CACHEOUTFILE both set to stdout

Self-explanatory.

Category E

Redirecting future diagnostic messages

You've used an ERRFILE command to change the destination of errors, warnings, debugging and PROGRESSFREQ diagnostics. This is just warning you so that you don't miss any messages.

Category F

Can't auto-detect format of logfile

The LOGFORMAT is set to automatic detection, but the first line of the logfile is not in any of the standard formats. This error can often be generated when you try and specify your own LOGFORMAT but put it after the LOGFILE command so that it is not in effect for that logfile.

DNS lock file already exists

To stop two copies of analog trying to write the DNS file at the same time, an empty "lock file" is created, which tells the second copy of analog to use DNS LOOKUP instead of DNS WRITE. If analog crashes, it may not delete its lock file. So if you get the "already exists" message even though no other copy of analog is running, you may need to delete the lock file yourself.

Category L

Large number of corrupt lines

This could indicate a problem with the logfile, or with the LOGFORMAT specification. The possible causes are described in the page about Choosing a logfile.

Logfiles overlap: possible double counting

Two logfiles which were counting the same type of item overlapped in time. Maybe you read two copies of the same logfile. Or maybe the LOGFORMAT specification should have told analog to ignore some of the items. Or it could be that the logfiles are in fact disjoint and there wasn't really a problem: analog only checks the dates of the logfiles, not the details.

Category M

Logfile contains lines with no [whatevers], which are being filtered

This is usually harmless. It is perhaps best explained by example. Suppose you are excluding certain files from the analysis, but that you are also analysing a browser log which just contains information about the browsers used, not which files they read. Then we can't exclude the browsers which read the excluded files, because we don't know which they were, so all browsers will be included.

Logfile contains lines with no file names (or bytes): page (or byte) counts may be low

If a logfile line doesn't contain a file name, analog will assume that the request wasn't for a page. Similarly, if it doesn't give the number of bytes transferred, analog will make the bytes zero. So the number of page requests or bytes credited to the other items on that line will then be too low.

Category R

Broken Pipe

1. Getting Started
2. Basic Configuration
3. Understanding the Output
4. Advanced Usage
5. Form Interface
6. Design Decisions

A. Getting Started

1. Analog just flashes up a DOS window and then quits.
   This is the correct behaviour. It should have created a report called Report.html. See Starting to use analog under Windows.
2. When I try and compile analog, it gives me an error.
   Maybe you need to edit the Makefile. There are some platform-specific notes in the section Starting to use analog on other platforms, and in the Makefile itself.
3. Analog didn't write the logfile when I ran it.
   Analog doesn't write the logfiles. Your web server writes the logfiles, and analog just reads them. See Starting to use analog.
4. Analog is looking for files like /usr/local/etc/httpd/analog/analog.cfg which don't exist.
   You have to set the location of these files in analhead.h before compiling.
5. Analog won't read extended logfiles generated by IIS.
   This server writes the date only at the top of the logfile, not on every line. But it doesn't write a new date if the date changes during the logfile, so analog can't tell which date later entries in the log occurred on. More details, and what to do about it, are in the section on Choosing a logfile.
6. What does this error (or warning) mean?
   See the section on Errors and warnings.
7. I tried to run analog from my browser, but it didn't work.
   You can't run analog itself directly from a browser. You need to use the CGI program which outputs the correct HTTP headers.
8. Is analog Year 2000 compatible?
   Yes (and so are all previous versions). It interprets two-year dates in input as lying between 1970 & 2069 inclusive.

B. Basic Configuration

1. I want to make several different statistics pages. Do I have to install several copies of analog?
   No. Just install it once, and run it with different configuration files.
2. Why doesn't the Daily Report only show the last six weeks?
   This is controlled by the FULLDAYROWS command.
3. How do I get information on just my pages, not everybody's?
   Use the FILEINCLUDE command.
4. How do I ignore accesses from my site?
   Use the HOSTEXCLUDE command.
5. How do I get the Request Report to list files with fewer than 20 requests?
   Use the REQFLOOR command.
6. How can I combine all the hosts from each company in the Host Report?
   You can see this information in the Domain Report if you use the SUBDOMAIN command: for example SUBDOMAIN *.com. (Or you can several HOSTALIAS commands -- but I think the first solution is better).
7. I want to list (or not to list) referrers with their search arguments in the Referrer Report
   To see the search arguments you may need to set the REFARGSFLOOR lower. To avoid seeing them, you could set the REFARGSFLOOR higher, or alternatively use the REFARGSEXCLUDE command to ignore them either for all files or just for particular files.
8. Can I find out the number of hosts that have accessed each file?
   or Can I find out the number of hosts visiting on each day?
   No; it would require storing too much data (all host/file pairs, or all host/day pairs). If there's a particular file you're interested in, use FILEINCLUDE to restrict the analysis to only that file. If there's a particular day you're interested in, use FROM and TO to restrict the analysis to only that day. You can see the last date on which each host visited by using the HOSTCOLS command.
9. Can I get data on individual visitors, or visits, to my site?
   No, it's not technically possible, and don't believe any program which tells you it is. See the section on How the web works for details.

C. Understanding the Output

1. How do I find out the number of hits from your data?
   I don't use the word hits, because people use it in different ways, so it's misleading. I use requests for the number of transfers of any type of file (text, graphics, ...), and page requests for the number of transfers of HTML pages. See the section on Analog's definitions for more information.
2. Why are there so many referrers from my own site?
   These come from all the internal links on your site, and all the graphics on your pages. See the section on How the web works for more information. If you don't want to see them, you can use REFREPEXCLUDE to exclude them.
3. Why doesn't analog agree with the counter on my page?
   There are lots of possible reasons. Do they both start from the same date? Are you just looking at requests for that one page with analog, not for all your other pages and graphics? Also, analog will record all requests to that page; if it's a graphic, your counter will only measure requests from people on graphical browsers that reached that place on the page.
4. Why do I only get "unresolved numerical addresses" in the domain report?
   Your server only records the numerical IP address of the hosts that contact you, not their names. Read the section about DNS lookups, or turn DNS resolution on in your server..
5. Why are my CGI scripts not listed in the Request Report?
   If they cause a redirection to another page, they will be listed in the Redirection Report, rather than the Request Report.
6. Why are directories listed in the Request Report?
   They are not directories, they are pages with the same name as the directory. For example, I have both a directory called /analog/ and a page called /analog/ (which happens to be the same as /analog/index.html).
7. When someone reads one of my pdf files, it scores dozens of hits.
   PDF files are often downloaded and read one page at a time, and each page will then count as a separate request. Although this is not ideal, it's much less clear what to do about it. Analog has no way of knowing how many pages constituted a single download in the reader's mind. As usual, we can only reliably report how many requests there were at the server, not guess what users did with the file later.

D. Advanced Usage

1. How can I do such-and-such with a command line option?
   Use the +C option to put any configuration command on the command line.
2. I want a list of all command line arguments.
   There is a list in the index.
3. How can I run analog automatically every day?
   This depends on your particular machine. On Unix, you need to run analog as a cron job (see "man cron"). This is my cron command:
   20 1 * * * $HOME/misc/analog
   On Windows NT you can do the same with the at command, but only an administrator can run at. On Windows 98, it should be possible with the Task Scheduler, although I haven't tried it. On Windows 95 it's not possible as far as I know.
   On Mac, there are programs called Cron or CronoTask to do this.
4. I host lots of virtual domains. How should I set up analog?
   In my opinion, if you haven't got many virtual domains, the best thing is to log each one to a different logfile and analyse them independently. If you have got lots of virtual domains, you may have to log them all to the same logfile. In that case, make sure to log the virtual host name on each line. Tell analog the LOGFORMAT if necessary, and then use the VHOSTINCLUDE command to pick out the lines you want. Alternatively, split the single logfile into separate logfiles before running analog on it; this avoids making several passes through the whole logfile.
5. Can I make multiple reports with one pass through the logfile?
   Not at the moment. I want to do this in a future version, but it will require some considerable work.
6. I ran out of memory when trying to run analog. What can I do?
   See the section on Coping with low memory.
7. You're processing 20,000,000 requests in under 10 minutes. Why is mine much slower?
   or Analog appears to stall.
   If you have DNS lookups on, they are very slow. Otherwise, it probably depends on the speed of your computer and disks, and what other programs are running at the same time. You can use the PROGRESSFREQ command to see if it's really stalled or whether it's just being slow. If you are running out of memory, you might find analog's LOWMEM commands helpful.
8. How do I make a link on my page that runs analog?
   Link to the anlgform program, with the desired options. But be careful about the load on your server.
9. Do I have to save all my old logfiles?
   This is answered in the section about Cache files.

E. Form Interface

1. My browser asked me to save anlgform.exe, rather than running it.
   You have to tell the server to execute the CGI program, not just send it out like it would for a normal file. Often this is done by putting it in a special /cgi-bin/ directory.
2. Why does the form interface give "Document Returned no Data"?
   If it doesn't happen for a while, then probably the server is giving up before the analog process has finished running. Increase the timeout interval on the server.
3. The images don't appear when running analog from the form interface.
   You probably need to change the IMAGEDIR. If the images are in your /cgi-bin/ directory, the server will try to execute them instead of just sending them out.
4. How can I specify different logfiles from the form interface?
   Just add a new field to the form with name=lo
5. Why do I get some reports that weren't requested on the form?
   If a report is neither included nor excluded on the form, the system default will be used. This will depend on your configuration files and on compile-time settings.

F. Design Decisions

1. Why don't you just use one image, and scale it with the WIDTH and HEIGHT attributes?
   or Can I change the background colour of my output?
   or Why not use HTML tables?
   These things don't work in HTML 2.0, which is the sort of HTML analog writes.
2. Well then, why not use HTML 4.0?
   Unfortunately my bar charts don't work in HTML 4.0. Also, not all browsers can cope with some features of HTML 4.0, such as tables. HTML 2.0 works on all browsers.
3. Why not just do DNS resolution of the hosts that actually make it into the Host Report?
   There is one theoretical and one practical problem. Theoretically, the problem is that which hosts do make it into the Host Report can change when the DNS lookups have been done. And practically, this wouldn't help identify the busiest countries or organisations, which is usually what you really want to know. However, there is a Perl script on the helper applications page to do this.
4. Couldn't you do the DNS lookups faster with threads?
   The problem is, the standard commands for DNS lookups are not thread-safe on many platforms, so it would involve a lot of platform-specific code. Again, there are programs for specific platforms on the helper applications page.
5. Why doesn't analog analyse the error_log?
   This is answered in detail on the What's new? page. But in summary, it's too difficult because each server has a different format for its error log. The various failure reports are good enough for most purposes.
6. My server lists local names in the logfile. Can you put a common suffix on them automatically?
   This wouldn't be a good idea, because things like "unknown" would get the suffix. You can always add them using HOSTALIAS.
7. Can you extrapolate from the current month's partial data to produce a prediction for the whole month, based on the rate so far?
   No. There are too many problems in trying to produce anything sensible, especially near the beginning of the month. Different days of the week and different times of day cause lots of problems. I would prefer to produce raw accurate data than suspect derived data.
8. Can you extend the Domain Report to say which US states people visited from?
   No. Some programs pretend to do this, but you can actually only tell which state the computer the person was using is in, which may be quite different from where the user was for ISP's or other large organisations.
9. Why not use language codes instead of country codes for the names of the language files?
   People are more familiar with the country codes. And not all of my languages have language codes anyway.
10. Why don't you sell analog?
    I didn't write analog for the money, and I'm happy just to see people use it. Also, by making it freeware, lots of people send me ideas and code to include in future versions. How do you think I got all those languages? (Of course, if you want to send me money, or gifts in kind, or even just postcards...).

There are three mailing lists for analog.

analog-announce

Announcements about analog. I post to this when there are new versions, for example. Usually only gets a few messages a year.

analog-help

Getting help with analog from experienced users. This is the place to go if you have trouble setting up or configuring the program. There is also a searchable web archive of this list.

analog-author

This just goes to me. Use for private comments.

To subscribe to the analog-announce mailing list, send a message to analog-announce-request@lists.isite.net with the word subscribe in the main body of the message. Note that the word has to be in the body of the message, not the subject. Also please note the -request part of the address.

If you want to get help with analog, please check the following simple things first.

1. Read the FAQ. Maybe I've answered your question already. If I have, I'll just direct you to the FAQ, not answer it again.
2. Read the list of known bugs at my site, to see if your bug is already known about.
3. Read the other relevant pages of the Readme, particularly the sections on Starting to use analog and Customising analog. You may also find the index useful. I don't appreciate people who are too lazy to read the documentation. (If the documentation is unclear, or the relevant paragraph is too well hidden, then that's a different matter. Of course I want to know about that.)
4. Have a look in the web archive of the mailing list to see if your question has already been answered there.
5. If analog isn't doing what you thought you asked it to, then run it with the PRINTVARS ON configuration command, and see what options it thinks it's meant to be using.

If you still need help, write to the analog-help mailing list. First you have to subscribe (you can't send mail without subscribing) by sending a message to analog-help-request@lists.isite.net with the word subscribe in the main body of the message. (Note that the word has to be in the body of the message, not the subject.) After you've received an acknowledgement that you have subscribed, you can send mail to analog-help@lists.isite.net. Don't try and use this address for subscribing though. It won't work!

Please do the following when you send mail to the list.

1. Describe exactly what you did, what you expected, and what the computer did. Include the exact text of any error messages, not a précis.
2. Mention which version of analog you are using, on which operating system.
3. Give your mail a subject line which indicates immediately what aspect of analog it is about. (This is useful for the archive).
4. Do not send long files or attachments unless you're asked to. We do not want to see your configuration file, your header file, your output file, or any logfile over 10 lines long. They are almost always useless to us. And anyway, excessively long messages will be rejected by the mailing list server.

If you want to send a private message to me, you can send it to me at analog-author@lists.isite.net.

Many thanks to ISite for providing these mailing lists for me, and to The Mail Archive for archiving the analog-help list.

These helper applications are all listed at the analog site. The list is growing quite quickly, so I'm not distributing it with the program. But I strongly recommend you go to the analog home page (or even better, to your local mirror site) and check it out.

Thanks are due to the author of getstats, Kevin Hughes. In the days before analog there were only three serious logfile analysis programs, and only one of them, getstats, had attractive output. I wrote analog when getstats stopped being able to cope with the size of our logfile, but my output still looks somewhat similar to his.

Thanks are also due to all those who helped in the early stages of writing this program, and gave me the encouragement to continue with analog and to release it publicly. Those who made helpful suggestions during beta testing are numerous, but I must mention particularly Dan Anderson, Martyn Johnson, Joe Ramey, Chris Ritson, Quentin Stafford-Fraser and Dave Stanworth. Above all Gareth McCaughan gave me lots of programming advice. The program would have run much more slowly without him.

Many people have provided mirror sites for analog, starting with Dave Stanworth (again!). The full list of mirror sites is listed elsewhere; thanks to all of them. Many thanks also to ISite for providing the mailing lists, and to The Mail Archive for archiving the analog-help list.

Mark Roedel first suggested porting analog to different platforms, and made the original DOS port. Shortly afterwards, Jason Linhart made the Mac port, and has continued to contribute lots of extra code for that platform and for the program in general. The Mac version also includes code contributed by Stephan Somogyi and Nigel Perry, and uses the ZLib library by Jean-loup Gailly & Mark Adler. Later ports were made by Dave Jones (VMS), Magnus Hagander (Win32), Nick Smith (Acorn RiscOS), Scott Tadman (BeOS), and Martin Kraemer & Holger Schranz (BS2000/OSD). Ivan Martinez compiles the OS/2 version. The BS2000/OSD port includes code developed by the Apache Group for use in the Apache HTTP server project. If NEED_MEMMOVE is defined at compile time, then this product includes software developed by the University of California, Berkeley and its contributors.

Thanks to all the other people who have contributed bits of code too. I apologise for not having room to name all of them. And thanks to those who have written standalone helper applications, for making analog more usable.

For the translations into other languages, many thanks are due to the following: Francesc Rocher, M. Mercè Llauge & Francesc Burrull i Mestres (Catalan), Yang Meng (Chinese), Jan Simek (Czech), Adrian Price (Danish), Ferry van het Groenewoud & Joost Baaij (Dutch), Henrik Huhtinen, Steve Kelly & Andrew Staples (Finnish), Patrice Lafont, Lucien Vieira, Jean-Marc Coursimault & Lionel Delaude (French), Mario Ellebrecht, Martin Kraemer, Holger Schranz & Thomas Jacob (German), Dimitris Xenakis (Greek), Laszlo Nemeth (Hungarian), Gustaf Gustafsson (Icelandic), Furio Ercolessi (Italian), Takayuki Matsuki, Stephen Obenski and Motonobu Takahashi (Japanese), Byungkwan Kim (Korean), Jurijs Turjanskis (Latvian), Ingrid (Lithuanian), Jan-Aage Bruvoll, Espen Bjarnø & Pâl Løberg (Norwegian Bokmâl), Magni Onsøien (Norwegian Nynorsk), Wlodek Lapot & Tomek Wozniak (Polish), Ivan Martinez (Brazilian Portuguese), Jaime Carvalho e Silva (European Portugese), Alex Mihaila (Romanian), San Sanych Timofeev & Boris Litvinenko (Russian), Stefan Billik (Slovak), Andrej Zizmond (Slovene), Javier Solis, Alexander Velasquez, Alfredo Sola & Martin Perez (Spanish), Björn Malmberg (Swedish), and Nezih Erkman (Turkish).

Finally, thanks to all of you for using the program!

3.3 (19-May-99)

New commands ERRFILE, DNSLOCKFILE, APACHELOGFORMAT and APACHEDEFAULTLOGFORMAT.
Can include the date in the name of the OUTFILE and the CACHEOUTFILE.
Support for WebSite logfiles.
New token %U in log formats for "Unix time" (seconds since 1970).
Won't overwrite old cache files.
Now works properly on SunOS 4.
Fix for occasional crashes on Windows.
Checks language files are not too long.
"Last seven days" data now calculated more accurately and displayed more clearly.
Computer-readable output now reports SORTBY's as well as floors.
Revised Makefile will work with older make's.
Corrected Catalan language files.
Includes form interfaces in French and Japanese.
LOGFORMAT documentation now includes the LOGFORMAT commands for all built-in log formats.

3.2 (04-May-99)

Bug fixes: in particular REFLINKINCLUDE pages now works; and cache files now include all items even if they're not wanted for the main report.
Lines without a particular item now work properly with INCLUDE and EXCLUDE commands. This can cause differences in results from previous versions.
New version of form interface to work round bug in Microsoft Internet Information Server.
New command NOROBOTS.
Backslashes are now coerced to forward slashes in filenames and user names. While not always technically correct, it often is, and it makes them behave correctly in other parts of the program.
User names are now treated as case insensitive. Let me know if this causes a problem on any system.
Computer-readable output style now reports floors.
Rewritten Unix Makefile, and VMS build script. Let me know of any compilation problems.
New languages: Catalan, Icelandic, Japanese, Korean, Latvian, Lithuanian. Corrected Spanish language files and French domains file.
LANGUAGE now selects local domains file automatically, where available.
Removed support for NetPresenz logs. The reasons are in the section on how to upgrade.
Form interface documentation rewritten; FAQ broken into sections; sections on logfiles and log formats separated and rewritten; new section on helper applications; and dozens of other improvements to the documentation.

3.11 (26-Nov-98)

Bug fix version.
Microsoft's attempt at W3 extended format is now understood even if there is a second #Fields: line in the logfile.
There is also a fix for a new Microsoft bug which results in an non-standard common format.
Intermittent crashes under Windows fixed.
Mailing lists announced.

3.1 (17-Oct-98)

Understands Microsoft's attempt at W3 extended format.
Several bugs fixed, including one that caused occasional crashes and one that caused the output to grow and grow.
Form interface works on Windows.
Allows aliases with two or more *'s on left hand side, if right hand side contains no *'s.
Aliases work properly with CASE INSENSITIVE.
Numerical SUBDOMAINs fixed.
Understands more WebSTAR and Netscape tokens.
Accents in domains file work.
LOGFORMAT removed from form interface as security risk.
Several warning messages improved.
Report aliases and in/exclusions shown in settings output.
Character set declared at top of output.
Spanish, Dutch, Norwegian (Bokmål and Nynorsk), Finnish, Turkish, Greek, Polish, Russian & Chinese language files included.

3.0 (15-Jun-98)

Corrected W3 extended format.
Fix for broken strcmp() function on SunOS 5.
Portuguese, Brazilian Portuguese, Danish and Hungarian language files included.
Precompiled executable for OS/2 available.

2.91beta1 (04-Jun-98)

Form interface included.
Uses less memory when compiling reports.
New operating system, BS2000/OSD, and code for EBCDIC character set.
New command DEFAULTLOGFORMAT.
LASTSEVEN and BASEURL reinstated.
More information added to PRINTVARS output.
AppleScript support for Unix-style command lines added to Mac version.
Now works on SunOS 4, and other small bug fixes.
French, German, Swedish, Czech, Slovak, Slovene and Romanian language files included.
One page version of the Readme included in the documentation.

2.90beta4 (09-Apr-98)

Mended DNS cache file reading, which I broke in yesterday's release.

2.90beta3 (08-Apr-98)

Fixed bug that caused a crash while giving warning messages on SunOS; bug that caused configuration files that called other configuration files not to be completed; and other smaller bugs.
Italian language files included.

2.90beta2 (03-Apr-98)

Separate LOGFORMATs for North American and international date formats, when using Microsoft or Netpresenz logs.
Understands the AppleShare IP server's attempt at the WebSTAR format.
Directory report now works properly even if you use the second argument to the LOGFILE command.
Wild cards in filenames work properly on the Mac.
Other small bug fixes.
One speed improvement (I gain about 3%).
Several corrections and clarifications to the documentation.

2.90beta1 (27-Mar-98)

This version is a completely rewritten version. Every single line of code is new. The whole code is shorter despite considerable improvements in functionality. Several people have reported that it is significantly faster. The most important new features are:

• Eleven new reports (Quarter-Hour, Five-Minute, Redirection, Failure, File Size, Referring Site, Redirected Referrer, Failed Referrer, Virtual Host, User, User Failure).
• Reads logfiles in user-customisable format.
• Analyses user and virtual host data, and failed requests.
• Hierarchical reports list subdirectories under directories, and allow analysis of browser version numbers.
• Faster sorting of long reports.
• Floor and sort method made independent.
• "Last date" column in reports, and can floor and sort by date.
• Busiest time period at bottom of time reports.
• "Not listed" line at bottom of other report.
• Knows HTTP/1.1 status codes.
• General Summary can go anywhere in the report.
• General Summary and "Go To"s can now be turned on and off independently.
• Status Code Report can be sorted in different ways.
• Time offset commands.
• Much better checking of invalid configuration options and invalid logfile lines.
• Only reads logfiles it might need.
• Improvements in DNS functionality: can now read the DNS file without further lookups: also, separate recheck intervals for successful and failed lookups.
• Hash sizes now chosen automatically.
• More flexible language support.
• Mac version reads gzipped logfiles.
• Mac version supports drag-and-drop onto program icon.
• Readme files completely re-written. Broken into lots of files, and new sections on Starting to use analog and What the results mean, as well as an index.

The following features have been abolished.

• No Error Report. The error log was always intended for humans rather than computers to read. Moreover, its format varied from server to server, and even between different versions of the same server. The place of the Error Report has largely been taken by the new reports, particularly the Failure Report.
• The approximate host counting has been abolished for the time being. I can put it back if there is a significant demand for it.
• Only one * can now appear on the left-hand side of aliases. This is to avoid ambiguities.
• For changes in the names and syntax of configuration options and command line arguments, see the page about upgrading.

The following features are not yet present, but will be added by version 3.

• The form interface.
• Most of the languages.

What was new in version 2?

What was new in version 1?

Upgrading from 3.2 and earlier

• In the computer-readable output style, the line L7, the time the last seven days begins after, has been replaced by E7, the time the last seven days ends. This is for consistency with the other output styles.
• Also in the computer-readable output, there is a new line reporting the floor and the SORTBY for the report. In 3.11 and earlier, this didn't exist, and in 3.2 it only reported the floor, not the SORTBY.
• %R (Mac-style filename) has been abolished in the LOGFORMAT. Just use plain %r instead.
• It is no longer allowed to set the CACHEOUTFILE to be the same as a previous cache file.
• The definition of the common log format and related formats changed between 3.11 & 3.2, and again between 3.2 & 3.3. This could cause differences in output, but they are likely to be only very minor.

Upgrading from 3.11 and earlier

• Lines without a particular item now work properly with INCLUDE and EXCLUDE commands. For example, if you do an INCLUDE to look at only certain lines, then lines without that type of item at all will not now be included, whereas previously they would have been. This can make the results lower than in these earlier versions.
• I have reluctantly removed support for NetPresenz logs. This hasn't worked for some time, and I have already been advising NetPresenz users not to use newer versions of analog because they could get wrong results. Unfortunately, fixing it would require a complete rewrite of the entire parsing code, which isn't going to happen any time soon. So my advice remains the same: continue to use version 2.11 or (even better) pre-process your logfiles into a form which analog can handle safely.

Upgrading from 3.0, Win32 form interface

• If using the form interface on Windows, it is now necessary to put the analog executable at \analog\analog.exe instead of \Program Files\analog\analog.exe

Upgrading from 2.90beta1

• LOGFORMAT MICROSOFT has been replaced by LOGFORMAT MICROSOFT-NA and LOGFORMAT MICROSOFT-INT; and similarly for LOGFORMAT NETPRESENZ.

Upgrading from 2.11 and earlier

• It is possible that there may be small discrepancies between the results from previous versions and the results from this version because the parsing code has changed, but any such differences should be minor. However...
• If you used to use REFEXCLUDE or BROWEXCLUDE, you most likely now want REFREPEXCLUDE or BROWREPEXCLUDE instead, or you will exclude lots of lines that were previously included.
• It is possible that this version may not automatically parse a logfile that previous versions could parse, because it checks more carefully that the logfile is in the format claimed. If so, you will have to use a LOGFORMAT command.
• Approximate host counting has been abolished, unless there's a significant demand for it.
• The Error Report has been abolished (together with the configuration commands ERROR, ERRLOG and ERRMINOCCS). See the What's new? page.
• The BROWLOG and REFLOG commands have also been abolished: just use LOGFILE instead.
• The HASHSIZE commands have been abolished: analog now chooses the size of the hash tables itself.
• The MINREQS and similar options have been replaced by the FLOOR commands.
• Only one * is now allowed on the left-hand side of aliases, to avoid ambiguities.
• Automatic detection of log type is now on a per-file rather than a per-line basis.
• ISPAGE is now called PAGEINCLUDE.
• WITHARGS and REFWITHARGS are now called ARGSINCLUDE and REFARGSINCLUDE.
• MONTHLYBACK is now called MONTHBACK.
• FULLHOSTS is now just called HOST.
• LOGOURL is now called LOGO, and is assumed to be within the IMAGEDIR.
• The UNIT commands have been abolished. They weren't very useful, and they didn't make much sense with the different ways of displaying the time report bar charts. The unit is now always chosen automatically.
• DIRLEVEL has been abolished, because the SUBDIR command is more general. Use SUBDIR */* or whatever instead.
• Comments aren't allowed in the domains file. I don't think this should cause a problem.
• GRAPHICAL is abolished. Instead, use lower case letters with the GRAPH commands.
• NUMLOOKUP has been replaced by DNS, and DNSFRESHHOURS by the commands DNSGOODHOURS and DNSBADHOURS.
• DNS cache files from previous versions are not compatible with this version.
• You can't use PAGES in the columns or SORTBY or FLOOR for the Request Report. Use REQINCLUDE pages instead.
• - as a synonym for none has been abolished in some places, e.g., HOSTURL.
• The following command line arguments have been abolished from earlier versions, many of the letters getting new meanings: 7, l, n, p, s, u, v, w. (-v has moved to -settings.) Others have been changed since version 1.2 as well.

Upgrading from 2.0, Win32 users

• Filenames for logfiles etc. should now be given DOS-style, with backslashes, rather than Unix-style with forward slashes.

Upgrading from 1.92 and earlier, Mac users

• There is no longer an automatic progress report. Use the PROGRESSFREQ command instead.

Upgrading from 1.9beta's

• Use REQINCLUDE and LINKINCLUDE instead of REQTYPE and PAGELINKS.

Upgrading from 1.2's and earlier

• Use *INCLUDE and *EXCLUDE instead of *ONLY and *IGNORE.
• The syntax of the *FLOOR commands has changed.
• Use *SORTBY REQUESTS or BYTES instead of *SORTBY BYREQUESTS or BYBYTES.

What's new in version 3?

2.11 (14-Mar-97)

Minor bug fixes to yesterday's release.

2.1 (13-Mar-97)

Language support rewritten, causing reduction in code size of 2200 lines.
New configuration command LANGFILE.
New Acorn RiscOS version.
Page requests per day reported.
Bug fix: CASE INSENSITIVE could cause %7E-type conversions not to take place.

2.0.2 (04-Mar-97)

DNS lookups and wildcards should now work in the Win32 version.
New configuration command PRINTVARS.
Fix for zero length hostnames after DNS lookups.
Minor corrections in French and Spanish translations.

2.0 (10-Feb-97)

New native Win32 version.
Wildcards allowed in filenames on Mac.
Ignores browser "-".

1.93beta (18-Jan-97)

New commands BROWALIAS, CONFIGFILE and PROGRESSFREQ.
Form program can now call configuration files.
Form program now uses the default choices if none specified.
Domain report prints correctly in preformatted output.
Specifying +1 and +V2 doesn't crash the program.
-v reports dates correctly.
Trailing dots on hostnames removed.
Second argument to LOGFILE command can't be obliterated by /../

1.92beta (08-Oct-96)

DNS lookups added on Mac.
Netpresenz format understood on Mac.
New languages: Spanish, Italian and Danish.
Extra information when debugging turned on.
*.htm are now pages on all machines.
A few small bugs fixed.

1.91beta4 (13-Jul-96)

Cache file now includes page request information.
DNS bug fixed.
New command DNSHASHSIZE.
Bug in browser reports fixed.

1.91beta3 (09-Jul-96)

BSD/OS compilation bug believed fixed.
Fixed HOSTALIAS which I broke yesterday.
DNS bug (causing too many lookups) identified, although not yet fixed.

1.91beta2 (08-Jul-96)

Some bug fixes (including: HOSTEXCLUDE and CASE INSENSITIVE didn't work properly; selecting "no links" failed on the form; less fussy about what can appear on the form).
Mac version no longer includes source code, so is much shorter.

1.91beta1 (05-Jul-96)

Now DNS code doesn't look up a name twice, even if one is a failed request.

1.91beta (05-Jul-96)

Will now output in any of several languages.
Preformatted output introduced.
New File Type Report.
Can limit the number of rows in the time reports.
Number of requests for pages (as opposed to raw requests) now calculated throughout.
DNS lookup returns, with cacheing across runs.
Logfiles can include wildcards.
Wildcards can include multiple *'s.
Can process case insensitive logfiles.
OUTPUTALIAS commands introduced.
New commands to specify exactly what is included, and what linked, in the request report and referrer report.
FILEALIAS a a and FILEALIAS a b; FILEALIAS b c now work.
New ALLOW options to cancel INCLUDES.
REPSEPCHAR and DECPOINT introduced.
DIRSUFFIX introduced.
Debugging reports number of corrupt lines in other logs.
Hash sizes can now be allocated at run time.
stdin can now be used for any input file, but not for two.
Macintosh version now quits automatically if no warnings have been issued.
Form interface made more secure.
"Mozilla (compatible)" separated out in Browser Summary.
Major internal changes should improve speed.
Code for non-Unix platforms integrated into main code.
"Referrer" spelled correctly.
Licence introduced.
Update file introduced.
Readme updated to include non-Unix instructions.

(19-Apr-96)

First Mac version.

1.9beta6

Two bug fixes (number of bytes was incorrectly reported in some cases, and -v would overwrite the OUTFILE).
Documentation improved.

1.9beta5

More bug fixes...

1.9beta4

One important bug fix (I broke GRAPHICAL OFF in 1.9beta3).
New form cgi options: ch, gr and ou=3.
Code shortened.

(05-Mar-96)

First DOS version.

1.9beta3

Mainly bug fixes and improved documentation.
Browser and referer reports now include failed requests.
The WARNINGS option can now be specified on the form.

1.9beta2

Small bug fixes

1.9beta (06-Feb-96)

Lots of changes. The most important new features are

• Six new reports (hourly report, browser report, browser summary, referer report, status code report and error report).
• Analysis of NCSA/Apache referer log, agent log and combined log formats.
• Graphical time reports that still work on text-based browsers.
• Configurable columns in the time reports.
• Time reports can run backwards.
• Time graphs can be plotted by bytes instead of by requests.
• Can cache old data so that old logfiles need not be kept.
• Can process several logfiles.
• Can combine logfiles from several different hosts.
• Will uncompress compressed logfiles.
• All configuration options can now be specified on the commandline.
• Mandatory configuration file added.
• Lots of new options in the form processing program.
• Wildcards greatly improved throughout.
• Alphabetical host report right-aligned.
• Bytes now quoted as MBytes etc. instead of long number.
• Produces HTML2.0 compliant output.
• New sort method RANDOM (saves time for long reports).
• Floors for reports now work properly.
• Can now specify a report FROM 100 or more days ago.
• Option to turn off warnings.
• Considerable savings in code length over previous versions.

What was new in version 1?

What's new in version 3?

What was new in version 2?

1.2.6

Minor bug fix; will only affect those with corrupt logfiles.

1.2.5

Minor bug fix for weekly report.

1.2.4

Patch for Spyglass server logfile format.

1.2.3

A couple of bug fixes (wild subdomains sometimes caused crashes).
-v option now gives the version number.

1.2.2

Patch for proxy servers: http:// not translated to http:/

1.2 (11-Nov-95)

Can configure columns in reports to give percentage requests and number of bytes.
Wild subdomains (e.g., *.com).
Nameless subdomains.
Subdomains now listed in alphabetical order.
Proper support for numerical hostnames in HOSTIGNORE, HOSTONLY, SUBDOMAIN and alphabetical sorting.
New BASEURL command allowing statistics to be displayed on other servers.
Output always says how things are sorted.
"Last 7 days" now behaves sensibly with TO.
Filenames containing /../, /./ and // translated.
Header and footer options removed from form (for security reasons).

1.1 (02-Oct-95)

Form interface introduced.
ASCII output now possible as well as HTML.
Output file can now be specified in the configuration file.
FROM and TO commands more powerful.
DEBUG and BACKGROUND introduced.
One bug fix: alphabetical sorting doesn't now swap some hostnames.
List of primes included in distribution.

1.0 (12-Sep-95)

Only minor changes since 0.94beta.

0.94beta (30-Aug-95)

New configuration variables SEPCHAR and REPORTORDER.
New configuration commands WITHARGS and WITHOUTARGS.
New commandline options +-A and +-x. (Config.: ALL and GENERAL).
Logfile entries with - as the return code are now regarded as successes, not corrupt entries.
Fixed bugs in host report when aliases or numerical hosts are present.
Documentation rewritten.

0.93beta (27-Jul-95)

Approximate hostname counting now possible in fixed memory.
New configuration commands ISPAGE and ISNOTPAGE.
New commandline option -v.
New configuration command WEEKBEGINSON.
Proper error message when memory exceeded.
Program split into several files.

0.92beta (11-Jul-95)

New reports introduced: hostname, full daily, and weekly.
FROM and TO commands introduced.
Header and footer files introduced.
More helpful warning messages.
Ability to read configuration instructions from stdin.
Subdomain commands moved from domains file to configuration file.
Makefile provided.

0.91beta (04-Jul-95)

Configuration file introduced, enabling many new options.
Some bug fixes and speed improvements.
Ability to print "top n" reports (rather than "everything higher than n").
Request report can print only pages.
Ability to try and resolve numerical addresses.
Now less fussy about the format of the domains file.
Logo added.
Readme converted to HTML.

0.9beta

More speed improvements, and some bug fixes.
Introduced -u option.
Introduced subdomain analysis.
Included "not modified" replies as successes, not redirects.
First public release at 0.9beta3. (29-Jun-95)

0.89beta (21-Jun-95)

Commandline arguments.
Efficiency improvements.
Host count and "last 7 day" statistics.

0.8beta (14-Jun-95)

Initial program, just default options.

This is the index for this Readme. Follow the numbers after each name to find references to that command or concept. Note that families of commands are indexed under the second part of the name: for example, HOSTEXCLUDE is under *EXCLUDE, not under HOST. This index includes all of analog's configuration commands: if a command you used in previous versions is not here, see the page on Upgrading from earlier versions.

Acknowledgements [1]
Addresses, numerical [1]
*ALIAS [1]
Aliases [1]
ALLBACK [1]
ALLGRAPH [1]
analhead.h [1]
analog.cfg [1][2][3][4]
Announcements [1]
APACHEDEFAULTLOGFORMAT [1]
APACHELOGFORMAT [1]
ARGSEXCLUDE [1]
*ARGSFLOOR [1]
ARGSINCLUDE [1]
*ARGSSORTBY [1]
*BACK [1]
Bar charts [1]
BASEURL [1]
Basic commands [1]
Broken pipe [1][2]
BROW* commands - see under second part of name
BROWSER [1]
Browser Report [1][2][3]
Browser Summary [1][2]
BROWSUM* commands - see under second part of name
Bugs, reporting [1]
Bytes, how displayed [1]
Cache files [1]
CACHEOUTFILE [1]
CACHEFILE [1]
CASE [1]
CGI program [1]
Colours [1]
*COLS [1][2]
Command line arguments [1][2][3][4]

• logfile name (LOGFILE) [1]
• - (LOGFILE stdin) [1]
• 4 (Quarter-Hour Report) [1]
• 5 (Five-Minute Report) [1]
• A (All reports) [1]
• a (HTML/ASCII output) [1]
• B (Browser Report) [1][2]
• b (Browser Summary) [1][2]
• c (Status Code Report) [1][2]
• C (Arbitrary configuration command) [1]
• D (Daily Report) [1]
• d (Daily Summary) [1]
• E (Redirection Report) [1][2]
• F (FROM date) [1]
• f (Referrer Report) [1][2]
• G (Default configuration file) [1]
• g (Other configuration files) [1]
• H (Hourly Report) [1]
• h (Hourly Summary) [1]
• I (Failure Report) [1][2]
• i (Directory Report) [1][2]
• J (Failed User Report) [1][2]
• K (Failed Referrer Report) [1][2]
• k (Redirected Referrer Report) [1][2]
• m (Monthly Report) [1]
• O (Output file) [1]
• o (Domain Report) [1][2]
• q (Warnings) [1]
• r (Request Report) [1][2]
• S (Host Report) [1][2]
• s (Referring Site Report) [1][2]
• settings (Settings of all variables) [1][2]
• T (TO date) [1]
• t (File Type Report) [1][2]
• U (Cache file) [1]
• u (User Report) [1][2]
• V (Debugging) [1]
• v (Virtual Host Report) [1][2]
• W (Weekly Report) [1]
• X (Goto's) [1]
• x (General Summary) [1]
• z (File Size Report) [1][2]

[ A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z ]