Browser, Spider and OS Definitions

Previous Next

Automated program that "crawls" the Web, generally for the purpose of indexing web pages for use by search engines. Because most web pages contain links to other pages, a spider can start almost anywhere. As soon as it sees a link to another page, it goes off and fetches it. Large search engines have many spiders working in parallel.

Deep Log Analyzer can recognize most of modern browsers, operating systems and search engine spiders.

But you can add custom browser, search engine spider and OS definitions in user_agents.ini file. Look at Unrecognized Browsers report. It shows all User Agent fields found in imported log files that are not described properly in user_agents.ini. If you get lots of hits from some kind of user agent (for example new download manager) that Deep Log Analyzer does not recognize, you can add custom definition for this user agent. It will be then recognized as a standard browser and will appear in all browser-related reports. Definitions should be entered in the following format in [Browsers] section:

BROWSERXX=[Name],[Log file stamp used to find it]

For example: BROWSER22=ReGet(downloader),ReGet

The same format is used for search engine robots (search spiders). Add records to [Spiders] section on user_agents.ini file in the following format:

SPIDERXX=[Name],[Log file stamp used to find it]

For example: SPIDER1=AltaVista,vscooter

You may also add new Operating System definitions to [Platforms] section in user_agents.ini in the following format:

PLATFORMXX = [Name],[Log file stamp used to find it]

PLATFORM1 = Windows XP,Windows NT 5.1

Notes

- Make sure that browsers, spiders and OSs numbering in their respective sections are continuous with no gaps

- Log file stamp is checked for match against user agent filed in the log file

- Do not change the order of the standard definitions provided with the program

- Back up edited ini-file as it can be overwritten when you install new version of Deep Log Analyzer

A website that enables users to locate information on the World Wide Web. Search engines use keywords entered by users to find Web sites which contain the information sought

Requests for files from visitors. Each HTML document and graphic file counts as a separate hit, so hits are not always an accurate representation of the number of different visitors to your site. Hits are also referred to as requests.

Fields in an extended web server log file identifying the browser and platform used by a visitor. This includes graphical browsers, text-only browsers and non-visual browsers such as audio or braille browsers, search engine robots.