freebsdzine.org
Graduate life: It's not just a job. It's an indenture.

[ Home  · Contribute  · Mailing Lists  · Contact Us  · Site Statistics  · Latest BSD News  · Submit an Article  ]

Search freebsdzine.org

FreeBSD 'zine Polls

Where should #freebsdzine live?

Undernet (irc.undernet.org)
Slashnet (irc.slashnet.org)
Openprojects (irc.openprojects.net)
DALnet (irc.dal.net)
IRCnet (us.ircnet.org)
at moses` house

Results  · More polls


Sections
· Wanted articles
· About the site
· The staff
· Copyright info
· Privacy policy
· Change log

Resources
· The FreeBSD Project
· The FreeBSD Diary
· BSD Today
· Daemon News
· Daily Daemon News
· Slashdot BSD
· FreshPorts
· The FreeBSD Mall
· BSDVault
· The FreeBSD Browser

FreeBSD Books
· Complete FreeBSD
· FreeBSD Handbook
· FreeBSD Corporate
Networker's Guide


Runs on FreeBSD
The FreeBSD Browser
Rich Morin <[email protected]>

FreeBSD is a big system; a complete distribution contains roughly 100,000 files. Consequently, it's essentially impossible for anyone to remember which files are related to each other, and why. So, we all find ourselves trying to remember (or find out) which files we should be examining, in order to solve a given problem. The FreeBSD Browser isn't a panacea, but I think it is a worthwhile addition to the existing set of system investigation tools.

Specifically, the browser is a specialized "search engine" for information on FreeBSD. It can help you find directories, files, manual pages, papers, and other useful resources. Let's try solving a typical problem with and without the browser, to see how it helps.

Using Existing Tools

FreeBSD provides a number of tools for system investigation. Some of my favorites for interactive use are apropos, grep, info, locate, ls, man, more, whatis, whereis, and which. If I need to start writing specialized tools, of course, I start using things like awk, find, sed, sh, and (for serious work :-) perl.

Let's think about how these might be used to find out which parts of the file system are relevant to a command, assuming that we already know its name (e.g., vi). man vi is a good place to start; in particular, the last part of a man page tends to contain some useful pointers:

FILES
  /bin/sh
    The default user shell.

  /etc/vi.exrc
    System-wide vi startup file.
  ...

SEE ALSO
  ctags(1), more(3), curses(3), dbopen(3)

  The ``Vi Quick Reference'' card.
  ...

Unfortunately, the list of pointers is seldom as complete as we might wish. For instance, although the man page for vi is really very well done, it says nothing about the /usr/share/vi or /var/preserve directories. It mentions some papers, but doesn't indicate where they might be found on the system.

In short, we have more commands to run. One possibility, at this point, would be to use the locate command. Typing locate vi | more will immediately show us the names of the desired directories. Unfortunately, it will also list about 4400 other items, most of which are totally irrelevant:

/etc/mail/RCS/virtusertable,v
/etc/mail/virtusertable
/etc/mail/virtusertable.db
/etc/mail/virtusertable.sample
/etc/services
/modules/vinum.ko
/sbin/vinum
/stand/help/network_device.hlp.gz
/usr/X11R6/bin/xvidtune
...

In short, locate isn't exactly a finely-tuned solution. whereis, on the other hand, is:

% whereis vi
vi: /usr/bin/vi /usr/share/man/man1/vi.1.gz /usr/src/usr.bin/vi

We now know the locations of the executable binary, man page (troff source version), and source directory. That sounds interesting; let's go look at the source code:

% cd /usr/src/usr.bin/vi
% ls -F
Makefile        config.h        pathnames.h     port.h

Uh-oh; there aren't any *.c files. Where's all the source code? Hmmmm; let's take a look at the Makefile. As expected, it contains a vital clue:

SRCDIR= ${.CURDIR}/../../contrib/nvi

Now we can cruise over to the real source directory:

% cd /usr/src/contrib/nvi
% ls -F
FAQ build/ common/ ip/ tcl_api
LAYOUT catalog/ docs/ ip_cl/ tcl_scripts/
LICENSE cl/ ex/ perl_api/ tk/
README clib/ include/ perl_scripts/ vi/

Because we had the tools we needed and knew how to use them, this exercise only took us a few minutes. Still, we had to do a lot of typing and we may have missed a few things along the way. Not bad, presuming we didn't miss too much, but we might be able to do better.

Using the Browser

Let's try using the FreeBSD Browser on the problem. I'll include some output snippets for hard-copy readers and the terminally lazy, but the best thing for you to do is to bring up this link in a separate window:

http://www.cfcl.com/md/fb/42/usr/bin/vi

If the machinery of the Internet (including my poor, beleaguered server :-) is functioning properly, you should have the relevant page on your screen in a matter of seconds. The "Description" section may or may not contain much of interest, depending on the item being examined. In some cases, however, it can be extremely informative.

/usr/bin/vi is an item of type "plain file". 

file(1) calls it: "ELF 32-bit LSB executable, Intel 80386, version 1
(FreeBSD), dynamically linked (uses shared libs), stripped".

Looking at the rest of the page, we see all of the information that our previous exercise gained us, and more. The following lines tell us about all of the files and directories we found in our exercise, adding a bit of useful contextual annotation:

File Tree Extract

?  / the system's root directory
?  :  home@ user home directories
?  :  :  username sample user home directory
?  :  :  :  .exrc rc file for ex(1) and vi(1)
?  :  :  :  .nexrc rc file for nex(1) and nvi(1)
?  :  tmp/ temporary files
?  :  usr/ non-critical system files
?  :  :  bin/ "add-on set" of user commands
?  :  :  :  edit* executable command (link)
?  :  :  :  ex* executable command (link)
?  :  :  :  nex* executable command (link)
?  :  :  :  nvi* executable command (link)
?  :  :  :  nview* executable command (link)
?  :  :  :  vi* executable command (link)
?  :  :  :  view* executable command (link)
?  :  :  obj target tree for system builds
?  :  :  :  usr/ system build target for /usr
?  :  :  :  :  src/ system build target for /usr/src
?  :  :  :  :  :  usr.bin/ build target for for /usr/bin
?  :  :  :  :  :  :  vi/ build target for for /usr/src/usr.bin/vi
?  :  :  share/ architecture-independent files
?  :  :  :  doc/ documentation files
?  :  :  :  :  usd/ User's Supplementary Documents
?  :  :  :  :  :  10.exref/ "Ex Reference Manual", etc.
?  :  :  :  :  :  :  paper.ascii.gz ASCII "Ex Reference Manual"
?  :  :  :  :  :  :  summary.ascii.gz ASCII "Ex/Edit Command Summary"
?  :  :  :  :  :  11.edit/ "Edit: A Tutorial"
?  :  :  :  :  :  :  paper.ascii.gz ASCII "Edit: A Tutorial"
?  :  :  :  :  :  12.vi/ vi(1) papers
?  :  :  :  :  :  :  paper.ascii.gz ASCII "An Intro. to Display Editing"
?  :  :  :  :  :  :  summary.ascii.gz ASCII "Ex Quick Reference"
?  :  :  :  :  :  :  vipwh.ascii.gz ASCII "Vi Command & Function Ref."
?  :  :  :  vi/ source code for /usr/bin/vi
?  :  :  :  :  catalog/ vi(1) message catalogs
?  :  :  :  :  perl/ vi(1) perl scripts
?  :  :  :  :  tcl vi(1) tcl scripts
?  :  : src/ source code files
?  :  :  :  contrib/ contributed source code
?  :  :  :  :  nvi/ used by /usr/src/usr.bin/vi
?  :  :  :  usr.bin/ source code for /usr/bin
?  :  :  :  :  vi/ source code for /usr/bin/vi
?  :  var/ transient system data
?  :  :  preserve/ preserved editor buffers
?  :  :  tmp@ temporary files
?  :  :  :  vi.recover vi(1) session recovery files

Documentation

?  csh(1) tcsh - C shell with file name completion and command line editing
?  ctags(1) ctags - create a tags file
?  ed(1) ed, red - text editor
?  edit(1) ee - easy editor
?  ee(1) ee - easy editor
?  ex(1) ex, vi, view - text editors
?  lp(1) lp - front-end to the print spooler
?  nex(1) ex, vi, view - text editors
?  nvi(1) ex, vi, view - text editors
?  nview(1) ex, vi, view - text editors
?  red(1) ed, red - text editor
?  ree(1) ee - easy editor
?  rs(1) rs - reshape a data array
?  tcsh(1) tcsh - C shell with file name completion and command line editing
?  vi(1) ex, vi, view - text editors
?  view(1) ex, vi, view - text editors
?  re_comp(1) re_comp, re_exec - regular expression handler
?  re_comp(1) re_comp, re_exec - regular expression handler
?  re_exec(1) re_comp, re_exec - regular expression handler

 

?  regexp(3) regcomp, regexec, regsub, regerror - regular expression handlers
?  regsub(3) regcomp, regexec, regsub, regerror - regular expression handlers

 

?  termcap(5) termcap - terminal capability database
?  terminfo(5) terminfo - terminal capability database

 

?  environ(7) environ - user environment

The browser also tells us about some items that our manual process missed, including some hard-linked commands (e.g., /usr/bin/nvi), the target directory for the build (/usr/obj/usr/src/usr.bin/vi), some other intriguing directories (/usr/share/vi and /var/preserve), and a slew of man pages. Finally, it gives us the exact location of the papers that the man page mentioned, in case we want to take a look at them.

The hard link information tells us that several commands share the same executable image. This is useful information, to be sure, but it also serves another purpose. If we had started our "traditional" inquiry with the "ex" command, whereis wouldn't have been nearly as helpful:

% whereis ex
ex: /usr/bin/ex /usr/share/man/man1/ex.1.gz /usr/src/contrib/nvi/ex

Although whereis found .../contrib/nvi/ex, it failed to find .../usr.bin/vi. Consequently, we would have had to do a bit more looking to find the "official" source directory. Because the browser is able to integrate information from a variety of sources, it tends to sail past gaps in the linkage information. A reference to a man page or a file might never be seen in the traditional method, but the browser uses it automatically.

The browser also eliminates a great deal of typing. Although it isn't obvious from the samples above, each listed item (whether a file, directory, or man page) is a link to another browser page. So, if your curiosity is aroused by an entry, a simple mouse click will bring instant gratification.

In addition, each item's line starts with a ? link, leading to a "linkage trail" page. This page details the links that were followed to get from the original item to the item in question. Aside from satisfying the user's curiosity, this page can allow the user to analyze and report linkage errors.

Fancier Tricks

Going back up to the top of the page, we see a few other items of interest. The "Target" form already contains the current path (or manual page) name, as a useful starting point. Edit this as desired and hit "return"; you'll be taken to the appropriate page.

Above the "Target" form, you should see a few more links. There may well be more over time, but for now, there are "search", help", and "comment" links.

The "search" link brings up a separate window for keyword-based searching. The searching algorithm is roughly the same as that used by whatis(1), but multiple keywords may be required. And, of course, the returned man page names are clickable links.

The "help" link brings up a separate window, with some fairly minimal online help information and a link to some background information on the browser and its motivations. You may find this interesting to review, but this article covers many of the same points in rather more detail.

The "comment" link is just a mailto, with the subject line already filled in. Use this if a page contains an error, is missing some information that it should contain, or whatever. The browser's current "knowledge base" is my own work, based on system documentation, some investigation, and my own (all too fallible) memory. As users tell me about problems they notice, I'll be able to improve the browser's annotations, link information, and built-in rules.

While we're on the subject of "built-in rules", let's consider the question of /dev. Everyone knows what a mess it is (and devfs actually promises to clean it up quite a bit), but finding the relevant information for a given device node can still be an ordeal. For starters, there's the problem of finding the device's "base name", so we can bring up the relevant man page.

With hundreds of device families and dozens of name-formation conventions, even looking up the man page for a given device (e.g., /dev/esa0) can be a hassle. Figuring out the implications of a given device name and finding the driver sources, of course, is even harder. So, what can the FreeBSD Browser tell us about it?

Description

/dev/esa0 is a member of the sa ((SCSI) Sequential Access devices)
family.  This device node has the attributes "eject on close, unit 0".
Related device nodes have names that match the Perl regular expressions
/^[en]?r?sa[0-9]+$/ and /^[en]?r?sa[0-9]+\.[0-9]+$/.

File Tree Extract

?  / the system's root directory
?  :  dev/ device nodes
?  :  :  esa0 device node
?  :  :  sa0 sample device node
?  :  :  src/ source code files
?  :  :  :  sys/ source code files for the kernel
?  :  :  :  :  cam/ CAM (Common Access Method) sources
?  :  :  :  :  :  scsi/ SCSI CAM (Common Access Method) sources
?  :  :  :  :  :  :  scsi_sa.c source code for /dev/esa0
?  :  :  :  :  :  :  scsi_sa.h source code for /dev/esa0

Documentation

?  mt(1) mt - magnetic tape manipulating program

 

?  CAM(4) SCSI, CAM - CAM SCSI subsystem
?  SCSI(4) SCSI, CAM - CAM SCSI subsystem
?  adv(4) adv - Advansys ISA/VL/EISA/PCI 8bit SCSI host adapter driver
?  adw(4) adw - Advansys PCI 16bit SCSI host adapter driver
?  aha(4) aha - Adaptec SCSI host adapter driver
?  ahb(4) ahb - Adaptec EISA SCSI host adapter driver
?  ahc(4) ahc - Adaptec VL/EISA/PCI SCSI host adapter driver
?  aic(4) aic - Adaptec AIC-6260 and AIC-6360 SCSI driver
?  amd(4) amd - amd53c974 PCI SCSI driver
?  bt(4) bt - Buslogic/Mylex MultiMaster SCSI host adapter driver
?  cam(4) SCSI, CAM - CAM SCSI subsystem
?  ch(4) ch - SCSI media-changer (jukebox) driver
?  dpt(4) dpt - DPT RAID controller SCSI driver
?  isp(4) isp - Qlogic based SCSI and FibreChannel SCSI host adapters
?  mtio(4) mtio - FreeBSD magtape interface
?  ncr(4) ncr - NCR 53C8xx SCSI driver
?  sa(4) sa - SCSI Sequential Access device driver
?  scsi(4) SCSI, CAM - CAM SCSI subsystem
?  st(4) st - SCSI tape driver
?  sym(4) sym - NCR/Symbios/LSI Logic 53C8xx PCI SCSI host adapter driver

This time, the "Description" is a bit more useful. It tells us the nature of the device, which modes are being specified, and which other device nodes are related to it. More precisely, it lists some Perl regular expressions that will match the names of related devices. Not everyone reads these things easily, however, so I have added a "regular expressions" link. This link brings up a page that explains each of the regular expressions, in excruciating detail:

The Perl regular expression /^[en]?r?sa[0-9]+$/ can be expanded as:

/^ # start of text string
[en]? # possibly the letter "e" or "n"
r? # possibly the letter "r"
sa # the device's base name
[0-9]+ # one or more decimal digits
$/x # end of text string

The Perl regular expression /^[en]?r?sa[0-9]+\.[0-9]+$/ can be expanded as:

/^ # start of text string
[en]? # possibly the letter "e" or "n"
r? # possibly the letter "r"
sa # the device's base name
[0-9]+ # one or more decimal digits (unit number)
\. # the special character "." (period)
[0-9]+ # one or more decimal digits (mode bits)
$/x # end of text string

The prefix letters "e" and "n" stand, respectively, for "eject on close" and "no rewind on close". The prefix letter "r" stands for "raw mode", but it is ignored; this device is always accessed in raw mode. The first set of digits is used to specify device units; The second set (mode bits) provides an alternative way to encode the "e" and "r" information.

Limitations

The browser occasionally brings up irrelevant material. This may be the result of bad input data (e.g., a misleading "SEE ALSO" reference), a collision in the man page "name space", or a poorly-implemented linkage rule.

A recent version of the browser, for instance, picked up a false trail from sa(4) to /sbin/sa (a system accounting program). Following this lead, it brought in a number of other irrelevant items (e.g., sa(8), acct(2), and the source and build directories for /sbin/sa).

Once a false trail has been discovered (and reported!), it can usually be resolved with little effort. Finding them all, however, will be quite a challenge. The browser's database contains about 100,000 nodes and a million or so links, with more on the way. So, there is a lot of room for things to go awry. A few "false positives" can be tolerated, however, if the alternative is to lose out on real information.

The browser really should be able to display the contents of items (e.g., directories, files, and man pages). The code for this isn't particularly hard (in fact, I already have most of it on hand), but the resource drain might well swamp my poor server, to say nothing of my Internet connection. I hope, however, to resolve this limitation fairly soon.

Nor does the browser cover the Ports Collection, systems other than FreeBSD, or (generally) information about the contents of files. These are, however, goals of the Meta Project. In fact, the FreeBSD Browser is only the current version of a much larger "Meta Demo". That, in turn, is only a minor subset of the full Meta proposal. Still, the FreeBSD Browser is a reasonable starting point; I hope you find it useful...

- Rich

Return to the March 2001 Issue



Issues
2001
· March
· February
· January

2000
· December
· August
· July
· June
· May
· April
· March
· February

1999
· January

Other issues from 1999 are available in the attic for now.

Other News
· Slashdot
· FreeBSD Diary
· BSD Today
· FreshPorts
· Daemon News
· OS Online
· Rootprompt
· Maximum BSD

Miscellaneous
· Jim's site

IRC
#freebsdzine
If you'd like to hang out with us and talk about the site, join us in #freebsdzine on Undernet.

Backend
You can add a list of our latest issue's articles to your site by using our RDF/RSS file. You can also add it to your My Netscape page, or add our slashbox once you log in over at Slashdot.

[ Home  · Contribute  · Mailing Lists  · Contact Us  · Site Statistics  · Latest BSD News  · Submit an Article  ]

Copyright © 1998-2001 · The FreeBSD 'zine · All rights reserved.