Tom Stambaugh Debugging Wiki Clones

'TomStambaugh notes: I've created MuWebWeb, a place to discuss MuItself, MuWeb(s), ?WikiItself, WikiWeb(s), WikiClone(s), and so on.

15-Dec-97

AustinDavid asks:

Could you use a text-searching engine in the back, like glimpse? I've hacked wiki to use files (no DBM at all), and am using it at work to support high-level design discussions. It's been working well, and I haven't seen any performance hit, but it's still under 50 pages or so.

It shouldn't be hard at all to integrate a search engine in the back (like glimpse) and have it rebuild the database nightly, or even as often as pages are changed (can glimpse update the db incrementally?). Perl is good for this level of interoperability . . .

TomStambaugh answered:

As far as using files directly, I noticed that as well when I did mine. I chose to keep the DBM entries in, since I'm evolving a different naming approach.

Yes, you could certainly generate indexes each time the page changes using a search engine. Presumably even in Wiki read accesses are more frequent than writes, and each page is short enough that it shouldn't take long to generate indexes for that page.

Don't forget, though, that DBM probes probably involve some filesystem access too. I think I'll wait until I see a performance problem before I worry to much about how to solve it, because I don't trust my intuitions and guesses about where those searches might be spending their time. At the moment, I just do a brute-force traversal of each file on each search.

I'm pursuing other more fundamental changes to the way I represent and persist the information, and my intent is to come back to the search problem after I finish those changes.

December 11th, 1997.

Q: DaveHarris tried to add a question to an early version, but it appeared to hang - it didn't come back to him and as far as he could tell, didn't create the page.

A: The ISP who hosts my site has had a problem in the past with spoofers, who fake an ip address and then do bad things. They therefore do a reverse NS lookup on the inbound IP Address, and insist that it be consistent with the request they received. In some cases, other ISP's do not properly register (using the internic SWIP protocol) the reverse NS lookups for IP addresses that they allocate or assign. The result is (when contacting my ISP) you get a long delay during "waiting for reply", until the NS lookup times out. The "normal" time for accessing my site should be about the same as any other...at best less than a second, at worst, a few seconds. Never anything approaching a minute.

December 10th, 1997.

A dialogue with DaveHarris:

Q: What's the difference between Mu and WikiWiki? Is it just a port, or will you try to improve it as ?ThoughtWeaver (et al) has tried?

A: I intend to improve it (but I have limited time to do so). I view the current Mu (I think of it as ?MuZero) as equivalent to WikiWiki. I intend to add things like persistent pages, some form of non-intrusive user authentication and security, and probably revenue generation to it. -- TomStambaugh

Q: I'm more likely to see replies in WikiWiki than WikiBase. (I think it's pretty awkward to have so many different systems. One aim for a WikiWikiClone should be to find a better way of managing this.)

A: Yes, one extension I'm writing for Mu is a more deeply flexible naming schema. In the spirit of the pattern community, I'd rather make something work first and then describe it and invite your feedback for changes.

Everything sort-of works.

I need...

• A security and persistence mechanism

I had to work around the length restriction on DBM entries. The path I chose (admittedly not the best) was to put each entry in its own file (an ?EntryFile) in a known directory, then put the name of the ?EntryFile as the value in the DBM.

This breaks the wiki search code, which requires full-text access to each entry. The ?BruteForceSolution is to traverse each ?EntryFile, one at a time, while executing the search. This has obvious performance implications. Better solutions involve building indexes across the data as its being created, but that gets well beyond the limits of my tiny mind (or at least my knowledge of search engine technology).