Are you looking for a rare or hard to find quake file?
I’ve created an index of about 33+ million URLs across about 2,000+ quake websites on archive.org, and the index is growing by the moment.
Using a simple web interface, we can search across the index to:
discover what versions of a file were released, by filename.
discover where they were hosted, historically.
attempt to download archived versions of the file from archive.org.
It’s like discmaster, but for archived quake websites. Basically, I create long lists of old quake websites (and sites that hosted quake files), download all pages from the archive and crawl the downloaded pages for URLs to add to the archive.
I’m keen to try it out for you, to gauge general interest in a site like this. Post filenames you’re looking for below (or dm me or shoot an email to jason.brownlee05@gmail.com ) and I’ll try to locate files for you.
If there’s strong interest, I’ll look into hosting the index and webapp somewhere public.
I’m also open to ideas on how it could be made more useful, and if you’re aware of other similar indexes/websites like this.
This sounds pretty neat. Though, my problem would be, I wouldn’t actually remember any names for maps that I would want to search. Only through playing would I go “Oh yeah, I remember this from when I was a kid!”.
But I’m sure a bunch of people will get use out of it.
The databases behind that are in the same directory: fileplanet_by_shaqfu.db and fileplanet.sqlite. The schema should be straightforward but just shout if they are interesting and you need help in using them. It sounds like you might have that same data in your database already.
Have you tried to download those files (at least unique by filename, so probably much less than 30 million )?
For the new Quaddicted database my vision is to really add any and all Quake-related files to it. It is indexed by their sha256 checksum and key=value tagging is used for organisation (e.g. “game_mode=singleplayer”, “type=map”, “title=foo” etc), so matching files to it is easy, as is filtering for specific things (if we foster the metadata well).
I was aware of the fileplanet search (you had shared it with me previously), but not the db/sqlite db files, thanks.
I have considered downloading all quake zip/txt files I come across and hosting them somewhere. I’m not convinced I should, yet. Instead, I am leaning toward providing the index and allowing users to download files themselves. On the flip side, archive.org (or portions thereof) may go away some day…
Your proposed system sounds great, my thoughts were similar. A file hash and name is required. Other meta data could be inferred at query time or pre-processed after being added to the index (e.g. .bsp and .map are maps), text can be tokenized and indexed, etc. Unpacking zips into the archive may also be required in some way to expose hidden gems (e.g. some mods that combine mods include original data/maps/txt files from both).
I ran the wanted list through my index and got hits, but I can clearly see many false positives (duke3d files, doom files, etc.) and dead links (not archived). I expanded the search to include .txt and .bsp files, just in case. It helped.
Some links may be interesting to the original requestees e.g. exclude from future searches and/or provide hints for future research and digging.
================================================================
Title : Pathway to hell
Filename : p2hell.bsp
Author : Sam Allen
Email Address : Tonylallen@msn.com
Description : A small 1 player level in a sorta
hell kind of setting.
Additional Credits to : Ben Morris for Worldcraft
ID software for THE BEST EVER GAME
created!
================================================================
...
)?