A weekend of Quaddicted future?

Spirit · November 12, 2018, 9:58pm

Since the Quaddicted map system is old and broken, new releases never get added and I have too much control on the site:

How about we find a weekend some time this year to focus on the shortcomings of the site and grand visions plans for the future? I have lots of ideas how a vastly different system could ensure that new releases are available in an instant, automated tools can become way more powerful, etc.

But alone it is not much fun to work this step by step.

Who would be up for some focused collaboration?

shamblernaut · November 13, 2018, 8:00am

I’d be up for this.

MikeTaylor · November 13, 2018, 1:44pm

I don’t think such a grand plan is necessary. To me, Quaddicted is 97% of what I want from a Quake site, and the missing 3% is just timely addition of new maps. I think you could just appoint three or four trusted lieutenants and given them the necessary privileges to do this, and we’d be fine.

Spirit · November 13, 2018, 4:23pm

The issue with adding the new maps on time is that people want free filenames and the possibility to release multiple versions of something using the same filename(s). The current system does not support that.

shamblernaut: Yay!

dumptruck_ds · November 13, 2018, 5:59pm

Spirit I can not actually help with code or anything but I am good at outreach IMO. What kind of skills should interested parties have. Python development I assume. What other disciplines would be helpful for this brainstorming?

Spirit · November 14, 2018, 9:26am

Oh but you can!

It should be more about data structures and organisation, tagging and other things that require knowledge of how and what kinds of things are released in the Quake (or general modding) scene and imagination how to put those into “rules”/models.

Python and all that comes later, once we have the concept right, the rest should be fun.

topher · November 15, 2018, 3:22pm

I can help too. I am in a bit of a mess in my life now; but i have some time to spare.

ideas:

one thing that can be done is making a package manager, each map or mappack has multiple versions. The old versions remain hosted. And the versions are generated internally if necesary.

the problem is expecting or enforcing rules on the releases…

otherwise we will need a more complex script to handle different things, or someone manually checking the packages.
or make a testing thing, that checks if a package is malformed, handling automatically the other ones, and making scripts/rules for whatever comes up. or, maybe, giving the rules and tools for the mappers so they do it manually themselves.

adib · November 15, 2018, 5:24pm

Long long time corporate boring stuff developer here, Java developer for more than 15 years now. Would like to help with “data structures and organization” or anything else. It would be nice to have someone more experienced in portals and content management in this discussion. Do we have someone?

Guest1 · November 27, 2018, 9:33pm

I am willing to help with this. For my job I do a lot of work in Python, R and SQL (mostly analysis, but also data transformation etc.). I have no experience with web development, but am interested to learn. I have limited time available (job, 3 small children, etc) but can review and comment on proposals, do some coding where needed and can work on documentation. My main motivation is the importance of an archive like Quaddicted for future popular culture historians (I am one by training

Guest1 · November 27, 2018, 9:40pm

My current line of thought for the near future is to have a submission script running which can check the validity of incoming submissions, do some minor metadata transformation if needed / possible and submits the package to the archive maintainers. Submissions with too many packaging errors are rejected and a helpful rejection message is returned.

Such a script could tide the archive over until a more robust solution is available.

Spirit · December 10, 2018, 7:21pm

Would this sunday work for people? I’d set us up a zulip instance to chat and share the logs afterwards (if not keep it online).

dumptruck_ds · December 10, 2018, 11:16pm

That’s a busy day personally, so it’s a maybe.

topher · December 11, 2018, 1:44am

i can this sunday.

Spirit · December 14, 2018, 1:27pm

I’ll try to set it up early tomorrow and dump my thoughts and ideas out. Then we have ample time for asynchronous discussion

Maxi · December 16, 2018, 9:02am

Hi. I’m relatively new to Quake community (several months), learning mapping and going to learn QuakeC too.
I have experience in web development and some experience with C++ with Qt framework.
Also, I have little obsession with archiving/saving information and I’m very interested in making map archive for Quake maps better. Not sure if my programmer’s knowledge will be useful, but I’m willing to help with what I can, at least with ideas.
I had a thought to slowly build some enhanced version of Quake Injector myself but realized that my ideas quickly became too ambitious for my skill in software development. And I think people need updated Quake Injector (or at least similar tool), not some grandiose steam-like app. So, it will be much better if that will be “official” tool, made by people with greater programming experience, than me.
Maybe I’ll do my ambitious tool someday anyway. Having alternatives is good.
Anyway, I’d like to share some of my ideas, that may fit Quake Injector and to join this discussion, if you don’t mind.

Spirit · December 16, 2018, 9:09am

Oh poop, I forgot an important appointment today so I won’t be online much.

I did set up https://chat.quaddicted.com and here is what I posted so far:

qfuture > what's bad:

quaddicted was built on the assumption that filenames are unique, that there would only be one “filename.zip” and that this file was immutable (other versions of the same map/mod/whatever would use a different filename)

so I decided to use the zip’s basename as identifier both in the database and in URLs for user-friendliness

that was not the case…

only i can upload to quaddicted so new releases becoming available in the review system or injector always depends on my attention (and willingness…)

quaddicted was foremost made for singleplayer maps but then many turned out to be mods and there is no clear line

the system does not support sidecar files like optional music and stuff like that. every thing in the database gets a webpage

there is no concept of a main name for things like quoth, dependencies are pointing to specific filename’s (the identifiers)

screenshots also follow the identifier rule, there can only be one screenshot per release

File types are restricted to zip files, but in reality everything from bsp to exe gets released

qfuture > metadata:

Quaddicted is basically metadata plus community stuff around files

metadata: what is this file, who made this and when, what is included, how big is it, etc

Currently this is stored in a SQL database with a column (or table for 1:n relationships like the ‘included files’) per metadata thing

That is not very flexible

And it requires dedicated editors. we have a website editor thanks to Sajt but it is rudimentary and not fit to be opened up for everyone to use

The current system is also dependant on me uploading files and the filenames abiding to strict rules, as it will automatically determine some metadata and inspect the zip file contents (failing on some zip dialects/versions)

My idea here is to define a json format for describing the file itself (hash, size, filename(s))

Everything that describes the kind or contents of the file would be a list of strings of key=value pairs, with duplicates allowed. For example gamemode=coop, gamemode=singleplayer, author=czg, author=BigFluffy, releasedate=2018-12-15, style=outrun, dependency=quoth>=2, title=The Title, etc

That would also potentially include machine readable installation instructions

Such json file would exist for each hash, maybe in files/by-sha512/$firstCharOfHash/$hash.json

The json way could also be a way of providing multiple URLs from where the file itself could be downloaded, we could even include magnet or ipfs urls

Those json files would be handled and tracked in a public git repository, with lots of trusted maintainers and the possibility for everyone to propose changes

Quaddicted would then use this main repository of file data and metadata, sync and parse it regularly and build its portals from it (like the badly URLed /reviews/ for sp maps) based on the tags it finds

So this would easily enable specific ‘views’ on the vast landscape of releases

And even enable tracking of arbitrary files currently not covered by any surviving website, e.g. movies, skins etc

The community aspects, comments, ratings, screenshots () would remain in a db at Quaddicted

since that require more interactivity and immediate action

Not sure about screenshots…

topher · December 16, 2018, 2:42pm

ok i don’t know how to login in that zulip webapp

seems solid. i see that currently you have 128bit hashes in the download metadata, between parens.
i would go directly with two chars, that will give 255 folders, that’s what git repositories use.

about the json file, most implementations take only the last entry of duplicated keys. Luckily the json format is not complex.
also we will need validators and other things

the good thing about the json is that anyone that can map can write a json and understand the rules pretty quickly.

about the metadata and filecontents i’m thinking two things:
a) guidelines
b) code to handle the most commons cases
i’m guessing that everything that is already in the archives are well formated and with valid metadata, so the guidelines/code are for new releases

for example, some people put a map file and gfx in the root of the zip file, and in the instructions explain where they go.

They can be community managed, if they’re in a github repo. for all purposes, well formed zip files are preferable to instructions to rename and move files.

Spirit · December 16, 2018, 8:13pm

Sorry, I accidentally left it invite-only (the default). Total fail on my end… It’s open now. There is also a desktop client: https://github.com/zulip/zulip-electron

Spirit · December 16, 2018, 9:40pm

Maxi, welcome! I am sure you can contribute in many ways, even if they are just comments here and there. Hell, I learned PHP and SQL when building this site originally and it still works (debatebly well ).

Great idea to follow git there, thanks! The 128bit hashes are md5, those are only good for validation of downloads, not as identifiers and someone could just hack a matching one to be funny.

I use Python and thus would treat these as dictionaries, duplicate keys definitely should not be there. Instead we will need to use lists for those things that might be multiplicities(?).

yeeee, human-editable is a major benefit. Validator (and some tool to help authors to create these files themselves) would come once we fixed a first version of the format.

[quote=topher]
i’m guessing that everything that is already in the archives are well formated and with valid metadata, so the guidelines/code are for new releases[/quote]
It depends :} But I would use the existing db to generate such json files as a start. Generally I would want them to exist for any Quake file imaginable, retroactively created by the community. All the mods, engines, whatnot. The tagging system should be able to handle this (I like https://wiki.openstreetmap.org/wiki/Map_Features a lot).

[quote=topher]
for example, some people put a map file and gfx in the root of the zip file, and in the instructions explain where they go.

They can be community managed, if they’re in a github repo. for all purposes, well formed zip files are preferable to instructions to rename and move files.[/quote]
People will release files in any way they like and there is no central instance (like idgames) to force anything. People release badly packaged maps to this day. Thus this idea of being able to supplement their files with metadata. One important key would be unpacking/installation instructions.

Thanks for your awesome input!

ericw · December 17, 2018, 6:37am

Hi Spirit, I wasn’t able to sign up at https://chat.quaddicted.com/
I clicked sign up, entered my email address and got: