New Map Design Thread

[AHWULF]

So you can see what's happening when you are not playing. For recruitment purposes. Cause it looks cool.

[AHWULF]

I have the data, not live yet, just to work with.

Looks like I can see town locations, connections, ownership, contention and a0 status, all within easy reach, to start.

Brigade moves are there as well as well as current locations. Fb locations. What buildings are damaged. In fact there are 78,000 "buildings" in the game. Every facility as well. I could almost make maps of each town but that might be a stretch.

There have been 717,000 brigade moves since 2009.

Lots of work to do...

[jwilly]

I don't have a suggestion as to what would be better, but when I hear "map" I think a static world map that shows me the tile array with the towns/cities, maps, forests, airfields, coastlines and so forth.

The kind of thing that Five has made, and other folks along the way as well.

Speaking of which: CRS really needs this whatever-you're-making, but it also really needs a good world editor.

Edited May 17, 2014 by jwilly

[AHWULF]

It will be "maps". A world editor is larger orders of magnitude more difficult. The current game world is crazy hard to modify, and a testament to how crazy doc is.

The biggest problem currently is having the right dynamic data. Wiretap was never a good solution. The future is making all sorts of data available including map related static data and current state and also dynamic stats data plus code you can use to make something different. The more stuff people have to build on the more cool stuff will get built, which only helps the game.

[dog3]

The future is making all sorts of data available including map related static data and current state and also dynamic stats data plus code you can use to make something different. The more stuff people have to build on the more cool stuff will get built' date=' which only helps the game.[/quote']

Ahmen! Please, please, try to make the data format and access api as regular as you can. It will become the foundation for everything built on it.

s!

[lungar]

Please, please, do what you can to regularize the data formats and api so that I don't have so many "special" cases to code and test. Wiretap had some good stuff but it suffered from it's haphazard design. I don't think that it matters much whether or not the data is provided in xml , json, or whatever is most convenient for you, as long as its consistent and orthogonal.

Once again, good luck and thanks for taking this task on.

s!

It was less haphazard than you might think, and I got relatively little feedback from consumers. But you can ask Xiper, etc; the folks who came back with feedback or observations about extra complexity generally got traction and changes.

This would be a good time to step up and be specific about what could be improved, but bear in mind that there are deliberate limitations on what the rats wanted to expose and there are also serious bandwidth considerations - one of the few common pieces of feedback I got was "it would be nice if I could get X and Y in the same request" where 'X' would be a static piece of information and 'Y' would be dynamic. That's a bandwidth hit.

[jwilly]

The future is making all sorts of data available including map related static data and current state and also dynamic stats data plus code you can use to make something different.

Will your "maps" solution be broken if eventually world objects (and ideally tiles, too) become four-state instead of one-state or two-state?

Edited May 18, 2014 by jwilly

[AHWULF]

Will your "maps" solution be broken if eventually world objects (and ideally tiles' date=' too) become four-state instead of one-state or two-state?[/quote']

Not sure what that means. I assume I would know about changes long before it mattered.

[dog3]

This would be a good time to step up and be specific about what could be improved, but bear in mind that there are deliberate limitations on what the rats wanted to expose and there are also serious bandwidth considerations - one of the few common pieces of feedback I got was "it would be nice if I could get X and Y in the same request" where 'X' would be a static piece of information and 'Y' would be dynamic. That's a bandwidth hit.

I think that kfs1 did a great job with wiretap but if changes are going to me made these comments are based on my recollections of writing a package providing an api (Objective-C) to the wiretap stream.

In an earlier comment I used the word "haphazard", which I now regret, to describe the api. I think that it was more like the natural accretion of features that just "happens" as result of the interaction between producers and consumers when a new service is provided. I think that kfs1 was on the right track using xml and made a great start but, with hindsight, I would add:

1) make the DTD for the xml available.

2) make element tags map 1-to-1 with primitive data objects. What are the primitives in this domain? For instance, a "" element in a 'cpdetail' report has attributes id,afs,depots,nbs, fbs,... but in a 'cplist' report a element has attributes name, type, orig-country,x,absx, and so on and different yet again in for a 'cp status' report. Yes, you can parse them but it is not as "orthogonal" as it could be. Formalizing the primitive data objects will help with this.

3) I think the issue that you were getting at above with regard to inquiry composition is related to the desire to only receive the data that you actually need. For instance the 'linkage' report or the cp list 'static' report, respectively, send data on all the links and all the cps whereas you may only be interested in links and cps on the frontline or near ao's. This impacts the storage requirements and transmitter/receiver power usage on resource-limited mobile devices.

Anyway, this is just my $0.02 based on recollections of my experience writing some wiretap code.

s!

[lungar]

I think that kfs1 did a great job with wiretap but if changes are going to me made these comments are based on my recollections of writing a package providing an api (Objective-C) to the wiretap stream.

In an earlier comment I used the word "haphazard", which I now regret, to describe the api. I think that it was more like the natural accretion of features that just "happens" as result of the interaction between producers and consumers when a new service is provided. I think that kfs1 was on the right track using xml and made a great start but, with hindsight, I would add:

1) make the DTD for the xml available.

2) make element tags map 1-to-1 with primitive data objects. What are the primitives in this domain? For instance, a "" element in a 'cpdetail' report has attributes id,afs,depots,nbs, fbs,... but in a 'cplist' report a element has attributes name, type, orig-country,x,absx, and so on and different yet again in for a 'cp status' report. Yes, you can parse them but it is not as "orthogonal" as it could be. Formalizing the primitive data objects will help with this.

3) I think the issue that you were getting at above with regard to inquiry composition is related to the desire to only receive the data that you actually need. For instance the 'linkage' report or the cp list 'static' report, respectively, send data on all the links and all the cps whereas you may only be interested in links and cps on the frontline or near ao's. This impacts the storage requirements and transmitter/receiver power usage on resource-limited mobile devices.

Anyway, this is just my $0.02 based on recollections of my experience writing some wiretap code.

s!

#1 DTDs were a little outside my comfort zone, and I'd contemplated autogenerating them in the dbframe script, but your point #2 would possibly have addressed that.

#2 I've had a lot more experience being on the other end of an xml api now, and I can see your point, but again, bandwidth often won out as the bigger concern. That's why some of them have crappy tags like "r". I don't know which side of that fence I'd come down on now.

#3 When I was working with xiper he was always very concerned about download times. Meanwhile the in-house concern was that my little toddy apache/tomcat service might get popular when something like bge got released and eat all our bandwidth. I really wasn't that concerned about mobile devices at that point (course, now that I've worked on the WoWs armory...)

-Oliver

[lungar]

Apologies for any weird spelling. Doing this on the kindle.

[dog3]

S! Oliver,

I didn't recognize you.

I always find it interesting to hear the history of design choices.

As you suggest the design parameters may have shifted since the original wiretap.

Do you have any comments on the particular factors affecting server loading, as opposed to client loading, based on your experience? I am just thinking that in this case reducing server loading may be more critical than the resource limitations of the clients.

For instance, would/could "pushing" data be more efficient than responding to client polling?

It's good to see you hanging around.

s!

[lungar]

S! Oliver,

I didn't recognize you.

I always find it interesting to hear the history of design choices.

As you suggest the design parameters may have shifted since the original wiretap.

Do you have any comments on the particular factors affecting server loading, as opposed to client loading, based on your experience? I am just thinking that in this case reducing server loading may be more critical than the resource limitations of the clients.

For instance, would/could "pushing" data be more efficient than responding to client polling?

It's good to see you hanging around.

s!

Server load wasn't really a concern, bandwidth and db load were. So a lot of the queries were actually static pages, the ones that didn't take arguments. I had a Perl script called dbframe which processed .dbf files which were basically little Perl hashes that provided a query and info on how to process the result set.

These were split into directories which identified the run frequency. Some ran once a minute, others ran once an hour. If the result didn't change, I left the file alone, allowing it to take advantage of http HEAD functionality.

The other concern was just flat byte by byte bandwidth usage. One of the problems with text xml is that it's very, very inefficient. 159 uses 9 bytes of markup vs 3 bytes of data, compared to csv or json where you pay a little header cost but the array versions just have 3 or 4 bytes for this field.

One tool author actually complained that the lack of whitespace made the xml files hard to read, so I toyed with human readable versions on a switch, but I saw those getting hammered and in some cases the data was 2/3rds syntactic sugar.

Today... I'd probably look at protobufs, amq or something first.

Oliver

[AHWULF]

Our plan is to dump the data periodically and my server will pick it up and it will serve up whatever itself, so load on the prod db won't an issue. Not as real time but close enough.

I am not a fan of xml. Since everything I will do is web json is my choice.

The biggest issue atm is that the xy of each town in the db doesn't exactly match the BGE map locations. Close, but randomly outside the boxes.

[jwilly]

A world editor is larger orders of magnitude more difficult. The current game world is crazy hard to modify

In your estimation, if someday down the road the resources became available, would a world editor be within the capability of a single architect/coder, or would it have to be a managed team project?

[dog3]

The biggest issue atm is that the xy of each town in the db doesn't exactly match the BGE map locations. Close' date=' but randomly outside the boxes.[/quote']

All the facilities locations should probably be checked as well. As I recall there are discrepancies between Five's maps and a few FBs. I don't know whether or not that was a problem with Five's map renderings or the source data.

I am not a fan of xml. Since everything I will do is web json is my choice.

My issue is not with the specific language. Whatever meets your requirements and "convenience" is an important parameter in this case. I'm just asking that as you tackle the server-side issues please keep the poor clients in mind as well.

The biggest issue atm is that the xy of each town in the db doesn't exactly match the BGE map locations. Close' date=' but randomly outside the boxes.[/quote']

As I recall there were only 5 significant digits (possibly fewer) given for each dimension in the geodetic transformation. That might be part of the problem.

s!

[AHWULF]

I may have to manually adjust the "location" of each item to match the dot on the map if there is one. It's just tedious. It might be because the address of the items is an octet location which is 600 meters across.

I hope to have something to show by rat chat time.

@jwilly - No, that's incredibly difficult to do. The data is, to be honest, about as simple as the US tax code.

[lungar]

Our plan is to dump the data periodically and my server will pick it up and it will serve up whatever itself, so load on the prod db won't an issue. Not as real time but close enough.

I am not a fan of xml. Since everything I will do is web json is my choice.

The biggest issue atm is that the xy of each town in the db doesn't exactly match the BGE map locations. Close, but randomly outside the boxes.

I think you're making a great call there: There are plenty of good json engines for most languages, and there's always bson.

You might want to look at the option of low-tier AWS virtual machines as a way to reduce the round trip times at a *very* low cost (I'm running kfs.org on a tiny Windows Server 2003 instance and so far it's cost me $0.05 for 3 months). I'd *had* a Playnet AWS account set up but I don't remember the login details.

Couple of time/hassle reducers:

1/ Bring the box up as a higher-spec system so it's not slow to configure, tune etc, make an image when you're ready so you can easily re-roll, then bring it down, change it to a tiny for production runs. You can set up load-balancers so that people always access the most local one to them, which would help you avoid bandwidth charges on any given instance.

2/ Use vagrant (https://docs.vagrantup.com/v2/) to setup/configure the desired configuration locally and then worry about deploying them: http://www.iheavy.com/2014/01/16/how-to-deploy-on-amazon-ec2-with-vagrant/

There's an AWS feature called Elastic IP, you get one free elastic per instance, and I think that ties in to load balancers so that you can have a US-West and US-East instance that both "appear" to have the same virtual IP, or maybe the do that via DNS, not sure

[lungar]

I may have to manually adjust the "location" of each item to match the dot on the map if there is one. It's just tedious. It might be because the address of the items is an octet location which is 600 meters across.

I hope to have something to show by rat chat time.

@jwilly - No, that's incredibly difficult to do. The data is, to be honest, about as simple as the US tax code.

Octet is 800 meters across, that's the "oct" part

[lungar]

Take a look at the following JS scripts:

http://www.kfs.org/wiretap/wwiiol-meta.js

If you're working with the 1.35 code, the country enums are no-longer correct (we dropped a couple so that country fit in 4 bits instead of 5 so we could support neutrality)

And if you look at http://www.kfs.org/wiretap/wwiiol-latlon.js you'll find functions, for example, for generating a Google Maps query for a given facility/cp. (z is zoom, t is type, m being map as opposed to satellite, streetview, hybrid, etc)

[lungar]

All the facilities locations should probably be checked as well. As I recall there are discrepancies between Five's maps and a few FBs. I don't know whether or not that was a problem with Five's map renderings or the source data.

That's a discrepancy between the CP or Facility marker in the terrain database vs the actual location of the FB in terms of building objects. It happens with some towns, you click the town and it takes you to a flag miles away from the facilities.

[jwilly]

My take over the years has been that the game has good enough gameplay, and would get more customers if it had freshness...preferably a regular stream of toy changes, but map alternation would work too. I already gather than the work input requirements of the weapon/object development process are irreducible and wouldn't lend themselves to toolization.

I think it's clear that the world development process could be much faster and easier with good tools. It's a drag that the development of such world development tools is knowledgably described as "incredibly difficult to do".

As used here, does "incredibly difficult" mean four coders for three months, or fifty coders for two years?

Edited May 24, 2014 by jwilly

[jwilly]

That's a discrepancy between the CP or Facility marker in the terrain database vs the actual location of the FB in terms of building objects. It happens with some towns' date=' you click the town and it takes you to a flag miles away from the facilities.[/quote']

The terrain of course is built from a tileset. Towns of course are manually constructed, but their locations are defined by the location of a road feature on a particular tile. It sounds as if you're describing that the world database has an addressing or measurement basis other than the tile grid and then a location on that tile...?

[AHWULF]

Octet is 800 meters across' date=' that's the "oct" part [/quote']

Typo. I thought it was because there are 8 triangles in one :-)

I'm using an AWS instance, just a micro, with an EIP built with Turnkey Linux node.js/nginx image + Redis.

@jwilly "incredibly difficult" i.e. rocket science - ask DOC how painful it all is right now. A better tool would help some but would take a lot of work, I don't have a good estimate. Basically a start from scratch project.

[lungar]

The terrain of course is built from a tileset. Towns of course are manually constructed' date=' but their locations are defined by the location of a road feature on a particular tile. It sounds as if you're describing that the world database has an addressing or measurement basis other than the tile grid and then a location on that tile...?[/quote']

One day, Doc will be able to describe the full horror that is the terrain editor. Until he does, he gets to look crazy and insane.

CP and Facility positions are based on manually placed fake objects (markers), and those don't always match where the facilities/buildings were ultimately placed.

Remember the odd apple tree you used to find near FBs? That's the actual FB marker. That's where parts of the game think the FB actually is.