One of the first things I did on this project was to make the map data files. Without the map, you can't test or build any of the core functionality of HAE-Eye that defines the majority of HAE Stack's gameplay. Map files are kind of weird. There's a lot of different formats for maps that are used for a bunch of different things. Often you can convert from one format to another (but not always), and different libraries and tools are only compatible with certain formats.
For my situation I ended up using a .pmtiles file for the map's visuals, and a .geojson file for place information and search. I ended up with these files by downloading an extract of Open Street Maps data, and then running it through a bunch of tools to generate the files necessary to render them in game. There were some issues, but for a prototype they worked fine:
* Note how the water just abruptly cuts off at a weird angle
So there we go! I had my files, they rendered in game, and I could display detailed place information in search: time to start building the game! And for over a year I just used these same files, with the knowledge that eventually I would need to update them. But eventually comes eventually, and as the game quality increased, the janky-ness of the map became an obvious issue.
More than a year removed from the initial file creation, I realized that I had almost no memory of how I'd made the .pmtiles file I was using in the game. I knew I'd installed some tools on my Macbook (I usually develop on my Windows PC, but some of the tools only ran on Mac/Linux). Did I use tippecanoe or tilemaker or something else? Did I export from Geofabrik, BBBike, Planet.osm? Was the original file a pbf, osm, xml, geojson? I'm using pmtiles, but did I convert to an mbtiles file first or go directly to pmtiles? During the original file generation I'd experimented with pretty much all of those tools and technologies, but the exact sequence of steps I used to end up with workable files is something I'd completely forgotten.
If you're thinking: "well it can't be that confusing", Here's a rough step-by-step breakdown of the steps I used to create the current version of the map.
Download a map region as a .osm.pbf from a source like Geofabrik or BBBike.
Generate a .pmtiles map from that .osm.pbf using tilemaker.
Check that the .pmtiles file works using pmtiles.io or in-game by replacing the old map.pmtiles file.
Determine the bounding box longitude & latitude to limit the final map to in-game. (This will significantly reduce the final file size).
Crop the osm.pbf file using Osmium.
Regenerate the .pmtiles and test it to make sure nothing is wrong.
Realize something is wrong: In my case, a certain large body of water was completely missing from the cropped file (presumably because it extended out past the bounds.
Go to Open Street Map and find the ID of the missing "feature" by exploring the map.
Download the feature directly using the Open Street Map API with wget.
Merge that new data into the cropped .osm.pbf, but first use Osmium to create a new ID for the feature that's higher than the highest feature ID that exists in the cropped .osm.pbf to avoid conflicts, then use Osmium to merge them.
Generate a .pmtiles file again and check that the feature appears now. Repeat Steps 7-11 until satisfied.
Rename things that you want renamed. For example if you want to change the name of the city, or any other features, you can do that as a one-off by using Osmium to convert the cropped .osm.pbf to a .osm, and then running sed with a replacement string, and then converting back to .osm.pbf.
Generate the "final" .pmtiles file using Tilemaker. Replace the old map.pmtiles file in game.
Generate the .geojson file using Osmium.
But we're not done! The generated .geojson file will include a bunch of junk, including duplicate entries for various IDs, so open up the GeoJSONFixer.html file I wrote, and drop in the .geojson file to get a fixed.geojson file. Replace the map.geojson file in game with that new file.
Now you're done... (except not really).
So, just a simple... 16 step process (if you ignore all the installations, set up, and various configuration files, and most of the console commands) to go from a download to working game files! It works, it's reproducible, and I actually took notes about how to do it this time!
Something that bugged me a little is that even with the name of the city changed, all the other map data is still identical to the real city this data comes from. I'm not a fan of that, since the implications are that you're viewing personal, potentially embarrassing information about people who live or work at specific locations. I'd been saying for a while that I should shuffle or rename all the streets in the game to more thoroughly distinguish between the real city and this fake HAE Stack version, so for the last couple of days I've been working on that.
Now after step 12 in the above list of steps, you need to add:
Open the ShuffleOsmStreetNames.html file.
Convert the .osm.pbf to a .osm and drop it into that webpage.
If you haven't already, generate the shuffle-names-list.json file. This will take all the street names listed in building addresses and export them into a big list.
Go through the list of street names and delete anything that shouldn't be shuffled.
Load that street names file into the shuffler and generate the shuffled-names-mapping.json file.
Click the shuffle button and wait while several thousand find/replace operations run on your 100MB .osm file.
Run the steps to generate the .pmtiles file and test it in the game. Amend the mappings and street names files and repeat steps 3-7 as necessary.
This is still a work-in-progress and I haven't done a full QA pass to test it in game. The street name shuffle has resulted in some pretty dumb looking stuff before I iterated on it, for example, a neighborhood being renamed "Chesapeake Harbor Harry S Trumans", because I was replacing the word "Hill" with "Harry S Trumans".
Once I get all the bugs worked out and update the map, I'll also have to change any text that references specific addresses. Fortunately in the game data files I use location IDs that aren't affected by this street name change.
All of this so that the fake people living at the fake versions of the real places in this fake city won't be confused with the real people living at the real places in the real city.
One of the other reasons it was important for me to formalize and document the map and data creation process is that eventually other people may be able to make their own content for the game. I've implemented much of the game such that you could load in your own story and people, and fully author everything from the chat system, the tasks, and people's position histories. But that's not quite ready for the public to start working on, so I'll save the discussion about that for a later Dev Diary post.