# duckOSM > OpenStreetMap to a routable road network in one DuckDB file, with edge IDs that survive every rebuild, clip and export. duckOSM turns OpenStreetMap data into routable road networks (driving, walking, cycling) in one DuckDB file, with stable edge ids, legal turns, and exports to SUMO, MATSim, GMNS and more. An agent skill with the essentials: https://github.com/Khoshkhah/duckOSM/blob/main/skills/duckosm/SKILL.md # Get started # duckOSM OpenStreetMap → a routable road network in one DuckDB file, with **edge IDs that survive every rebuild, clip and export**. [Get started](https://khoshkhah.github.io/duckOSM/first_network/index.md) [View on GitHub](https://github.com/Khoshkhah/duckOSM) ## Why duckOSM Most OSM-to-network tools number their edges `0, 1, 2, …`. Rebuild with fresher OSM data, cut out a smaller area, or export to a simulator, and the numbers change, so every table keyed on them (traffic counts, speeds, map matches, demand) has to be matched to the network again. duckOSM's `edge_id` is a **content hash** of three ids that OpenStreetMap already gives the road segment: the OSM **way** it comes from (`osm_id`) and the OSM **nodes** where it starts and ends (`source`, `target`): ``` edge_id = hash(osm_id, source, target) ``` The order of `source` and `target` is the direction, so the two directions of a two-way street get different ids. None of the three is a number duckOSM makes up, so the id stays the same: - **Across rebuilds** ______________________________________________________________________ The same OSM data gives the same ids. An id only changes if that way or its end nodes change in OSM. - **Across clips** ______________________________________________________________________ Build a country once, cut out a city in seconds: the city keeps the country's ids. - **Across exports** ______________________________________________________________________ SUMO, MATSim, GMNS, GeoPackage and networkx all use the `edge_id` as their own id. [How the id is built, in detail](https://khoshkhah.github.io/duckOSM/concepts/edge-ids/index.md) ## Quick start ``` pip install "duckosm[routing]" curl -LO https://download.geofabrik.de/europe/monaco-latest.osm.pbf duckosm build --pbf monaco-latest.osm.pbf --output monaco.duckdb --modes driving ``` ``` import duckdb from duckosm import route con = duckdb.connect("monaco.duckdb", read_only=True) q = "SELECT min(edge_id) FROM driving.edges WHERE name = ?" a = con.execute(q, ["Boulevard du Larvotto"]).fetchone()[0] b = con.execute(q, ["Avenue Princesse Grace"]).fetchone()[0] r = route(con, a, b) # fastest path; weight="length" for distance r["time_s"], r["length_m"], r["edges"] # ~114 s, ~1.5 km, the ordered edge_ids ``` Next: [Your first network](https://khoshkhah.github.io/duckOSM/first_network/index.md) walks through it step by step, and the [guides](https://khoshkhah.github.io/duckOSM/guides/build/index.md) cover each task. ## What you get - **A driving, a walking and a cycling network** in one file, one schema each, split at the same junctions so a road has the same `edge_id` in every mode. OSM access tags are respected: private roads are kept for the map but never routed ([details](https://khoshkhah.github.io/duckOSM/concepts/networks/#access-private-and-forbidden-roads)). - **The graph of legal turns** (`edge_graph`), with OSM turn restrictions, for routing in SQL, Python or networkx ([Route](https://khoshkhah.github.io/duckOSM/guides/route/index.md)). - **Base-map layers** (`features.*`: water, land use, buildings, POIs) in the same file ([Base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md)). - **Maps in one offline HTML page:** `duckosm viz` draws the network and the base map from the database ([Draw a map](https://khoshkhah.github.io/duckOSM/guides/draw-map/index.md)); `duckosm route-map` is a route planner (drive, walk, cycle, walk + drive) where you drag a start and an end and get turn-by-turn directions ([try it](https://khoshkhah.github.io/duckOSM/guides/route/#on-a-map)). The pages are drawn by [mapstyle](https://khoshkhah.github.io/mapstyle/). - **The drawing order of the roads:** `duckosm levels` stores which road is painted over which (casing and fill numbers) in the file ([Store the drawing order](https://khoshkhah.github.io/duckOSM/guides/drawing-order/index.md)). - **Any area:** a GeoJSON boundary, a place name, a box or an H3 cell; `duckosm boundary` finds an area's border by name, and `duckosm extract` cuts a city out of a country build in seconds ([Prepare an area](https://khoshkhah.github.io/duckOSM/guides/prepare-area/index.md)). - **Your own fixes for OSM errors**, applied on every build ([Fix OSM errors](https://khoshkhah.github.io/duckOSM/guides/fix-osm-errors/index.md)). ## Exports | Format | Command | How it's verified | | -------------------------------------------------------------------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------- | | [**SUMO**](https://khoshkhah.github.io/duckOSM/exports/sumo/index.md) | `duckosm sumo` | SUMO's own `netconvert` assembles the network; turn restrictions become explicit connections | | [**MATSim**](https://khoshkhah.github.io/duckOSM/exports/matsim/index.md) | `duckosm matsim`, `matsim-lanes` | validated against MATSim's official DTD and XSD schemas | | [**GMNS**](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md) | `duckosm gmns` | follows the GMNS table spec | | [**GeoPackage / shapefile**](https://khoshkhah.github.io/duckOSM/exports/gis/index.md) | `duckosm export-gis` | `duckosm gis-debug` reads the file back through GDAL and diffs every `edge_id` | | [**networkx**](https://khoshkhah.github.io/duckOSM/exports/networkx/index.md) | `duckosm export-graph` | unit tests; GraphML or lossless gpickle | **Experimental:** OpenDRIVE, Lanelet2, railML, lane-level routing, GMNS meso/micro networks, intermodal routing, elevation, admin boundaries and base-map layers. They work and are structure-tested, but haven't been checked in the target tools yet. Which export for which tool: [Exports](https://khoshkhah.github.io/duckOSM/exports/index.md). ______________________________________________________________________ duckOSM is MIT-licensed and built on [DuckDB](https://duckdb.org) and its spatial extension. It is an independent project, not affiliated with or endorsed by the DuckDB Foundation. # Install duckOSM needs Python 3.10 or newer. ``` pip install duckosm duckosm --version # duckosm, version 0.2.0 ``` ## Extras Some features need more packages. Add the extras you want in brackets, for example `pip install "duckosm[routing,viz]"`. | Extra | Adds | For | | ----------- | ------------------- | ------------------------------------------------------------ | | `routing` | networkx | `route()`, `Router`, `to_networkx()`, `duckosm export-graph` | | `viz` | roadstyle | `duckosm viz` maps | | `sumo` | SUMO's `netconvert` | `duckosm sumo` | | `elevation` | rasterio, pyproj | `duckosm elevation` | ## Tools outside Python Two command-line tools that pip can't install make some steps better. duckOSM works without them. | Tool | Used for | Without it | Install | | -------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | **osmium** | cutting a PBF to a boundary (`build -b`, `clip-pbf`) | `build` uses the whole PBF (with a warning); `clip-pbf` fails | `apt install osmium-tool` · `brew install osmium-tool` | | **GDAL** (`ogr2ogr`) | reading borders from a PBF (`boundary`, `admin`), one-file GeoPackage (`export-gis`) | `boundary` looks the border up online; `export-gis` writes one file per layer; `admin` fails | `apt install gdal-bin` · `brew install gdal` | ## Where files go Every relative path (outputs, configs, reports) is relative to the folder you run `duckosm` in. Next: [build your first network](https://khoshkhah.github.io/duckOSM/first_network/index.md). # Your first network This builds the road network of Monaco from scratch, then queries it, routes on it and draws it. It takes a few minutes. You need duckOSM [installed](https://khoshkhah.github.io/duckOSM/install/index.md) with `pip install "duckosm[routing,viz]"`. ## 1. Choose your area Search for **Monaco** on [openstreetmap.org](https://www.openstreetmap.org) and click a result whose border is drawn on the map. The page address ends in `relation/`: that number is the area's **OSM id**. Monaco has two: the country, [relation/1124039](https://www.openstreetmap.org/relation/1124039), whose border reaches far out to sea (its territorial waters), and its land area, [relation/2220322](https://www.openstreetmap.org/relation/2220322). For roads we want the land: **2220322**. ## 2. Download the map data OpenStreetMap data comes as `.osm.pbf` files. [Geofabrik](https://download.geofabrik.de) offers free extracts of every continent, country and many regions, updated daily. Monaco has its own: ``` curl -LO https://download.geofabrik.de/europe/monaco-latest.osm.pbf # 0.7 MB ``` For your own area, download the smallest region that contains it. ## 3. Make the boundary file ``` duckosm boundary --osm-id 2220322 --pbf monaco-latest.osm.pbf ``` ``` wrote monaco.geojson: Monaco (admin level 8, OSM relation 2220322, 2.39 km²), from PBF monaco-latest.osm.pbf ``` ## 4. Build the network ``` duckosm build --pbf monaco-latest.osm.pbf -b monaco.geojson \ -m driving -m walking -m cycling ``` About ten seconds later you have **`monaco.duckdb`**: one file holding a driving, a walking and a cycling network, each in its own schema, plus the [base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md) (water, buildings, POIs…). Roads that cross the border are kept whole, and small pieces that don't connect to the rest are removed. ## 5. Query it The database is a normal DuckDB file. Which kinds of road make up Monaco's driving network? ``` import duckdb con = duckdb.connect("monaco.duckdb", read_only=True) con.sql("""SELECT highway, count(*) AS edges, round(sum(length_m) / 1000, 1) AS km FROM driving.edges GROUP BY highway ORDER BY km DESC LIMIT 5""").show() ``` | highway | edges | km | | ----------- | ----- | ---- | | residential | 919 | 31.7 | | service | 781 | 20.1 | | secondary | 428 | 14.9 | | tertiary | 387 | 12.5 | | primary | 197 | 11.2 | ## 6. Find a route Routes go from one edge (a directed piece of road) to another, and only take legal turns: ``` from duckosm import route q = "SELECT min(edge_id) FROM driving.edges WHERE name = ?" a = con.execute(q, ["Boulevard du Larvotto"]).fetchone()[0] b = con.execute(q, ["Avenue Princesse Grace"]).fetchone()[0] r = route(con, a, b) r["time_s"], r["length_m"] # (114.1, 1513.0): about 2 minutes, 1.5 km r["edges"] # the 29 edge_ids along the way ``` Every `edge_id` stays the same when you rebuild, cut out a smaller area, or export: see [Stable edge ids](https://khoshkhah.github.io/duckOSM/concepts/edge-ids/index.md). ## 7. Draw it ``` duckosm viz monaco.duckdb -m driving # -> reports/monaco_driving_network.html (drawn by mapstyle) ``` Open the file in a browser. This is that file, live: zoom, hover a road, click one to see its `edge_id`, toggle road classes in the legend: ## Next - Build your own area: the same steps, with your area's id and region file. - [Guides](https://khoshkhah.github.io/duckOSM/guides/prepare-area/index.md): prepare other areas, build options, fix OSM errors, query, route, maps, elevation. - Exports: [SUMO](https://khoshkhah.github.io/duckOSM/exports/sumo/index.md), [MATSim](https://khoshkhah.github.io/duckOSM/exports/matsim/index.md), [GMNS](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md), [GIS](https://khoshkhah.github.io/duckOSM/exports/gis/index.md), [networkx](https://khoshkhah.github.io/duckOSM/exports/networkx/index.md). # Guides # Prepare an area Before a build you need two things: the map data (a `.osm.pbf` file) and, for a city or district, its boundary. [Your first network](https://khoshkhah.github.io/duckOSM/first_network/index.md) walks through this once for Monaco; this page has the details. ## Get the map data [Geofabrik](https://download.geofabrik.de) offers free `.osm.pbf` extracts of every continent, country and many regions, updated daily. Download the **smallest region that contains your area**: ``` curl -LO https://download.geofabrik.de/europe/monaco-latest.osm.pbf # 0.7 MB curl -LO https://download.geofabrik.de/europe/sweden-latest.osm.pbf # 835 MB ``` Any `.osm.pbf` works, up to the [whole planet](https://planet.openstreetmap.org). ## Find the area's id Search for the area on [openstreetmap.org](https://www.openstreetmap.org) and click the result whose border is drawn on the map. The page address ends in `relation/`: that number is the **OSM id**. Seeing the outline is the surest way to pick the right area: there is a Södermalm in Stockholm and one in Sundsvall, and a coastal country's border can reach far out to sea. Monaco's country border ([relation/1124039](https://www.openstreetmap.org/relation/1124039)) includes its territorial waters; its land area is [relation/2220322](https://www.openstreetmap.org/relation/2220322). ## Make the boundary file ``` duckosm boundary --osm-id 2220322 --pbf monaco-latest.osm.pbf # -> monaco.geojson ``` ``` wrote monaco.geojson: Monaco (admin level 8, OSM relation 2220322, 2.39 km²), from PBF monaco-latest.osm.pbf ``` With `--pbf`, the border is read from the PBF itself (offline; needs [GDAL](https://khoshkhah.github.io/duckOSM/install/#tools-outside-python)). Without it, or if GDAL is missing, it's fetched from OpenStreetMap online. **By name instead of id.** `duckosm boundary Monaco --pbf monaco-latest.osm.pbf` searches by name: case, accents and hyphens don't matter, and English names (`name:en`) match too. If the name matches more than one border, it takes the exact name with the largest area and lists the others with their ids, so you can rerun with the right `--osm-id`. For "Monaco" it takes the country (with its sea) and lists the land area: ``` also matched (pick one with --osm-id): Monaco (admin level 8, OSM relation 2220322, 2.39 km²) Monaco-Ville (admin level 10, OSM relation 2220207, 0.2 km²) ``` | Option | Does | | ------------- | --------------------------------------- | | `--osm-id ID` | take this exact OSM relation | | `--pbf FILE` | read the border from this PBF (offline) | | `--offline` | never go online | | `-o FILE` | output file (default `.geojson`) | Any GeoJSON polygon you already have works as a boundary too. ### Or let the build make it In a [config file](https://khoshkhah.github.io/duckOSM/guides/build/#with-a-config-file), give the area instead of a file: ``` boundary: place: Monaco # found like `duckosm boundary Monaco` # bbox: [7.40, 43.72, 7.44, 43.75] # or a box: west, south, east, north # h3_cell: 883969a403fffff # or one H3 cell (also: build --h3-cell) ``` The build writes it to `.boundary.geojson` next to the output and uses that file. ## Several areas from one region To build many areas of one country, build the country once and cut the areas out of it. Cutting takes seconds, re-reads no OSM, and every `edge_id` stays the same as in the parent, so data keyed on the parent's ids works on the areas as they are: ``` duckosm build --pbf sweden-latest.osm.pbf -m driving # slow, once -> sweden-latest.duckdb duckosm boundary --osm-id 2017432 -o sodermalm.geojson # Södermalm, Stockholm duckosm extract --source sweden-latest.duckdb --db sodermalm.duckdb --boundary sodermalm.geojson ``` `extract` keeps edges that cross the border whole. It can also find the area by `--name` or `--osm-id` if the parent has [administrative boundaries](https://khoshkhah.github.io/duckOSM/guides/admin-boundaries/index.md). The same cut can run as a build from a config file with `source.type: duckdb`; see [Configuration](https://khoshkhah.github.io/duckOSM/reference/configuration/index.md). ## Cut a smaller PBF To get just the area's map data as a PBF, for example for another tool: ``` duckosm clip-pbf sweden-latest.osm.pbf sodermalm.geojson # -> sodermalm.osm.pbf ``` It needs [osmium](https://khoshkhah.github.io/duckOSM/install/#tools-outside-python). By default, ways that cross the border are kept whole (`--strategy complete_ways`); `--strategy smart` also keeps rivers and other areas whole, which the [base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md) need. # Build a network ``` duckosm build --pbf monaco-latest.osm.pbf -b monaco.geojson -m driving -m walking -m cycling ``` This writes `monaco.duckdb`: one schema per mode (`driving`, `walking`, `cycling`), each with its edges, nodes, the graph of legal turns and, for driving, the turn restrictions; and the [base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md) (`features`: water, land, buildings, POIs). With a boundary (`-b`), the build first cuts the PBF to it (a copy is kept in `./pbf/` for the next build, until the boundary or the PBF changes) and removes small pieces of network that don't connect to the rest. What each step does: [How a build works](https://khoshkhah.github.io/duckOSM/concepts/build/index.md). ## Options | Option | Does | Default | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------- | | `-p`, `--pbf` | input `.osm.pbf` | | | `-b`, `--boundary` | GeoJSON area to build ([make one](https://khoshkhah.github.io/duckOSM/guides/prepare-area/index.md)) | none: the whole PBF | | `-m`, `--modes` | `driving`, `walking` or `cycling`; repeat for several | `driving` | | `-o`, `--output` | output file | `.duckdb`, else `.duckdb` | | `-c`, `--config` | a config file (below) | | | `--source-db` | cut from a built db instead of a PBF ([how](https://khoshkhah.github.io/duckOSM/guides/prepare-area/#several-areas-from-one-region)) | | | `--graph` / `--no-graph` | build the graph of legal turns | on | | `--h3-index` / `--no-h3-index` | add H3 cell ids to nodes and edges | on | | `--h3-resolution` | H3 resolution, 0–15 | `8` | | `--no-features` | skip the [base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md) (water, land, buildings, POIs) | built | | `--fixes` | a rules file of [fixes for OSM errors](https://khoshkhah.github.io/duckOSM/guides/fix-osm-errors/index.md) | none | | `--log-file` | also write the log to a file | console only | ## With a config file For repeatable builds, keep the settings in a file. `duckosm init-config` writes a fully commented template: ``` duckosm init-config monaco.yaml # edit it, then: duckosm build -c monaco.yaml ``` ``` name: monaco # output: /monaco.duckdb output_path: . source: pbf_path: monaco-latest.osm.pbf boundary: path: monaco.geojson modes: [driving, walking, cycling] validation: enabled: true # check the result (see "Check a build") ``` A config file also reaches settings the command line doesn't have: [validation](https://khoshkhah.github.io/duckOSM/guides/check-build/index.md), [fixes for OSM errors](https://khoshkhah.github.io/duckOSM/guides/fix-osm-errors/index.md), memory limits, and more. Every field: [Configuration](https://khoshkhah.github.io/duckOSM/reference/configuration/index.md). In a clone of the repo, `config/sample_monaco.yaml` is a ready example. ## Large areas A whole country takes time and memory (Sweden: 835 MB of PBF). The build processes the network in batches sized to fit; if it still runs out of memory, set `options.memory_limit` (e.g. `"16GB"`) and `options.threads` in the config. Build a big region once and [cut areas from it](https://khoshkhah.github.io/duckOSM/guides/prepare-area/#several-areas-from-one-region) instead of rebuilding each one. # Fix OSM errors OpenStreetMap is sometimes wrong for a road you care about: a one-way street tagged as two-way, a wrong lane count, a missing turn ban. The best fix is [editing OpenStreetMap](https://www.openstreetmap.org/edit), but that takes time to reach your downloads. Meanwhile, duckOSM can apply the correction itself, on every build, from a rules file. ## The rules file ``` overrides: # fixes to a way (keyed by its OSM id) - osm_id: 4230100 # Avenue Delphine: make it one-way oneway: true - osm_id: 4230099 # Avenue Saint-Romain: not for bikes exclude_modes: [cycling] turn_restrictions: # turns OSM should forbid but doesn't - from_way: 4230100 # no turn from Avenue Delphine onto Avenue Saint-Romain via_node: 21924057 to_way: 4230099 restriction: no_left_turn ``` Give it to the build, on the command line or in your [config file](https://khoshkhah.github.io/duckOSM/guides/build/#with-a-config-file): ``` duckosm build --pbf monaco-latest.osm.pbf -b monaco.geojson -m driving -m cycling --fixes fixes.yaml ``` ``` osm_overrides: fixes.yaml ``` The build log says how many rules matched, per network: ``` OSM fixes (fixes.yaml, driving): 1 of 2 way rules matched this area OSM fixes (fixes.yaml): 1 of 1 turn rule matched this area OSM fixes (fixes.yaml, cycling): 2 of 2 way rules matched this area ``` On Monaco, the file above gives: | | Without | With | | --------------------------------------------------- | ------- | ----------- | | driving edges of Avenue Delphine | 2 | 1 (one-way) | | cycling edges of Avenue Saint-Romain | 18 | 0 | | turns from Avenue Delphine onto Avenue Saint-Romain | 1 | 0 | A rule for a way or junction that isn't in the area does nothing, so one file can hold fixes for every area you build. Fixes are only applied when you name the file. ## Way fixes (`overrides`) | Field | Does | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `osm_id` | the way to fix (required) | | `oneway` | `true` / `false`: force one-way or two-way | | `lanes` | lanes per direction, both directions | | `lanes_forward`, `lanes_backward` | lanes per direction, each separately | | `layer` | vertical level (fixes a wrong `layer` tag, e.g. a ramp at ground level tagged `layer=-1`) | | `exclude_modes` | remove the way from these networks: `driving`, `walking`, `cycling` | | `remove` | `true`: remove the way from the build (no road, building or feature) | | `tags` | set these OSM tags on the way before the build reads it, e.g. `{highway: unclassified}` for a car road that OSM tags `pedestrian` | | `keep_nodes` | `[from, to]`: keep only this part of the way (OSM node ids). To keep the other part, add it under `new_ways` with `tags_from` | | `turn_lanes_forward`, `turn_lanes_backward` | the painted arrows, in OSM `turn:lanes` syntax (`through;left\|right`) | | `note` | free text, for your own record | ## New ways (`new_ways`) A road that OSM does not have: ``` new_ways: - id: 9000001049107 source: nvdb:1049107 nodes: [180929, [18.063835, 59.328511], 180933] tags: {highway: tertiary, oneway: "yes", name: "Rödbodtorget"} ``` | Field | Does | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `id` | the way's id (required): 9,000,000,000,000 or more, used one time only. Do not change it: the edge ids come from it | | `nodes` | the nodes in the way's direction: an OSM node id, or `[lon, lat]` for a new node. Make both ends OSM nodes, so that the way connects | | `tags` | the OSM tags of the way | | `tags_from` | an OSM way whose tags (before any fix) the new way starts with; `tags` goes on top | | `source`, `note` | free text, for your own record | A new way that ends on a node inside another way cuts that way there, so that way gets new edge ids. ## Turn fixes (`turn_restrictions`) | Field | Does | | -------------------------------- | -------------------------------------------------------------------------------------- | | `from_way`, `via_node`, `to_way` | the turn: from this way, through this junction node, onto that way (OSM ids, required) | | `restriction` | `no_u_turn` (default), `no_left_turn`, `no_right_turn`, `no_straight_on`, or `only_*` | | `note` | free text | The turn disappears from the driving network everywhere: routing, and every export (SUMO, MATSim, GMNS, OpenDRIVE). To find the ids, look the road up on [openstreetmap.org](https://www.openstreetmap.org) (click it: the address ends in `way/`; junctions are `node/`), or use [`duckosm way`](https://khoshkhah.github.io/duckOSM/guides/query/#look-up-one-osm-way). Only fix what you've checked on the ground or in imagery: a wrong `oneway` silently removes a direction of travel. # Query the database A build is one DuckDB file. Open it from Python, the [DuckDB command line](https://duckdb.org/docs/installation/), or any tool that reads DuckDB (DBeaver, …). Every example on this page was run on the Monaco network from [Your first network](https://khoshkhah.github.io/duckOSM/first_network/index.md). ``` import duckdb con = duckdb.connect("monaco.duckdb", read_only=True) con.execute("LOAD spatial") # for the ST_ functions below con.sql("SELECT count(*) FROM driving.edges").show() ``` ## Where the tables are Each mode has its own schema: `driving.edges`, `walking.edges`, `cycling.edges`, and so on. Always name the schema, or switch to it with `USE driving;`. A tool that shows no tables is usually looking at the empty default schema `main`. Builds from a PBF also keep the parsed OSM data in `raw`. Private roads (driveways, gated streets) and, in driving, bus lanes are in `driving.private_edges` and the like, not in `edges`: you can see them but not route on them ([why](https://khoshkhah.github.io/duckOSM/concepts/networks/#access-private-and-forbidden-roads)). Every table and column: [Database schema](https://khoshkhah.github.io/duckOSM/reference/database/index.md). To see what one database holds (modes, counts, time zone, build date), run `duckosm info monaco.duckdb`. ## The size of each network ``` SELECT 'driving' AS mode, count(*) AS edges, round(sum(length_m) / 1000, 1) AS km FROM driving.edges UNION ALL SELECT 'walking', count(*), round(sum(length_m) / 1000, 1) FROM walking.edges UNION ALL SELECT 'cycling', count(*), round(sum(length_m) / 1000, 1) FROM cycling.edges; ``` | mode | edges | km | | ------- | ----- | ----- | | driving | 2765 | 92.6 | | walking | 10952 | 244.1 | | cycling | 10274 | 245.5 | ## Roads ``` -- the fastest roads SELECT name, highway, maxspeed_kmh, round(length_m) AS length_m FROM driving.edges WHERE maxspeed_kmh >= 70 ORDER BY maxspeed_kmh DESC, length_m DESC LIMIT 5; -- the longest edges SELECT name, highway, round(length_m) AS length_m FROM driving.edges ORDER BY length_m DESC LIMIT 5; -- one-way and two-way SELECT oneway, count(*) AS edges FROM driving.edges GROUP BY oneway; ``` ## Turns ``` -- where you can go next from one edge (the graph of legal turns) SELECT e.edge_id, e.name, e.highway FROM driving.edge_graph g JOIN driving.edges e ON e.edge_id = g.to_edge WHERE g.from_edge = 2226047604433257818; -- the end of Avenue Delphine ``` | edge_id | name | highway | | ------------------- | ------------------- | ----------- | | 8072509973910824634 | Avenue Saint-Romain | residential | | 3664387098764709418 | Avenue Delphine | residential | ``` -- junctions with the most turn restrictions SELECT via_node, count(*) AS restrictions FROM driving.turn_restrictions GROUP BY via_node ORDER BY restrictions DESC LIMIT 5; ``` ## Places Coordinates are longitude/latitude (EPSG:4326). DuckDB's distance function on the sphere expects latitude first, so flip the geometry: ``` -- the roads nearest a point (the Casino de Monte-Carlo: lat 43.7397, lon 7.4270) SELECT name, highway, round(ST_Distance_Sphere(ST_FlipCoordinates(ST_Centroid(geometry)), ST_Point(43.7397, 7.4270))) AS metres FROM driving.edges ORDER BY metres, edge_id LIMIT 3; ``` | name | highway | metres | | ---------------------- | ----------- | ------ | | Allées des Boulingrins | service | 19.0 | | Allées des Boulingrins | residential | 19.0 | | Avenue de Monte-Carlo | residential | 21.0 | ``` -- edges per H3 cell (the cell of each edge's start) SELECT from_cell, count(*) AS edges FROM driving.edges GROUP BY from_cell ORDER BY edges DESC LIMIT 3; ``` ## Raw OpenStreetMap data Builds from a PBF keep every OSM node, way and relation in `raw`, with all their tags, so you can reach things that aren't roads (areas cut from a bigger build don't have `raw`): ``` SELECT osm_id, tags['name'] AS name, tags['amenity'] AS amenity FROM raw.nodes WHERE tags['amenity'] IN ('hospital', 'pharmacy') AND tags['name'] IS NOT NULL LIMIT 5; ``` ## Look up one OSM way `duckosm way` shows all the data a build has for one OSM **way** (the OpenStreetMap object for a street, or a stretch of one): first its raw OSM tags and node list, then every edge built from it, in every mode, as one table. ``` duckosm way monaco.duckdb 4230100 # Avenue Delphine; -m driving for one mode, --geom for WKT duckosm way monaco.duckdb 4230100 -o way.csv # write the table to a file (.csv / .parquet / .json) ``` ``` raw way 4230100: 10 node refs tags: {'highway': 'residential', 'name': 'Avenue Delphine'} refs: [21923931, 12421715488, 21924090, 25243156, 3625098865, ...] ``` | mode | edge_id | edge_ref | source → target | length_m | maxspeed_kmh | cost_s | walk_type / cycle_type | | ------- | ------------------- | ---------- | ------------------- | -------- | ------------ | ------ | ---------------------- | | cycling | 2226047604433257818 | 4230100#1f | 21923931 → 21924057 | 88.3 | 15 | 21.2 | mixed_traffic | | driving | 2226047604433257818 | 4230100#1f | 21923931 → 21924057 | 88.3 | 30 | 10.6 | | | walking | 2226047604433257818 | 4230100#1f | 21923931 → 21924057 | 88.3 | 5 | 63.6 | shared_road | | cycling | 3664387098764709418 | 4230100#1r | 21924057 → 21923931 | 88.3 | 15 | 21.2 | mixed_traffic | | driving | 3664387098764709418 | 4230100#1r | 21924057 → 21923931 | 88.3 | 30 | 10.6 | | | walking | 3664387098764709418 | 4230100#1r | 21924057 → 21923931 | 88.3 | 5 | 63.6 | shared_road | (A selection of the columns; the real table has every `edges` column.) This two-way street became two edges, one per direction (`#1f` forward, `#1r` reverse). Each has the same `edge_id` in all three modes, with that mode's own speed and travel time. ``` from duckosm.query import way_table way_table(con, 4230100).show() # the same table in Python / a notebook ``` Use it when a particular street looks wrong: how it was split, which edges came from it, what speed and cost each got. A negative `osm_id` looks up the short connector edges duckOSM adds to join dangling paths. # Draw a map ``` pip install "duckosm[viz]" duckosm viz monaco.duckdb # in reports/: monaco_map.html (every mode) + one page per mode ``` The maps are drawn by [mapstyle](https://khoshkhah.github.io/mapstyle/), duckOSM's map library, from the database itself: water, land, buildings and the sea ([base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md)) under the roads, so the page needs no background from the web (one can still be picked). The roads are coloured by class, with a hover tooltip and a 2D/3D button; click a road to copy its `edge_id`. Tunnels are drawn under the roads above them, bridges on top. Private roads are grey and bus lanes a muted blue: you see them, but they are not in the network ([why](https://khoshkhah.github.io/duckOSM/concepts/networks/#access-private-and-forbidden-roads)). In the Roads box, the road classes and the Bridges, Tunnels, Private roads and Bus lanes rows hide or show those roads. `monaco_map.html` has every mode; each `monaco__network.html` brings one mode to the front. Try it: this is Monaco with every mode, as `duckosm viz` wrote it. | Option | Does | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `-m driving` | only that mode's page (repeat for several); without `-m`: every mode's page plus `_map.html` | | `--basemap voyager` | a background from the web under the map: `voyager`, `positron`, `osm`, `satellite` (default: none, the database's own layers) | | `--no-arrows` | no direction arrows (on by default, on one-way roads; zoom in to see them) | | `--no-boundary` | hide the dashed outline of the area | | `--out-dir maps` | output folder (default `reports`) | A page holds the whole network, so keep it to a city. For other looks, or to build your own dashboard, use mapstyle directly (`pip install mapstyle`; its themes, path styles and dashboard page are in its [docs](https://khoshkhah.github.io/mapstyle/)). **Other palettes, or colour by a column** (needs a clone of the repo): ``` python scripts/roadstyle_map.py --db monaco.duckdb --palette mono python scripts/roadstyle_map.py --db monaco.duckdb --color-by maxspeed_kmh ``` The order in which crossing and meeting roads are painted can be stored in the file: [Store the drawing order](https://khoshkhah.github.io/duckOSM/guides/drawing-order/index.md). # Store the drawing order ``` pip install "duckosm[levels]" duckosm levels monaco.duckdb # the level area monaco.levels/, solved, written into the file duckosm levels-edit monaco.duckdb # optional: fix places by hand, with mapstyle's base map and every road; each Apply writes into the file ``` When roads cross or meet, a map must decide which is painted over which: a bridge over the road under it, a street over its sidewalk, a junction without a ring between the roads that meet. The map does it with two numbers for every road: a **casing number** (for the outline, in three parts: start, main, end) and a **fill number** (for the colour). The casings and fills are painted number by number, lowest first, and at each number all casings before all fills. 0 is the ground. `duckosm build` hands the roads to [roadstyle](https://khoshkhah.github.io/roadstyle/guides/levels/) as its last step, which works them out exactly as for any roadstyle map, and stores the result in the schema `visualization` of the file, so that a map or another tool reads it instead of computing it again. The file is always current: **a rebuild keeps your edits** (they are tables in the file) and solves again with them. For a big area the step takes long: `options.levels: false` (`build --no-levels`) skips it, and `duckosm levels` makes it later. ## The level area The roads of **all the modes together** (`edges` and `private_edges` of `driving`, `walking` and `cycling`) go into one **level area**, a folder next to the file (`monaco.levels/`, or `--area DIR`): | File | Whose | Holds | | ------------------------------------ | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `roads.parquet`, `pairs.csv` | made by each run | the solver's input: one row per road, the pairs of roads that cross or meet | | `edits.csv`, `heads.csv`, `caps.csv` | **yours**; a copy of the tables in the file | your changes: pairs added or switched off, head lengths and end shapes per road end. The tables are the master: each run writes these files from them | | `levels.csv`, `levels_info.json` | made by each solve | the result, and what the solver says about it | | `area.json` | made once | the file the area belongs to | Your edits are in the file itself (`visualization.level_*`, below): a rebuild carries them over, and the folder is only a work area. A file of the folder that differs from the table is kept as `*.csv.before` when it is written again. A row that names a road that is not in the file any more is ignored with a warning and kept in `visualization.level_unmatched`; the next build tries it again. How the solver decides, and the editor: [Which road is on top](https://khoshkhah.github.io/roadstyle/guides/levels/). ## What it writes | Table | Holds | | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `visualization.edge_levels` | one row for every `edge_id`: `casing_start`, `casing_level`, `casing_end`, `fill_level`, and its ends as drawn: `head_start_m`, `head_end_m`, `cap_start`, `cap_end` | | `visualization.edge_levels_meta` | one row: where it came from (the area), the number of edges and a hash of their ids, the roadstyle version, the time | | `visualization.level_edits`, `level_heads`, `level_caps` | your `edits.csv`, `heads.csv` and `caps.csv`, copied at each solve, so the database holds everything. A new area folder gets them back from here (needs a roadstyle that has `restore_area_inputs`) | Both tables are replaced at each solve, by `duckosm levels` and by the editor. A stack of roads that cannot all be kept is counted in the message, never hidden. ## Read it ``` import duckdb, roadstyle as rs from duckosm.levels import load_roads con = duckdb.connect("monaco.duckdb", read_only=True) levels = rs.load_area_levels(con, load_roads("monaco.duckdb")) ``` `load_area_levels` checks that the edges are those the numbers were solved for, and stops with a message if not. It never recomputes. After a rebuild of the file, run `duckosm levels` again. Options: [`duckosm levels`](https://khoshkhah.github.io/duckOSM/reference/cli/#levels). # Route A route goes from one edge (a directed piece of road) to another over the graph of legal turns, so it never takes a banned turn. Needs `pip install "duckosm[routing]"`. ``` import duckdb from duckosm import route, Router con = duckdb.connect("monaco.duckdb", read_only=True) r = route(con, from_edge, to_edge) # fastest, by car r["edges"], r["time_s"], r["length_m"] # ordered edge_ids, seconds, metres r["path"] # per edge: name, highway, length_m, cost_s, geometry route(con, from_edge, to_edge, weight="length") # shortest instead of fastest route(con, from_edge, to_edge, mode="walking") # on the walking network (or "cycling") router = Router(con, mode="driving") # many routes: build the graph once router.route(from_edge, to_edge) ``` The graph is held in memory: fine for a city, heavy for a country. ## On a map ``` duckosm multimodal monaco.duckdb # optional: the tables Walk + drive needs duckosm route-map monaco.duckdb -m walk+drive # -> reports/monaco_route_map.html ``` One HTML page, no server: drag the two markers to set the start and the end, and pick Drive, Walk, Cycle, or Walk + drive (walk to the car, drive, walk from it; it needs `duckosm multimodal` first). A marker joins the roads within the off-road distance by a walk, and the first and last roads count only the part travelled, as in [`route_points()`](#between-two-points). The panel lists the directions ("Turn left onto …", "At the roundabout, take the 2nd exit …"), the same as [`directions()`](#turn-by-turn-directions); click one to go to it on the map. Click a road to copy its `edge_id`; the Street View button shows the street. The Roads box hides road classes, bridges, tunnels, private roads or bus lanes, as on the [drawn maps](https://khoshkhah.github.io/duckOSM/guides/draw-map/index.md). The map under the route is drawn by mapstyle; the routing is duckOSM's. Needs `pip install "duckosm[viz]"`. Everything is inside the page, so keep it to a city: above 100,000 edges the command warns. `-m driving` puts that mode in front and picks it first; `-m walk+drive` shows every mode and opens on Walk + drive, as the map above does. ## Find the edges to route between **By street name:** ``` q = "SELECT min(edge_id) FROM driving.edges WHERE name = ?" from_edge = con.execute(q, ["Boulevard du Larvotto"]).fetchone()[0] ``` **Nearest to a point** (latitude first, see [Query the database](https://khoshkhah.github.io/duckOSM/guides/query/#places)): ``` con.execute("LOAD spatial") near = """SELECT edge_id FROM driving.edges ORDER BY ST_Distance_Sphere(ST_FlipCoordinates(ST_Centroid(geometry)), ST_Point(?, ?)) LIMIT 1""" from_edge = con.execute(near, [43.7285, 7.4155]).fetchone()[0] # Fontvieille to_edge = con.execute(near, [43.7480, 7.4400]).fetchone()[0] # Larvotto route(con, from_edge, to_edge)["time_s"] # 287 s, 3.7 km by car ``` **On the map:** in the route map above, click a road to copy its `edge_id`. Each mode has its own edges: for `mode="walking"`, take the ids from `walking.edges`. ## Between two points `route_points` routes from one place to another, not from edge to edge. Each point joins the network at its nearest road point within `radius_m`; the straight walk to it costs walking time (`access_kmh`), whatever the mode, and never crosses another road (one of the mode, or any road for cars: it joins the first road in its way); and the first and last edges count only the part you travel: starting in the middle of a 300 m street costs 150 m of it. ``` from duckosm import route_points r = route_points(con, (7.4155, 43.7285), (7.4400, 43.7480), mode="driving", # (lon, lat) radius_m=200, access_kmh=4.5) # Larvotto's point is on the beach, ~100 m from a road r["time_s"], r["length_m"] # door to door: the walks to the road + the parts of edges used r["start"] # {point, road_point, edge_id, fraction, access_m, access_s} r["path"][0]["from_fraction"] # where on the first edge the trip begins (0 to 1) ``` Every road of the mode within the radius is tried, both directions of a two-way street, so a point between a street and a footpath takes whichever gives the faster trip. A point with no road within the radius raises `ValueError` ("no road within 50 m of the start"). For walk → drive → walk, `route_multimodal_points(con, a, b, radius_m=50)` does the same over the `mm` tables (below). Design: [point routing](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/point_routing.md). ## Turn-by-turn directions `directions(con, route)` turns any route (from `route`, `route_points`, `route_multimodal` or `route_multimodal_points`) into steps, the ones the route map lists: ``` from duckosm import directions for s in directions(con, r): # r from route_points above print(s["text"], round(s["length_m"])) ``` ``` Head northwest on Avenue des Castelans 23 At the roundabout, take the 1st exit onto Avenue Albert II 179 At the roundabout, take the 3rd exit onto Avenue Albert II 189 Continue onto Tunnel Rocher Palais 207 … At the roundabout, take the 4th exit onto Avenue Princesse Grace 172 Arrive at your destination 0 ``` Each step also has `type` (`depart`, `turn`, `fork`, `new name`, `roundabout`, `mode`, `arrive`), `modifier` (`left`, `slight right`, `uturn`, …), `name`, `mode`, `at` (the `(lon, lat)` where it happens) and, for a roundabout, `exit`. A step's `length_m` is the distance to the next one. The rules are route-guidance's (the OSRM model); the route map runs the same rules in the page. ## Across modes (walk → drive → walk) `duckosm multimodal` joins the walking, cycling and driving networks at the junctions they share, adding the `mm` tables to the database. `route_multimodal` then finds the fastest trip that may change mode on the way. *Experimental.* ``` duckosm multimodal monaco.duckdb # --transfer-cost 60: seconds per change of mode ``` It goes from one junction to another, so it takes OSM node ids, not edge ids: ``` from duckosm import route_multimodal near_node = """SELECT node_id FROM walking.nodes WHERE node_id > 0 ORDER BY ST_Distance_Sphere(ST_FlipCoordinates(geom), ST_Point(?, ?)) LIMIT 1""" start = con.execute(near_node, [43.7285, 7.4155]).fetchone()[0] # Fontvieille end = con.execute(near_node, [43.7480, 7.4400]).fetchone()[0] # Larvotto r = route_multimodal(con, start, end) [(leg["mode"], round(leg["time_s"])) for leg in r["legs"]] # [('walking', 7), ('driving', 265)] r["time_s"], r["transfers"] # 393 s including 2 mode changes ``` | Argument | Does | Default | | ------------------------ | ------------------------------------------------- | ----------- | | `start_mode`, `end_mode` | the mode at the start and at the end | `"walking"` | | `allowed_modes` | limit the modes, e.g. `["walking", "cycling"]` | all | | `enforce_sequence` | at most one vehicle leg, reached and left on foot | `True` | It returns `None` if the two points aren't connected. Changing mode is allowed at any shared junction, so a trip can "pick up a car" anywhere: good for comparing modes, not a real trip planner. Public transport isn't included. How it works: [Routing across modes](https://khoshkhah.github.io/duckOSM/concepts/multimodal/index.md). # Add elevation `duckosm elevation` adds the ground height to every node of a finished build, in place. Needs `pip install "duckosm[elevation]"`. ``` duckosm elevation monaco.duckdb ``` ``` source=auto → copernicus (bbox 7.41, 43.73, 7.44, 43.75) [cycling] 4151 nodes sampled (0 nodata/fill) -> ele [driving] 1719 nodes sampled (0 nodata/fill) -> ele [walking] 4116 nodes sampled (0 nodata/fill) -> ele elevation added from Copernicus GLO-30 — 9986 nodes (0 nodata/fill) across 3 mode(s) -> ele ``` It adds: - `nodes.ele`: the height at each node, in metres; - `edges.z_from`, `edges.z_to`: the height at each edge's two ends; - a row in `main.elevation_metadata` saying where the heights came from (source, resolution, vertical datum, licence). The [MATSim](https://khoshkhah.github.io/duckOSM/exports/matsim/index.md), [OpenDRIVE](https://khoshkhah.github.io/duckOSM/exports/opendrive/index.md), [GIS](https://khoshkhah.github.io/duckOSM/exports/gis/index.md) and [GMNS](https://khoshkhah.github.io/duckOSM/exports/gmns_tables/index.md) exports then include the heights (GMNS: `node.z_coord`, `link.grade`). ## Where the heights come from With no options it picks a free global elevation model for your area and streams just the part it needs: **Copernicus GLO-30** (30 m, worldwide, no account). Inside Europe it uses **EU-DTM** instead when you set an [OpenTopography](https://opentopography.org) key in `OPENTOPOGRAPHY_API_KEY`. For better accuracy, point it at your own file, e.g. a national 1 m model: ``` duckosm elevation monaco.duckdb --dem my_dtm.tif # any GDAL raster ``` | Option | Does | Default | | --------------- | --------------------------------------------------------------------------- | ---------- | | `--dem FILE` | use this elevation file (GeoTIFF, VRT, … any format GDAL reads) | | | `--source` | `auto`, `copernicus`, or `eudtm` (needs the key) | `auto` | | `-m MODE` | only this mode; repeat for several | every mode | | `--nodata-fill` | height written where the model has no value | `0` | | `--suffix dsm` | store as a second surface (`ele_dsm`, `z_from_dsm`, …) instead of replacing | | Where to get elevation data, and how the sources compare: [Elevation sources](https://khoshkhah.github.io/duckOSM/reference/elevation-sources/index.md). ## What the heights mean The height is the **surface at the node's position**, not the road itself: - A **tunnel** gets the height of the ground above it, and a **bridge** the ground below. - **Copernicus is a surface model**: it includes buildings and trees, at 30 m resolution. In a dense city, short streets between buildings can look steep. A bare-earth model (EU-DTM, or a national one) gives better road grades. So leave tunnels and bridges out when you look at gradients: ``` SELECT name, round(z_from) AS z_from, round(z_to) AS z_to, round(length_m) AS length_m, round(100 * (z_to - z_from) / length_m, 1) AS grade_pct FROM driving.edges WHERE length_m > 100 AND name IS NOT NULL AND tunnel IS NULL AND bridge IS NULL ORDER BY abs(z_to - z_from) / length_m DESC LIMIT 5; ``` | name | z_from | z_to | length_m | grade_pct | | ---------------------------- | ------ | ----- | -------- | --------- | | Boulevard de Belgique | 86.0 | 111.0 | 110.0 | 22.5 | | Boulevard de Belgique | 111.0 | 86.0 | 110.0 | -22.5 | | Boulevard du Jardin Exotique | 117.0 | 97.0 | 123.0 | -16.5 | | Boulevard du Jardin Exotique | 97.0 | 117.0 | 123.0 | 16.5 | | Avenue de La Quarantaine | 32.0 | 8.0 | 144.0 | -16.1 | With Copernicus, the tunnels of Monaco would top this list: Tunnel Albert II "falls" from 336 m to 32 m, the height of the hill above it. # Add administrative boundaries `duckosm admin` adds every administrative area of a region (country, counties, municipalities, districts, …) to a built database, as the table `main.admin_boundaries`, with each area's parent so the hierarchy can be walked. It needs [GDAL](https://khoshkhah.github.io/duckOSM/install/#tools-outside-python). ``` duckosm admin --pbf monaco-latest.osm.pbf --db monaco.duckdb ``` ``` admin_boundaries: 11 features admin_level 2: 1 admin_level 8: 1 admin_level 10: 9 ``` Use it to tag roads by district, or to cut areas by name with [`extract --name`](https://khoshkhah.github.io/duckOSM/guides/prepare-area/#several-areas-from-one-region). (To get one area's boundary as a file, [`duckosm boundary`](https://khoshkhah.github.io/duckOSM/guides/prepare-area/#make-the-boundary-file) is simpler.) Reading the borders of a whole country takes a while; `--gpkg FILE` reuses an earlier extraction. ## What `admin_level` means `admin_level` is a tag on OSM `boundary=administrative` relations: an integer (roughly 1–11) giving a feature's position in the administrative hierarchy. **Lower number = larger / higher-level unit**; `2` is the country, and each higher number is a progressively smaller subdivision. The **exact meaning of each number is country-specific** — `admin_level=4` is a *county* in Sweden but a *state* in the USA — because administrative structures differ between countries. The authoritative mapping is the per-country table in the OSM wiki: . ### Sweden Sweden's formal hierarchy is shallow — **Country → 21 Län (counties) → 290 Kommuner (municipalities)**. Below the municipality there is no general administrative subdivision; the levels below `7` are a mix of the statistical *distrikt* (created 2016, based on the old parishes/*socknar*) and city-internal *stadsdelar*. | `admin_level` | Meaning | Swedish term | Notes | | ------------- | ------------------------------ | -------------------------- | --------------------------------------------- | | 2 | Country | Land / Riket | "Sverige" | | 4 | County | Län | 21 of them | | 6 | Sami reindeer-herding district | Sameby | Special; sits inside a län, not a normal tier | | 7 | Municipality | Kommun | 290 official | | 8 | District (former parish) | Distrikt / socken | Partial OSM coverage | | 9 | District / city borough | Distrikt / Stadsdelsområde | Partial coverage | | 10 | Neighborhood / locality | Stadsdel / delområde | Partial coverage | | 11 | Minor locality | Köping / småort | Rare | Caveats: - **Levels 1, 3 and 5 are not used** in Sweden. - **Coverage below kommun (8–11) is incomplete.** Sweden has ~2,500 official *distrikt*, but only a fraction are mapped in OSM, so do not treat levels 8–11 as exhaustive. - A few boundaries carry `boundary=administrative` with **no `admin_level`** tag; these land with `admin_level = NULL`. ## Table schema `admin_boundaries` | Column | Type | Description | | --------------- | -------- | ------------------------------------------------------------------------------------------------ | | `osm_id` | BIGINT | OSM id of the boundary relation | | `name` | VARCHAR | Boundary name in the local language (may be NULL) | | `name_en` | VARCHAR | English name (OSM `name:en`); present for the country, counties and big cities, NULL otherwise | | `admin_level` | INTEGER | OSM admin level (see above); NULL if untagged | | `parent_osm_id` | BIGINT | `osm_id` of the immediate enclosing boundary (see below); NULL for the root and a few edge cases | | `geometry` | GEOMETRY | MultiPolygon, EPSG:4326 (same CRS as the edge geometries) | ## The `parent_osm_id` hierarchy OSM does not store an explicit parent link, so `parent_osm_id` is derived **spatially**: a guaranteed-interior point of each boundary (`ST_PointOnSurface`) is tested against the geometry of candidate parents (those with a lower `admin_level`), bounding-box pre-filtered for speed. The **immediate parent** is the containing boundary with the *highest* `admin_level` below the child's (smallest by area to break ties). `parent_osm_id` is NULL for: - the country root (`admin_level = 2`), and - a small number of boundaries whose interior point isn't covered by any parent — typically coastline/border artifacts (e.g. villages on the Sweden–Finland border) or unnamed fragments. On the full Sweden extract this links **1,456 of 1,472** boundaries, e.g. 21 counties → Sweden, 289 kommuner → county, districts → kommun. ## Example queries On Monaco: ``` -- how many areas at each level SELECT admin_level, count(*) AS areas FROM admin_boundaries GROUP BY 1 ORDER BY 1; -- each district with its parent SELECT c.name, c.admin_level, p.name AS parent FROM admin_boundaries c LEFT JOIN admin_boundaries p ON c.parent_osm_id = p.osm_id WHERE c.admin_level = 10 ORDER BY c.name; -- which district contains a point (lon, lat)? -> Monte-Carlo SELECT name FROM admin_boundaries WHERE admin_level = 10 AND ST_Contains(geometry, ST_Point(7.4270, 43.7397)); -- driving edges per district SELECT d.name AS district, count(*) AS edges FROM driving.edges e JOIN admin_boundaries d ON d.admin_level = 10 AND ST_Contains(d.geometry, ST_PointOnSurface(e.geometry)) GROUP BY 1 ORDER BY edges DESC; ``` | district | edges | | ------------ | ----- | | Monte-Carlo | 636 | | Fontvieille | 494 | | La Condamine | 469 | | Larvotto | 269 | | … | | The full ancestry of an area (district → municipality → country): ``` WITH RECURSIVE up AS ( SELECT osm_id, name, admin_level, parent_osm_id FROM admin_boundaries WHERE name = 'Monte-Carlo' UNION ALL SELECT b.osm_id, b.name, b.admin_level, b.parent_osm_id FROM admin_boundaries b JOIN up ON b.osm_id = up.parent_osm_id ) SELECT admin_level, name FROM up ORDER BY admin_level DESC; ``` # Check a build ## Validation during the build Switch validation on in your [config file](https://khoshkhah.github.io/duckOSM/guides/build/#with-a-config-file). The build then checks its own result and fails if something is wrong: ``` validation: enabled: true ``` It's off by default (and a build from the command line alone can't switch it on), but the [config template](https://khoshkhah.github.io/duckOSM/guides/build/#with-a-config-file) turns it on. On Monaco it prints, per mode: ``` validate[driving] single_component: OK — 1 component(s), largest 2,765/2,765 (100.0%) validate[driving] no_stranded_named: OK — 0 named edge(s) outside the largest component validate[driving] unique_node_id: OK — 0 duplicate node_id(s) in nodes validate[driving] way_length_conserved: OK — 0 deleted parallel arc(s) (way stretch missing while a kept edge bridges its endpoints) validate[driving] layer_without_structure: WARN — 11 edge(s) with layer≠0 but no bridge/tunnel tag (OSM tagging; drawn grade-separated, no 3D deck) ``` | Check | Fails the build when | | ------------------------- | ----------------------------------------------------------------------------------------- | | `single_component` | the network isn't one connected piece | | `no_stranded_named` | a named road sits outside the main network | | `unique_node_id` | two nodes share an id | | `way_length_conserved` | part of an OSM way went missing when it was split into edges | | `layer_without_structure` | never: a warning about OSM tagging (a road on another level with no bridge or tunnel tag) | The first two only run when the build has a boundary: a whole region can legitimately have separate pieces (islands, for example). Each check can be switched off; see [Configuration](https://khoshkhah.github.io/duckOSM/reference/configuration/index.md). Set `fail_on_error: false` to be warned instead of stopped. ## A report of the build With `report: {enabled: true}` in the config, the build also writes a summary to `reports/` as Markdown and HTML: counts per mode, what the road filter kept and dropped, the network pieces, and the validation results. ## Check the geometry In a clone of the repo, `scripts/validate_geometry.py` checks that every edge's geometry starts and ends at its nodes, and that there are no loops or zero-length edges (driving and walking networks): ``` python scripts/validate_geometry.py --db monaco.duckdb ``` ``` Validating Geometry for Mode: driving ✓ No self-loops found. ✓ All edge endpoints match their topological nodes. ✓ No degenerate (zero-length) geometries. ... ALL GEOMETRY CHECKS PASSED! ``` To look at one street in detail, see [Look up one OSM way](https://khoshkhah.github.io/duckOSM/guides/query/#look-up-one-osm-way). # Exports # Exports Every export reads a built database and keeps the **`edge_id` as the id in the target format** (SUMO edge id, MATSim link id, GMNS `link_id`, OpenDRIVE road id, a GIS attribute). So anything you keyed on an `edge_id` in duckOSM (counts, speeds, sensor data) joins straight onto the exported network. ## Which export for which tool | You use | Export | Command | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | SUMO | [SUMO](https://khoshkhah.github.io/duckOSM/exports/sumo/index.md) `.net.xml` | `duckosm sumo` | | MATSim, BEAM, eqasim | [MATSim](https://khoshkhah.github.io/duckOSM/exports/matsim/index.md) `network.xml`, plus lanes and signals | `duckosm matsim`, `matsim-lanes` | | DTALite, Path4GMNS, other GMNS tools | [GMNS](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md) tables (DuckDB or CSV), plus meso / micro networks | `duckosm gmns` | | QGIS, ArcGIS, FME, Visum, Aimsun importers | [GeoPackage or shapefile](https://khoshkhah.github.io/duckOSM/exports/gis/index.md) | `duckosm export-gis` | | Python analysis | [networkx](https://khoshkhah.github.io/duckOSM/exports/networkx/index.md) graph, GraphML, gpickle | `duckosm export-graph` | | CARLA, esmini, Vissim | [OpenDRIVE](https://khoshkhah.github.io/duckOSM/exports/opendrive/index.md) `.xodr` (experimental) | `duckosm opendrive` | | Autoware, the lanelet2 library | [Lanelet2](https://khoshkhah.github.io/duckOSM/exports/lanelet2/index.md) `.osm` (experimental) | `duckosm lanelet2` | | OpenTrack, RailSys, Viriato | [railML 2.4](https://khoshkhah.github.io/duckOSM/exports/railml/index.md) (experimental) | `duckosm railml` | | Your own lane-level routing | [a lane graph](https://khoshkhah.github.io/duckOSM/exports/lane-routing/index.md) (experimental) | `duckosm lane-graph`, `route-lanes` | **Experimental** exports work and are tested for structure, but haven't been loaded in the target tools yet. If you try one, an issue saying how it went helps. ## Two starting points Most exports read the database `duckosm build` writes. The lane-level ones need the lanes and turns per lane first, so they read a **GMNS database** made by `duckosm gmns`: ``` flowchart LR db[("monaco.duckdb")] --> sumo["sumo"] db --> matsim["matsim"] db --> gis["export-gis"] db --> nx["export-graph"] db --> xodr["opendrive"] db --> rail["railml"] db --> gmns[("monaco_gmns.duckdb
duckosm gmns")] gmns --> lanes["matsim-lanes"] gmns --> xodrj["opendrive --junctions"] gmns --> ll2["lanelet2"] gmns --> lg["lane-graph, route-lanes"] ``` ## Coordinates Most exports write longitude / latitude (EPSG:4326). MATSim and OpenDRIVE need metres, so they reproject to the **UTM zone of the data** (EPSG:32632 for Monaco); `--crs` picks another, e.g. `EPSG:3006` for Sweden's national grid. ## Elevation After [`duckosm elevation`](https://khoshkhah.github.io/duckOSM/guides/elevation/index.md), the heights go along: GIS layers get `ele` / `z_from` / `z_to` attributes, MATSim nodes a `z`, and OpenDRIVE roads an elevation profile. # SUMO ``` pip install "duckosm[sumo]" # brings SUMO's netconvert duckosm sumo monaco.duckdb # -> sumo/monaco.net.xml ``` duckOSM writes the network in SUMO's plain-XML format and runs SUMO's own `netconvert` to build the `.net.xml`, in metres in the UTM zone of the data. Every SUMO edge id is the duckOSM `edge_id`, and each junction allows only the turns in `edge_graph`, so turn restrictions are kept. netconvert leaves out a few U-turns it can't build (Monaco: 37 of 1,628); it never adds a turn. In a driving network each edge's lanes come from `driving.lane_profile` ([design](https://khoshkhah.github.io/duckOSM/design/gmns_lane_profile/index.md)): one SUMO lane per lane, with its vehicle classes (a bus lane `bus`, a bike lane `bicycle`, a shared one `bus bicycle`, a car lane every road vehicle) and its width, numbered from the right as SUMO does. `duckosm.sumo_compare.lane_diff(db, osm_file, work_dir)` builds the same area with SUMO's own OSM import too and lists, per OSM way and direction, every lane difference with its cause (an override, the contraflow bus lane, a default where no `lanes` tag is ...). Painted turn arrows (`turn:lanes`) are not in this network: SUMO's plain XML has no field for them. The GMNS export applies them ([GMNS tables](https://khoshkhah.github.io/duckOSM/exports/gmns_tables/index.md), `lane_check`): where a link's arrows fit its exits, read left to right by order, netconvert's OSM import connects that link's lanes by them, and those lane connections go into the plain XML. A walking network is for pedestrians only (`allow="pedestrian"`, with SUMO walking areas at the junctions), a cycling network for bicycles only. A driving network allows every vehicle class. A walking or cycling edge is one path lane, 2 m or 1.5 m wide (not the road's lane count at 3.2 m). A one-way road's lanes are centred on its line (`spreadType="center"`); the two directions of a two-way road lie either side of it. | Option | Does | | --------------------- | -------------------------------------------------------------------------------------------- | | `--no-netconvert` | only write the plain-XML inputs (`.nod.xml`, `.edg.xml`, `.con.xml`) | | `--no-connections` | let netconvert work out the turns itself | | `-c my.netccfg` | your own netconvert settings instead of duckOSM's (it still sets the input and output files) | | `-m walking` | another mode (default `driving`) | | `--out-dir`, `--name` | output folder (default `sumo`) and file name (default: the db's name) | ``` import duckdb from duckosm import to_sumo con = duckdb.connect("monaco.duckdb", read_only=True) out = to_sumo(con, "sumo/") # out["net"] = "sumo/network.net.xml" to_sumo(con, "sumo/", config={"junctions.join": "true"}) # change one netconvert option to_sumo(con, "sumo/", edge_attrs={eid: {"numLanes": 2, "width": 3.1}}) # your own lanes and widths per edge (id as number or text) ``` **Joining close junctions** (`junctions.join`, off by default). A crossroads mapped as two or three close nodes becomes one junction. netconvert reads the turns (`.con.xml`) before it joins, and silently drops every turn onto an edge the join swallows (the short edge between two joined nodes): a road whose straight or left turn ran over that edge lost it. So when joining is on, duckOSM asks netconvert which nodes it joins (a first run with `--junctions.join-output`), merges those nodes itself (`cluster_`, as netconvert names them), writes every legal path through the swallowed edges as one turn (never back through a node it passed) and runs netconvert again without joining. Turn restrictions still hold: a path is legal only if each of its steps is in `edge_graph`. # MATSim ``` duckosm matsim monaco.duckdb # -> monaco_network.xml.gz: 1,719 nodes, 2,765 links ``` A MATSim `network.xml` (the `network_v2` format), which MATSim, BEAM and eqasim read. Each duckOSM edge becomes one directed link: | Link attribute | From | | -------------- | -------------------------------------------------------------------------------------------------- | | `id` | `edge_id` | | `from`, `to` | `source`, `target` (node ids) | | `length` | `length_m` | | `freespeed` | `maxspeed_kmh` / 3.6 (m/s) | | `permlanes` | `lanes` (per direction) | | `capacity` | a per-lane value for the road class × `permlanes` (veh/h; motorway 2,000 … service 300, paths 800) | | `modes` | `car`, `bike` or `walk` | Node coordinates are in metres, in the UTM zone of the data (written in the file's attributes); `--crs` picks another. | Option | Does | Default | | -------------- | ------------------------------------------------- | ----------------------- | | `-m`, `--mode` | a mode, a comma list (`driving,cycling`) or `all` | `driving` | | `--crs` | the projected CRS for node coordinates | UTM zone of the data | | `--no-gzip` | write plain `.xml` | gzipped | | `-o`, `--out` | output file | `_network.xml.gz` | ``` from duckosm import to_matsim to_matsim("monaco.duckdb", "network.xml.gz", mode="driving") # -> {"nodes", "links", "crs"} ``` ## All modes in one network ``` duckosm matsim monaco.duckdb -m all -o network.xml.gz # 4,481 nodes, 12,789 links ``` A road used by several modes is one link with every mode on it (`modes="car,bike,walk"`), since its `edge_id` is the same in each network. MATSim routes each leg of a trip on the links that allow its mode, and the links are shared, so no transfer links are needed. ## Lanes and signals ``` duckosm gmns monaco.duckdb # the lanes and turns first duckosm matsim-lanes monaco_gmns.duckdb # -> ./lanes.xml + signalSystems / signalGroups / signalControl.xml (-o: folder) ``` Reads a [GMNS database](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md). The link ids are the same `edge_id`s, so these files go with the `network.xml` from the same build. - **`lanes.xml`**: for each link, its lanes and which links each lane leads to, so only legal turns are possible. Turning back on the same road is allowed only at a dead end. Where OSM tags `turn:lanes`, each lane gets its own turns; elsewhere every lane may take every legal turn. - **Signals**: one signal system per junction with traffic lights in OSM (Monaco: 1). **The timing is a placeholder**, a fixed 90 s cycle in two phases, since OSM has no signal plans: calibrate it. `--cycle` sets the cycle length, `--no-signals` skips the signal files. ## Limits - `network.xml` alone has no turns: MATSim lets a car take any link at a junction, banned turns included. Add `lanes.xml` for legal turns only. - `freespeed` and `capacity` come from class defaults where OSM has no speed limit or lane count. - No demand, plans, transit or `config.xml`: those are your scenario. - Checked against MATSim's official DTD and XSD schemas (in the tests). # GMNS ``` duckosm gmns monaco.duckdb # -> monaco_gmns.duckdb duckosm gmns monaco.duckdb --to-csv gmns/ # also the spec CSVs ``` [GMNS](https://zephyr-data-specs.github.io/GMNS/) (General Modeling Network Specification) is the open table format of DTALite, Path4GMNS and other network-modelling tools. duckOSM writes every GMNS table that OSM has data for into **one new DuckDB file** that stands alone: it doesn't need the source database, and its tables carry real geometry, so you can query and draw them. `--to-csv DIR` also writes the standard `node.csv`, `link.csv`, `lane.csv`, … into `DIR//`. The CSVs have only the columns the spec defines, so the geometries and each lane's turns stay in the DuckDB file. ## What's in it Every table, column by column, with where each value comes from: [GMNS tables](https://khoshkhah.github.io/duckOSM/exports/gmns_tables/index.md). One schema per mode, `gmns_driving`, `gmns_walking`, `gmns_cycling`. Monaco, driving: | Table | Rows | What | | --------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `node` | 1,719 | junctions; `ctrl_type = 'signal'` at traffic lights | | `link` | 2,765 | directed roads (in driving also the 18 bus-only directions, one `bus` lane each: [link](https://khoshkhah.github.io/duckOSM/exports/gmns_tables/#link)): **`link_id` = `edge_id`**, length, speed, lanes, capacity (a per-lane default by road class), `facility_type` (the OSM `highway`), name, and (DuckDB only) `bridge` / `tunnel` / `layer` | | `lane` | 3,182 | one row per lane, with its turns, allowed uses and width where OSM tags them | | `movement` | 3,500 | legal turns (from `edge_graph`; turning back along the same road only at a dead end, or where it's the only way into the road's other direction): type (left, thru, right, uturn, merge, diverge), code (`NBL`, `EBT`, …), the lanes it starts from and ends in (equal-length ranges read in order: from `turn:lanes` where tagged, else osm2gmns's rule, [design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_lane_movements.md)), a curved turn path | | `crossing`, `lane_crossing` | duckOSM extension | the pedestrian crossings painted on the driving lanes, and which stretch of each lane they cover (in `gmns_driving` only): from the OSM crossing ways and the crossing nodes that have no way ([design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_crossings.md)) | | `link_along` | duckOSM extension | one road a footpath runs along (a footpath often runs along several): its `link_id`, the road's `along_link_id` and network, the metres along it and the gap; the roads are ones cars can use, of the footpath's own level. The link also has `along_link_id` / `along_mode` / `along_gap_m` / `along_kind` for the road with the longest stretch ([columns](https://khoshkhah.github.io/duckOSM/exports/gmns_tables/#link_along)) | | `lane_connector` | duckOSM extension | one path per lane pair of a movement: a curve from the end of one lane to the start of the next (`from_lane_id`, `to_lane_id`, `mvmt_id`, `width`, `geom`); lanes stop where a junction starts or where they'd jump sideways, and the connector joins them; a U-turn at a road end is a half-circle as wide as its lanes; with driving and walking exported, `gmns_walking.lane_connector` also holds a footway's join to a road lane at a shared node ([design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_lane_connectors.md)) A road's pieces are placed as one run, so a lane going on through a junction or a bend is one continuous line ([design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_lane_runs.md)). | | `geometry` | 2,765 | link shapes | | `signal_controller` | 1 | where the traffic lights are (OSM has no timings) | | `curb_seg` | 2 | on-street parking, where OSM tags `parking:*` | | `location` | 1,803 | OSM points on a link: crossings, give-way and stop signs, signals, parking entrances | | `zone` | 1 | the area: every node is in it; its outline when the database was built with a boundary | | `config`, `use_definition`, `use_group` | | units (metres, km/h), CRS (EPSG:4326), modes | Tables GMNS defines but OSM has no data for (signal timing, time-of-day) are left out, as the spec intends: see [GMNS tables](https://khoshkhah.github.io/duckOSM/exports/gmns_tables/#what-is-not-written-and-why). ## Lanes from OSM tags Every link gets its lanes from its lane count; where OSM has the details, they're filled in: | OSM tag | Gives | | ------------------------------------------ | ----------------------------------------------------- | | `lanes`, `lanes:forward`, `lanes:backward` | the number of lanes each way | | `turn:lanes` | each lane's turns, and which lanes feed each movement | | `width:lanes` | lane width (empty where untagged) | | `bicycle:lanes`, `psv:lanes` | bike and bus lanes | Most ways have none of these (Monaco: `turn:lanes` on 0.4 % of roads), so most lanes are general lanes with no width. Lane details need the raw OSM tags, which a PBF build keeps; an area clipped from a bigger build (`source.type: duckdb`, `duckosm extract`) gets lane counts only. Each lane also has a centre line drawn beside the road, on the traffic side, 3.25 m apart (`--drive-side left` for left-hand traffic). A road mapped as two one-way ways, one per direction, drawn closer together than its lanes need, is placed as one road: each direction's lanes start from the line midway between the two ways, so the directions never overlap (`--no-pair-carriageways` turns this off; [design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_paired_carriageways.md)). In the walking network a footpath (`footway`, `path`, `steps`, …) is one 2 m strip for both of its directions, centred on its line (people walk both ways on it). A **sidewalk** (`footway=sidewalk`) is placed from the cross-section of the road it runs along, not from its own OSM line: its lane lies just outside the road's kerb-side lane (the road's lanes from `gmns_driving` where it is there), so it never lies on the carriageway (`--no-walk-frame` turns this off, `--walk-clearance M` leaves a gap to the kerb; only a mapped sidewalk moves, no footpath is put where OSM has none; a sidewalk that still lies on a road (its road on its far side, too far, or none found) is pushed out across the road to the nearer kerb, half its width beyond it, so a street does not lose a side; [design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_walking_frame.md)). ## Meso and micro networks For simulators that work below the road level: ``` duckosm gmns monaco.duckdb --meso --micro ``` - **Meso** (`meso_driving`): each road becomes a section, and each legal turn a connector between sections. Monaco: 2,765 sections + 3,500 connectors. - **Micro** (`micro_driving`): each lane cut into 7 m cells, with lane-change links between side-by-side cells and turn links across junctions. Monaco: 25,423 links. Both are modelled on osm2gmns' (not identical: see [GMNS tables](https://khoshkhah.github.io/duckOSM/exports/gmns_tables/#meso-and-micro-networks)); their ids are built from `edge_id` (`M` for a section, `X-` for a meso connector), so they stay the same across rebuilds and lead back to the road. Driving by default; `--meso-mode cycling` / `--micro-mode cycling` for cycling. ## All modes in one network `--combined` adds `gmns_all`: one `link` table for every mode, a road shared by several modes being one link with `allowed_uses = 'auto,bike,walk'`. Monaco: 12,789 links. ## See it ``` duckosm gmns-viz monaco_gmns.duckdb # inspect: lanes (+ meso / micro if built), hover for ids duckosm gmns-map monaco_gmns.duckdb # present: every lane at its width, on a base map ``` Each writes one HTML file. `-m` picks the mode. `gmns-map` draws with [lanestyle](https://github.com/Khoshkhah/lanestyle) (`pip install "duckosm[viz]"`). Monaco's lanes, as `gmns-map` draws them (scroll to zoom, drag to move): each lane is a strip of road with its lane lines and an arrow for its direction, and bridges and tunnels keep their order. **Click a lane**: it turns red, and the lanes its movements lead into turn green. Bus and bike lanes get their own colour. ## Options | Option | Does | Default | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | `-m`, `--mode` | modes to export; repeat for several | every mode in the db | | `-o`, `--out` | output file | `_gmns.duckdb` | | `--to-csv DIR` | also write the spec CSVs | | | `--meso`, `--micro` | also build the meso / micro networks | off | | `--meso-mode`, `--micro-mode` | their modes | `driving` | | `--combined` | also write `gmns_all` | off | | `--drive-side` | `right` or `left` traffic | `right` | | `--no-pair-carriageways` | centre each one-way way's lanes on it, even beside its opposite direction | paired | | `--walk-frame` | move a mapped sidewalk off the carriageway (placed from its road's kerb, or pushed to the nearer kerb) and join the footways that meet it | off: footways stay as OSM maps them | | `--walk-clearance` | gap in metres between the kerb and a footpath placed from its road | 0 | | `--no-lane-geometry` | skip the lane centre lines | computed | | `--gtfs FEED` | a GTFS feed (`.zip` or folder, repeatable): its stops become `location` rows with their `gtfs_stop_id` ([how](https://khoshkhah.github.io/duckOSM/exports/gmns_tables/#transit-stops-from-a-gtfs-feed)) | none | | `--gtfs-max-m` | how far from a link a GTFS stop may be and still be put on it | 30 | | `--csv-extensions` | keep duckOSM's extra columns in the CSVs, as `u_`-prefixed user-defined fields | standard columns only | | `--check` | check the values of the result (see Conformance) and exit 1 if one fails; also prints how connected each network is (information only) | off | ``` from duckosm import to_gmns, to_meso, to_micro to_gmns("monaco.duckdb", "monaco_gmns.duckdb") to_meso("monaco_gmns.duckdb") # on an existing GMNS db to_micro("monaco_gmns.duckdb", cell_length_m=7.0) ``` ## Conformance to the GMNS standard The CSV output (`--to-csv`) is checked against the spec's own schemas, release v0.97, with [frictionless](https://framework.frictionlessdata.io/): `tests/test_gmns_spec.py` runs it on every test build, using the schemas in `tests/data/gmns_spec/`. It also checks the values the spec lists as categories (`ped_facility`, `bike_facility`, movement `type`, `ctrl_type`). Run it on your own output: ``` duckosm gmns area.duckdb --to-csv gmns/ # put the spec's datapackage.json and *.schema.json (release v0.97) next to the CSVs, then frictionless validate datapackage.json ``` The spec's validators check the shape of the tables, not what the values say about each other. `duckosm gmns ... --check` does (`duckosm.check_gmns(path)` from Python): a link starts and ends on its nodes; `link.length` is its geometry's length; `lane_num` runs 1..n and `link.lanes` is the number of lane rows; a movement turns at the node where its inbound link ends, and its lane ranges exist and are as long on both sides. Run on Monaco, Södermalm, Tartu and Granville Island, in every mode: all pass. What differs from the spec, on purpose: the DuckDB file has extra columns (`geom` on `node`, `link`, `geometry`, `lane`, `location` and `zone`; `lane.turn`; `link.osm_id`, `bridge`, `tunnel`, `layer`; `location.osm_id`; `signal_controller.node_id`, `control_type`) and a `lane_connector` table; the CSV leaves them out. A U-turn has no `mvmt_code` (the spec's code has no U; `type` is `uturn`). OSM `sidewalk` and `cycleway` are mapped to the spec's categories, so which side a sidewalk is on is lost. The optional spec tables `segment*`, `*_tod` and the signal timing tables are not written. Walking and cycling links that cars cannot use have no `lanes` or `capacity` (the standard defines both for motor vehicles). Design and audit: [gmns_spec_conformance](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_spec_conformance.md). ## duckOSM and osm2gmns [osm2gmns](https://github.com/jiawlu/OSM2GMNS) is the established OSM-to-GMNS tool. Run on the same Södermalm PBF: | | duckOSM | osm2gmns 0.7 | | ----------------- | ---------------------------------------------------------- | --------------------------- | | Ids | `edge_id`, the same on every rebuild | numbered from 0 on each run | | Lanes | per-lane turns, bus and bike lanes, widths from OSM tags | lane geometry | | Turns | from the graph of legal turns (OSM turn restrictions kept) | from geometry | | Complex junctions | kept as in OSM | merged into one | | Modes | all in one file, sharing ids | one network type per run | osm2gmns goes further for simulation (junction consolidation, a mature micro network); duckOSM's network keeps stable ids and more of OSM's lane data. They work well together. # GMNS tables, one by one A reference for every table in the GMNS database that `duckosm gmns` writes: what a row is, how it is made from OSM, and what each column holds. For the command and the options, see [GMNS](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md). For the standard itself, see the [GMNS specification](https://zephyr-data-specs.github.io/GMNS/spec/) (release 0.97). Numbers are from Monaco, driving. ## How to read this page **Where the tables are.** One schema per mode: `gmns_driving`, `gmns_walking`, `gmns_cycling`. With `--combined` there is also `gmns_all`; with `--meso` / `--micro`, `meso_` and `micro_`. ``` SELECT count(*) FROM gmns_driving.link; -- ATTACH the file, or open it directly ``` **Column marks.** The *Spec* column says where a column comes from (`--csv-extensions` keeps the ➕ columns in the CSVs too, as the spec's user-defined fields, with a `u_` prefix: `u_osm_id`, `u_bridge`, `u_turn`, …; the native `geom` columns never go to a CSV): | Mark | Meaning | In the CSVs (`--to-csv`)? | | ---- | -------------------------------------------------------------------- | -------------------------------------- | | ✅ | a column of the GMNS standard, filled from OSM | yes | | ∅ | a column of the standard that OSM has no data for, so it stays empty | yes, empty | | ➕ | a duckOSM extension: not in the standard | no (the CSV has only standard columns) | **Conventions that hold in every table:** - **Ids are integers.** `link_id` is duckOSM's `edge_id`, a content hash (`BIGINT`, up to 19 digits). It is the same on every rebuild and the same in every mode, so you can join GMNS to your own data on it. Keep it `BIGINT`: as a float or a JavaScript number it loses digits. `node_id` is the OSM node id. - **Units:** lengths and widths in metres, speeds in km/h, coordinates in longitude / latitude (EPSG:4326). The `config` table says so. - **Geometry twice.** Where a table has a shape, the standard column `geometry` holds it as WKT text, and the extension column `geom` holds the same shape as a native `GEOMETRY` you can query with the DuckDB `spatial` extension (`ST_*`) and draw. Only `geometry` goes to the CSVs. - **One link per direction.** A two-way road is two links with swapped nodes and reversed geometry; a one-way road is one. Every link has `directed = true`. - **Lane numbers** run from the left in the direction of travel: lane 1 is the leftmost lane, whichever side traffic drives. (With right-hand traffic, the default, the leftmost lane is next to the road's centre line; with `--drive-side left` the lanes lie left of it and the leftmost is the farthest.) OSM's `turn:lanes` and `bicycle:lanes` list lanes in the same order, so entry *n* is lane *n*. ## The tables at a glance | Table | Standard? | One row is | Rows | Read it for | | ----------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------ | ----------- | --------------------------------------------- | | [`config`](#config) | yes | the dataset | 1 | units, CRS, spec version | | [`use_definition`](#use_definition) | yes | a way of using a lane (auto, bus, …) | 3 | what `allowed_uses` values mean | | [`use_group`](#use_group) | yes | a named group of uses | 1 | the mode's uses | | [`node`](#node) | **required** | a junction or end of a link | 1,904 | where links meet | | [`link`](#link) | **required** | one direction of a road | 3,092 | the routable network | | [`geometry`](#geometry) | yes | the shape of a link | 3,092 | link shapes, keyed by id | | [`lane`](#lane) | yes | one lane of a link | 3,520 | lane use, width, turns | | [`movement`](#movement) | yes | one legal turn at a junction | 3,957 | which link may follow which, from which lanes | | [`lane_connector`](#lane_connector) | ➕ extension | one lane-to-lane path across a junction (driving), and a footway's [join](#footway-joins) to a road lane (walking) | 1,813 + 542 | drawing lanes through junctions | | [`lane_check`](#lane_check) | ➕ extension | something the lanes could not be given: a lane with no way in or out, a turn with no lane connection | — | what to check (in Street View) | | [`crossing`](#crossing) | ➕ extension | one pedestrian crossing painted on the driving lanes | 741 | drawing zebra crossings | | [`lane_crossing`](#lane_crossing) | ➕ extension | one lane a crossing covers, and which stretch of it | 909 | painting the stripes lane by lane | | [`link_along`](#link_along) | ➕ extension | one road a footpath runs along (a footpath often runs along several) | — | matching a sidewalk to its whole street | | [`curb_seg`](#curb_seg) | yes | a stretch of kerb with a rule | 2 | on-street parking | | [`signal_controller`](#signal_controller) | yes | a traffic signal | 1 | where the signals are | | [`location`](#location) | yes | a point along a link | 1,803 | crossings, signs, parking entrances | | [`zone`](#zone) | yes | an area | 1 | the area's name and, when known, outline | | [`gmns_all`](#gmns_all-all-modes-in-one) | | all modes in one `node` + `link` | | one network for every mode | | [`meso_*`, `micro_*`](#meso-and-micro-networks) | osm2gmns layout | sections, connectors, cells | | simulators | How they point at each other: ``` erDiagram node ||--o{ link : "from_node_id / to_node_id" link ||--|| geometry : "geometry_id" link ||--o{ lane : "link_id" link ||--o{ curb_seg : "link_id" link ||--o{ location : "link_id" node ||--o{ movement : "node_id" link ||--o{ movement : "ib_link_id / ob_link_id" movement ||--o{ lane_connector : "mvmt_id" lane ||--o{ lane_connector : "from_lane_id / to_lane_id" node ||--o| signal_controller : "node_id" use_definition ||--o{ lane : "allowed_uses" ``` ## config One row that says how to read the other tables. It is what a tool needs to know first. | Column | Type | Spec | Value | | ----------------------- | ------- | ---- | ---------------------------------------------------------------- | | `dataset_name` | VARCHAR | ✅ | the output file's name plus the mode, e.g. `monaco_gmns_driving` | | `short_length` | VARCHAR | ✅ | `meter`: lane widths, linear references | | `long_length` | VARCHAR | ✅ | `meter`: link lengths | | `speed` | VARCHAR | ✅ | `kmh` | | `crs` | VARCHAR | ✅ | `EPSG:4326` | | `geometry_field_format` | VARCHAR | ✅ | `wkt` | | `currency` | VARCHAR | ∅ | empty (OSM has no toll prices) | | `version_number` | DOUBLE | ✅ | `0.97`, the GMNS release the output follows | | `id_type` | VARCHAR | ✅ | `integer` | ## use_definition What each use in `allowed_uses` means, so a tool can tell a bus lane from a car lane. | Column | Type | Spec | Holds | | ----------------------------------- | ------- | ---- | ----------------------------------------------------------------------------- | | `use` | VARCHAR | ✅ | the name: `auto`, `bus`, `bike` (driving); `walk` (walking); `bike` (cycling) | | `persons_per_vehicle` | DECIMAL | ✅ | `auto` 1, `bus` 25, `bike` 1, `walk` 1 | | `pce` | DECIMAL | ✅ | passenger-car equivalents: `auto` 1.0, `bus` 2.0, `bike` 0.2, `walk` 0 | | `special_conditions`, `description` | VARCHAR | ∅ | empty | Every use that appears in `link.allowed_uses`, `lane.allowed_uses` or `movement.allowed_uses` is defined here, as the standard requires. ## use_group A name for the uses of a mode, to keep `allowed_uses` lists short. | Column | Type | Spec | Holds | | ------------- | ------- | ---- | ----------------------------------------- | | `use_group` | VARCHAR | ✅ | the mode: `driving`, `walking`, `cycling` | | `uses` | VARCHAR | ✅ | its use: `auto`, `walk`, `bike` | | `description` | VARCHAR | ∅ | empty | ## node A point where links start or end: a junction, a dead end, or a point where a road was cut. This is one of the two required tables, with `link`. A node is made wherever two OSM ways meet or a way ends (as in the source network's `nodes`; a bus-only road's own nodes, which `nodes` lacks, are its links' end points, with no height). A road that loops back on itself is cut in two at its midpoint; that point is a node with a **negative** id, because it is not an OSM node. | Column | Type | Spec | Holds | | ------------------------ | -------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `node_id` | BIGINT | ✅ key | the OSM node id; negative for the cut points above | | `x_coord`, `y_coord` | DOUBLE | ✅ | longitude, latitude | | `ctrl_type` | VARCHAR | ✅ | `signal` where the OSM node is a `highway=traffic_signals`; else from the stop and give-way signs on its approaches: `stop_4_way` (every approach stops, four or more), `stop` (every approach, fewer), `stop_2_way` (some approaches), `yield`; empty where there is none. See [`movement`](#movement) for which signs count | | `z_coord` | DOUBLE | ✅ | metres: the node's height, when the source was given heights with [`duckosm elevation`](https://khoshkhah.github.io/duckOSM/guides/elevation/index.md); empty otherwise | | `zone_id` | BIGINT | ✅ | `1`: every node is in the [`zone`](#zone). (Tools such as Path4GMNS refuse nodes with no zone.) | | `node_type` | VARCHAR | ✅ | OSM names, as the spec's FAQ recommends: the OSM `highway` value for a `turning_circle`, `mini_roundabout` or `motorway_junction`; else `intersection` (three or more neighbouring nodes), `dead_end` (one); empty where a road is merely cut | | `name`, `parent_node_id` | | ∅ | empty | | `geom` | GEOMETRY | ➕ | the point | ## link The routable network: **one row per direction of a road**. A footpath is one link too (two, if it can be walked both ways). In driving, a direction only buses may use is a link too, of one lane (`allowed_uses` `bus`): a bus lane against a one-way street and a bus-only road, the source's `driving.private_edges` with `access = 'bus'` ([design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/bus_only_edges.md)). Their turns are the source `edge_graph`'s rows with `uses = 'bus'`, so [movements](#movement) lead into and out of them, for buses only. Monaco: 18 (Boulevard Charles III's bus lane back, `1449981121#1r`, is one). | Column | Type | Spec | Holds | | ---------------------------------------------------------- | -------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `link_id` | BIGINT | ✅ key | the `edge_id`: a hash of the OSM way and the two nodes; stable, same in every mode | | `name` | VARCHAR | ✅ | the street name (`name` tag), often empty | | `from_node_id`, `to_node_id` | BIGINT | ✅ | the two [`node`](#node)s, in the direction of travel | | `directed` | BOOLEAN | ✅ | always `true` | | `geometry_id` | BIGINT | ✅ | the same as `link_id`: the row of [`geometry`](#geometry) | | `geometry` | VARCHAR | ✅ | the shape, WKT `LINESTRING`, from `from_node` to `to_node` | | `dir_flag` | INTEGER | ✅ | always `1`: the shape runs from `from_node` to `to_node` | | `length` | DOUBLE | ✅ | metres: the length of the shape on the ellipsoid, so it always agrees with `geometry` | | `facility_type` | VARCHAR | ✅ | the OSM `highway` value as it is: `residential`, `primary_link`, `footway`, … | | `capacity` | DOUBLE | ✅ | saturation capacity, passenger cars per hour **per lane**: a default per road class (motorway 2000, trunk 1800, primary 1600, secondary 1400, tertiary 1200, unclassified 1000, residential 800, living_street 300, service 300, anything else 800). Empty on links that cars cannot use, see below | | `free_speed` | FLOAT | ✅ | km/h: the `maxspeed` tag, else a default per road class; walking 5, cycling 15 | | `lanes` | INTEGER | ✅ | the number of permanent **motor-vehicle** lanes in this direction, from the `lanes` / `lanes:forward` / `lanes:backward` tags, **1 where untagged**; 1 on a bus-only link (a one-way street's `lanes` are its own direction's). Not the number of rows in `lane`: the standard excludes turn pockets, bike lanes, shoulders and parking lanes. Empty on links that cars cannot use | | `bike_facility` | VARCHAR | ✅ | from OSM `cycleway`, in the standard's words: `unseparated bike lane`, `separated bike lane`, `shared lane`, `counter-flow bike lane`, `paved shoulder`, `none`, `other` | | `ped_facility` | VARCHAR | ✅ | from OSM `sidewalk`: `sidewalk` (either side or both: which side is lost), `offstreet_path` (`separate`), `none`, `unknown` | | `allowed_uses` | VARCHAR | ✅ | the mode's use: `auto`, `walk` or `bike`; on a bus-only link `bus`, or `bus,bike` where bikes may use it too (the way has `oneway:bicycle=no`, or `cycleway` / `cycleway:left` / `cycleway:right` = `share_busway`, or the edge is in cycling) | | `grade` | DOUBLE | ✅ | percent, negative downhill: `100 × (height at the end − height at the start) / length`, when the source has heights (`duckosm elevation`). Empty on a bridge or in a tunnel (there the height is the ground below or above, not the road) and where the slope would pass 100 %, the spec's limit. A coarse elevation model makes short links noisy: see the [elevation guide](https://khoshkhah.github.io/duckOSM/guides/elevation/#what-the-heights-mean) | | `parent_link_id` | BIGINT | ✅ | for a sidewalk (an OSM way with `footway=sidewalk`) in the walking and cycling schemas, the road link it runs along: the nearest, roughly parallel one within 30 m, and of a two-way road the link the sidewalk is on the kerb side of (its right with right-hand traffic). Empty on roads and on a sidewalk with no road beside it | | `parking`, `toll`, `jurisdiction`, `row_width` | | ∅ | empty (parking is in [`curb_seg`](#curb_seg)) | | `osm_id` | BIGINT | ➕ | the OSM way the link comes from: one way gives several links (one per piece and direction). A way duckOSM had to split carries a *virtual* negative id in its edges (`edge_ref` `-1525349445#1f`); this column is always the real way (its absolute value), and the way's tags (`highway`, `footway`, `crossing`, lane tags) are read from it | | `edge_ref` | VARCHAR | ➕ | duckOSM's readable id of the link, `#` (`503633032#2f`): `n` counts a way's pieces along it, `f` / `r` is the direction. The two directions of one two-way piece share `#` (`#2f` and `#2r`), so two links with the same way and number are twins; links of different ways (`503633032#2f`, `503633034#3f`) never are. Not a key (`link_id` is) | | `footway` | VARCHAR | ➕ | the OSM `footway` tag (`sidewalk`, `crossing`, `link`, …), empty without raw tags: a `crossing` lies on the road it crosses (a reader draws it over the road) | | `crossing`, `crossing_markings` | VARCHAR | ➕ | on a `footway=crossing` link, the OSM `crossing` (`marked`, `zebra`, `uncontrolled`, `unmarked`, `traffic_signals`, …) and `crossing:markings` (`yes`, `zebra`, `no`, …) tags; empty on every other link | | `along_link_id`, `along_mode`, `along_gap_m`, `along_kind` | BIGINT, VARCHAR, DOUBLE, VARCHAR | ➕ | on a footpath link, the road it runs along: the nearest roughly parallel road of the footpath's own level (within 30 m, on its kerb side), even where that road is only in `gmns_driving` (`link_id` is the same hash in every schema); the network it is in (`walking` / `driving`); the metres between the two lines; and `along_kind`: `sidewalk` (a `footway=sidewalk` way) or `adjacent` (any other footway, path, pedestrian or cycleway link, not a link or steps, that lies within 5 m of the road's edge: a label of closeness, nothing is moved; a `footway=crossing` link only when it runs along the road for most of its length and is either 10 m or longer (a footpath OSM tagged by mistake) or shares a node with a matched footpath that runs along the same road (a sidewalk's crossing of a side street, which also runs along the main road)). The road is one cars can use: a road only people walk on is not one to run along. Empty on every other link. `parent_link_id` is the same road for a `sidewalk` only, and only when it is a link of this table | | `bridge`, `tunnel`, `layer` | VARCHAR | ➕ | the OSM tags as they are (`yes`, `-1`, …): which links are on a bridge or in a tunnel, and the drawing order | | `geom` | GEOMETRY | ➕ | the shape | **Links that cars cannot use.** In the walking and cycling schemas, a link whose OSM `highway` is `footway`, `path`, `cycleway`, `steps`, `pedestrian`, `bridleway`, `corridor` or `platform` has `lanes` and `capacity` empty: both are motor-vehicle quantities, and the standard treats foot and bike links as uncapacitated. A road that is walked or cycled on keeps its motor `lanes` and `capacity`. ``` -- the longest bridges SELECT name, round(length) AS metres FROM gmns_driving.link WHERE bridge = 'yes' ORDER BY length DESC LIMIT 5; ``` ## geometry The shape of every link, in its own table keyed by `geometry_id`: for tools that keep shapes apart from links. It repeats `link.geometry`, row for row. | Column | Type | Spec | Holds | | ------------- | -------- | ------ | ---------------- | | `geometry_id` | BIGINT | ✅ key | the `link_id` | | `geometry` | VARCHAR | ✅ | WKT `LINESTRING` | | `geom` | GEOMETRY | ➕ | the same, native | ## lane One row per lane of a link. Without OSM lane tags the lanes are just numbered; with them, each lane knows its use, width and turns. | Column | Type | Spec | Holds | | ------------------------ | -------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `lane_id` | VARCHAR | ✅ key | `_`, e.g. `8511077704723192952_2` | | `link_id` | BIGINT | ✅ | the [`link`](#link) | | `lane_num` | BIGINT | ✅ | 1 = the leftmost lane in the direction of travel, up to the number of lanes. An on-road bike lane on the right (`cycleway:right=lane`) is the last; one on the LEFT of a one-way road (`cycleway:left=lane`) is -1, left of lane 1 (the spec numbers lanes left of the left-most through lane negative), so the motor lanes keep 1..n | | `allowed_uses` | VARCHAR | ✅ | `auto`; `bus` where `psv:lanes` / `bus:lanes` says `designated`; `bike` where `bicycle:lanes` does, or for an on-road bike lane (see below); the one lane of a bus-only link as its link's, `bus` or `bus,bike` (driving). `walk` / `bike` in the walking / cycling schemas | | `width` | DOUBLE | ✅ | metres, from `width:lanes`; else the class default (service 2.5, residential / living street / unclassified 3.0, others 3.25); a bus-only link's lane 3.25; a bike lane 1.5; **empty on a walk lane without a width** | | `r_barrier`, `l_barrier` | VARCHAR | ∅ | empty | | `turn` | VARCHAR | ➕ | the lane's `turn:lanes` value (`left`, `through;right`, …), empty where untagged | | `geom` | GEOMETRY | ➕ | the lane's centre line, offset from the link line by the widths of the lanes before it. A two-way road's lanes sit on the traffic side; a road mapped as two one-way ways is placed as one road (no overlap). It runs to the link's nodes | | `geom_cut` | GEOMETRY | ➕ | driving: the lane's line where SUMO netconvert ends it, at the edge of the junction; the `lane_connector` paths meet it ([design](https://khoshkhah.github.io/duckOSM/design/gmns_lane_profile/index.md)). `geom` is the line to the node | How many rows: the lane count, plus a lane for each extra entry of `turn:lanes` (a turn pocket), plus one for an on-road bike lane. That is why a link can have `lanes = 2` and three `lane` rows. **On-road bike lanes.** Where OSM tags `cycleway=lane` (or `cycleway:right=lane` / `:both`) on the right of the direction of travel (for the way's reverse direction: `cycleway:left` / `:both`), the link gets one more lane row, `allowed_uses = 'bike'`, the *last* lane, to the right of the motor lanes; its `width` is the `cycleway:right:width` tag, 1.5 m if untagged. The GMNS FAQ says on-road bike lanes belong in the lane table. It is not a place to turn into: movements, forks and merges use the motor lanes only, and `link.lanes` stays the motor lane count. Not done for left-hand traffic (`--drive-side left`: the lane would be on the left and renumber the motor lanes), for separated tracks (they are their own links in the cycling schema) or where `bicycle:lanes` already marks a lane. ``` -- the lanes of one street, with their uses SELECT l.name, k.link_id, k.lane_num, k.allowed_uses FROM gmns_driving.link l JOIN gmns_driving.lane k USING (link_id) WHERE l.name = 'Boulevard Charles III' ORDER BY k.link_id, k.lane_num; ``` ## movement One row per legal turn: **from this link, onto that link, at this node**. It says which links may follow which, and from which lanes. The turns are the legal ones: OSM turn restrictions are applied. Turning back onto the road's other direction (a U-turn) is kept only at a dead end, or where it is the only way into that road's other direction. | Column | Type | Spec | Holds | | ------------------------------ | ------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `mvmt_id` | VARCHAR | ✅ key | `-` | | `node_id` | BIGINT | ✅ | the junction where the turn is made: the end of the inbound link and the start of the outbound | | `ib_link_id`, `ob_link_id` | BIGINT | ✅ | the inbound and outbound [`link`](#link) | | `type` | VARCHAR | ✅ | `left`, `right`, `thru`, `uturn` (from the change of heading at the junction), `merge`, `diverge` (the road joins or splits within 45°) | | `mvmt_code` | VARCHAR | ✅ | direction of travel and turn, e.g. `NBL` = northbound, left (`NB`/`EB`/`SB`/`WB` + `L`/`T`/`R`). **Empty for a U-turn**: the standard's code has no U; `type` says it | | `start_ib_lane`, `end_ib_lane` | INTEGER | ✅ | the inbound lanes the turn starts from, `lane_num` from..to | | `start_ob_lane`, `end_ob_lane` | INTEGER | ✅ | the outbound lanes it ends in. The two ranges are the same length and match in order: the first inbound lane goes to the first outbound lane. Where the lanes of one turn do not go on side by side (one lane into two), the turn has more rows: `-2`, ...; a bike lane's own, `-b` (`allowed_uses` bike) | | `allowed_uses` | VARCHAR | ✅ | the mode's use; into or out of a bus-only link (driving), `bus`, or `bus,bike` where every bus-only link of the turn takes bikes | | `ctrl_type` | VARCHAR | ✅ | `signal` at a signalised junction; otherwise `stop` or `yield` where an OSM `highway=stop` / `give_way` sign applies to the inbound link (cars and bikes, not walking). A sign lies on a way node on the approach, so it counts when it is within 50 m of the end of the inbound link, before the junction, and its `direction` (`forward`, `backward` or `both`, along the way) matches the link's. Signs with no `direction`, or exactly on the junction node, say nothing about which approach and are left out | | `geometry` | VARCHAR | ✅ | a short curve from the end of the inbound link to the start of the outbound link, WKT | | `name`, `penalty`, `capacity` | | ∅ | empty | A bus's turn into or out of a bus-only link (the source `edge_graph`'s rows with `uses = 'bus'`, the same U-turn rule) is a movement too. Monaco: 27 such turns. Where the lane ranges come from ([design](https://khoshkhah.github.io/duckOSM/design/gmns_lane_profile/index.md)): driving and cycling, SUMO netconvert's lane connections, given duckOSM's lanes and legal turns; walking (a footpath is one strip), the lanes in order, lane 1 into lane 1. A turn netconvert gives no lanes keeps its row without lanes and is listed in [`lane_check`](#lane_check). A range is never half empty: both ends are set, or none. ``` -- from which lanes can a vehicle turn left? SELECT i.name, k.lane_num FROM gmns_driving.movement m JOIN gmns_driving.link i ON i.link_id = m.ib_link_id JOIN gmns_driving.lane k ON k.link_id = m.ib_link_id AND k.lane_num BETWEEN m.start_ib_lane AND m.end_ib_lane WHERE m.type = 'left'; ``` ## lane_connector duckOSM's own table (in `gmns_driving`; a footway has no turning path through a junction in OSM, its lane runs to its node, so `gmns_walking` has only the short [footway joins](#footway-joins)), for drawing and for lane-level routing: the **path from one lane to the next** across a junction. A movement from lanes 1-2 into lanes 1-2 has two connectors. | Column | Type | Holds | | ---------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `connector_id` | VARCHAR | `>` | | `mvmt_id` | VARCHAR | the [`movement`](#movement) it belongs to | | `from_lane_id`, `to_lane_id` | VARCHAR | the two [`lane`](#lane)s | | `width` | DOUBLE | metres (the lane width, 3.25 m by default; a U-turn is as wide as its lanes, never more than the gap between them) | | `geom` | GEOMETRY | the path through the junction: SUMO netconvert's internal lane, from the lane's `geom_cut` end to the next lane's `geom_cut` start | Which lane goes to which lane, and the path between them, come from SUMO netconvert, given duckOSM's lanes (`driving.lane_profile`) and legal turns ([design](https://khoshkhah.github.io/duckOSM/design/gmns_lane_profile/index.md)). Not in the standard, so never in the CSVs. ## lane_check duckOSM's own table, in `gmns_driving`, `gmns_cycling` and `gmns_walking`: what the lanes could not be given, listed instead of filled in by a guess. Empty when everything maps. | Column | Type | Holds | | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kind` | VARCHAR | `no way out` (no movement leaves the lane: it ends before a narrowing), `no way in`, `no lane connection` (netconvert built none for a legal turn), `lane count` (SUMO built a link with other lanes than duckOSM's), `against its arrow` (a lane that goes to an exit its `turn:lanes` arrow does not name, though SUMO was given the arrows: a link's arrows name its exits left to right, by order, not by angle; where they fit, netconvert's OSM import connects the lanes by them, `sumo._arrow_connections`), `arrows not matched` (a link whose arrows name more or fewer directions than it has exits: a turn the arrows leave out, or arrows meant for a junction further on; SUMO is not given them) | | `id` | VARCHAR | the `lane_id`, `mvmt_id` or `link_id` | | `detail` | VARCHAR | what was found | Most rows are OSM data to check (a `lanes=2` on a one-lane road leaves a lane with no way out); fix them with `osm_overrides.yaml`. ### Footway joins When both `driving` and `walking` are exported, `gmns_walking.lane_connector` holds one straight connector per footway end that meets a road at a node both links share, from the nearest road lane's end to the footway lane's end, when the two are 0.3-6 m apart (a road lane sits beside its link's line, a footway lane on it). `from_lane_id` is the road lane (the same id in `gmns_driving` and `gmns_walking`), `to_lane_id` the footway lane, `width` the footway's, `mvmt_id` the walking movement between the two links, or NULL. Nothing moves: the footway still runs to its OSM node. See the [design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_walk_joins.md). ## crossing duckOSM's own table (only in `gmns_driving`), for drawing: **one pedestrian crossing painted on the road**, from the OSM `footway=crossing` ways and from the `highway=crossing` nodes that no crossing way has. The zebra is the **best rectangle for the links it crosses** (below), then projected on each lane. Rows exist only where the crossing lies across a driving lane **of its own level** (a ground crossing over a road in a tunnel is no crossing of that road; level: the OSM `layer`, else bridge 1 / tunnel -1) (30 degrees or more from the lane, not winding); a crossing way and the nodes on it are one crossing, and two on one place are one (the longer wins). | Column | Type | Holds | | --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `crossing_id` | VARCHAR | `w` (a crossing way; `w#2` for its second piece, across a dual carriageway) or `n` (a node) | | `source` | VARCHAR | `way` (its line is the zebra's) or `node` (a crossing node with no way: the line is square across the road, through the node) | | `crossing_type` | VARCHAR | the OSM `crossing` tag (`marked`, `zebra`, `uncontrolled`, `unmarked`, `traffic_signals`, …) | | `markings` | VARCHAR | the OSM `crossing:markings` tag (`yes`, `zebra`, `no`, …) | | `painted` | BOOLEAN | are stripes painted: false for `crossing=unmarked`, `crossing:markings=no`, signals alone and no tag at all | | `width` | DOUBLE | the zebra's width along the road, metres: the OSM `width` if tagged, else estimated from the width of the road it crosses (`length`): 0.4 × that, between 2.5 and 4 m | | `length` | DOUBLE | the rectangle's size across the road, metres, from the first covered lane's outer edge to the last | | `osm_id` | BIGINT | the OSM way or node | | `geom` | GEOMETRY | the zebra's **rectangle** (a polygon): `width` along the road, `length` across it | ## lane_crossing duckOSM's own table (only in `gmns_driving`): **one row per crossing for each driving lane it covers**, so a zebra is painted lane by lane and follows a curved lane. Only `auto` and `bus` (and `bus,bike`) lanes: a bike lane is no part of a zebra. | Column | Type | Holds | | -------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `crossing_id` | VARCHAR | the [`crossing`](#crossing) | | `lane_id` | VARCHAR | the [`lane`](#lane), or the `connector_id` of a [`lane_connector`](#lane_connector) (road surface too: a zebra over the joint of two lanes is on both lanes and the connector between them) | | `link_id` | BIGINT | its [`link`](#link) (a connector's: that of the lane it leaves) | | `start_lr`, `end_lr` | DOUBLE | the stretch of this lane the rectangle covers, metres from the lane's start: where the rectangle really overlaps the lane's own shape, so a lane that is short or ends inside the zebra still has its part (`end_lr - start_lr` is up to `width`) | | `across_from`, `across_to` | DOUBLE | the extent of that overlap across the rectangle: metres from the rectangle's first side to the overlap's two sides. Stripes laid in that one frame stay continuous from lane to lane | | `to_left` | BOOLEAN | which way `across` grows in this lane: true to the lane's left (in its direction of travel), false to its right | To paint: cut the lane's geometry between `start_lr` and `end_lr`, and put in it the stripes (pitch `stripe + gap` from `across = 0`) whose centre lies between `across_from` and `across_to`, each a strip along the lane (so parallel to it) at the lateral position that `across` and `to_left` give (a lane that is not square to the rectangle's axis: the part of it). Not in the standard, so never in the CSVs. See the [design](https://github.com/Khoshkhah/duckOSM/blob/main/docs/design/gmns_crossings.md). ## link_along duckOSM's own table (filled in `gmns_walking`, where the footpaths are; empty in the other networks): **one row per footpath link for each road it runs along**. A sidewalk usually follows a street made of several edges, not one, so [`link`](#link)'s `along_link_id` (the road along the longest stretch) is only the first of them. A road is in the route when the footpath runs along it for 4 m or more (its edge within 8 m of the footpath's and parallel to it, same level). Not in the standard, so never in the CSVs. | Column | Type | Holds | | --------------- | ------- | ---------------------------------------------------------------------------------------------------------------- | | `link_id` | BIGINT | the footpath's [`link`](#link) | | `along_link_id` | BIGINT | the road's link: of `gmns_walking` or, `along_mode` `driving`, of `gmns_driving` (the same hash in every schema) | | `along_mode` | VARCHAR | the network the road is in: `walking` or `driving` | | `covered_m` | DOUBLE | about how many metres of the footpath run along this road (sampled every 2 m) | | `gap_m` | DOUBLE | the metres between the two lines at their nearest | ## curb_seg A stretch of kerb with a rule. duckOSM writes on-street parking, where OSM tags `parking:left` / `parking:right` (or `parking:lane:`). Monaco has two. | Column | Type | Spec | Holds | | -------------------- | ------- | ------ | ------------------------------------------------------------ | | `curb_seg_id` | VARCHAR | ✅ key | `_` | | `link_id` | BIGINT | ✅ | the [`link`](#link) | | `ref_node_id` | BIGINT | ✅ | the link's `from_node_id`: where distance 0 is | | `start_lr`, `end_lr` | DOUBLE | ✅ | metres from `ref_node_id`: the whole link, `0` to its length | | `regulation` | VARCHAR | ✅ | the OSM tag value, e.g. `parallel` | | `width` | INTEGER | ✅ | empty | ## signal_controller Where traffic signals are. OSM has no signal timing, so there are no phases or plans: only the controllers. | Column | Type | Spec | Holds | | --------------- | ------- | ------ | ------------------------------ | | `controller_id` | VARCHAR | ✅ key | `sig_` | | `node_id` | BIGINT | ➕ | the signalised [`node`](#node) | | `control_type` | VARCHAR | ➕ | `signal` | ## location A point **along a link**: a crossing, a stop or give-way sign, a signal. It is how the standard places street furniture, and OSM has many of these (Monaco, driving: 1,521 crossings, 108 parking entrances, 100 give-way signs, 34 signals, 20 stop signs). A row is made for each OSM node that lies on a link (within about 0.2 m of its shape) and carries one of these tags: `highway` = `crossing`, `bus_stop`, `give_way`, `stop`, `traffic_signals`, `toll_gantry`, `mini_roundabout`, `speed_camera`; `traffic_calming`; `amenity=parking_entrance`; `railway=level_crossing`. A node that lies on several links (a junction) gets a row for each. A node beside the road, on no link, is not written: that is why Monaco has no bus stops here, its stops are platform nodes next to the road. The table needs the raw OSM tags, like lane details do: an area clipped from a bigger build has none. | Column | Type | Spec | Holds | | -------------------- | -------- | ------ | ----------------------------------------------------------------------------------------------------------------------- | | `loc_id` | VARCHAR | ✅ key | `_` | | `link_id` | BIGINT | ✅ | the [`link`](#link) it lies on | | `ref_node_id` | BIGINT | ✅ | the link's `from_node_id` | | `lr` | DOUBLE | ✅ | metres from `ref_node_id` along the link's shape | | `x_coord`, `y_coord` | DOUBLE | ✅ | longitude, latitude of the OSM node | | `loc_type` | VARCHAR | ✅ | the OSM tag value: `crossing`, `bus_stop`, `give_way`, … (the standard recommends OSM names) | | `z_coord` | DOUBLE | ✅ | metres, between the link's two end heights by distance along it (empty without heights, and on a bridge or in a tunnel) | | `zone_id` | BIGINT | ✅ | `1`, the [`zone`](#zone) | | `gtfs_stop_id` | VARCHAR | ✅ | the GTFS `stop_id`, on the rows made from a GTFS feed (`--gtfs`, below); empty on the OpenStreetMap rows | | `osm_id` | BIGINT | ➕ | the OSM node id; empty on GTFS rows | | `name` | VARCHAR | ➕ | the stop's name: OSM's `name` tag, or the GTFS `stop_name` | | `geom` | GEOMETRY | ➕ | the point | ### Transit stops from a GTFS feed The GMNS README recommends GTFS for transit and the spec has `location.gtfs_stop_id` to point at it. Give the feed to `duckosm gmns`: ``` duckosm gmns monaco.duckdb --gtfs monaco_gtfs.zip # a .zip or an unzipped folder; repeat for several feeds duckosm gmns monaco.duckdb --gtfs a.zip --gtfs b.zip --gtfs-max-m 40 ``` Only `stops.txt` is needed. With `stop_times.txt`, `trips.txt` and `routes.txt` the stops are told apart by the service that calls there (`route_type`): `bus_stop`, `tram_stop`, `train_station`, `subway_station`, `ferry_terminal`, … in OpenStreetMap's words, and `entrance` for a station entrance (`location_type` 2). Stations (1) and boarding areas (4) only group stops and are left out. Each stop becomes a `location` row: - **On the nearest link of each mode within 30 m** (`--gtfs-max-m`): in the driving schema the stops a **bus** calls at (a tram or train stop has no place on a road); in the walking and cycling schemas every stop, since people walk to a tram or train stop too. Of a two-way road's two links the stop is put on the one it is on the **kerb side** of (its right with right-hand traffic). A stop farther than that from any link of the mode is not written there (the log says how many). - `lr` is the distance along the link; `x_coord`, `y_coord` are the stop's own coordinates; `gtfs_stop_id` and `name` come from the feed. - An OpenStreetMap `bus_stop` node within 25 m of a GTFS bus stop is the same stop: its row is replaced by the GTFS one, which carries the id. Example: the Monaco bus network's feed ([Compagnie des Autobus de Monaco, on transport.data.gouv.fr](https://transport.data.gouv.fr/datasets/gtfs-3), Licence Ouverte 2.0) has 97 stops; 95 of the 96 that a bus serves go onto a road link of Monaco (OpenStreetMap alone has none there: its bus stops are platform nodes beside the road). ## zone An area. duckOSM writes **one** zone, and every node (and every [`location`](#location)) is in it. Its outline is the area you built, when the database was built with a boundary (`--boundary`, an H3 cell, an admin area; Tartu has one, Monaco does not); without one the zone has a name (the output file's) and no outline. A zone is needed because GMNS tools such as [Path4GMNS](https://github.com/jiaweizhang/path4gmns) refuse a network whose nodes are in no zone. [lanestyle](https://github.com/Khoshkhah/lanestyle) draws the outline. | Column | Type | Spec | Holds | | ------------ | -------- | ------ | ----------------------------------------------------------------------------------------- | | `zone_id` | BIGINT | ✅ key | `1` | | `name` | VARCHAR | ✅ | the boundary's name if it has one, else the area (the output file's name without `_gmns`) | | `boundary` | VARCHAR | ✅ | the outline as WKT `POLYGON` / `MULTIPOLYGON`; empty without a boundary | | `super_zone` | VARCHAR | ∅ | empty | | `geom` | GEOMETRY | ➕ | the same outline, native | ## gmns_all: all modes in one `--combined` adds a schema `gmns_all` with `config`, `node`, `link`, `use_definition` and `use_group` only. A road used by several modes is **one link** (they share `link_id`), with `allowed_uses` listing them all, e.g. `auto,bike,walk`. There are no lanes or movements here: they differ by mode, so they stay in the per-mode schemas. ## Meso and micro networks `--meso` and `--micro` build networks for simulators. They are **modelled on osm2gmns' meso and micro networks** (the same idea, and many of the same column names), but they are not its files and the standard has no meso or micro tables. Their ids are built from the `link_id`, so they lead back to the road. Lengths are in metres, measured on each link's own geometry, and **every link starts and ends exactly at its nodes**. - **Meso:** each road becomes a *section*, shortened by about 6 m at both ends (at most 35 % of the road), and each legal turn a *connector*: a smooth curve from the end of the inbound section to the start of the outbound one. - **Micro:** each lane is cut into cells of about 7 m. Cells in a row are linked; side-by-side cells of adjacent lanes are linked by a *lane change*; and each lane pair of a movement is a *turn* link from the end of the inbound lane to the start of the outbound lane, along the [`lane_connector`](#lane_connector) curve where there is one. | Schema.table | Rows (Monaco) | One row is | Main columns | | ------------------------- | ------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `meso_.meso_node` | 6,184 | a section end | `node_id` (`u` start, `d` end), `x_coord`, `y_coord`, `macro_node_id`, `macro_link_id`, `geom` | | `meso_.meso_link` | 7,049 | a road **section** (`meso_type = 'normal'`) or a turn **connector** (`'movement'`) | `link_id` (`M` / `X-`), `from_node_id`, `to_node_id`, `length`, `lanes`, `capacity` (empty: it follows from the lanes), `free_speed`, `facility_type`, `macro_link_id`, `movement_id`, `mvmt_txt_id`, `start_ib_lane`, `end_ib_lane`, `geometry`, `geom` | | `micro_.micro_node` | 22,553 | the start of a cell of a lane | `node_id` (`@`), `x_coord`, `y_coord`, `lane_id`, `cell_k`, `macro_link_id`, `lane_no`, `geom` | | `micro_.micro_link` | 27,123 | a cell, a lane change, or a turn (`cell_type` = `normal`, `lane_change`, `movement`) | `link_id` (`C#`, `H…`, `X…`), `from_node_id`, `to_node_id`, `length`, `width`, `lane_no`, `macro_link_id`, `meso_link_id`, `geometry`, `geom` | **Compared with osm2gmns 0.7.6** (run on the same Monaco PBF): osm2gmns names the road class `link_type_name` and codes it as a number `link_type`, and codes `cell_type` as 1 / 2; we write the OSM `highway` in `facility_type` and text in `cell_type`. It has `macro_node_id`, `zone_id`, `activity_type`, `is_boundary` and a micro `capacity` that we do not; we have `geom`, `meso_type`, the lane ranges, `width` and `cell_k` that it does not. Its network is also coarser (Monaco: 1,280 links against our 3,092, since it merges junctions). ## What is not written, and why The standard has 25 tables. duckOSM writes 12 of them (everything above except `lane_connector`, `crossing`, `lane_crossing` and `link_along`, which are its own) and leaves out the other 13: | Tables | Why | | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `segment`, `segment_lane` (a lane added or dropped part-way along a link) | not needed: an edge is cut at every node that two ways share or where a way ends, so wherever OSM's `lanes` changes, the edge is already cut | | `link_tod`, `lane_tod`, `movement_tod`, `segment_tod`, `segment_lane_tod`, `time_set_definitions` (time-of-day changes) | OSM's `*:conditional` tags are on a fraction of one percent to 7 % of ways, and are mostly parking and loading rules, not travel lanes | | `signal_timing_plan`, `signal_timing_phase`, `signal_phase_mvmt`, `signal_coordination`, `signal_detector` | OSM has no signal timing, phases or detectors; any value would be invented | ## Checking the output The CSVs are validated against the standard's own schemas on every test run, and `duckosm gmns --check` tests that the values agree with each other (a link starts and ends on its nodes, `length` is its shape's, lane numbers run 1..n, a movement turns where its inbound link ends and uses lanes that exist). See [Conformance](https://khoshkhah.github.io/duckOSM/exports/gmns/#conformance-to-the-gmns-standard). # GeoPackage, GeoParquet and shapefile ``` duckosm export-gis monaco.duckdb # -> monaco.gpkg duckosm export-gis monaco.duckdb -f shp # -> monaco_gis/monaco_edges_driving.shp, … (one per layer) duckosm export-gis monaco.duckdb -f geoparquet # -> monaco_geoparquet/edges_driving.parquet, … (one per layer) ``` For QGIS, ArcGIS, FME, or any tool that imports GIS files (Visum, Aimsun, …). GeoPackage uses GDAL's `ogr2ogr` to put every layer in one file; without it you get one `.gpkg` per layer. ## GeoParquet GeoParquet is a column file format for geodata. Use it for web and cloud tools. GeoLibre opens a file from a URL (`?data=`). RustGIS also opens it. DuckDB, GeoPandas and GDAL read it too. - `out` is a folder. The command writes one file per layer: `edges_.parquet`, `nodes_.parquet` and `boundary.parquet`. The layers are the same as in the GeoPackage. - DuckDB writes the GeoParquet `geo` metadata. Coordinates are longitude / latitude (EPSG:4326). - Column names stay as they are (no shapefile renames). `edge_id` stays a 64-bit integer. As in the GeoPackage, list columns such as `refs` are not written. - If the database has the table `visualization.edge_levels`, each `edges_.parquet` file also gets all its columns except `edge_id`. The join uses `edge_id`. An edge with no row there gets NULL values. The columns are `casing_start`, `casing_level`, `casing_end` and `fill_level`, and, if the table has them, `head_start_m`, `head_end_m`, `cap_start` and `cap_end`. The `nodes` and `boundary` files do not get them. duckOSM only reads that table. Without it, the command writes the edges as they are and logs one line. - Each `features.*` table (the base map) gets its own file, `features_.parquet`, with all its columns. mapstyle's look splits a file into its layers by `kind`. - The folder also gets `layers.json`, the list of the files: `file`, `kind` (`edges`, `nodes`, `boundary` or `features`), `mode` (edges and nodes) or `table` (features), and `count` for each. The roadstyle GeoLibre plugin reads it to open the whole folder in one step (roadstyle › Open a duckOSM folder). ## Layers One pair per mode, plus the area boundary. Monaco: | Layer | Features | | -------------------------------- | ------------- | | `edges_driving`, `nodes_driving` | 2,765, 1,719 | | `edges_walking`, `nodes_walking` | 10,952, 4,116 | | `edges_cycling`, `nodes_cycling` | 10,274, 4,151 | | `boundary` | 1 | Every column of the `edges` and `nodes` tables comes along as an attribute, except lists such as `refs`: `edge_id`, `source`, `target`, `osm_id`, `highway`, `name`, `oneway`, `lanes`, `length_m`, `maxspeed_kmh`, `cost_s`, … ([what each means](https://khoshkhah.github.io/duckOSM/reference/database/index.md)); with [elevation](https://khoshkhah.github.io/duckOSM/guides/elevation/index.md), `ele`, `z_from`, `z_to` too. `length_m` and `cost_s` are rounded to 2 decimals. Coordinates are longitude / latitude (EPSG:4326). Only the map layers are exported: the graph of legal turns and the turn restrictions are not GIS data. For those, use [networkx](https://khoshkhah.github.io/duckOSM/exports/networkx/index.md) or [SUMO](https://khoshkhah.github.io/duckOSM/exports/sumo/index.md). ## Shapefile limits Use a shapefile only for a tool that needs one: - **Ids are written as text.** Every 64-bit column (`edge_id`, `node_id`, `osm_id`, `source`, `target`, the H3 cells) is too big for a shapefile number; read them back as strings. GeoPackage keeps them as 64-bit integers. - Field names can have 10 characters, so `maxspeed_kmh` is renamed `spd_kmh`, `is_reverse` `is_rev`, `admin_level` `adm_level`. - One file set per layer; 2 GB per file. ## Options | Option | Does | Default | | ---------------- | ----------------------------------------------------- | ---------------------------------------------------- | | `-m`, `--mode` | modes to export; repeat for several | every mode in the db | | `-f`, `--format` | `gpkg`, `geoparquet` or `shp` | `gpkg` | | `-o`, `--out` | `.gpkg` file, or folder for shapefiles and GeoParquet | `.gpkg` / `_gis/` / `_geoparquet/` | | `--name` | the file name | the db's name | | `--no-boundary` | leave out the boundary layer | included | ``` import duckdb from duckosm import to_gis con = duckdb.connect("monaco.duckdb", read_only=True) to_gis(con, "monaco.gpkg") # -> {"layers": {name: count}, ...} to_gis(con, "monaco_geoparquet", fmt="geoparquet") # a folder of .parquet files ``` ## Check an export ``` duckosm gis-debug monaco.gpkg --source-db monaco.duckdb # -> monaco_gis_debug.html ``` Reads the file back through GDAL, as a GIS tool would, and writes a page with a map of every layer and a check: feature counts, CRS and, with `--source-db`, whether every `edge_id` in the file matches the database exactly. It ends with PASS, CHECK or FAIL. Needs `geopandas`. # networkx Load the network into [networkx](https://networkx.org) for your own analysis, or save it to a file. Needs `pip install "duckosm[routing]"`. | Function | Nodes | Edges | | ------------------------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `to_networkx(con)` | edges (`edge_id`), with `name`, `highway`, `length_m`, `maxspeed_kmh`, `cost_s`, `geometry` (WKT) | legal turns; `weight` is the time of the edge you leave (`weight="length"`: its metres) | | `to_networkx_nodes(con)` | junctions (`x`, `y` = lon, lat) | roads, keyed by `edge_id`, with every column (osmnx layout, but the length is `length_m`, not `length`) | Both take `mode=` too. To save either one to a file (the format comes from the extension): ``` duckosm export-graph monaco.duckdb # -> monaco_driving.graphml (junctions) duckosm export-graph monaco.duckdb -g edge -o routing.gpickle # the routing graph ``` GraphML opens in any tool but stores lists and geometry as text; gpickle keeps lists and types but only loads in Python (add `--geometry shapely` for shapely geometries on the junction graph). From Python: `write_graph(con, "monaco.graphml")`. ## Turns of the restricted edges The edge graph has only the mode's own turns. To add the turns into and out of the restricted edges (`private_edges`: `bus`, `private`, `pedestrian`, `delivery`), give `uses`: ``` G = to_networkx(con, "driving", uses="pedestrian,private") # CLI: duckosm export-graph --graph edge --uses pedestrian,private G.nodes[edge_id]["access"] # 'pedestrian' on a restricted edge; absent on a usable edge ``` A restricted edge is a node with its `cost_s` as the weight of its arcs. Do not route cars over them: use this to look at what meets at a junction. [Database reference](https://khoshkhah.github.io/duckOSM/reference/database/#edge_graph). # OpenDRIVE (experimental) ``` duckosm opendrive monaco.duckdb # -> monaco.xodr: 2,765 roads duckosm gmns monaco.duckdb # for junctions: a GMNS db first duckosm opendrive monaco_gmns.duckdb --junctions # -> monaco_gmns.xodr: + 1,744 connecting roads, 482 junctions ``` An ASAM OpenDRIVE 1.7 `.xodr`, for driving simulators (CARLA, esmini) and microsimulators (PTV Vissim, Aimsun). Each directed edge becomes one road (`road id` = `edge_id`): a centre line in metres (the UTM zone of the data, or `--crs`) and its driving lanes, each 3.25 m wide (OSM's `width:lanes` isn't used yet). With [elevation](https://khoshkhah.github.io/duckOSM/guides/elevation/index.md), roads get an elevation profile (not yet with `--junctions`). Without `--junctions`, roads are not linked to each other at all: good to look at, not to drive. With `--junctions` (reading a [GMNS database](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md)), each legal turn at an intersection becomes a connecting road along a smooth curve, grouped into junctions, so vehicles can drive through; where a road simply continues, the two roads link directly. **Limits:** checked for structure only, not against the ASAM schema, and not yet loaded in CARLA or esmini. A connecting road has one lane, linked to the rightmost lane of the roads it joins. It starts and ends near the road ends, usually about 2 m off but sometimes tens of metres (Monaco: 220 of 1,744 more than 5 m), which a strict simulator may report. No road type or speed limit, road markings, signals or banking. ``` from duckosm import to_opendrive to_opendrive("monaco.duckdb", "monaco.xodr") # -> {"roads", "crs"} to_opendrive("monaco_gmns.duckdb", "monaco.xodr", junctions=True) ``` # Lanelet2 (experimental) ``` duckosm gmns monaco.duckdb # the lanes first duckosm lanelet2 monaco_gmns.duckdb # -> monaco_gmns.lanelet2.osm: 3,182 lanelets ``` A [Lanelet2](https://github.com/fzi-forschungszentrum-informatik/Lanelet2) map, the lane-map format of Autoware and the `lanelet2` library. It is OSM XML, so it also opens in OSM tools. Each lane of a [GMNS database](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md) becomes a lanelet: a left and a right border (the lane's centre line ± half its width), tagged `subtype` (`road`, `bus_lane`, `bicycle_lane`), `location=urban`, `one_way=yes`, `speed_limit` (km/h) and `duckosm:edge_id`. It needs the lane centre lines that `duckosm gmns` draws by default (not with `--no-lane-geometry`). **It is a starting map, not a survey-grade HD map.** The borders come from OSM centre lines and a 3.25 m default width, accurate to about a metre, not centimetres. Lanelets touch where roads meet but mostly don't share both end points (Monaco: 4 % of the legal turns do), so the `lanelet2` router can't route across them yet. Not yet: routing between lanelets, lane-change neighbours, traffic lights and stop lines. Checked for structure only; not yet loaded in Autoware. ``` from duckosm import to_lanelet2 to_lanelet2("monaco_gmns.duckdb", "monaco.lanelet2.osm") # -> {"lanelets", "ways", "nodes"} ``` # railML (experimental) ``` duckosm railml monaco.duckdb # -> monaco.railml.xml: 13 tracks, 6 switches, 1 station ``` A railML 2.4 infrastructure file, for OpenTrack, RailSys and Viriato. duckOSM has no rail network, so this command reads the railways from the raw OSM data kept in the database (a build from a `.osm.pbf`; an area clipped from a bigger build has none) (`rail`, `light_rail`, `subway`, `tram`, `narrow_gauge`, `funicular`) and builds: - **tracks**, split at switches and where tracks meet, with their length; - **switches** (`railway=switch`, or where three or more tracks meet) and **signals** (`railway=signal`) at their position on the track; - **stations** (`railway=station` / `halt`), and **buffer stops** (`railway=buffer_stop` at a track end; other dead ends are open ends). **Limits:** infrastructure only (no timetable or rolling stock). OSM rarely tags electrification, gauge, speed or gradient, so they are left out. At a switch only two of the tracks are joined: the third branch is left as an open end, and the switch has no connections yet. Tracks have no geometry (only stations have coordinates), and a signal's direction isn't read. Checked for structure only (the railML schema needs registration), not yet loaded in a rail tool. ``` from duckosm import to_railml to_railml("monaco.duckdb", "monaco.railml.xml") # -> {"tracks", "switches", "signals", "ocps"} ``` # Lane-level routing (experimental) A route that says **which lane** to be in, not only which road. It runs on a [GMNS database](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md): ``` duckosm gmns monaco.duckdb duckosm lane-graph monaco_gmns.duckdb # 3,182 lanes, 3,890 turn + 834 lane-change links duckosm route-lanes monaco_gmns.duckdb -o route.geojson ``` ``` route: 115 lanes, cost 3876, maneuvers: right turn, right turn, merge, fork, lane change, … (25 in all), right turn, U-turn ``` `` and `` are lane ids (`_`, e.g. `4737533508550016661_1`), or `edge_id`s (then their lane 1). An unknown id gives "no lane route", as an unreachable lane does. The lane graph links each lane to the lanes it can turn into (legal turns only) and to the lanes beside it (a lane change). The cost is metres, plus a penalty for each turn and lane change. `lane-graph` saves the graph in the GMNS database (`lane_driving.lane_edges`) so that repeated routes are faster; `route-lanes` also works without it. Needs `pip install "duckosm[routing]"`. ``` from duckosm import build_lane_graph, route_lanes build_lane_graph("monaco_gmns.duckdb") r = route_lanes("monaco_gmns.duckdb", from_lane, to_lane) r["lanes"], r["cost"], r["maneuvers"], r["geometry"] ``` **Limits:** where OSM has no `turn:lanes` (most roads), any lane may take any legal turn, so a route can turn from a lane a real driver would leave first. Good for planning, not for lane-accurate control. Driving by default; `-m cycling` runs too, but cycling roads mostly have one lane. # Concepts # How a build works A build (`duckosm build`, or `DuckOSM(config).run()` in Python) runs a list of steps chosen from the config. There are two ways in: **build from a PBF**, or **clip from a parent build**. Each step is SQL that runs inside the output DuckDB file. ## Build from a PBF Dashed boxes only run under a condition (see the table below). ``` flowchart TB PBF[(".osm.pbf")] subgraph ONCE["① Once per build"] direction TB PRE["Pre-clip
osmium extract to the boundary"] LOAD["Parse OSM
ST_READOSM → raw.nodes · raw.ways · raw.relations"] GJ["Global junctions
one set of split points shared by every mode"] PRE --> LOAD --> GJ end subgraph MODE["② Per mode: driving · walking · cycling, each in its own schema"] direction TB NET["Network
filter roads · OSM fixes · build edges · simplify and merge"] FIX["Repair
connect dangling paths · mark dismount edges"] ATTR["Attributes
speeds · travel-time costs · walk / cycle type · private and bus-only roads apart"] ROUTE["Routing
turn restrictions · edge graph of legal turns"] CHECK["Clean and check
drop disconnected pieces · H3 · indexes · validate"] NET --> FIX --> ATTR --> ROUTE --> CHECK end subgraph AFTER["③ Once, after all modes"] direction TB MM["Intermodal graph mm.*"] FEAT["Base-map layers features.*"] FIN["Finish
edge_id_hash macros · metadata · report · map"] end DB[("network.duckdb")] PBF --> ONCE ONCE --> MODE MODE --> AFTER AFTER --> DB classDef optional stroke-dasharray: 5 4 classDef file stroke:#ffd43b,stroke-width:3px class PRE,FIX,MM optional class PBF,DB file ``` ## Clip from a parent build `source.type: duckdb` in a config (or `duckosm build --source-db`) reads no OSM. It copies every edge that touches the boundary, whole (`clip.predicate`, default `intersects`), out of an existing build, so every `edge_id` is the parent's. How to use it: [Several areas from one region](https://khoshkhah.github.io/duckOSM/guides/prepare-area/#several-areas-from-one-region). ``` flowchart LR PARENT[("parent build
e.g. sweden.duckdb")] BOUND["boundary
GeoJSON"] CLIP["DuckdbClipper
per mode: copy edges, nodes,
edge graph, restrictions"] COMP["Component filter
drop boundary stubs"] DONE["indexes · validate
metadata · report"] DB[("area.duckdb")] PARENT --> CLIP BOUND --> CLIP CLIP --> COMP --> DONE --> DB classDef optional stroke-dasharray: 5 4 classDef file stroke:#ffd43b,stroke-width:3px class COMP optional class PARENT,DB file ``` A clip build needs a boundary. It copies `edges`, `private_edges`, `nodes`, `edge_graph` and `turn_restrictions` only: no `raw.*`, `ways`, `edge_id_map`, `global_junctions` or `features.*`. ## Every step, in order Config keys are shown with their default. | Stage | Step | Runs when | | ----- | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ① | Pre-clip the PBF (`osmium extract`) | a boundary is set and `osmium` is installed (else the whole PBF is read, with a warning); the result is cached in `pbf/` in the current folder, and cut again when the boundary or the PBF changes | | ① | Parse OSM into `raw.*` (`ST_READOSM`) | always | | ① | Load the boundary into `main.boundary` | a boundary is set; `boundary.buffer_m` (0) grows it | | ① | Boundary cells (`main.boundary_cells`) | a boundary is set and `options.boundary_cells` (off) | | ① | Global junctions (`main.global_junctions`) | `options.global_junctions` (on) | | ② | Filter roads | always; the rules per mode: [What each network contains](https://khoshkhah.github.io/duckOSM/concepts/networks/index.md) | | ② | OSM fixes | a fixes file is named: `osm_overrides:` in the config, or `build --fixes`. There is no default file. See [Fix OSM errors](https://khoshkhah.github.io/duckOSM/guides/fix-osm-errors/index.md) | | ② | Build edges | always | | ② | Simplify and merge | always (cut ways into edges where they meet); merging: `options.merge_segments` (on) | | ② | Connect dangling paths | walking and cycling; `clip.connectivity_rescue` (on) | | ② | Mark dismount edges | cycling; `options.cycling_dismount` (on) | | ② | Speeds | `options.process_speeds` (on) | | ② | Travel-time costs | `options.calculate_costs` (on) | | ② | Walk / cycle type | walking and cycling; `options.functional_types` (on) | | ② | Move private roads (and, in driving, bus-only roads and bus lanes) to `private_edges` | always; see [Access](https://khoshkhah.github.io/duckOSM/concepts/networks/#access-private-and-forbidden-roads) | | ② | Turn restrictions | driving only; `options.extract_restrictions` (on) | | ② | Edge graph | `options.build_graph` (on) | | ② | Component filter | a boundary is set, `build_graph` is on, and `clip.keep_largest_component` (on) or `clip.min_component_edges` > 1 | | ② | H3 cells | `options.h3_indexing` (on) | | ② | Indexes | always | | ② | Validate | `validation.enabled` (off; on in the config template). See [Check a build](https://khoshkhah.github.io/duckOSM/guides/check-build/index.md) | | ③ | Intermodal graph `mm.*` | `multimodal.enabled` (off); needs walking plus another mode. See [Routing across modes](https://khoshkhah.github.io/duckOSM/concepts/multimodal/index.md) | | ③ | Base-map layers `features.*` | a build from a PBF and `options.build_features` (on); `build --no-features` turns it off | | ③ | Bus routes `bus.*` | a build from a PBF with driving and `options.bus_routes` (on); it stops the build if it fails. See [Bus routes](https://khoshkhah.github.io/duckOSM/concepts/networks/#bus-routes) | | ③ | `edge_id_hash` macros | always | | ③ | `main.visualization_metadata` and the time zone | always; a build fails if the area's time zone can't be found | | ③ | Report / map | `report.enabled` / `viz.enabled` (both off; the config template turns the report on) | The steps in ③ never stop a build: if one fails, the build logs a warning and goes on (the time zone and the bus routes are the exceptions). Each mode runs in its own schema (`driving`, `walking`, `cycling`), so the same tables exist once per mode. The steps in ② are explained in [Stable edge ids](https://khoshkhah.github.io/duckOSM/concepts/edge-ids/index.md) (build, simplify, merge), [What each network contains](https://khoshkhah.github.io/duckOSM/concepts/networks/index.md) (filter, speeds, costs, types, restrictions) and [Network clean-up](https://khoshkhah.github.io/duckOSM/concepts/cleanup/index.md) (connect, component filter, validate). ## One mode, step by step Monaco, driving (`duckosm build -c config/sample_monaco.yaml`, from the build log): | After | Result | | ------------------ | ------------------------------------------------------------------------------------------------- | | Filter roads | 1,126 OSM ways | | Build edges | 1,588 edges (one per way, plus the reverse of two-way ways; a closed way waits for the next step) | | Simplify and merge | 3,040 edges: ways split at junctions | | Private roads | 214 edges moved to `private_edges` (18 of them for buses only) | | Turn restrictions | 38 restrictions mapped to edges | | Component filter | 2,765 edges: 79 edges in 15 small pieces dropped | ## What stays in the database Besides the routing tables (`edges`, `nodes`, `edge_graph`, `turn_restrictions`), a PBF build keeps the parsed OSM data (`raw.*`), each mode's `private_edges`, filtered `ways` and `way_nodes` and `edge_id_map`, and `main.global_junctions`, `main.boundary` and the base-map layers `features.*`. Virtual nodes are in `nodes` (`node_id < 0`). Every table and column: [Database schema](https://khoshkhah.github.io/duckOSM/reference/database/index.md). # Stable edge ids ``` edge_id = (hash(osm_id, source, target) >> 1)::BIGINT -- DuckDB's hash() ``` - **`osm_id`**: the OSM way the edge comes from. - **`source`, `target`**: the OSM node ids where the edge starts and ends (or a [virtual node](#one-id-per-edge)). Their order is the direction, so the two directions of a two-way road have different ids. `edge_id` is not a row number. All three inputs come from OpenStreetMap ids, so an unchanged edge gets the same `edge_id` on every rebuild, in every area cut from the same build, and in every export. An id changes only when its edge changes: a new way id, new end nodes, a new junction that splits it, or, for a [merged edge](#merged-edges), a change in the pieces it is made of. The `is_reverse` column says whether an edge runs against the way's drawing direction (a way tagged `oneway=-1` is turned round first, so its one edge is not a reverse). For a merged two-way edge the forward direction is chosen by node id, not by the drawing. It is not part of the id. ## One id per edge `(osm_id, source, target)` must be unique. Where it would repeat (a way that ends where it starts, a way that runs between the same two junctions twice, or goes from A to B and back), the build splits the edge at its midpoint with a **virtual node**. A virtual node has `node_id < 0` and its id is a hash of the edge's content, so it is as stable as the rest. If a repeated triple still survives, the build stops with an error. Monaco: 1 virtual node in driving, 19 in walking, 21 in cycling. ## The same id in every mode Every mode cuts ways at one shared set of points, `main.global_junctions`: each node of a road that another highway way also uses (another road, or a footway, path or cycleway, such as a crosswalk), and each node where a road ends. It is made once from all highway ways in `raw.ways`, before any mode filters them. A road is cut at these points, and so is a footway or path that passes through one, even in a mode that doesn't have the road. So a way that is in two modes is cut into the same pieces and has the same `edge_id` in both. Monaco: 2,754 of the 2,765 driving edges have the same `edge_id` in cycling, 2,275 in walking. The others are roads that mode doesn't keep. So a crosswalk cuts the road it crosses in driving too, and the road cuts the crosswalk in walking too (where a road whose sidewalk is a separate way isn't in the walking network): walkers and cyclists connect to the road there, and the pieces are the same in every mode. Ways that don't exist on the ground (`highway=proposed`, `construction`, …) cut nothing. `options.global_junctions: false` makes each mode split at its own junctions, and the ids of a road can then differ between modes. Because the mode is not in the id, the same `edge_id` can be in several schemas. Across modes, the key is `(mode, edge_id)`. ## Merged edges With `options.merge_segments` (on), a chain of edges that is one road is joined into one edge: the edges must agree on `highway`, `name`, `oneway`, `maxspeed`, `lanes`, `surface`, `access`, `junction`, `layer`, `bridge`, `tunnel` and `service`. Both directions take the `osm_id` of the piece at the forward edge's `source` end (`seq` 1). A point in `main.global_junctions` is never merged away (with `options.global_junctions` on). So roads are not joined across OSM ways; only chains of footways, paths and the like are. Monaco: | Mode | Edges with merging | Edges with `merge_segments: false` | | ------- | ------------------ | ---------------------------------- | | driving | 2,765 | 2,765 | | walking | 10,952 | 11,374 | | cycling | 10,274 | 10,828 | ### `edge_id_map` Each mode with merging has an `edge_id_map` table: one row per piece of each merged edge. The reverse of a two-way road is a merged edge of its own, with its own rows. A piece and the merged edge it maps to always run the same way. | Column | Meaning | | ------------- | ---------------------------------------------------------------- | | `old_edge_id` | the piece's `edge_id` in a build without merging | | `new_edge_id` | the merged edge it is part of | | `seq` | the piece's position, 1 at the forward edge's `source` end | | `is_reverse` | whether the piece (`old_edge_id`) runs against its way's drawing | | `osm_id` | the piece's own OSM way | An edge that is not in the table was not merged. Rows of edges removed by the [component filter](https://khoshkhah.github.io/duckOSM/concepts/cleanup/#component-filter) are removed too. Monaco, walking: 796 rows for 372 merged edges. The OSM ways of one merged edge, in order: ``` SELECT seq, osm_id FROM walking.edge_id_map WHERE new_edge_id = ? ORDER BY seq; ``` ## Connector edges The edges that join a dangling path to the network have a negative `osm_id`: `-min(a, b)`, where `a` and `b` are the `osm_id`s of the path and of the road it joins. Their `edge_id` uses the same formula. Select them with `osm_id < 0`. See [Network clean-up](https://khoshkhah.github.io/duckOSM/concepts/cleanup/#connect-dangling-paths). ## Clips and exports A clip (`source.type: duckdb`, `build --source-db`, `duckosm extract`) copies edges as they are, so every `edge_id` is the parent's. A clip has no `edge_id_map`. The road [exports](https://khoshkhah.github.io/duckOSM/exports/index.md) keep `edge_id`: as the id (SUMO, MATSim, GMNS `link_id`, OpenDRIVE road id) or as an attribute or tag (GeoPackage, Lanelet2 `duckosm:edge_id`). railML (rail) has none. ## Compute or match an id Every database has two macros: | Macro | Formula | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------- | | `edge_id_hash(osm_id, source, target)` | the formula above | | `edge_id_hash_v1(osm_id, source, target, is_reverse)` | the older formula, which also hashed `is_reverse`: to map ids from a database built with it | ``` SELECT count(*) FROM driving.edges WHERE edge_id <> edge_id_hash(osm_id, source, target); -- 0 ``` In Python: `from duckosm import edge_id_hash, edge_id_expr` (one id, or the SQL expression). `hash()` is internal to DuckDB and may change in a new major version, so duckOSM requires `duckdb<2`. Outside DuckDB the id can't be recomputed. To give your own data the ids, join on the three columns and read `edge_id`: ``` SELECT s.*, e.edge_id FROM my_segments s JOIN driving.edges e USING (osm_id, source, target); ``` # What each network contains Each mode is its own schema with its own `edges`, `nodes` and `edge_graph`. Monaco: | Mode | Edges | Nodes | Speed | | --------- | ------ | ----- | -------------------------------------------- | | `driving` | 2,765 | 1,719 | the speed limit, or a default per road class | | `walking` | 10,952 | 4,116 | 5 km/h | | `cycling` | 10,274 | 4,151 | 15 km/h; 5 km/h on dismount edges | ## Which OSM ways each mode keeps | Mode | Keeps | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `driving` | `highway` = `motorway`, `trunk`, `primary`, `secondary`, `tertiary` (and their `_link`s), `unclassified`, `residential`, `service`, `living_street`, `road`. Also `highway=pedestrian` when `motorcar` or `motor_vehicle` is `yes` / `designated` / `permissive`, or `access` is `delivery` / `destination`: these become `living_street` | | `walking` | every way people may walk on: `highway` = `footway`, `path`, `pedestrian`, `steps`, `living_street`, `residential`, `service`, `platform`, `corridor`, `track`, `bridleway`, `road`, `unclassified`, `tertiary`, `secondary`, `primary`, `trunk` (and their `_link`s). Not a road tagged `motorroad=yes`, `foot=use_sidepath`, or `sidewalk` (or `sidewalk:both`, or both `sidewalk:left` and `sidewalk:right`) = `separate`: people walk on its separately mapped sidewalk. Any way tagged `foot` = `yes` / `designated` / `permissive` or `sidewalk` = `yes` / `both` / `left` / `right` is kept. Never `motorway` or `motorway_link`. [Walking](https://khoshkhah.github.io/duckOSM/guides/walking/index.md) | | `cycling` | `highway` = `cycleway`, `path`, `track`, `bridleway`, `living_street`, `residential`, `service`, `unclassified`, `tertiary`, `secondary`, `primary` (and their `_link`s); any way tagged `bicycle` = `yes` / `designated` / `permissive`, or with a `cycleway` tag other than `no` / `none` / `separate`. Plus [dismount edges](#dismount-edges) | Then the [access tags](#access-private-and-forbidden-roads) decide, per mode. Every mode needs a `highway` tag: a way with only `foot`, `bicycle` or `sidewalk` tags, or with only `area:highway` (the outline of a pedestrian area), is not a road and is in no network. The one exception is the ferry line (`route=ferry`, with `foot=yes` for walking or `bicycle=yes` for cycling): in Monaco it comes into walking, where it is walked at 5 km/h like a footpath, and its `highway` is `ferry` (its `walk_type` is `footpath`, its `cycle_type` `mixed_traffic`). `highway=service` (driveways, parking aisles, alleys) is part of the network like any other road. ## Access: private and forbidden roads For each mode, the most specific access tag decides: | Mode | Tags, most specific first | | --------- | --------------------------------------------------------------------------------------- | | `driving` | `motorcar`, `motor_vehicle`, `vehicle`, `access` | | `walking` | `foot`, `access` | | `cycling` | `bicycle`, `vehicle`, `access`; on a [dismount edge](#dismount-edges): `foot`, `access` | | Its value | The road in that mode | | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `no` (driving also `agricultural`, `forestry`, `emergency`) | **not there.** It stays in the other modes that may use it: a road tagged `motor_vehicle=no` is still walkable | | `private` | **visible, never routable**: in `.private_edges`, not in `edges` | | driving only: for buses only (below) | **visible, never routable**, like a private road: in `driving.private_edges` with `access = 'bus'` | | anything else (`yes`, `destination`, `delivery`, …) or none | a normal road | So `access=private` + `motor_vehicle=yes` is a normal road for driving, and `motor_vehicle=no` doesn't affect walking or cycling. The mode's value is kept in `edges.access`. **Private roads** (a driveway, a gated street, a company road) are built like every other road, with the same `edge_id` formula, then moved from `edges` to `private_edges`, which has the same columns except the H3 ones. The component filter doesn't prune `private_edges`. So routing, the graph of legal turns and every export use only roads you may use, and a road you can reach only through a private one is dropped with the [component filter](https://khoshkhah.github.io/duckOSM/concepts/cleanup/#component-filter). The maps still draw private roads, marked ([Draw a map](https://khoshkhah.github.io/duckOSM/guides/draw-map/index.md)). **Bus lanes and bus-only roads** are cars' private roads too: in driving they are built, then moved to `private_edges` with `access = 'bus'`. A bus-only road is one with `highway=busway`, or whose driving access is `psv`, or `no` / `private` while `bus` or `psv` is `yes` / `designated` (`psv` means buses and taxis; both count as `bus`). A **bus lane against a one-way street** (`oneway=yes` with `oneway:bus=no` or `oneway:psv=no`) adds the reverse edge, as a bus lane: its `edge_id` is the same as that direction in cycling and walking. Monaco: 18 bus edges, among them the contraflow bus lane of Boulevard des Moulins. ### Pedestrian streets in driving A `highway=pedestrian` way that cars may use (`motorcar` or `motor_vehicle` is `yes`, `designated` or `permissive`, or `access` is `delivery`, `destination` or `agricultural`) is a `living_street` in `driving.edges`. Every other pedestrian way is built in driving too (cut at the same points, with the same `edge_id` as in walking), then moved to `driving.private_edges` with `highway = 'pedestrian'`. `access` tells why cars cannot use it: | The most specific car tag (`motorcar`, `motor_vehicle`, `vehicle`, `access`) | `access` | | ---------------------------------------------------------------------------- | ------------ | | `delivery` (from `motorcar` or `motor_vehicle`) | `delivery` | | `private` | `private` | | none, `no`, or any other value | `pedestrian` | These ways are never routed or exported. The maps draw them. Other path classes (`footway`, `path`, `steps`, `cycleway`) are not in driving. Norrmalm: 831 edges (783 `pedestrian`, 32 `delivery`, 16 `private`); `driving.edges` is unchanged. Design: `docs/design/pedestrian_private_edges.md`. Monaco: | Mode | Routable edges | In `private_edges` | | --------- | -------------- | ----------------------------------------------- | | `driving` | 2,765 | 689 (473 pedestrian, 198 private, 18 for buses) | | `walking` | 10,952 | 154 | | `cycling` | 10,274 | 133 | A way a mode leaves out is still in `raw.ways`, with all its tags. To remove or keep one road yourself, use a [fix](https://khoshkhah.github.io/duckOSM/guides/fix-osm-errors/index.md). ## One-way roads A one-way edge has no reverse twin, so the road can only be used from `source` to `target`. | Mode | One-way when | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `driving` | `oneway` = `yes` / `1` / `true` / `-1`; `junction` = `roundabout` / `circular`; `motorway` and `motorway_link` unless `oneway=no` | | `cycling` | as driving (without the motorway rule), but `oneway:bicycle=no` makes it two-way and `oneway:bicycle=yes` one-way. Dismount edges are always two-way | | `walking` | only `oneway:foot` = `yes` / `1` / `true` / `-1` | `oneway=-1` (one-way against the way's drawing direction) is turned round, so its edge runs the legal way. ## Speeds and travel time `maxspeed_kmh` is never empty. For driving it is the `maxspeed` tag when it is a number, converted when it ends in `mph`, and otherwise (untagged, or a value like `RU:urban` or `walk`) a default: | Road class | km/h | | ----------------------------- | ---- | | `motorway`, `motorway_link` | 110 | | `trunk`, `trunk_link` | 90 | | `primary`, `primary_link` | 70 | | `secondary`, `secondary_link` | 60 | | `tertiary`, `tertiary_link` | 50 | | `unclassified`, `road` | 40 | | `residential` | 30 | | `living_street`, `service` | 20 | | any other class | 50 | Monaco: 899 of the 2,765 driving edges have a `maxspeed` tag. `cost_s`, the travel time in seconds, is `length_m / (maxspeed_kmh / 3.6)`. It uses the speed limit: no traffic, no delay at junctions. Routing and the edge graph use it. ## Lanes `lanes` is the number of lanes in the edge's own direction, never empty, except on [connector edges](https://khoshkhah.github.io/duckOSM/concepts/cleanup/#connect-dangling-paths) (walking, cycling), where it is NULL: 1. `lanes:forward` / `lanes:backward`, when tagged. 1. A one-way road: all of `lanes`. 1. A two-way road: `lanes` split in two, at least 1 each way; the drawing direction gets the larger half. 1. Nothing tagged: 2 for `motorway` and `trunk`, otherwise 1. A `lanes:reversible` lane is added to both directions. Monaco: 1,165 of the 2,765 driving edges are on a way with a lanes tag. ## Dismount edges Cycling also gets footways and pedestrian streets, so that cycleways joined only by them stay connected. On those where riding isn't allowed (`bicycle` isn't `yes` / `designated` / `permissive`), `dismount` is `TRUE`: the bike is pushed, at 5 km/h, in both directions, and `cycle_type` is `dismount`. Their access is the walking one (`foot`, then `access`): `no` leaves them out, `private` moves them to `private_edges`. A [connector edge](https://khoshkhah.github.io/duckOSM/concepts/cleanup/#connect-dangling-paths) that extends a dismount path is dismount too. Monaco: 7,240 of the 10,274 cycling edges. For a ride-only network, use `WHERE NOT dismount`; `options.cycling_dismount: false` leaves them out of the build. ## `walk_type` and `cycle_type` Walking and cycling edges get a type from OSM sub-tags. The first rule that matches wins. | `walk_type` | From | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | | `escalator` / `steps` | `highway=steps`, with / without `conveying` | | `sidewalk` / `crossing` / `footpath` | `highway=footway` with `footway=sidewalk` / `footway=crossing` / anything else | | `plaza` / `pedestrian_street` | `highway=pedestrian`, with / without `area=yes` | | `shared_street` | `living_street` | | `sidewalk` | any other road with a `sidewalk` tag (not `no` / `none` / `separate`): you walk its sidewalk | | `shared_road` | `residential`, `service`, `unclassified`, `road`, `tertiary`, `secondary`, `primary`, `trunk` (and their `_link`s): walked on the carriageway | | `corridor` | `highway=corridor`, or `indoor=yes` | | `platform` | `highway=platform` | | `path` | `path`, `track`, `bridleway` | | `footpath` | anything else | | `cycle_type` | From | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `dismount` | a [dismount edge](#dismount-edges) | | `cycleway` | `highway=cycleway` | | `cycle_track` / `cycle_lane` / `shared_lane` / `bus_cycle_lane` | the cycleway tag on the edge's side: `track` / `lane` / `shared_lane` / `share_busway` | | `segregated_path` / `shared_path` | `path`, `track`, `bridleway`, with / without `segregated=yes` | | `mixed_traffic` | anything else, including a footway or pedestrian street you may ride (`bicycle=yes`) | The side is `cycleway:right` for an edge in the drawing direction and `cycleway:left` for its reverse, then `cycleway:both`, then `cycleway`. ## Turn restrictions Only the driving network has them. They come from OSM relations `type=restriction` with a `from` way, a `via` node and a `to` way, and from the turn rules of a [fixes file](https://khoshkhah.github.io/duckOSM/guides/fix-osm-errors/index.md). A `no_*` restriction removes that one turn from `edge_graph`; an `only_*` restriction removes every other turn from the `from` edge. Monaco: 38 of the 43 restriction relations map onto edges. Not used: restrictions through a via way, the `except` tag, and tags like `restriction:hgv` or `restriction:conditional`. Any turn not removed is allowed, including a U-turn onto the reverse twin of a two-way road. ## Bus routes Buses have no network of their own: the OSM bus route relations (`route` = `bus`, `trolleybus`, `share_taxi`) are matched to the driving edges they travel, and stored in schema [`bus`](https://khoshkhah.github.io/duckOSM/reference/database/#bus-schema) (`bus.routes`, `bus.route_edges`). A build from a PBF with driving does it (`options.bus_routes`, on); [`duckosm bus DB`](https://khoshkhah.github.io/duckOSM/reference/cli/#bus) adds it to a file built before. A route relation lists its ways in travel order; its stops and platforms are left out. Each way is travelled from the node it shares with the way before to the node it shares with the way after, so its direction comes from its neighbours, and on a two-way street that picks the `#nf` or the `#nr` edge. The first or last way of a stretch, where only one neighbour is known, is travelled from or to its far end; when it is joined in its middle (or is a stretch alone), the role `forward` / `backward` decides, else the way's one-way. A roundabout is followed from where the route enters to where it leaves. The edges come from `driving.edges` and, for bus lanes and bus-only roads, `driving.private_edges` with `access = 'bus'`: a bus against a one-way street runs on its contraflow bus lane. Nothing is left out silently. `bus.routes.outside` counts the member ways outside the area; `bus.routes.gaps` lists every other way the route could not be followed on, as `' '`: | Kind | The way | | --------------- | ------------------------------------------------------------------------------------------------------- | | `no_connection` | shares no node with the next member way: the route breaks there | | `no_edge` | has no driving edge (nor bus lane) in the direction travelled | | `private` | is a private road (in `driving.private_edges`, not for buses) | | `role_conflict` | has a role (`forward` / `backward`) its neighbours contradict; the neighbours are followed | | `ambiguous` | the first or last way of a stretch, two-way, joined in its middle, no role: its direction can't be told | | `role` | has a role other than `''`, `forward`, `backward` | Monaco: 56 bus routes, none with gaps (38 run partly outside the area), 5,609 route edges on 1,034 driving edges, 14 of them bus lanes. The contraflow bus lane of Boulevard Charles III (`1449981121#1r`) carries lines 3, 607 and X1. The [level area](https://khoshkhah.github.io/duckOSM/guides/drawing-order/index.md) marks these edges: `bus` is in their `modes` and `bus_lines` lists the lines, e.g. `1, 2, 5`. # Network clean-up OSM data has small gaps: a footway that ends a metre short of the road it meets, a road cut in two by the area's border. Two steps deal with them, then an optional check. | Step | Modes | Runs when | | ------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------- | | [Connect dangling paths](#connect-dangling-paths) | walking, cycling | `clip.connectivity_rescue` (on), in a build from a PBF | | [Component filter](#component-filter) | all | a boundary is set, and `clip.keep_largest_component` (on) or `clip.min_component_edges` > 1 | | [Validation](#validation) | all | `validation.enabled` (off; on in the config template) | ## Connect dangling paths A **dangling end** is a node that touches only one edge, where that edge is a `cycleway`, `footway`, `path`, `steps`, `pedestrian` or `track`. For each one, the build adds a straight **connector edge** to the nearest node within `clip.connect_snap_m` metres (default 10), in both directions. Without it, the path would be a separate piece and the component filter would drop it. A connector edge has: - a negative `osm_id`, `-min(a, b)`: `a` and `b` are the `osm_id`s of the path and of the road at the node it joins. Select connectors with `osm_id < 0`. - the `highway` of the path and `oneway = FALSE`; no `name`, `lanes` or other tags (NULL). In cycling, a connector that extends a [dismount](https://khoshkhah.github.io/duckOSM/concepts/networks/#dismount-edges) path is dismount too. - an `edge_id` from the [usual formula](https://khoshkhah.github.io/duckOSM/concepts/edge-ids/index.md). Monaco: | Mode | Dangling ends joined | Connector edges added | Left after the component filter | | ------- | -------------------- | --------------------- | ------------------------------- | | walking | 254 | 508 | 490 | | cycling | 512 | 1,024 | 986 | Limits: it joins the nearest *node*, not the nearest point of an edge, so a connector can be longer than the real gap. The distance ignores levels, so it can join a path to a road on a bridge above it. An end with no node within the distance is not joined. Driving isn't repaired. ## Component filter The filter keeps the largest connected piece of each network and drops the rest: stubs cut off by the border and small isolated pieces. It counts pieces by edges, and ignores direction (weakly connected). Dropped edges also leave `edge_graph`, `turn_restrictions`, `nodes` and `edge_id_map`. It works on the routable `edges` only: `private_edges` are not filtered. | Key | Default | Does | | ----------------------------- | ------- | --------------------------------------------------------- | | `clip.keep_largest_component` | `true` | keep only the largest piece | | `clip.min_component_edges` | `1` | also keep every other piece with at least this many edges | It runs only when a boundary is set (a whole country has real islands) and, for a PBF build, when the edge graph is built. It runs in clip builds too. Monaco: | Mode | Edges dropped | Pieces dropped | | ------- | ------------- | -------------- | | driving | 79 | 15 | | walking | 268 | 51 | | cycling | 225 | 35 | Because direction is ignored, the kept network can still hold a one-way dead end: an edge you can reach but not leave. Monaco, driving: 25 such edges. ## Where the border cuts **Build from a PBF.** The PBF is cut with `osmium` (strategy `smart` by default, `complete_ways` when `options.build_features: false`; `options.clip_strategy` sets it): every way with a node inside the boundary is kept whole, so roads that cross the border stick out past it. Then the component filter removes what doesn't connect. `boundary.buffer_m` grows the boundary first. **Clip from a parent build.** `clip.predicate` decides which edges are copied: | `clip.predicate` | Keeps an edge when | | ---------------------- | ----------------------------------------------------------------------------- | | `intersects` (default) | it touches the boundary: roads that cross the border are kept whole | | `within` | it lies completely inside: no stubs, but roads that cross the border are lost | | `centroid` | its middle point is inside | `duckosm extract` always uses `intersects`. ## Validation With `validation.enabled`, each mode is checked once it is built. A failed check stops the build unless `validation.fail_on_error: false`. How to switch it on and what the output looks like: [Check a build](https://khoshkhah.github.io/duckOSM/guides/check-build/index.md). | Check (`validation.` key) | Passes when | | ------------------------------ | ---------------------------------------------------------------------------------------- | | `assert_single_component` | the largest piece holds at least 99.9% of the edges | | `assert_no_stranded_named` | no edge with a name is outside the largest piece | | `assert_unique_node_id` | no `node_id` appears twice in `nodes` | | `assert_way_length_conserved` | no stretch of a kept OSM way is missing while an edge of the same way joins its two ends | | `warn_layer_without_structure` | a warning only: edges with `layer` ≠ 0 but no `bridge` or `tunnel` tag | The first two run only when the component filter ran. `assert_way_length_conserved` is skipped in a clip build, which has no `ways` table. # Routing across modes `duckosm multimodal` joins the walking, cycling and driving networks of a database into one graph, so that a trip can change mode on the way: walk to the car, drive, walk from the car. How to run it and route with `route_multimodal`: [Route across modes](https://khoshkhah.github.io/duckOSM/guides/route/#across-modes-walk--drive--walk). *Experimental.* ## What it adds It writes a schema `mm` into the database (or the build does, with `multimodal.enabled: true`). It needs walking and at least one other mode; without them it only warns and writes an empty `mm.transfers`. `mm.edges` is a view, but `mm.transfers` is a table: run `duckosm multimodal` again after rebuilding a mode. | Table | Contents | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `mm.edges` | a view of every mode's `edges`, with a `mode` column: `mode`, `edge_id`, `source`, `target`, `cost_s`, `length_m`, `highway`, `name`, `geometry`. The key is `(mode, edge_id)`, because [the same road has the same `edge_id` in every mode](https://khoshkhah.github.io/duckOSM/concepts/edge-ids/#the-same-id-in-every-mode) | | `mm.transfers` | one row per change of mode: `node_id`, `from_mode`, `to_mode`, `cost_s`, `kind` | A transfer is made at every node that walking shares with another mode, in both directions. A `node_id` is the OSM node id (or a [virtual node](https://khoshkhah.github.io/duckOSM/concepts/edge-ids/#one-id-per-edge), `node_id < 0`), the same in every mode, so a shared node is the same junction. | From → to | `kind` | | ------------------------------------- | --------------------------- | | walking → driving / driving → walking | `park` / `retrieve` | | walking → cycling / cycling → walking | `bike_park` / `bike_unpark` | There is no transfer between driving and cycling: every change of mode goes through walking. Monaco: walking shares 1,382 nodes with driving and 3,786 with cycling, so `mm.transfers` has 10,336 rows (`duckosm multimodal monaco.duckdb`). ## The model A point in the graph is a pair `(node_id, mode)`. An edge of a mode moves you inside that mode; a transfer moves you to another mode at the same node. Everything is in seconds, so a trip's time is the sum of its edges' `cost_s` plus the sum of its transfers' `cost_s`. By default (`enforce_sequence=True`) a trip has this shape: ``` walk* (drive* | cycle*) walk* ``` It starts and ends in `start_mode` and `end_mode` (both `walking` by default), has at most one vehicle leg, and gets on and off that vehicle on foot. With `enforce_sequence=False` any sequence of modes the transfers allow is possible. ## Transfer cost Each transfer costs the same number of seconds: 60 by default, `--transfer-cost` on the command line. In a config file the costs can differ by direction: ``` multimodal: enabled: true transfer_s: 60 # every change of mode transfer_costs: # optional, per direction "walking->driving": 60 "driving->walking": 30 ``` `duckosm multimodal` takes only the one `--transfer-cost`; costs per direction need a build with the config above. ## Limits - **A vehicle is waiting at every shared junction.** Transfers aren't limited to car parks or bike stands, so a trip can "pick up a car" anywhere. Good for comparing modes, not a real trip planner. - **No turn restrictions.** `route_multimodal` routes over nodes, not over the graph of legal turns, so a driving leg can take a banned turn. For one mode, use `route()`, which obeys them. - **No public transport.** OSM has stops and lines but no timetables. - The graph is held in memory: fine for a city. # duckOSM and OSMnx [OSMnx](https://osmnx.readthedocs.io) is the standard Python library for street networks from OSM, so it is a useful independent check of what duckOSM extracts. The two don't use the same road filter or the same graph model, so compare **length of road per `highway` class**, not totals and not edge counts. ## Line them up first **Road filter.** Use OSMnx's `drive_service`, the closest to duckOSM's driving network: | Network | Service roads | | --------------------- | ------------------------------------------------------------------------------------ | | OSMnx `drive` | none | | OSMnx `drive_service` | yes, but not `service=parking`, `parking_aisle`, `emergency_access` or `private` | | duckOSM `driving` | all, including parking aisles and driveways; private ones are drawn but not routable | **Pedestrian streets.** OSMnx's `drive` and `drive_service` leave `highway=pedestrian` out. duckOSM keeps the pedestrian streets that are open to cars, as `living_street`. **Access tags.** duckOSM reads the most specific tag (`motorcar`, then `motor_vehicle`, `vehicle`, `access`). OSMnx drops any way with `motor_vehicle=no`, `motorcar=no` or `access=private`. duckOSM keeps `access=private` roads in `private_edges`, drawn on its maps but never routed (see [Access](https://khoshkhah.github.io/duckOSM/concepts/networks/#access-private-and-forbidden-roads)). **Border.** `graph_from_bbox` drops every edge that crosses the box. Pass `truncate_by_edge=True`, and cut both networks to the same area before measuring. Here the area is duckOSM's own boundary polygon, because the box around Monaco also holds French roads that duckOSM left out. **Length.** Measure in a metric CRS, the UTM zone of the data. Web Mercator (EPSG:3857) makes lengths about 1.5 times too long at this latitude. ## Recipe Run it in an environment that has `osmnx`, `duckdb` and `geopandas`. It needs a live connection to the Overpass API. ``` import osmnx as ox, duckdb, geopandas as gpd, pandas as pd con = duckdb.connect("area.duckdb", read_only=True) con.execute("LOAD spatial") # the area to measure: duckOSM's boundary area = gpd.GeoSeries.from_wkb( con.execute("SELECT ST_AsWKB(geom) FROM main.boundary").df().iloc[:, 0].map(bytes), crs=4326).union_all() utm = gpd.GeoSeries([area], crs=4326).estimate_utm_crs() # metres (EPSG:32632 for Monaco) cut = gpd.GeoSeries([area], crs=4326).to_crs(utm).iloc[0] def metres(gdf, name): gdf = gdf.to_crs(utm) gdf["highway"] = gdf["highway"].map(lambda h: h[0] if isinstance(h, list) else h) gdf["m"] = gdf.clip(cut).length return gdf.groupby("highway")["m"].sum().rename(name) def osmnx_m(network_type, truncate): G = ox.graph_from_bbox(area.bounds, network_type=network_type, truncate_by_edge=truncate) return metres(ox.convert.graph_to_gdfs(ox.convert.to_undirected(G), nodes=False), network_type) duck = gpd.GeoDataFrame(con.execute( "SELECT highway, ST_AsWKB(geometry) AS geometry FROM driving.edges WHERE NOT is_reverse").df()) duck["geometry"] = gpd.GeoSeries.from_wkb(duck["geometry"].map(bytes), crs=4326) duck = metres(duck.set_geometry("geometry"), "duckOSM") t = pd.concat([osmnx_m("drive", False), osmnx_m("drive", True), osmnx_m("drive_service", True), duck], axis=1).fillna(0).round() t.columns = ["drive", "drive + truncate", "drive_service + truncate", "duckOSM"] t = t[t.sum(axis=1) > 0].sort_values("duckOSM", ascending=False) t.loc["total"] = t.sum() print(t.to_string()) ``` When a simplified OSMnx edge merges several OSM ways, its `highway` is a list; the recipe takes the first element. `NOT is_reverse` keeps one direction of each two-way street, to match the undirected OSMnx graph. ## An example Measured on Monaco with the recipe above: metres of road inside the boundary of the duckOSM build (OSM data as of the run; a new run can differ by a few hundred metres). | `highway` | OSMnx `drive` | `drive` + truncate | `drive_service` + truncate | duckOSM | Difference | | -------------- | ------------- | ------------------ | -------------------------- | ---------- | ---------- | | residential | 20,108 | 20,132 | 19,993 | 19,810 | −183 | | service | 0 | 0 | 12,898 | 12,023 | −875 | | secondary | 10,451 | 10,451 | 10,451 | 10,554 | +103 | | tertiary | 8,492 | 8,492 | 8,492 | 8,492 | 0 | | primary | 6,139 | 6,486 | 6,486 | 6,486 | 0 | | primary_link | 1,018 | 1,018 | 1,018 | 1,018 | 0 | | unclassified | 427 | 427 | 554 | 427 | −127 | | secondary_link | 520 | 520 | 520 | 391 | −129 | | living_street | 89 | 89 | 89 | 89 | 0 | | tertiary_link | 45 | 45 | 45 | 45 | 0 | | **total** | **47,289** | **47,660** | **60,546** | **59,335** | **−2.0%** | The difference is duckOSM minus `drive_service` + truncate. - **Truncate** adds 371 m, 347 m of it on `primary`: the edges that cross the border. - **service −875 m**: mostly roads tagged `motor_vehicle=private` without an `access` tag (1,226 m in Monaco). duckOSM reads that tag and moves them to `private_edges`; OSMnx drops only `access=private`, so it keeps them. Smaller differences go the other way. - **residential, secondary, unclassified, secondary_link**: differences of 100 to 200 m each. Where OSMnx merges several ways into one edge, its `highway` is a list, and the recipe takes the first element, so a class can shift to a neighbour from run to run. - **living_street**: equal here. The extra `living_street` from pedestrian streets open to cars (see above) did not show up in Monaco. `tertiary`, `primary`, `primary_link`, `living_street` and `tertiary_link` match to the metre: where the filters agree, the geometry agrees. ## What it is good for Good for checking geometry and clipping, and for noticing a whole road class that went missing. Not good for counts of edges or nodes: OSMnx also splits and merges, but duckOSM cuts every mode at the same points, including where a footway or crosswalk meets a road, merges chains by its own rules, and adds virtual nodes, so its counts can't match OSMnx's. OSMnx downloads from the Overpass API, so keep the area small. # Reference # Command line Every `duckosm` command and its options. `duckosm --help` prints the same list; `duckosm --version` prints the version. `-h` is short for `--help` everywhere. | Group | Commands | | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | [Build and areas](#build-and-areas) | `build`, `init-config`, `boundary`, `clip-pbf`, `extract` | | [Query and inspect](#query-and-inspect) | `way` | | [Maps and routing](#maps-and-routing) | `viz`, `route-map` | | [Exports](#exports) | `sumo`, `matsim`, `matsim-lanes`, `gmns`, `gmns-map`, `gmns-viz`, `export-gis`, `gis-debug`, `export-graph` | | [Experimental and lane-level](#experimental-and-lane-level) | `opendrive`, `lanelet2`, `railml`, `lane-graph`, `route-lanes` | | [Enrich a built database](#enrich-a-built-database) | `elevation`, `admin`, `multimodal` | `DB` is a built duckOSM database; `GMNS_DB` is a database written by [`gmns`](#gmns). Commands that enrich a database (`elevation`, `admin`, `multimodal`) change it in place. ## Build and areas ### `build` Builds a network from a PBF, or cuts one from a built database. Guide: [Build a network](https://khoshkhah.github.io/duckOSM/guides/build/index.md). ``` duckosm build [OPTIONS] ``` | Option | Does | Default | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `-c`, `--config PATH` | a [config file](https://khoshkhah.github.io/duckOSM/reference/configuration/index.md). An option below that you type overrides the file | | | `-p`, `--pbf PATH` | input `.osm.pbf` | | | `-o`, `--output PATH` | output file | `.duckdb`, named after the boundary file, else the PBF, else the `--source-db` file | | `-b`, `--boundary PATH` | GeoJSON area to build | none: the whole PBF | | `--source-db PATH` | cut from this built database instead of a PBF ([how](https://khoshkhah.github.io/duckOSM/guides/prepare-area/#several-areas-from-one-region)) | | | `--h3-cell TEXT` | build the area of this H3 cell: its outline is the boundary | | | `--graph` / `--no-graph` | build `edge_graph`, the graph of legal turns | on | | `--h3-index` / `--no-h3-index` | add H3 cell ids to nodes and edges | on | | `--h3-resolution INTEGER` | H3 resolution, 0–15 | `8` | | `-m`, `--modes TEXT` | `driving`, `walking` or `cycling`; repeat for several | `driving` | | `--features` / `--no-features` | build the [base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md), `features.*` | on | | `--levels` / `--no-levels` | the last step: the [drawing order](https://khoshkhah.github.io/duckOSM/guides/drawing-order/index.md) of the roads (`visualization.edge_levels`), with your edits kept | on | | `--fixes FILE` | a rules file of [fixes for OSM errors](https://khoshkhah.github.io/duckOSM/guides/fix-osm-errors/index.md); overrides the config's `osm_overrides` key | none | | `--log-file FILE` | also write the log to this file | console only | With none of `-c`, `-p` and `--source-db`, it uses `config/default.yaml` if that file exists. ### `init-config` Writes the commented config template, to edit and build from. Guide: [With a config file](https://khoshkhah.github.io/duckOSM/guides/build/#with-a-config-file). ``` duckosm init-config [OPTIONS] [PATH] ``` | Option | Does | Default | | --------- | ----------------------------- | -------------- | | `PATH` | the file to write | `duckosm.yaml` | | `--force` | overwrite `PATH` if it exists | | ### `boundary` Finds the boundary of an area by name and writes it as GeoJSON: first in the PBF's own borders (offline), then with Nominatim (online). Guide: [Prepare an area](https://khoshkhah.github.io/duckOSM/guides/prepare-area/index.md). ``` duckosm boundary [OPTIONS] [NAME] ``` | Option | Does | Default | | ------------------ | ------------------------------------------------------------- | ---------------- | | `--pbf PATH` | search the borders in this PBF first; needs GDAL's `ogr2ogr` | | | `--osm-id INTEGER` | take this OSM boundary relation (when a name matches several) | | | `--offline` | never ask Nominatim | | | `-o`, `--out FILE` | output GeoJSON | `.geojson` | ### `clip-pbf` Cuts a PBF down to a GeoJSON area, with `osmium`. Guide: [Cut a smaller PBF](https://khoshkhah.github.io/duckOSM/guides/prepare-area/#cut-a-smaller-pbf). ``` duckosm clip-pbf [OPTIONS] PBF BOUNDARY ``` | Option | Does | Default | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | | `-o`, `--out FILE` | output PBF | `.osm.pbf` | | `--strategy` | `complete_ways`: a way that crosses the border is kept whole; `smart`: also completes multipolygons (rivers, land cover), for [base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md); `simple`: cuts ways at the border | `complete_ways` | ### `extract` Cuts one area out of a built database into a new database, by name, OSM id or GeoJSON. Edge ids are kept. Guide: [Several areas from one region](https://khoshkhah.github.io/duckOSM/guides/prepare-area/#several-areas-from-one-region). ``` duckosm extract --source SOURCE --db DB (--name NAME | --osm-id OSM_ID | --boundary BOUNDARY) ``` | Option | Does | Default | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | `--source SOURCE` | the built database to cut from | required | | `--db DB` | the new database (overwritten) | required | | `--name NAME` | the area, by name in the source's `admin_boundaries` ([add them](https://khoshkhah.github.io/duckOSM/guides/admin-boundaries/index.md)) | one of the three | | `--osm-id OSM_ID` | the area, by its `admin_boundaries` OSM id | one of the three | | `--boundary BOUNDARY` | the area, as a GeoJSON file | one of the three | ## Query and inspect ### `info` Prints what a built database holds: per mode its edges, nodes, private edges, km, legal turns (`edge_graph`) and turn restrictions; when and by which duckOSM version it was built; its time zone; and the other schemas (raw OSM data, base-map layers, `mm`, boundary, elevation, drawing order). Reads only. ``` duckosm info [OPTIONS] DB ``` | Option | Does | Default | | -------- | ---------------------------------------- | ------- | | `--json` | print one JSON object instead of a table | | ``` $ duckosm info monaco.duckdb monaco.duckdb (built 2026-10-01 18:20, duckOSM 0.1.0; time zone Europe/Monaco) mode edges nodes private_edges km edge_graph turn_restrictions driving 2,765 1,719 214 92.6 4,953 38 walking 10,952 4,116 154 244.1 32,476 - cycling 10,274 4,151 133 245.5 28,851 - also: raw (OSM data), features (11 layers), mm (across modes), boundary ``` A database built before this command existed has no build date; the rest is the same. ### `way` Prints everything the database knows about one OSM way: its raw row, then its edges in every mode, in order. A negative `OSM_ID` shows [connector edges](https://khoshkhah.github.io/duckOSM/concepts/cleanup/#connect-dangling-paths). Guide: [Look up one OSM way](https://khoshkhah.github.io/duckOSM/guides/query/#look-up-one-osm-way). ``` duckosm way [OPTIONS] DB OSM_ID ``` | Option | Does | Default | | ---------------------- | ------------------------------------------------------------------------------ | -------------------- | | `-m`, `--mode TEXT` | modes to include; repeat for several | every mode in the db | | `--geom` / `--no-geom` | add the edge geometry as WKT | off | | `-o`, `--out PATH` | write the table to a `.csv`, `.parquet` or `.json` file instead of printing it | | | `--json` | print one JSON object: the raw way (`tags`, `refs`) and its edges | | ## Maps and routing ### `viz` Draws the maps of a built network with [mapstyle](https://khoshkhah.github.io/mapstyle/): one page with every mode and one per mode, over the database's own base map. Needs `duckosm[viz]`. Guide: [Draw a map](https://khoshkhah.github.io/duckOSM/guides/draw-map/index.md). ``` duckosm viz [OPTIONS] DB ``` | Option | Does | Default | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | `-m`, `--mode TEXT` | only these modes' pages; repeat for several | every mode's page, plus `_map.html` with all modes | | `--basemap TEXT` | a background from the web: `voyager`, `positron`, `osm`, `satellite`, … | none: the database's own layers | | `--out-dir TEXT` | output folder; files are `_map.html` and `__network.html` | `reports` | | `--arrows` / `--no-arrows` | one-way arrows, shown when zoomed in | on | | `--boundary` / `--no-boundary` | draw `main.boundary`, if there is one | on | | `--google-3d` / `--no-google-3d` | a Google 3D window linked to the map. It needs a Google Maps key in `GOOGLE_MAPS_KEY`, and the page must be served over http at an address on the key's website list | on when `GOOGLE_MAPS_KEY` is set | ### `levels` Makes and solves the level area of the database and stores the drawing order in it (schema [`visualization`](https://khoshkhah.github.io/duckOSM/reference/database/#visualization-schema)): every road gets a casing number (start, main, end), a fill number and its ends as drawn. The roads of all modes go into one level area folder next to the database (`DB.levels`), where your `edits.csv`, `heads.csv` and `caps.csv` are written from the tables `visualization.level_*` of the database (they are the master); roadstyle solves it with them. `duckosm build` does this as its last step; this command makes it again, or later for a build with `--no-levels`. The database is changed in place. To fix places by hand: `roadstyle-levels edit DB.levels`. Needs `duckosm[levels]`. Guide: [Store the drawing order](https://khoshkhah.github.io/duckOSM/guides/drawing-order/index.md). ``` duckosm levels [OPTIONS] DB ``` | Option | Does | Default | | ------------ | --------------------- | -------------------------------- | | `--area DIR` | the level area folder | `DB.levels` next to the database | ### `levels-edit` The level editor of the database: fix which road is drawn over which (bridges, crossings) and the head and the cap of a road end. It is [roadstyle's editor](https://khoshkhah.github.io/roadstyle/guides/levels/), started on the area of the file, with mapstyle's base map (water, land, buildings, railways, signals) and **every road of the file**: the roads of its modes and their `private_edges` (bus, private, pedestrian, delivery). The roads that no mode can use are coloured as on the maps: private grey, bus lanes blue, pedestrian streets lavender, delivery streets tan. With `GOOGLE_MAPS_KEY` set, the editor has the Google 3D window and Street View Linked; the key's website list must have the editor's address (`http://localhost:8000/*`, the same as for the maps). A file with one mode gives an editor of that mode; with several modes, the editor's boxes show only some. The work folder is written from the tables `visualization.level_*` (they are the master), and solved first if the stored numbers are not those of the roads in the file. At each Apply the editor writes your edits and the new drawing order into the file. Needs `duckosm[levels]`; the layers need `duckosm[viz]`. Guide: [Store the drawing order](https://khoshkhah.github.io/duckOSM/guides/drawing-order/index.md). ``` duckosm levels-edit [OPTIONS] DB ``` | Option | Does | Default | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | `--area DIR` | the work folder, rewritten from the tables of the database | `DB.levels` next to the database | | `--port INTEGER` | the editor is at `http://localhost:PORT/`, and the maps of `--reports` on the same port: one port for ever, so a Google key's website list needs one address (`http://localhost:8000/*`) | `8000` | | `--reports DIR` | a folder of maps (`duckosm viz`), served as files at the same port; any other address is the editor | `reports` | | `--plain` | no base-map layers: the editor as roadstyle draws it | off | ### `bus` Matches the database's OSM bus route relations to its driving edges and writes schema [`bus`](https://khoshkhah.github.io/duckOSM/reference/database/#bus-schema): `bus.routes` (one row per route, with its gaps) and `bus.route_edges` (the edges each route travels, in order). A build from a PBF does it too (`options.bus_routes`); this adds it to a file built before. The database is changed in place. Needs `raw.*` and the driving network. How: [Bus routes](https://khoshkhah.github.io/duckOSM/concepts/networks/#bus-routes). ``` duckosm bus DB ``` ### `route-map` Writes duckOSM's route planner over a map drawn by mapstyle: drag a start and an end, pick Drive, Walk, Cycle or Walk + drive. Routing runs in the browser with the same answers as `route_points()` and `directions()`. Walk + drive needs [`multimodal`](#multimodal) first. Needs `duckosm[viz]`. Guide: [On a map](https://khoshkhah.github.io/duckOSM/guides/route/#on-a-map). ``` duckosm route-map [OPTIONS] DB ``` | Option | Does | Default | | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | `-m`, `--mode [driving\|walking\|cycling\|walk+drive]` | the first choice and the mode in front (`walk+drive`: every mode; needs `multimodal`); the page offers every mode | the first mode, walking in front | | `--basemap TEXT` | a background from the web: `voyager`, `positron`, `osm`, `satellite`, … | none: the database's own layers | | `--google-3d` / `--no-google-3d` | a Google 3D window linked to the map. It needs a Google Maps key in `GOOGLE_MAPS_KEY`, and the page must be served over http | on when `GOOGLE_MAPS_KEY` is set | | `-o`, `--out TEXT` | output HTML | `reports/_route_map.html` | ## Exports ### `sumo` Writes a SUMO network; the SUMO edge id is the `edge_id`. Page: [SUMO](https://khoshkhah.github.io/duckOSM/exports/sumo/index.md). ``` duckosm sumo [OPTIONS] DB ``` | Option | Does | Default | | ------------------------------------ | --------------------------------------------------------------------------- | ---------------- | | `-m`, `--mode TEXT` | the mode | `driving` | | `--out-dir TEXT` | output folder | `sumo` | | `--name TEXT` | file name base | the db file name | | `--connections` / `--no-connections` | write the allowed turns from `edge_graph`; off lets `netconvert` guess them | on | | `-c`, `--config PATH` | a `netconvert` config (`.netccfg`) instead of the built-in one | | | `--netconvert` / `--no-netconvert` | build the `.net.xml` with `netconvert`; off writes only the plain XML files | on | ### `matsim` Writes a MATSim `network.xml`; link ids are `edge_id`s. Page: [MATSim](https://khoshkhah.github.io/duckOSM/exports/matsim/index.md). ``` duckosm matsim [OPTIONS] DB ``` | Option | Does | Default | | ---------------------- | ------------------------------------------------------------------------------------ | ----------------------- | | `-m`, `--mode TEXT` | a mode, a comma list (`driving,cycling`) or `all`: one network with `modes` per link | `driving` | | `--crs TEXT` | projected CRS for node coordinates | UTM zone of the data | | `--gzip` / `--no-gzip` | gzip the file | on | | `-o`, `--out TEXT` | output file | `_network.xml.gz` | ### `matsim-lanes` Writes MATSim `lanes.xml` and signal files from a GMNS database. The signal timing is a placeholder. Page: [Lanes and signals](https://khoshkhah.github.io/duckOSM/exports/matsim/#lanes-and-signals). ``` duckosm matsim-lanes [OPTIONS] GMNS_DB ``` | Option | Does | Default | | ---------------------------- | --------------------------------------------------------------- | --------- | | `-m`, `--mode TEXT` | the GMNS mode | `driving` | | `--signals` / `--no-signals` | also write `signalSystems`, `signalGroups`, `signalControl.xml` | on | | `--cycle INTEGER` | signal cycle length, seconds | `90` | | `-o`, `--out TEXT` | output folder | `.` | ### `gmns` Writes a GMNS database: nodes, links, lanes, movements, signals; `link_id` is the `edge_id`. Page: [GMNS](https://khoshkhah.github.io/duckOSM/exports/gmns/index.md). ``` duckosm gmns [OPTIONS] DB ``` | Option | Does | Default | | ---------------------------------------- | ------------------------------------------------------- | -------------------- | | `-o`, `--out TEXT` | output database | `_gmns.duckdb` | | `-m`, `--mode TEXT` | modes; repeat for several | every mode in the db | | `--to-csv TEXT` | also write the standard GMNS CSV files to this folder | | | `--lane-geometry` / `--no-lane-geometry` | compute a line for each lane (needs shapely) | on | | `--meso` | also build the meso (lane-level) network, `meso_` | | | `--meso-mode TEXT` | modes for `--meso`; repeat for several | `driving` | | `--micro` | also build the micro (cell) network, `micro_` | | | `--micro-mode TEXT` | modes for `--micro`; repeat for several | `driving` | | `--combined` | also write `gmns_all`, one network for all modes | | | `--drive-side` | `right` or `left`: the side two-way lanes are drawn on | `right` | ### `gmns-map` Writes an HTML map of a GMNS database with [lanestyle](https://github.com/Khoshkhah/lanestyle): every lane at its width, on a base map. Needs `pip install "duckosm[viz]"`. Page: [See it](https://khoshkhah.github.io/duckOSM/exports/gmns/#see-it). ``` duckosm gmns-map [OPTIONS] GMNS_DB ``` | Option | Does | Default | | ------------------- | ------------------------------------------------------------------------------ | ------------------- | | `-m`, `--mode TEXT` | the GMNS mode | `driving` | | `--palette TEXT` | road colours: `mono`, `carto` or `highsat` | `mono` | | `--source-db PATH` | the core db, for bridges and tunnels in a GMNS file made before links had them | | | `-o`, `--out TEXT` | output HTML | `_lanes.html` | ### `gmns-viz` Writes an HTML viewer of a GMNS database: lanes, and the meso network when there is one. Hover a line for its attributes. Page: [See it](https://khoshkhah.github.io/duckOSM/exports/gmns/#see-it). ``` duckosm gmns-viz [OPTIONS] GMNS_DB ``` | Option | Does | Default | | ------------------- | ------------- | -------------------- | | `-m`, `--mode TEXT` | the GMNS mode | `driving` | | `-o`, `--out TEXT` | output HTML | `_viewer.html` | ### `export-gis` Writes a GeoPackage, GeoParquet files or shapefiles: edges, nodes and the boundary, `edge_id` kept. Page: [GeoPackage, GeoParquet and shapefile](https://khoshkhah.github.io/duckOSM/exports/gis/index.md). ``` duckosm export-gis [OPTIONS] DB ``` | Option | Does | Default | | ------------------------------ | ------------------------------------------------------ | ---------------------------------------------------- | | `-m`, `--mode TEXT` | modes; repeat for several | every mode in the db | | `-f`, `--format` | `gpkg` (GeoPackage), `geoparquet` or `shp` (shapefile) | `gpkg` | | `-o`, `--out TEXT` | `.gpkg` file, or folder for shapefiles and GeoParquet | `.gpkg` / `_gis/` / `_geoparquet/` | | `--boundary` / `--no-boundary` | include `main.boundary` as a layer | on | | `--name TEXT` | layer and file name base | the db file name | ### `gis-debug` Reads a GeoPackage or shapefile export back through GDAL and writes an HTML page: a map of every layer and a check of counts, CRS and `edge_id`s. Needs geopandas. Page: [Check an export](https://khoshkhah.github.io/duckOSM/exports/gis/#check-an-export). ``` duckosm gis-debug [OPTIONS] EXPORT_PATH ``` | Option | Does | Default | | ------------------ | ------------------------------------------------------------ | ----------------------- | | `--source-db PATH` | the database to compare against (every `edge_id` must match) | | | `-o`, `--out TEXT` | output HTML | `_gis_debug.html` | | `--name TEXT` | name shown on the page | the export file name | | `--json` | also print the check as JSON: verdict, summary, each layer | | ### `export-graph` Writes a networkx graph file (GraphML or gpickle). Needs networkx. Page: [networkx](https://khoshkhah.github.io/duckOSM/exports/networkx/index.md). ``` duckosm export-graph [OPTIONS] DB ``` | Option | Does | Default | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `-m`, `--mode TEXT` | the mode | `driving` | | `-o`, `--out TEXT` | output file; the extension picks the format | `_.graphml` | | `-g`, `--graph` | `node`: junctions and roads; `edge`: the edge-based routing graph | `node` | | `-f`, `--format` | `graphml` or `gpickle` | from `--out`, else `graphml` | | `--weight` | arc weight, `time` or `length`; `--graph edge` only | `time` | | `--geometry` | edge geometry, `wkt`, `shapely` or `none`; `--graph node` only; GraphML writes shapely as WKT | `wkt` | | `--uses TEXT` | more turns for `--graph edge`: a comma list of `bus`, `private`, `pedestrian`, `delivery`, the turns into and out of those restricted edges (`private_edges`); they get an `access` attribute | the mode's own turns only | ## Experimental and lane-level ### `opendrive` Writes an ASAM OpenDRIVE `.xodr`: roads and lanes. Page: [OpenDRIVE](https://khoshkhah.github.io/duckOSM/exports/opendrive/index.md). ``` duckosm opendrive [OPTIONS] DB ``` | Option | Does | Default | | ------------------- | ---------------------------------------------------------------------- | -------------------- | | `-m`, `--mode TEXT` | the mode | `driving` | | `--crs TEXT` | projected CRS for the road lines | UTM zone of the data | | `--junctions` | also write junctions and turn roads; `DB` must then be a GMNS database | | | `-o`, `--out TEXT` | output file | `.xodr` | ### `lanelet2` Writes a Lanelet2 map (`.osm`) from a GMNS database: one lanelet per lane. Page: [Lanelet2](https://khoshkhah.github.io/duckOSM/exports/lanelet2/index.md). ``` duckosm lanelet2 [OPTIONS] GMNS_DB ``` | Option | Does | Default | | ------------------- | ------------- | --------------------- | | `-m`, `--mode TEXT` | the GMNS mode | `driving` | | `-o`, `--out TEXT` | output file | `.lanelet2.osm` | ### `railml` Writes a railML 2.4 file of the rail network, read from `raw.*` (there is no rail mode). Page: [railML](https://khoshkhah.github.io/duckOSM/exports/railml/index.md). ``` duckosm railml [OPTIONS] DB ``` | Option | Does | Default | | ------------------ | ----------- | ------------------- | | `-o`, `--out TEXT` | output file | `.railml.xml` | ### `lane-graph` Builds a lane-level routing graph in a GMNS database: lane-to-lane turns and lane changes, in `lane_.lane_edges`. Page: [Lane-level routing](https://khoshkhah.github.io/duckOSM/exports/lane-routing/index.md). ``` duckosm lane-graph [OPTIONS] GMNS_DB ``` | Option | Does | Default | | ------------------- | ------------- | --------- | | `-m`, `--mode TEXT` | the GMNS mode | `driving` | ### `route-lanes` Finds a lane-level route between two lanes and prints the lanes, cost and manoeuvres. Each of `FROM_LANE` and `TO_LANE` is a `lane_id`, or an `edge_id` (its lane 1). Uses the table from [`lane-graph`](#lane-graph), or builds the graph in memory without it. Page: [Lane-level routing](https://khoshkhah.github.io/duckOSM/exports/lane-routing/index.md). ``` duckosm route-lanes [OPTIONS] GMNS_DB FROM_LANE TO_LANE ``` | Option | Does | Default | | ------------------- | ----------------------------------------------------------------------- | --------- | | `-m`, `--mode TEXT` | the GMNS mode | `driving` | | `-o`, `--out TEXT` | also write the route as GeoJSON | | | `--json` | print the route as JSON: `lanes`, `cost`, `maneuvers`, `geometry` (WKT) | | ## Enrich a built database ### `elevation` Adds the ground height to every node (`nodes.ele`) and edge end (`edges.z_from`, `z_to`), and a row in `main.elevation_metadata`. Running it again overwrites. Needs `duckosm[elevation]`. Guide: [Add elevation](https://khoshkhah.github.io/duckOSM/guides/elevation/index.md); sources: [Elevation sources](https://khoshkhah.github.io/duckOSM/reference/elevation-sources/index.md). ``` duckosm elevation [OPTIONS] DB ``` | Option | Does | Default | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | | `--dem TEXT` | an elevation file or URL, any format GDAL reads; wins over `--source` | | | `--source TEXT` | `auto`, `copernicus` or `eudtm` (needs `OPENTOPOGRAPHY_API_KEY`). `auto` picks EU-DTM in Europe when the key is set, else Copernicus | `auto` | | `-m`, `--mode TEXT` | modes; repeat for several | every mode in the db | | `--nodata-fill FLOAT` | height written where the model has no value | `0.0` | | `--suffix NAME` | store as a second surface: `--suffix dsm` writes `ele_dsm`, `z_from_dsm`, `z_to_dsm` | | ### `admin` Adds `main.admin_boundaries`: every OSM administrative boundary in the PBF, with its parent. Replaces the table if it exists. Needs GDAL's `ogr2ogr`. Guide: [Add administrative boundaries](https://khoshkhah.github.io/duckOSM/guides/admin-boundaries/index.md). ``` duckosm admin --pbf PBF --db DB [--gpkg GPKG] ``` | Option | Does | Default | | ------------- | -------------------------------------------------------------------------------------------------------------- | ---------------- | | `--pbf PBF` | the OSM PBF | required | | `--db DB` | the database to add the table to | required | | `--gpkg GPKG` | GeoPackage of the boundaries: reused if it exists, else written here (skips the slow `ogr2ogr` step next time) | a temporary file | ### `multimodal` Builds `mm.edges` and `mm.transfers`, so a route can change mode (walk, drive, walk). Needs walking and at least one other mode. Page: [Routing across modes](https://khoshkhah.github.io/duckOSM/concepts/multimodal/index.md). ``` duckosm multimodal [OPTIONS] DB ``` | Option | Does | Default | | ----------------------- | ------------------------------------ | ------- | | `--transfer-cost FLOAT` | seconds added at each change of mode | `60.0` | | `--schema TEXT` | schema for the two tables | `mm` | # Configuration A build reads its settings from a YAML file. `duckosm init-config` writes a commented template; edit it, then build: ``` duckosm init-config my_area.yaml duckosm build -c my_area.yaml ``` - A key you leave out takes the **default** in the tables below. An unknown key anywhere (at the top level, a section name, or inside a section) is ignored with a warning. That covers a typo and a key that no longer exists. - Relative paths resolve against the folder you run `duckosm` in, not the config file's folder. - With `-c`, a `build` option you type overrides the file (e.g. `-c monaco.yaml -m walking --no-features`); options you don't type leave the file's values. The **Flag** column shows the option for each field ([Command line](https://khoshkhah.github.io/duckOSM/reference/cli/#build)). - The template sets a few keys to a value other than the default; the **Template** column shows them. It turns on `validation` and `report`, and gives example values for `name`, `output_path` and `source.pbf_path`. In a clone of the repo, `config/sample_monaco.yaml` is a short working example. ## Top level | Key | Default | Template | Flag | Does | | --------------- | --------------- | ----------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | `default` | `my_import` | stem of `-o`, else of `-b`, else of `-p` or `--source-db` | the area's name: the output file is `.duckdb` when `output_path` is a folder; also names the cached PBF cut (below) and the report files | | `output_path` | `output.duckdb` | `.` | `-o` | a path ending in `.duckdb` is the output file; anything else is a folder, and the file is `/.duckdb` | | `modes` | `[driving]` | `[driving]` | `-m` | the networks to build: any of `driving`, `walking`, `cycling`, one schema each | | `osm_overrides` | none | none | `--fixes` | a rules file of [fixes for OSM errors](https://khoshkhah.github.io/duckOSM/guides/fix-osm-errors/index.md). Used only when named | | `pbf_path` | none | | | short for `source.pbf_path` | | `boundary_path` | none | | | short for `boundary.path` | | `h3_cell` | none | | `--h3-cell` | short for `boundary.h3_cell` | With a boundary, the PBF is first cut to it and the cut is kept as `pbf/...osm.pbf`, where `` is a fingerprint of the boundary and the PBF. The next build reuses it while both are unchanged; after a change it cuts again and removes the old cut of the same name and strategy. ## `source` Where the network comes from. [How a build works](https://khoshkhah.github.io/duckOSM/concepts/build/index.md) describes both kinds. | Key | Default | Flag | Does | | ------------------ | ------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `source.type` | `pbf` | `--source-db` sets `duckdb` | `pbf`: build from OSM data. `duckdb`: cut an area out of a built database, keeping its edge ids | | `source.pbf_path` | none | `-p` | the `.osm.pbf` file; required for `pbf` | | `source.source_db` | none | `--source-db` | the built database to cut from; required for `duckdb`, together with a boundary (`path`, `place`, `bbox` or `h3_cell`) | ## `boundary` The area to build. | Key | Default | Flag | Does | | ------------------- | ------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `boundary.path` | none | `-b` | a GeoJSON polygon. With none of the four boundary keys, the whole PBF is built and the component filter doesn't run | | `boundary.buffer_m` | `0` | | grow the boundary outwards by this many metres first (measured in the local UTM zone) | | `boundary.place` | none | | a place name, e.g. `Monaco`: looked up in the PBF's own borders, else on Nominatim, as [`duckosm boundary`](https://khoshkhah.github.io/duckOSM/reference/cli/#boundary) does | | `boundary.bbox` | none | | a box, `[west, south, east, north]` in degrees | | `boundary.h3_cell` | none | `--h3-cell` | an H3 cell id: its outline is the area | Give one of `path`, `place`, `bbox`, `h3_cell`; `path` wins. The other three are written to `.boundary.geojson` next to the output, and the build uses that file. To make a boundary file: [Prepare an area](https://khoshkhah.github.io/duckOSM/guides/prepare-area/index.md). ## `clip` The clean-up after the area is cut. Details: [Network clean-up](https://khoshkhah.github.io/duckOSM/concepts/cleanup/index.md). | Key | Default | Does | | ----------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `clip.predicate` | `intersects` | `duckdb` source only: which edges are copied. `intersects`: every edge that touches the area; `within`: only edges completely inside; `centroid`: edges whose middle point is inside. A `pbf` build always keeps whole ways | | `clip.keep_largest_component` | `true` | keep only the largest connected piece of each network. Runs only with a boundary and, for a `pbf` build, with `options.build_graph` | | `clip.min_component_edges` | `1` | with `keep_largest_component: false`, keep every piece with at least this many edges | | `clip.connectivity_rescue` | `true` | walking and cycling, `pbf` build: [join dangling path ends](https://khoshkhah.github.io/duckOSM/concepts/cleanup/#connect-dangling-paths) to the nearest node. Runs with or without a boundary | | `clip.connect_snap_m` | `10.0` | the largest gap, in metres, that `connectivity_rescue` joins | ## `options` Build options. They apply to a `pbf` build; a `duckdb` build uses only `memory_limit`, `threads` and `build_graph` (for the indexes). | Key | Default | Flag | Does | | ----------------------------------- | ---------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `options.build_graph` | `true` | `--graph` / `--no-graph` | build `edge_graph`, the graph of legal turns. The component filter needs it | | `options.h3_indexing` | `true` | `--h3-index` / `--no-h3-index` | add `nodes.h3_cell` and `edges.from_cell`, `to_cell`, `lca_res` | | `options.h3_resolution` | `8` | `--h3-resolution` | H3 resolution, 0–15, for those columns and for `boundary_cells` | | `options.merge_segments` | `true` | | join a chain of edges that is one road into one edge; writes `.edge_id_map`. [Merged edges](https://khoshkhah.github.io/duckOSM/concepts/edge-ids/#merged-edges) | | `options.global_junctions` | `true` | | split every mode's roads at the same junctions, so a road has the same `edge_id` in every mode. [The same id in every mode](https://khoshkhah.github.io/duckOSM/concepts/edge-ids/#the-same-id-in-every-mode) | | `options.infer_separate_sidewalks` | `true` | | a road with no sidewalk tag whose both sides are covered (a `footway=sidewalk` line beside it, or the other half of a dual carriageway) gets `sidewalk=separate` (and `duckosm:sidewalk=inferred`) in `raw.ways`, so walking uses its sidewalks, not its carriageway. Needs the walking mode | | `options.functional_types` | `true` | | add `walk_type` (walking) and `cycle_type` (cycling). [`walk_type` and `cycle_type`](https://khoshkhah.github.io/duckOSM/concepts/networks/#walk_type-and-cycle_type) | | `options.cycling_dismount` | `true` | | add footways and pedestrian streets to cycling as [dismount edges](https://khoshkhah.github.io/duckOSM/concepts/networks/#dismount-edges) | | `options.process_speeds` | `true` | | add `maxspeed_kmh` to edges. [Speeds and travel time](https://khoshkhah.github.io/duckOSM/concepts/networks/#speeds-and-travel-time) | | `options.calculate_costs` | `true` | | add `cost_s`, the travel time in seconds. Routing and `multimodal` need it | | `options.extract_restrictions` | `true` | | driving: build `turn_restrictions` and remove the forbidden turns from `edge_graph`. [Turn restrictions](https://khoshkhah.github.io/duckOSM/concepts/networks/#turn-restrictions) | | `options.boundary_cells` | `false` | | write `main.boundary_cells`, the H3 cells that cover the boundary. Needs a boundary | | `options.boundary_cell_resolutions` | none | | resolutions for `boundary_cells`, e.g. `[6, 7, 8]`; none means `[h3_resolution]` | | `options.build_features` | `true` | | build the [base-map layers](https://khoshkhah.github.io/duckOSM/reference/features/index.md), `features.*` (`build --no-features` turns it off). Needs `raw.*`, so a `duckdb` build skips it | | `options.sea` | `overture` | | the sea, [`features.ocean`](https://khoshkhah.github.io/duckOSM/reference/features/index.md), from Overture Maps (needs the network; skipped offline); `false` = none | | `options.levels` | `true` | | the last step: the drawing order of the roads, [`visualization.edge_levels`](https://khoshkhah.github.io/duckOSM/reference/database/#visualization-schema), solved with your edits kept in `visualization.level_*` (`build --no-levels` turns it off). Needs the `levels` extra. A clip takes the edits of its parent for the roads it has. [Guide](https://khoshkhah.github.io/duckOSM/guides/drawing-order/index.md) | | `options.bus_routes` | `true` | | match the OSM bus routes to the driving edges: schema [`bus`](https://khoshkhah.github.io/duckOSM/reference/database/#bus-schema). Needs driving and `raw.*`, so a `duckdb` build skips it | | `options.clip_strategy` | none | | how `osmium` cuts the PBF: `complete_ways`, `smart` or `simple` ([`clip-pbf`](https://khoshkhah.github.io/duckOSM/reference/cli/#clip-pbf)). None means `smart` with `build_features`, else `complete_ways` | | `options.memory_limit` | none | | DuckDB memory limit, e.g. `"16GB"`; none is DuckDB's default | | `options.threads` | none | | number of DuckDB threads; none is DuckDB's default | | `options.simplify_batches` | `0` | | split the simplify step into this many batches to save memory; `0` sizes them from the node count, `1` is one pass | ## `validation` Checks each mode at the end of the build. What each check means: [Check a build](https://khoshkhah.github.io/duckOSM/guides/check-build/index.md). | Key | Default | Template | Does | | ----------------------------------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `validation.enabled` | `false` | `true` | run the checks | | `validation.fail_on_error` | `true` | `true` | stop the build when a check fails; `false` only warns | | `validation.assert_single_component` | `true` | `true` | one connected network. Skipped when no component clean-up runs: no boundary, no edge graph, or `clip.keep_largest_component: false` with `clip.min_component_edges: 1` | | `validation.assert_no_stranded_named` | `true` | `true` | no named road outside the main network. Skipped when no component clean-up runs, as above | | `validation.assert_unique_node_id` | `true` | | no two nodes with the same `node_id` | | `validation.assert_way_length_conserved` | `true` | | no part of an OSM way lost when it was split into edges. Skipped in a `duckdb` build | | `validation.warn_layer_without_structure` | `true` | `true` | warn (never fail) about edges with a `layer` but no `bridge` or `tunnel` tag | ## `multimodal` Builds `mm.edges` and `mm.transfers` at the end of the build, the same as [`duckosm multimodal`](https://khoshkhah.github.io/duckOSM/reference/cli/#multimodal). Needs walking and at least one other mode. [Routing across modes](https://khoshkhah.github.io/duckOSM/concepts/multimodal/index.md). | Key | Default | Does | | --------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------- | | `multimodal.enabled` | `false` | build the `mm` tables | | `multimodal.transfer_s` | `60.0` | seconds added at each change of mode | | `multimodal.transfer_costs` | none | seconds per direction, e.g. `{"walking->driving": 60, "driving->walking": 30}`; a pair not listed uses `transfer_s` | ## `report` and `viz` | Key | Default | Template | Does | | ---------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `report.enabled` | `false` | `true` | write `reports/_