Skip to content

Commit 0d09eaf

Browse files
pettirosspettiross
authored
Update for home page Solutions Notebook link (#87)
* Update for home page Solutions Notebook link - Match style guide - Make it run in tiny by removing unneeded map geometries slowing Kepler.gl rendering - Increase number of routes matched from 10 to 100 - Remove filtered map view * Text edits for clarity * - Pretty print and improve maps config json
1 parent d2fcc4f commit 0d09eaf

File tree

2 files changed

+386
-195
lines changed

2 files changed

+386
-195
lines changed

‎Analyzing_Data/GPS_Map_Matching.ipynb‎

Lines changed: 74 additions & 194 deletions
Original file line numberDiff line numberDiff line change
@@ -5,25 +5,13 @@
55
"id": "a6418743-3dd6-48e4-83ab-b1bb0c0689d0",
66
"metadata": {},
77
"source": [
8-
"![](https://wherobots.com/wp-content/uploads/2023/12/Inline-Blue_Black_onWhite@3x.png)\n",
8+
"![Wherobots logo](../assets/img/header-logo.png)\n",
99
"\n",
10-
"# WherobotsAI Map Matching Example\n",
10+
"# WherobotsAI Map Matching\n",
1111
"\n",
12-
"In this notebook we introduce Wherobots Map Matching, a library for creating map applications with large scale geospatial data, and explore the task of matching noisy GPS trajectory data to underlying road segments using OpenStreetMap road network data. [Read more about Wherobots Map Matching in the Wherobots documentation.](https://docs.wherobots.com/latest/tutorials/sedonamaps/introduction/)"
13-
]
14-
},
15-
{
16-
"cell_type": "code",
17-
"execution_count": null,
18-
"id": "58006583",
19-
"metadata": {},
20-
"outputs": [],
21-
"source": [
22-
"import json\n",
23-
"from shapely.geometry import LineString\n",
24-
"from pyspark.sql.window import Window\n",
25-
"from pyspark.sql.functions import col, expr, udf, collect_list, struct, row_number, lit\n",
26-
"from sedona.spark import *"
12+
"In this notebook we introduce Wherobots Map Matching, a library for creating map applications with large scale geospatial data. Map matching is a crucial step in many transportation analyses, aligning a sequence of observed user positions (usually from GPS) onto a digital map. This identifies the most likely sequence of roads that a vehicle has traversed. \n",
13+
"\n",
14+
"We will explore matching noisy GPS trajectory data to road segments using OpenStreetMap (OSM) road network data. [Read more about Wherobots Map Matching in the Wherobots documentation.](https://docs.wherobots.com/latest/tutorials/wherobotsai/map-matching/map_matching/)\n"
2715
]
2816
},
2917
{
@@ -41,31 +29,24 @@
4129
"metadata": {},
4230
"outputs": [],
4331
"source": [
32+
"import json\n",
33+
"from shapely.geometry import LineString\n",
34+
"from pyspark.sql.window import Window\n",
35+
"from pyspark.sql.functions import col, expr, udf, collect_list, struct, row_number, lit\n",
36+
"from sedona.spark import *\n",
37+
"\n",
4438
"config = SedonaContext.builder().getOrCreate()\n",
4539
"sedona = SedonaContext.create(config)"
4640
]
4741
},
48-
{
49-
"cell_type": "markdown",
50-
"id": "b6305680",
51-
"metadata": {},
52-
"source": [
53-
"## Map Matching\n",
54-
"Map matching is a crucial step in many transportation analyses. It involves aligning a sequence of observed user positions (usually from GPS) onto a digital map, identifying the most likely path or sequence of roads that a user has traversed. \n",
55-
"\n",
56-
"In this section, we will use Wherobots Map Matching for our map matching tasks."
57-
]
58-
},
5942
{
6043
"cell_type": "markdown",
6144
"id": "da17a10b",
6245
"metadata": {},
6346
"source": [
64-
"### Load Ann Arbor, Michigan Road Network Data from OSM File into Spatial Dataframe\n",
65-
"We are utilizing the OpenStreetMap (OSM) data specific to the Ann Arbor, Michigan region to provide the foundational road network for our analysis. OpenStreetMap offers detailed and open-sourced road network data, making it a prime choice for transportation studies.\n",
47+
"## Load OpenStreetMap road data\n",
6648
"\n",
67-
"The step load_OSM is executed only once to load this road network data. Given the granularity and detail of OSM datasets, this process might take some time.\n",
68-
"<br><br>"
49+
"We will call `load_osm` for the road network we will match against. Whereobots Map Matcher uses OSM's XML file format to load detailed, open source road network data. We've got a sample dataset for the Ann Arbor, Michigan area that we will use. The `[car]` parameter tells `matcher` to filter anything out of the network that is not big enough for motor vehicle traffic."
6950
]
7051
},
7152
{
@@ -76,22 +57,30 @@
7657
"outputs": [],
7758
"source": [
7859
"from wherobots import matcher\n",
79-
"dfEdge = matcher.load_osm(\"s3://wherobots-examples/data/osm_AnnArbor_large.xml\", \"[car]\")\n",
80-
"dfEdge.show(5)"
60+
"\n",
61+
"roads_df = matcher.load_osm(\"s3://wherobots-examples/data/osm_AnnArbor_large.xml\", \"[car]\")\n",
62+
"\n",
63+
"roads_df.show(10)"
8164
]
8265
},
8366
{
8467
"cell_type": "markdown",
8568
"id": "2d293de7",
8669
"metadata": {},
8770
"source": [
88-
"### Load GPS Tracks Data from VED Dataset\n",
89-
"For this analysis, we're leveraging the Vehicle Energy Dataset (VED). VED is a comprehensive dataset capturing GPS trajectories of 383 vehicles (including gasoline vehicles, HEVs, and PHEV/EVs) in Ann Arbor, Michigan, USA, from Nov 2017 to Nov 2018. The data spans ~374,000 miles and includes details on fuel, energy, speed, and auxiliary power usage. Driving scenarios cover diverse conditions, from highways to traffic-dense downtown areas, across different seasons.\n",
71+
"### Load sample GPS tracking data from VED\n",
72+
"\n",
73+
"For this analysis, we're leveraging the [Vehicle Energy Dataset (VED)](https://github.com/gsoh/VED). VED is a comprehensive dataset capturing one year of GPS trajectories for 383 vehicles (including gasoline vehicles, HEVs, and PHEV/EVs) in the Ann Arbor area. The data spans about 374,000 miles/600,000 km and includes details on fuel, energy, speed, and auxiliary power usage. Driving scenarios cover diverse conditions, from highways to traffic-dense downtown areas and across four seasons.\n",
9074
"\n",
91-
"Source: \"Vehicle Energy Dataset (VED), A Large-scale Dataset for Vehicle Energy Consumption Research\" by Geunseob (GS) Oh, David J. LeBlanc, Huei Peng. Published in IEEE Transactions on Intelligent Transportation Systems (T-ITS), 2020.\n",
75+
"> Source: \"Vehicle Energy Dataset (VED), A Large-scale Dataset for Vehicle Energy Consumption Research\" by Geunseob (GS) Oh, David J. LeBlanc, Huei Peng. Published in IEEE Transactions on Intelligent Transportation Systems (T-ITS), 2020.\n",
9276
"\n",
93-
"GitHub: https://github.com/gsoh/VED\n",
94-
"<br><br>"
77+
"Each row in the dataset represents a spatial-temporal point of one vehicle's journey. We are going to use these five columns:\n",
78+
"\n",
79+
"- VehId — Vehicle ID\n",
80+
"- Trip — Trip ID; unique per vehicle\n",
81+
"- Timestamp(ms)\n",
82+
"- Latitude[deg]\n",
83+
"- Longitude[deg]"
9584
]
9685
},
9786
{
@@ -101,63 +90,26 @@
10190
"metadata": {},
10291
"outputs": [],
10392
"source": [
104-
"df = sedona.read.csv(\"s3://wherobots-examples/data/VED_171101_week.csv\", header=True, inferSchema=True)"
105-
]
106-
},
107-
{
108-
"cell_type": "markdown",
109-
"id": "c935a3e9-7989-4d49-9edb-d2b521c5ab62",
110-
"metadata": {},
111-
"source": [
112-
"<br>For the purpose of this analysis, we are specifically extracting the columns representing the vehicle id, trip id, timestamp, latitude, and longitude. Each row in the dataset represents a spatial-temporal point of a vehicle's journey, with columns detailing:\n",
93+
"gps_tracks_df = sedona.read.csv(\"s3://wherobots-examples/data/VED_171101_week.csv\", header=True, inferSchema=True)\n",
94+
"gps_tracks_df = gps_tracks_df.select(['VehId', 'Trip', 'Timestamp(ms)','Latitude[deg]', 'Longitude[deg]'])\n",
11395
"\n",
114-
"**VehId**: Vehicle Identifier.<br>\n",
115-
"**Trip**: Trip Identifier for a vehicle. It helps distinguish between different journeys of the same vehicle.<br>\n",
116-
"**Timestamp(ms)**: Timestamp of the data point, typically represented in milliseconds.<br>\n",
117-
"**Latitude[deg]**: Latitude coordinate of the vehicle at the given timestamp.<br>\n",
118-
"**Longitude[deg]**: Longitude coordinate of the vehicle at the given timestamp.\n",
119-
"<br><br>"
120-
]
121-
},
122-
{
123-
"cell_type": "code",
124-
"execution_count": null,
125-
"id": "2f60a128-e3b4-4c3b-8fff-3283f3e375b2",
126-
"metadata": {},
127-
"outputs": [],
128-
"source": [
129-
"df = df.select(['VehId', 'Trip', 'Timestamp(ms)','Latitude[deg]', 'Longitude[deg]'])"
130-
]
131-
},
132-
{
133-
"cell_type": "code",
134-
"execution_count": null,
135-
"id": "07e545a6-7ab7-462a-a1c6-834a74029014",
136-
"metadata": {},
137-
"outputs": [],
138-
"source": [
139-
"df.show(10)"
140-
]
141-
},
142-
{
143-
"cell_type": "markdown",
144-
"id": "742540f5-e5f6-41b8-bf02-04fc90b108bc",
145-
"metadata": {},
146-
"source": [
147-
"<br>The combination of VehId and Trip together form a unique key for our dataset. This combination allows us to isolate individual vehicle trajectories. Every unique pair signifies a specific trajectory of a vehicle. Raw GPS points, while valuable, can be scattered, redundant, and lack context when viewed independently. By organizing these individual points into coherent trajectories represented by Linestrings, we enhance our ability to interpret, analyze, and apply the data in meaningful ways."
96+
"gps_tracks_df.show(10)"
14897
]
14998
},
15099
{
151100
"cell_type": "markdown",
152101
"id": "1b515bc7-0a40-4914-a647-b3645718a560",
153102
"metadata": {},
154103
"source": [
155-
"### Create LineString Geometries from GPS tracks\n",
104+
"## Aggregate GPS points into LineString geometries\n",
105+
"\n",
106+
"The combination of VehId and Trip together form a unique key for our dataset. This combination allows us to isolate individual vehicle trajectories. Every unique pair signifies a specific trajectory of a vehicle. Raw GPS points, while valuable, can be scattered, redundant, and lack context when viewed independently. By organizing these individual points into coherent trajectories represented by LineString geometries, we enhance our ability to interpret, analyze, and apply the data in meaningful ways.\n",
156107
"\n",
157-
"A groupBy operation is performed on 'VehId' and 'Trip' columns to isolate individual trajectories. The resulting LineString essentially captures the responding vehicle's trajectory over time. The rows are first sorted by their timestamps to ensure the LineString follows the chronological order of the GPS data points.\n",
108+
"A `groupBy` operation on 'VehId' and 'Trip' isolates each trip, a LineString representing the vehicle's course. We sort the rows by timestamps so the LineString follows the correct order of the GPS data points.\n",
158109
"\n",
159-
"A User Defined Function (UDF) is created for Spark that utilizes the function below to process Spatial DataFrame rows into LineString geometries.\n",
160-
"<br><br>"
110+
"We'll write a `rows_to_linestring` function for Spark to process Sedona DataFrame rows into LineString geometries, then collect them in a new DataFrame, `trips_df`.\n",
111+
"\n",
112+
"Finally, we'll give each trip a unique ID using `row_number`."
161113
]
162114
},
163115
{
@@ -173,53 +125,35 @@
173125
" linestring = LineString(coords)\n",
174126
" return linestring\n",
175127
"\n",
176-
"linestring_udf = udf(rows_to_linestring, GeometryType())"
177-
]
178-
},
179-
{
180-
"cell_type": "code",
181-
"execution_count": null,
182-
"id": "5207571f-9928-4645-b96c-f2aee95ad7f4",
183-
"metadata": {},
184-
"outputs": [],
185-
"source": [
186-
"# Group by VehId and Trip and aggregate\n",
187-
"dfPath = (df\n",
188-
" .groupBy(\"VehId\", \"Trip\")\n",
189-
" .agg(collect_list(struct(\"Timestamp(ms)\", \"Latitude[deg]\", \"Longitude[deg]\")).alias(\"coords\"))\n",
190-
" .withColumn(\"geometry\", linestring_udf(\"coords\"))\n",
191-
" )"
192-
]
193-
},
194-
{
195-
"cell_type": "markdown",
196-
"id": "b12b3607-ba5b-468b-9b64-5d21ef3bdbcc",
197-
"metadata": {},
198-
"source": [
199-
"### Create a Spatial DataFrame of GPS Tracks"
200-
]
201-
},
202-
{
203-
"cell_type": "code",
204-
"execution_count": null,
205-
"id": "452ac443-2c60-4497-a892-74cecf7ef3a1",
206-
"metadata": {},
207-
"outputs": [],
208-
"source": [
209-
"# Using row_number to generate unique IDs\n",
210-
"window_spec = Window.partitionBy(lit(5)).orderBy(\"VehId\", \"Trip\") # Ordering by existing columns to provide some deterministic order\n",
211-
"dfPath = dfPath.withColumn(\"ids\", row_number().over(window_spec) - 1)\n",
212-
"dfPath = dfPath.filter(dfPath['ids'] < 10)\n",
213-
"dfPath = dfPath.select(\"ids\", \"VehId\", \"Trip\", \"coords\", \"geometry\")\n",
214-
"dfPath.show()"
128+
"linestring_udf = udf(rows_to_linestring, GeometryType())\n",
129+
"\n",
130+
"trips_df = (gps_tracks_df\n",
131+
" .groupBy(\"VehId\", \"Trip\")\n",
132+
" .agg(collect_list(struct(\"Timestamp(ms)\", \"Latitude[deg]\", \"Longitude[deg]\")).alias(\"coords\"))\n",
133+
" .withColumn(\"geometry\", linestring_udf(\"coords\"))\n",
134+
" )\n",
135+
"\n",
136+
"window_spec = Window.partitionBy(lit(5)).orderBy(\"VehId\", \"Trip\")\n",
137+
"trips_df = trips_df.withColumn(\"ids\", row_number().over(window_spec) - 1)\n",
138+
"trips_df = trips_df.filter(trips_df['ids'] < 100) # Filter to 100 trips because this is an example notebook; no need to be exhaustive\n",
139+
"trips_df = trips_df.select(\"ids\", \"VehId\", \"Trip\", \"coords\", \"geometry\")\n",
140+
"\n",
141+
"trips_df.show()"
215142
]
216143
},
217144
{
218145
"cell_type": "markdown",
219146
"id": "9176442c-3e34-4ca1-97b7-229c95a87b59",
220147
"metadata": {},
221148
"source": [
222-
"## Perform Map Matching"
149+
"## Perform Map Matching\n",
150+
"\n",
151+
"Finally, we will pass the road network and the aggregated trips into `matcher`, and tell it the name of the relevant columns (`geometry` in both tables).\n",
152+
"\n",
153+
"- **ids**: A unique identifier for each trajectory, representing a distinct vehicle journey.\n",
154+
"- **observed_points**: Represents the original GPS trajectories. These are the linestrings formed from the raw GPS points collected during each vehicle journey.\n",
155+
"- **matched_points**: The processed trajectories post map-matching. These linestrings are aligned onto the actual road network, correcting for any GPS inaccuracies.\n",
156+
"- **matched_nodes**: A list of node identifiers from the road network that the matched trajectory passes through. These nodes correspond to intersections, turns, or other significant points in the road network."
223157
]
224158
},
225159
{
@@ -233,93 +167,39 @@
233167
"sedona.conf.set(\"wherobots.tools.mm.maxdistinit\", \"100\")\n",
234168
"sedona.conf.set(\"wherobots.tools.mm.obsnoise\", \"40\")\n",
235169
"\n",
236-
"dfMmResult = matcher.match(dfEdge, dfPath, \"geometry\", \"geometry\")"
237-
]
238-
},
239-
{
240-
"cell_type": "markdown",
241-
"id": "a6c65c9b-b15a-40bd-a9bb-f59c41a2ad9d",
242-
"metadata": {},
243-
"source": [
244-
"<br>The dataframe showcases the results of a map matching process on GPS trajectories:\n",
170+
"matched_routes_df = matcher.match(roads_df, trips_df, \"geometry\", \"geometry\")\n",
245171
"\n",
246-
"**ids**: A unique identifier for each trajectory, representing a distinct vehicle journey.<br>\n",
247-
"**observed_points**: Represents the original GPS trajectories. These are the linestrings formed from the raw GPS points collected during each vehicle journey.<br>\n",
248-
"**matched_points**: The processed trajectories post map-matching. These linestrings are aligned onto the actual road network, correcting for any GPS inaccuracies.<br>\n",
249-
"**matched_nodes**: A list of node identifiers from the road network that the matched trajectory passes through. These nodes correspond to intersections, turns, or other significant points in the road network.\n",
250-
"<br><br>"
251-
]
252-
},
253-
{
254-
"cell_type": "code",
255-
"execution_count": null,
256-
"id": "8707a991-d327-4c90-88e3-7e4df3e46128",
257-
"metadata": {},
258-
"outputs": [],
259-
"source": [
260-
"dfMmResult.show()"
172+
"matched_routes_df.show()"
261173
]
262174
},
263175
{
264176
"cell_type": "markdown",
265177
"id": "2f54472c-d309-4e95-ae8a-44a5a61a0e5b",
266178
"metadata": {},
267179
"source": [
268-
"## Visualize the result using SedonaKepler"
269-
]
270-
},
271-
{
272-
"cell_type": "code",
273-
"execution_count": null,
274-
"id": "6440a667-09b0-41c0-b53d-d44cfa0b8306",
275-
"metadata": {},
276-
"outputs": [],
277-
"source": [
278-
"with open('assets/conf/map_config.json', 'r') as file:\n",
279-
" map_config = json.load(file)"
280-
]
281-
},
282-
{
283-
"cell_type": "code",
284-
"execution_count": null,
285-
"id": "edb95cfd-4dcb-4e94-bb8e-b6dfcc198adc",
286-
"metadata": {},
287-
"outputs": [],
288-
"source": [
289-
"mapAll = SedonaKepler.create_map()\n",
180+
"## Visualize the result using SedonaKepler\n",
290181
"\n",
291-
"SedonaKepler.add_df(mapAll, dfEdge, name=\"Road Network\")\n",
292-
"SedonaKepler.add_df(mapAll, dfMmResult.selectExpr(\"observed_points AS geometry\"), name=\"Observed Points\")\n",
293-
"SedonaKepler.add_df(mapAll, dfMmResult.selectExpr(\"matched_points AS geometry\"), name=\"Matched Points\")\n",
294-
"mapAll.config = map_config\n",
295-
"\n",
296-
"mapAll"
297-
]
298-
},
299-
{
300-
"cell_type": "markdown",
301-
"id": "adf3d9e6-40fd-4d51-912e-33babe98f90e",
302-
"metadata": {},
303-
"source": [
304-
"<br>In this visualization, we are focusing on displaying the data corresponding to 'id' value 2. To visualize data for a different 'id' value, simply change the filter condition to the desired 'id' value.\n",
305-
"<br><br>"
182+
"The `map_config.json` file specifies the bounding box and how to draw the road network and the source and matched routes."
306183
]
307184
},
308185
{
309186
"cell_type": "code",
310187
"execution_count": null,
311-
"id": "49d05638-b35e-4079-83ad-ba433b7d5fc9",
188+
"id": "edb95cfd-4dcb-4e94-bb8e-b6dfcc198adc",
312189
"metadata": {},
313190
"outputs": [],
314191
"source": [
315-
"mapFil = SedonaKepler.create_map()\n",
192+
"with open('assets/conf/map_config.json', 'r') as file:\n",
193+
" map_config = json.load(file)\n",
194+
" \n",
195+
"viz = SedonaKepler.create_map()\n",
316196
"\n",
317-
"SedonaKepler.add_df(mapFil, dfEdge, name=\"Road Network\")\n",
318-
"SedonaKepler.add_df(mapFil, dfMmResult.filter(dfMmResult['ids']==2).selectExpr(\"observed_points AS geometry\"), name=\"Observed Points\")\n",
319-
"SedonaKepler.add_df(mapFil, dfMmResult.filter(dfMmResult['ids']==2).selectExpr(\"matched_points AS geometry\"), name=\"Matched Points\")\n",
320-
"mapFil.config = map_config\n",
197+
"SedonaKepler.add_df(viz, roads_df.select(\"geometry\"), name=\"Road Network\")\n",
198+
"SedonaKepler.add_df(viz, matched_routes_df.selectExpr(\"observed_points AS geometry\", \"ids AS trip_id\"), name=\"Observed Points\")\n",
199+
"SedonaKepler.add_df(viz, matched_routes_df.selectExpr(\"matched_points AS geometry\", \"ids AS trip_id\"), name=\"Matched Points\")\n",
200+
"viz.config = map_config\n",
321201
"\n",
322-
"mapFil"
202+
"viz"
323203
]
324204
}
325205
],

0 commit comments

Comments
 (0)