Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Mobility Feed API - AI Coding Assistant Instructions
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file adds context to copilot and other AI agents.


## Project Architecture

This is a **mobility data API service** built with FastAPI, serving open mobility data from across the world. The architecture follows a **code-generation pattern** with clear separation between generated and implementation code.

### Core Components

- **`api/`**: Main FastAPI application with spec-first development using OpenAPI Generator
- **`functions-python/`**: Google Cloud Functions for data processing (batch jobs, validation, analytics)
- **`web-app/`**: Frontend application
- **PostgreSQL + PostGIS**: Database with geospatial support for mobility data

### Key Generated vs Implementation Split

- **Generated code** (never edit): `api/src/feeds_gen/` and `api/src/shared/database_gen/`
- **Implementation code**: `api/src/feeds/impl/` contains actual business logic
- **Schema source**: `docs/DatabaseCatalogAPI.yaml` drives code generation

## Critical Development Workflows
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to copilot itself: Keep .github/copilot-instructions.md short and link to human‑oriented docs in the repo.
Maybe this section should be linked to https://github.com/MobilityData/mobility-feed-api?tab=readme-ov-file#installation--usage?
I don't really know enough about this to say if it makes a big difference.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this came after the merge. We should keep enhancing the copilot-instructions files as we learn more about AI's interactions with our codebase. Let's create a PR with this suggestion.


### Initial Setup
```bash
# One-time OpenAPI setup
scripts/setup-openapi-generator.sh

# Install dependencies
cd api && pip3 install -r requirements.txt -r requirements_dev.txt

# Start local database
docker-compose --env-file ./config/.env.local up -d --force-recreate

# Generate API stubs (run after schema changes)
scripts/api-gen.sh
scripts/db-gen.sh
```

### Common Development Commands
```bash
# Start API server (includes Swagger UI at http://localhost:8080/docs/)
scripts/api-start.sh

# Run tests with coverage
scripts/api-tests.sh
# Run specific test file
scripts/api-tests.sh my_test_file.py

# Lint checks (Flake8 + Black)
scripts/lint-tests.sh

# Reset and populate local database
./scripts/docker-localdb-rebuild-data.sh --populate-db
# Include test datasets
./scripts/docker-localdb-rebuild-data.sh --populate-db --populate-test-data
```

## Project-Specific Patterns

### Error Handling Convention
- Use `shared.common.error_handling.InternalHTTPException` for internal errors
- Convert to FastAPI HTTPException using `feeds.impl.error_handling.convert_exception()`
- Store error messages as Finals in `api/src/feeds/impl/error_handling.py`
- Error responses follow: `{"details": "The error message"}`

### Database Patterns
- **Polymorphic inheritance**: `Feed` base class with `GtfsFeed`, `GbfsFeed`, `GtfsRTFeed` subclasses
- **SQLAlchemy ORM**: Models in `shared/database_gen/sqlacodegen_models.py` (generated)
- **Session management**: Use `@with_db_session` decorator for database operations
- **Unique IDs**: Generate with `generate_unique_id()` (36-char UUID4)

### API Implementation Structure
- Endpoints in `feeds/impl/*_api_impl.py` extend generated base classes from `feeds_gen/`
- Filter classes in `shared/feed_filters/` for query parameter handling
- Model implementations in `feeds/impl/models/` extend generated models

### Code Generation Workflow
1. Modify `docs/DatabaseCatalogAPI.yaml` for API changes
2. Run `scripts/api-gen.sh` to regenerate FastAPI stubs
3. Run `scripts/db-gen.sh` for database schema changes
4. Implement business logic in `feeds/impl/` classes

### Testing Patterns
- Tests use empty local test DB (reset with `--use-test-db` flag)
- Coverage reports in `scripts/coverage_reports/`
- Python path configured to `src/` in `pyproject.toml`

### Functions Architecture
- **Google Cloud Functions** in `functions-python/` for background processing
- Shared database models via `database_gen/` symlink
- Each function has its own deployment configuration
- Tasks include: validation reports, batch datasets, GBFS validation, BigQuery ingestion

### Authentication
- **OAuth2 Bearer tokens** for API access
- Refresh tokens from mobilitydatabase.org account
- Access tokens valid for 1 hour
- Test endpoint: `/v1/metadata` with Bearer token

## Integration Points

- **BigQuery**: Analytics data pipeline via `big_query_ingestion/` function
- **PostGIS**: Geospatial queries for location-based feed filtering
- **Liquibase**: Database schema migrations in `liquibase/` directory
- **Docker**: Multi-service setup with PostgreSQL, test DB, and schema documentation

## File Exclusions for AI Context
- Skip `src/feeds_gen/*` and `src/shared/database_gen/*` (generated code)
- Skip `data/` and `data-test/` (database volumes)
- Skip `htmlcov/` (coverage reports)
- Black formatter excludes these paths automatically
2 changes: 1 addition & 1 deletion .github/workflows/api-deployer.yml
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ jobs:
- uses: actions/download-artifact@v4
with:
name: feeds_operations_gen
path: functions-python/operations_api/src/feeds_operations_gen/
path: functions-python/operations_api/src/feeds_gen/

- name: Build python functions
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,5 +140,5 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: feeds_operations_gen
path: functions-python/operations_api/src/feeds_operations_gen/
path: functions-python/operations_api/src/feeds_gen/
overwrite: true
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -80,4 +80,7 @@ tf.plan
functions-python/**/*.csv

# Local emulators
.cloudstorage
.cloudstorage

# Project files
*.code-workspace
2 changes: 1 addition & 1 deletion api/src/shared/db_models/gtfs_dataset_impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from shared.db_models.bounding_box_impl import BoundingBoxImpl
from shared.db_models.validation_report_impl import ValidationReportImpl
from feeds_gen.models.gtfs_dataset import GtfsDataset
from utils.model_utils import compare_java_versions
from shared.db_models.model_utils import compare_java_versions


class GtfsDatasetImpl(GtfsDataset):
Expand Down
6 changes: 3 additions & 3 deletions api/src/shared/db_models/gtfs_rt_feed_impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def from_orm(cls, feed: GtfsRTFeedOrm | None) -> GtfsRTFeed | None:
gtfs_rt_feed: GtfsRTFeed = super().from_orm(feed)
if not gtfs_rt_feed:
return None
gtfs_rt_feed.locations = [LocationImpl.from_orm(item) for item in feed.locations]
gtfs_rt_feed.entity_types = [item.name for item in feed.entitytypes]
gtfs_rt_feed.feed_references = [item.stable_id for item in feed.gtfs_feeds]
gtfs_rt_feed.locations = [LocationImpl.from_orm(item) for item in feed.locations] if feed.locations else []
gtfs_rt_feed.entity_types = [item.name for item in feed.entitytypes] if feed.entitytypes else []
gtfs_rt_feed.feed_references = [item.stable_id for item in feed.gtfs_feeds] if feed.gtfs_feeds else []
return gtfs_rt_feed
2 changes: 1 addition & 1 deletion api/src/shared/db_models/latest_dataset_impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from shared.db_models.bounding_box_impl import BoundingBoxImpl
from feeds_gen.models.latest_dataset import LatestDataset
from feeds_gen.models.latest_dataset_validation_report import LatestDatasetValidationReport
from utils.model_utils import compare_java_versions
from shared.db_models.model_utils import compare_java_versions


class LatestDatasetImpl(LatestDataset):
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion api/tests/utils/test_compare_java_versions.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import unittest
from utils.model_utils import compare_java_versions
from shared.db_models.model_utils import compare_java_versions


class TestCompareJavaVersions(unittest.TestCase):
Expand Down
Loading