System Architecture

User Services Online (USO) is a modular, high-performance information management portal developed by the Canadian Light Source (CLS). It is designed to administer operations at a large-scale scientific research facility, managing user proposals, experiments, sample safety approvals, scheduling shifts on beamlines, tracking research publications, and compiling user feedback surveys.

This document provides a comprehensive analysis, visualization, and documentation of the USO system architecture, serving as a primary reference for system architects, developers, and onboarding engineers.

1. System Context Diagram (C4 Level 1)

This diagram shows the high-level boundary of the USO application, how different personas interact with it, and the external dependencies integrated into the platform.

        flowchart TD
    subgraph Users ["Facility Personas"]
        user["Facility User (Scientist)<br/><i>Submits proposals, samples, agreements</i>"]
        reviewer["Scientific Reviewer<br/><i>Reviews & scores proposals</i>"]
        staff["Facility Staff / User Office<br/><i>Schedules shifts, reviews safety</i>"]
        admin["System Administrator<br/><i>Configures cycles & settings</i>"]
    end

    uso["User Services Online (USO)<br/><b>[Core Web Portal System]</b>"]

    subgraph External ["External Integrations"]
        cas["Central Authentication Service (CAS)<br/><i>Single Sign-On</i>"]
        peopledir["People Directory API<br/><i>Synchronizes user roles & permissions</i>"]
        email["SMTP Email Service<br/><i>Sends notifications</i>"]
        crossref["Crossref / Open Citations API<br/><i>Fetches publication metadata</i>"]
        weather["OpenWeather API<br/><i>Provides weather observations</i>"]
    end

    user -->|Submits proposals & samples| uso
    reviewer -->|Reviews & scores proposals| uso
    staff -->|Schedules beamtime & reviews safety| uso
    admin -->|Configures settings & cycles| uso

    uso -->|Authenticates users via CAS Protocol| cas
    uso -->|Syncs accounts, roles, & perms via REST/JSON| peopledir
    uso -->|Triggers outbound emails via SMTP| email
    uso -->|Resolves DOIs & citations via REST/JSON| crossref
    uso -->|Fetches weather info via REST/JSON| weather

2. Container Diagram (C4 Level 2)

This diagram breaks down the technology choices, service boundaries, and data storage solutions within the USO ecosystem.

        flowchart TD
    actor["Portal Users<br/><i>Researchers, Reviewers, Staff, Admins</i>"]
    cas["CAS Service Provider<br/><i>External Single Sign-On</i>"]
    peopledir["People Directory API<br/><i>External Roles & Perms Sync</i>"]

    subgraph app_boundary ["USO Production Boundary"]
        web_ui["Web Portal Frontend<br/><i>(Bootstrap 5, jQuery, HTML/CSS/JS)</i>"]
        django_app["Django Core Application Server<br/><i>(Python 3, Django 5.x)</i>"]
        isocron["Isocron Background Scheduler<br/><i>(Django Command CLI)</i>"]
        database[("Relational Database<br/><i>(PostgreSQL / SQLite)</i>")]
    end

    actor -->|Accesses via HTTPS| web_ui
    web_ui -->|HTTP/AJAX requests| django_app
    django_app -->|Redirects for SSO authentication| cas
    django_app -->|Queries/updates user data| peopledir
    django_app -->|Reads/Writes SQL| database
    isocron -->|Queries task queue & writes logs| database
    django_app -.->|Monitors/triggers tasks via database logs| isocron

3. Modular Monolith Component Directory (C4 Level 3)

USO is architected as a decoupled modular monolith inside a single Django project. Domain boundaries are strictly separated into discrete Django applications placed under the apps/ directory.

The diagram below illustrates how components interact, showing the flow from authentication through proposal submission, project creation, sample safety clearance, scheduling, and finally to publication mapping.

        graph TD
    subgraph Authentication & Core Shared Layers
        users[users app: Custom User & Institution Profile]
        roleperms[roleperms app: Dynamic Regex Roles/Permissions]
        misc[misc app: Shared GFK, DateSpan, and Attachment Mixins]
        notifier[notifier app: Email & Notification Dispatcher]
    end

    subgraph Scientific Operations Lifecycle
        proposals[proposals app: Submissions, Review Cycles & Scores]
        projects[projects app: Approved Proposals, Sessions & Materials]
        beamlines[beamlines app: Facilities, Labs & Workspaces]
        samples[samples app: Sample Hazards, GHS Pictograms & GHS H/P Statements]
        scheduler[scheduler app: Beamtime Event Scheduling & Shifts]
        agreements[agreements app: User Agreements & Acceptance Checklists]
        publications[publications app: Research Paper Citation Mapping]
        surveys[surveys app: Post-run User Feedback Surveys]
    end

    users --> roleperms
    proposals --> users
    proposals --> beamlines
    projects --> proposals
    projects --> samples
    projects --> beamlines
    scheduler --> projects
    scheduler --> beamlines
    agreements --> users
    publications --> beamlines
    publications --> users
    surveys --> beamlines
    surveys --> users

Module Breakdown

Here is a detailed catalogue of the 14 custom applications defined within apps/:

Module

Core Purpose

Primary Models / Classes

Key Integrations

users

Manages custom user authentication profiles, contact details, and institutional directories.

User, Institution, Address, Country, Region

Syncs user details and institutional data with the external “People Directory” API.

roleperms

Implements fine-grained role-based access control (RBAC) extending Django permissions.

RolePermsUserMixin

Validates roles & permissions synchronized from the “People Directory”.

proposals

Coordinates the entire research proposal submission and peer review lifecycle.

Proposal, Submission, ReviewCycle, Reviewer, ReviewStage, Review

Dynamic form definition via django-dynforms.

projects

Holds approved proposal states, experimental sessions, safety materials, and allocations.

Project, Material, Session, LabSession, Allocation, ShiftRequest

Links projects back to cycles, teams, and beamlines.

beamlines

Defines structural models for the facility’s experimental setups and physical footprint.

Facility (beamline/sector/department), Lab, LabWorkSpace, Ancillary

Tracks spot size, flux, and support details.

samples

Controls experimental sample declarations, chemical hazards, and GHS indicators.

Sample, Hazard, HStatement, PStatement, Pictogram, SafetyPermission

Evaluates sample hazard classes against safety guides.

scheduler

Governs the planning grids, shift durations, operating modes, and allocation bounds.

ShiftConfig, Schedule, Event, Mode, ModeType

Combines with projects allocations to schedule runs.

agreements

Handles user legal/safety acknowledgments, tracking digital acceptance signatures.

Agreement, Acceptance

Records timestamp, signature hash, and remote IP.

notifier

Manages reusable communication templates and asynchronous email transmissions.

Notification, MessageTemplate

Supports background queueing for notification delivery.

isocron

Scheduled background task execution wrapper mimicking cron logs.

BackgroundTask, TaskLog

Runs loops for cleanup, syncing, and reminders.

publications

Monitors publication listings originating from experiments conducted at the facility.

Publication, Journal, JournalMetric, FundingSource, ArticleMetric

Queries Crossref / Open Citations API.

surveys

Gathers operational feedback from user groups at the close of their experiment runs.

Feedback, Rating, Category

Rendered via django-dynforms custom templates.

weather

Caches local weather patterns to aid in monitoring local outdoor setups.

Weather

Periodically hits OpenWeather API.

misc

Offers centralized utility tools, Generic Foreign Key (GFK) mappings, and activity logs.

Clarification, Attachment, ActivityLog, DateSpanMixin

Core mixin dependencies across all apps.

4. Key Architectural Design Patterns & Features

1. Extended Role-Based Access Control (RBAC) System

Unlike standard Django apps which rely on many-to-many lookup tables for permissions (auth_user_groups), USO stores list arrays representing role hierarchies directly on the custom User model using database JSON fields:

  • Core mixin: RolePermsUserMixin (located in [models.py](file:////apps/roleperms/models.py))

  • Permission verification uses regular expressions (roles__iregex) to evaluate cascading role templates (e.g., matching wildcard facility roles like staff:* or specific facility levels like staff:contracts).

  • Check helpers include custom class view decorators: AdminRequiredMixin, StaffRequiredMixin, and OwnerRequiredMixin (found in [views.py](file:////apps/roleperms/views.py)).

2. Dynamically Discovered Applications Routing

To prevent monolithic route listings, the primary URL dispatcher [urls.py](file:////apps/usonline/urls.py) implements the dynamic iterator utility iterload (from [utils.py](file:////apps/misc/utils.py)).

  • This automatically searches settings INSTALLED_APPS directories for any modules exposing api_urls.py or user_urls.py, auto-routing them onto namespaces /api/v1/ and /user/ respectively.

3. Deep Core Framework Dependencies

USO heavily leverages specialized custom libraries written specifically for the system’s modular architecture:

  • django-dynforms: For dynamic questionnaire definition. Proposals, reviews, and surveys do not have static hardcoded HTML inputs; instead, their schemas are defined dynamically in database models and mapped to forms.

  • django-itemlist: Standardizes all listing dashboards with uniform search bars, pagination, dynamic CSV export links, and filter lists.

  • django-crisp-modals: Wraps Bootstrap 5 inputs inside floating overlay dialog fields without requiring manual JavaScript trigger implementations on standard actions.

  • django-reportcraft: Facilitates advanced database reports and SQL aggregation queries (such as counting shifts or mapping age groups).

5. Developer Onboarding Quick-Start

Local Environment Setup

To get a local instance of USO running, execute the following steps in your terminal:

# 1. Clone the repository and navigate into it
git clone https://github.com/michel4j/uso.git
cd uso

# 2. Set up a Python virtual environment and activate it
python -m venv .venv
source .venv/bin/activate

# 3. Install packages
pip install -r requirements.txt

# 4. Prepare local folder layouts, configuration templates, and database
# This script copies the settings template, makes static directories, and runs migrations
./deploy/prepare-instance.sh

# 5. Populate local development database with Faker-generated data
python deploy/generate-data.py

# 6. Start the local server
python manage.py runserver

The application will be accessible at: http://localhost:8000/.