Skip to main content

Persistence

Subato stores structured data (e.g. tasks and exercises) in PostgreSQL. Semi-structured data, such as solutions and course files, is stored using a combination of PostgreSQL and the file system. Thus, metadata that benefits from relational access (e.g. a solution's submission timestamp) is stored in PostgreSQL while the actual file contents are stored on disk.

File System Structure​

Subato uses two directories for file-based storage:

  • subato2-data: persistent application data, such as solution files.
  • subato2-tmp: temporary data that can be easily recreated, such as local copies of Git repositories used by task pools.

subato2-data​

- subato2-data
    - solutions
        - 2023
            - 01
                - 05
                    - 717
                        - First.java
                        - Second.java
    - task
        - 1
            - files
            - text
            - meta.xml
        - 2
    - courseFile
        - 1
            - Test.pdf
        - 2
    - bundle
        - 1.zip
        - 2.zip

Each top-level directory corresponds to a different entity type. For tasks (task) and course files (courseFile), files are stored in directories named after the corresponding database ID of the entities. Solutions (solutions) are organized by submission timestamp and their ID. This avoids a single directory containing potentially thousands of submissions, which could make manually inspecting the file system infeasible. Moreover, each task has a generated bundle (bundle) that packages all associated files for download into a zip archive. Bundle files are named after the task's ID in the database.

There is currently no consistency check on the subato2-data directory. Thus, a directory may exist even if the corresponding database record has been removed. While Subato ensures that the subato2-data is updated when entities are deleted from the database, there may be scenarios where inconsistencies may emerge. For example, this may occur when the database is reset manually.

subato2-tmp​

- subato2-tmp
    - pool
        - 1
            - ...
    - 31f3f56f-ae9c-475c-9ab7-76057f0816fc
    - ...

Task pool repositories are stored under pool, but the imported tasks themselves live in the subato2-data/task directory. Thus, these pool directories are just a cache/copy of the repository and can be deleted. If a pool directory is deleted, the next synchronization recreates it automatically.

The remaining directories are named with randomly generated UUIDs and are used for temporary checks. For example, when validating a task pool repository URL, Subato clones the repository into a temporary directory before checking it.

File Tree Cache​

Subato maintains a cached representation of the file hierarchy for unstructured data stored on the file system. The cache is stored as a tree.json file inside relevant directories under subato2-data. It allows efficient file lookup and enables features such as displaying the structure of solutions containing nested directories and multiple files.

Database / ORM​

Database access follows the repository pattern, with repository implementations provided by Spring Data JPA. For each entity that serves as an aggregate root, define an <Entity>Repository interface that extends JpaRepository. If more fine-granular access is required beyond that what Spring Data JPA can express, define an <Entity>CustomRepository interface containing the required methods and implement them in a <Entity>RepositoryImpl class. Then, this class may inject <Entity>Repository to implement the required methods.

Validation​

Entities should be annotated with constraints such as @Size and @NotBlank from the jakarta.validation.constraints package. As the project was configured to disable automatic validation on entity changes, validation must be performed manually by injecting jakarta.validation.Validator and calling its validate method. When there are any violations, a ValidationException should be thrown. For each violation, pass a reference to the field that caused the error and the automatically generated error message returned by validate to the exception object. Then, this will send an appropriate error response to clients which allows them to display the violations.

Migrations​

The database schema evolves through migrations defined and run with Liquibase, which should be placed under src/main/resources/db. The changelog directory contains the migration history, organized into <year>/<month> subdirectories.

A new migration should be created whenever entity definitions change. Each migration should:

  • reflect the corresponding entity changes
  • define appropriate default values where required
  • ensure that production databases can be upgraded during deployment

For IntelliJ, the JPA-Buddy plugin can assist with generating migrations. Alternatively, migrations can be generated automatically using Claude Code.

"Waiting for changelog lock...."

If you interrupt the backend during startup while migrations are running, the lock on the database may not be released. In that case, run the following query manually:

UPDATE DATABASECHANGELOGLOCK SET LOCKED=false, LOCKGRANTED=null, LOCKEDBY=null where ID=1;

Specifications and Dynamic Queries​

Spring's Specification API allows queries to be composed dynamically. This is especially useful for implementing filters and reusing query logic across repositories. However, specifications normally reference entity attributes through strings, which causes invalid references to be only detected at runtime. Metamodels provide statically typed alternatives. Instead of referring to attributes by name, queries use generated classes, allowing invalid references to be detected at compile time. Thus, the project was setup to automatically generate metamodel classes using the JPA Static Model Generator during compilation. For an entity class named <Entity>, the associated metamodel class is generated as <Entity>_.

To use the Specification API, extend the repository interface of an entity with JpaSpecificationExecutor. Then, create an <Entity>Filter class and provide a builder for constructing filter instances with the builder pattern. The <Entity>Filter class should then provide a toSpecification() method for creating the Specification object based on the filter criteria. Filter objects can then be combined with a Sort object and the current PolicyContext into an <Entity>Query object, which allows custom repository methods to assemble a dynamic query. For an example, see CourseFilter, CourseQuery, CourseSpecifications, and CourseRepositoryImpl.