* feat: calendar cache and sync - wip * Add env.example * refactor on CalendarCacheEventService * remove test console.log * Fix type checks errors * chore: remove pt comment * add route.ts * chore: fix tests * Improve cache impl * chore: update recurring event id * chore: small improvements * calendar cache improvements * Fix remove dynamic imports * Add cleanup stale cache * Fix tests * add event update * type fixes * feat: add comprehensive tests for new calendar subscription API routes - Add tests for /api/cron/calendar-subscriptions-cleanup route (9 tests) - Add tests for /api/cron/calendar-subscriptions route (10 tests) - Add tests for /api/webhooks/calendar-subscription/[provider] route (11 tests) - Add missing feature flags for calendar-subscription-cache and calendar-subscription-sync - All 30 tests pass with comprehensive coverage of authentication, feature flags, error handling, and service instantiation Tests cover: - Authentication scenarios (API key validation, Bearer tokens, query parameters) - Feature flag combinations (cache/sync enabled/disabled states) - Success and error handling (including non-Error exceptions) - Service instantiation with proper dependency injection - Provider validation for webhook endpoints Co-Authored-By: Volnei Munhoz <volnei.munhoz@gmail.com> * feat: add comprehensive tests for calendar subscription services, repositories, and adapters - Add unit tests for CalendarSubscriptionService with subscription, webhook, and event processing - Add unit tests for CalendarCacheEventService with cache operations and cleanup - Add unit tests for CalendarSyncService with Cal.com event filtering and booking operations - Add unit tests for CalendarCacheEventRepository with CRUD operations - Add unit tests for SelectedCalendarRepository with calendar selection management - Add unit tests for GoogleCalendarSubscriptionAdapter with subscription and event fetching - Add unit tests for Office365CalendarSubscriptionAdapter with placeholder implementation - Add unit tests for AdaptersFactory with provider management and adapter creation - Fix lint issues by removing explicit 'any' type casting and unused variables - All tests follow Cal.com conventions using Vitest framework with proper mocking Co-Authored-By: Volnei Munhoz <volnei.munhoz@gmail.com> * fix: improve calendar-subscriptions-cleanup test performance by adding missing mocks - Add comprehensive mocks for defaultResponderForAppDir, logger, performance monitoring, and Sentry - Fix slow test execution (933ms -> <100ms) caused by missing dependency mocks - Ensure consistent test performance across different environments Co-Authored-By: Volnei Munhoz <volnei.munhoz@gmail.com> * Fix tests * Fix tests * type fix * Fix coderabbit comments * Fix types * Fix test * Update apps/web/app/api/cron/calendar-subscriptions/route.ts Co-authored-by: Alex van Andel <me@alexvanandel.com> * Fixes by first review * feat: add database migrations for calendar cache and sync fields - Add CalendarCacheEventStatus enum with confirmed, tentative, cancelled values - Add new fields to SelectedCalendar: channelId, channelKind, channelResourceId, channelResourceUri, channelExpiration, syncSubscribedAt, syncToken, syncedAt, syncErrorAt, syncErrorCount - Create CalendarCacheEvent table with foreign key to SelectedCalendar - Add necessary indexes and constraints for performance and data integrity Fixes database schema issues causing e2e test failures with 'column does not exist' errors. Co-Authored-By: Volnei Munhoz <volnei.munhoz@gmail.com> * only google-calendar for now * docs: add Calendar Cache and Sync feature documentation - Add comprehensive feature overview and motivation - Document feature flags with SQL examples - Include SQL examples for enabling features for users and teams - Reference technical documentation files Addresses PR #23876 documentation requirements Co-Authored-By: Volnei Munhoz <volnei.munhoz@gmail.com> * docs: update calendar subscription README with comprehensive documentation - Undo incorrect changes to main README.md - Update packages/features/calendar-subscription/README.md with: - Feature overview and motivation - Environment variables section - Complete feature flags documentation with SQL examples - SQL examples for enabling features for users and teams - Detailed architecture documentation Addresses PR #23876 documentation requirements Co-Authored-By: Volnei Munhoz <volnei.munhoz@gmail.com> * fix docs * Fix test to available calendars * Fix test to available calendars * add migration and sync boilerplate * fix typo * remove double log * sync boilerplate --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Alex van Andel <me@alexvanandel.com> Co-authored-by: Keith Williams <keithwillcode@gmail.com>
116 lines
5.4 KiB
Markdown
116 lines
5.4 KiB
Markdown
# Calendar Cache and Sync
|
|
|
|
The **Calendar Cache and Sync** feature provides efficient calendar synchronization with intelligent caching to reduce API calls and ensure real-time updates across your Cal.com instance.
|
|
|
|
## Feature Overview
|
|
|
|
This feature introduces two complementary capabilities:
|
|
|
|
- **Calendar Cache**: Stores calendar availability data locally to reduce external API calls and improve performance
|
|
- **Calendar Sync**: Uses webhooks to automatically listen for calendar updates and apply changes in real-time
|
|
|
|
**Key Benefits:**
|
|
- **Efficiency**: Reduces API calls with optimized caching strategies
|
|
- **Reliability**: Guarantees updates through webhook event delivery
|
|
- **Real-Time Sync**: Ensures calendars are always up-to-date with minimal latency
|
|
- **Scalability**: Supports multiple calendars and handles high-volume updates seamlessly
|
|
|
|
**Motivation:**
|
|
By subscribing to calendars via webhooks and implementing intelligent caching, you gain a smarter, faster, and more resource-friendly way to keep your data in sync. This eliminates the need for constant polling and reduces the load on external calendar APIs while ensuring data consistency.
|
|
|
|
## Environment Variables
|
|
|
|
- **GOOGLE_WEBHOOK_URL**: Optional, only used for local tests. Default points to application url.
|
|
- **GOOGLE_WEBHOOK_TOKEN**: Required, token to validate Google Webhook incoming.
|
|
- **MICROSOFT_WEBHOOK_URL**: Optional, only used for local tests. Default points to application url.
|
|
- **MICROSOFT_WEBHOOK_TOKEN**: Required, token to validate Microsoft Webhook incoming.
|
|
|
|
## Feature Flags
|
|
|
|
This feature is controlled by three feature flags that can be enabled independently:
|
|
|
|
### 1. calendar-subscription-cache
|
|
Enables calendar cache recording and usage through calendars. This flag should be managed individually by teams.
|
|
|
|
```sql
|
|
INSERT INTO "Feature" ("slug", "enabled", "description", "type", "stale", "lastUsedAt", "createdAt", "updatedAt", "updatedBy")
|
|
VALUES ('calendar-subscription-cache', false, 'Allow calendar cache to be recorded and used through calendars.', 'OPERATIONAL', false, NULL, NOW(), NOW(), NULL)
|
|
ON CONFLICT (slug) DO NOTHING;
|
|
```
|
|
|
|
### 2. calendar-subscription-sync
|
|
Enables calendar sync globally for all users regardless of team or organization.
|
|
|
|
```sql
|
|
INSERT INTO "Feature" ("slug", "enabled", "description", "type", "stale", "lastUsedAt", "createdAt", "updatedAt", "updatedBy")
|
|
VALUES ('calendar-subscription-sync', false, 'Enable calendar sync for all calendars globally.', 'OPERATIONAL', false, NULL, NOW(), NOW(), NULL)
|
|
ON CONFLICT (slug) DO NOTHING;
|
|
```
|
|
|
|
## Enabling Features for Specific Users
|
|
|
|
To enable calendar cache features for specific users, add entries to the `UserFeatures` table:
|
|
|
|
```sql
|
|
-- Enable calendar-subscription-cache for user ID 123
|
|
INSERT INTO "UserFeatures" ("userId", "featureId", "assignedAt", "assignedBy", "updatedAt")
|
|
VALUES (123, 'calendar-subscription-cache', NOW(), 'admin', NOW())
|
|
ON CONFLICT ("userId", "featureId") DO NOTHING;
|
|
|
|
-- Enable calendar-subscription-sync for user ID 123
|
|
INSERT INTO "UserFeatures" ("userId", "featureId", "assignedAt", "assignedBy", "updatedAt")
|
|
VALUES (123, 'calendar-subscription-sync', NOW(), 'admin', NOW())
|
|
ON CONFLICT ("userId", "featureId") DO NOTHING;
|
|
```
|
|
|
|
## Enabling Features for Specific Teams
|
|
|
|
To enable calendar cache features for specific teams, add entries to the `TeamFeatures` table:
|
|
|
|
```sql
|
|
-- Enable calendar-subscription-cache for team ID 456
|
|
INSERT INTO "TeamFeatures" ("teamId", "featureId", "assignedAt", "assignedBy", "updatedAt")
|
|
VALUES (456, 'calendar-subscription-cache', NOW(), 'admin', NOW())
|
|
ON CONFLICT ("teamId", "featureId") DO NOTHING;
|
|
|
|
-- Enable calendar-subscription-sync for team ID 456
|
|
INSERT INTO "TeamFeatures" ("teamId", "featureId", "assignedAt", "assignedBy", "updatedAt")
|
|
VALUES (456, 'calendar-subscription-sync', NOW(), 'admin', NOW())
|
|
ON CONFLICT ("teamId", "featureId") DO NOTHING;
|
|
```
|
|
|
|
## Architecture
|
|
|
|
The calendar cache and sync system consists of several key components:
|
|
|
|
### Database Schema
|
|
- **CalendarCacheEvent Table**: Stores cached calendar events with status tracking
|
|
- **SelectedCalendar Extensions**: Additional fields for sync state and webhook management including:
|
|
- `channelId`: Webhook channel identifier
|
|
- `channelResourceId`: Resource ID for webhook subscriptions
|
|
- `channelResourceUri`: URI for webhook notifications
|
|
- `channelKind`: Type of webhook channel
|
|
- `channelExpiration`: Webhook subscription expiration time
|
|
- `syncToken`: Token for incremental sync
|
|
- `syncedAt`: Last successful sync timestamp
|
|
- `syncErrorAt`: Last sync error timestamp
|
|
- `syncErrorCount`: Number of consecutive sync errors
|
|
- `syncSubscribedAt`: Webhook subscription timestamp
|
|
|
|
### Core Services
|
|
- **CalendarCacheEventRepository**: Manages cached event storage and retrieval
|
|
- **CalendarSubscriptionService**: Orchestrates webhook subscriptions and event processing
|
|
- **Provider-specific Adapters**: Handle calendar-specific sync logic (Google, Office365)
|
|
|
|
### Background Processes
|
|
- **Cron Jobs**: Automated processes for cache cleanup and calendar watching
|
|
- **Webhook Handlers**: Real-time event processing for calendar updates
|
|
|
|
### Integration Points
|
|
- **Calendar Providers**: Google Calendar, Office365, and other supported integrations
|
|
- **Webhook Endpoints**: Receive real-time notifications from calendar providers
|
|
- **Cache Layer**: Optimized storage for frequently accessed calendar data
|
|
|
|
For detailed technical implementation, see:
|
|
- Database migrations in `packages/prisma/migrations/`
|