Docker Setup for ODAPI

This guide covers running ODAPI in Docker for both development and production environments.

Status: ✅ Fully functional local Docker development environment

Prerequisites
Development Setup
Architecture & Configuration
Production Setup
Common Operations
Troubleshooting
Known Issues & Solutions

Prerequisites

Docker Engine 20.10+ installed
Docker Compose 2.0+ installed
At least 4GB of RAM available for Docker

Check your versions:

docker --version
docker compose version

Development Setup

Quick Start

Environment file is already configured: The .env file is pre-configured with correct defaults. You can optionally set your INPI API key.
Build and start all services:
```
docker compose up -d --build
```
This will start:
- Redis 7 (port 6379)
- Elasticsearch 8.11 (port 9200)
- Rails API (port 3000)
- Sidekiq background processor
Note: PostgreSQL is NOT included in Docker. The application connects to your local development database configured in config/database.yml via Rails encrypted credentials.
Wait for services to be healthy (30-60 seconds):
```
docker compose ps
```
Look for (healthy) status on redis and elasticsearch.
Ensure your local PostgreSQL database is running with the primary database (e.g., odapi) configured in your Rails credentials.

Run database migrations (first time only):

docker compose exec rails bundle exec rails db:migrate

Seed essential data (first time only):

docker compose exec rails bundle exec rails db:seed

Verify everything is running:
```
curl http://localhost:3000/up
```
Should return: <body style="background-color: green"></body>
Access the application:
- API: http://localhost:3000
- Health check: http://localhost:3000/up
- Redis: localhost:6379
- Elasticsearch: http://localhost:9200
- PostgreSQL: Your local database (configured in Rails credentials)

Development Services

The development stack includes:

rails - Rails API server (port 3000)
sidekiq - Background job processor
redis - Redis cache and job queue
elasticsearch - Elasticsearch 8.11 search engine

External (not in Docker):

PostgreSQL - Your local development database (configured in Rails credentials)

Development Workflow

View logs:

# All services
docker compose logs -f

# Specific service
docker compose logs -f rails
docker compose logs -f sidekiq

Run Rails commands:

# Rails console
docker compose exec rails rails console

# Run migrations
docker compose exec rails rails db:migrate

# Run tests
docker compose exec rails bundle exec rspec

# Run RuboCop
docker compose exec rails bundle exec rubocop

Run Rake tasks:

# Background job tasks
docker compose exec rails rake parser_jobs:process[10]
docker compose exec rails rake parser_jobs:stats

# Cache management
docker compose exec rails rake cache:set_all
docker compose exec rails rake cache:status

Install new gems:

# After updating Gemfile
docker compose exec rails bundle install

# Or rebuild the container
docker compose up -d --build rails

Restart services:

# Restart all services
docker compose restart

# Restart specific service
docker compose restart rails
docker compose restart sidekiq

Stop services:

# Stop all services
docker compose down

# Stop and remove volumes (⚠️ deletes all data)
docker compose down -v

Code Changes

The development setup uses volume mounts, so code changes are reflected immediately without rebuilding:

Ruby code changes: Restart Rails server (docker compose restart rails)
Gem changes: Reinstall bundle (docker compose exec rails bundle install)
Database schema changes: Run migrations

Debugging

Attach to Rails for debugging:

# Stop detached Rails container first
docker compose stop rails

# Run Rails in foreground with attached terminal
docker compose run --rm --service-ports rails

Now you can use binding.pry or debugger in your code.

Architecture & Configuration

Multi-Database Setup

ODAPI uses a single PostgreSQL database:

Primary Database (e.g., odapi):
- Companies, reference data, statistics
- ParserJob records with status tracking and raw API data
- Configured in Rails encrypted credentials

Database Configuration

The application uses Rails encrypted credentials for database configuration in development:

# config/database.yml (development section)
host: <%= Rails.application.credentials.dig(:development, :database, :host) %>
database: <%= Rails.application.credentials.dig(:development, :database, :database) %>
username: <%= Rails.application.credentials.dig(:development, :database, :username) %>
password: nil  # or from credentials if needed

To configure your database:

rails credentials:edit --environment development

Important: The Docker containers connect to your local PostgreSQL database on your host machine, not a containerized database.

Redis Configuration

The application expects BR_REDIS_HOST environment variable in development:

# config/initializers/redis.rb
url = Rails.env.development? ? ENV["BR_REDIS_HOST"] : "redis://localhost:6379/2"

Docker Compose sets both:

BR_REDIS_HOST=redis://redis:6379/0 (for application code)
REDIS_URL=redis://redis:6379/0 (for Sidekiq)

Environment Variables

Environment variables are defined in .env and used by docker-compose.yml:

# Redis
REDIS_PORT=6379

# Elasticsearch
ELASTICSEARCH_PORT=9200

# Rails
RAILS_PORT=3000
RAILS_ENV=development

# # INPI API (optional - set your own key)
# INPI_API_KEY=your_api_key_here
# INPI_API_URL=https://api.gouv.fr/
# 

**Note:** Database credentials are NOT in `.env` - they're configured in Rails encrypted credentials (`config/credentials/development.yml.enc`). Edit with:
```bash
rails credentials:edit --environment development

PostgreSQL Version Notes

Important: This application uses your local PostgreSQL database (not containerized).

The db/structure.sql and db/parser_jobs_structure.sql files should be compatible with your PostgreSQL version. If using PostgreSQL 17+, features like SET transaction_timeout = 0; are supported. For older versions, you may need to remove unsupported commands from structure files.

Sidekiq Bootstrap Requirement

Sidekiq requires a sidekiq_cron_schedule setting in the database to start. This is automatically created by db/seeds.rb:

unless Setting.exists?(key: "sidekiq_cron_schedule")
  Setting.create!(
    key: "sidekiq_cron_schedule",
    value: "{}",
    value_type: "hash"
  )
end

Always run db:seed after creating a fresh database, or Sidekiq will crash on startup with:

ActiveRecord::RecordNotFound: Couldn't find Setting with [WHERE "settings"."key" = $1]

Docker Compose Service Dependencies

Services start in this order:

Infrastructure (redis, elasticsearch) - start first, wait for healthy
Rails - starts after infrastructure is healthy
Sidekiq - starts after infrastructure is healthy

Note: PostgreSQL is not managed by Docker Compose - ensure your local database is running before starting the stack.

All containerized services have health checks defined to ensure proper startup order.

Production Setup

Architecture

Production uses:

Traefik - Reverse proxy with automatic HTTPS (Let's Encrypt)
Rails (Puma) - API server
Sidekiq - Background jobs
Redis - Cache and queue
Elasticsearch - Search engine
External PostgreSQL (Neon DB)

Deployment

Build production image:

docker build -f Dockerfile.production -t odapi:latest .

Create production environment file:

cp .env.example .env.production
# Edit .env.production with production values

Start production stack:

docker compose -f docker-compose.production.yml --env-file .env.production up -d

Run database migrations:

docker compose -f docker-compose.production.yml exec rails rails db:migrate
docker compose -f docker-compose.production.yml exec rails rails db:migrate:parser_jobs

Production Environment Variables

Required variables in .env.production:

# Database URL (Neon)
NEON_PRIMARY_URL=postgresql://user:pass@host/odapi_production

# Security
RAILS_MASTER_KEY=your_master_key_from_config/master.key
SECRET_KEY_BASE=generate_with_rails_secret

# Redis (generated password)
REDIS_PASSWORD=your_secure_redis_password

# Elasticsearch (generated password)
ELASTIC_PASSWORD=your_secure_elastic_password


# Domain and SSL
DOMAIN=api.yourdomain.com
ACME_EMAIL=admin@yourdomain.com

Generate secure values:

# SECRET_KEY_BASE
docker compose -f docker-compose.production.yml run --rm rails rails secret

# Redis/Elastic passwords
openssl rand -base64 32

Common Operations

Database Operations

Reset development database: Since PostgreSQL is not in Docker, use standard Rails commands:

docker compose exec rails rails db:drop
docker compose exec rails rails db:create
docker compose exec rails rails db:migrate
docker compose exec rails rails db:seed

Create a database backup: Use your local PostgreSQL tools:

pg_dump -U your_username your_database_name > backup.sql

Restore from backup:

psql -U your_username your_database_name < backup.sql

Access PostgreSQL directly: Use your local psql client:

psql -U your_username your_database_name

Redis Operations

Access Redis CLI:

docker compose exec redis redis-cli

Clear all Redis data:

docker compose exec redis redis-cli FLUSHALL

Monitor Redis operations:

docker compose exec redis redis-cli MONITOR

Elasticsearch Operations

Check cluster health:

curl http://localhost:9200/_cluster/health?pretty

View indices:

curl http://localhost:9200/_cat/indices?v

Reindex from Rails:

docker compose exec rails rails searchkick:reindex:all

Monitoring

Check container health:

docker compose ps

View resource usage:

docker stats

View disk usage:

docker system df

Known Issues & Solutions

Issue: Sidekiq Crashes on Startup

Symptom:

ActiveRecord::RecordNotFound: Couldn't find Setting with [WHERE "settings"."key" = $1]

Cause: Missing sidekiq_cron_schedule setting in database.

Solution:

docker compose exec rails bundle exec rails db:seed
docker compose restart sidekiq

Issue: PostgreSQL Version Mismatch

Symptom:

ERROR:  unrecognized configuration parameter "transaction_timeout"

Cause: Structure files contain PostgreSQL 17+ features, Docker uses PostgreSQL 15.

Solution: Already fixed in db/structure.sql and db/parser_jobs_structure.sql. If you regenerate these files from a newer PostgreSQL version, remove the SET transaction_timeout = 0; line.

Issue: Redis Connection Refused

Symptom:

Redis::CannotConnectError: Connection refused - connect(2) for 127.0.0.1:6379

Cause: Missing BR_REDIS_HOST environment variable.

Solution: Already configured in docker-compose.yml. If you see this error:

# Verify environment variable is set
docker compose exec rails env | grep BR_REDIS

# If missing, recreate container
docker compose up -d --force-recreate rails

Issue: Rails Can't Connect to Database

Symptom:

connection to server failed: Connection refused
# or
FATAL:  database "xyz" does not exist
# or
FATAL:  password authentication failed

Cause: Database credentials in Rails encrypted credentials don't match your local PostgreSQL setup.

Solution:

Check your local PostgreSQL is running:
```
pg_isready
```
Edit your Rails credentials with correct database info:
```
rails credentials:edit --environment development
```

Verify the credentials match your local PostgreSQL setup:

development:
  database:
    host: localhost  # or your PostgreSQL host
    database: odapi  # your actual database name
    username: your_pg_username

Restart Rails and Sidekiq:
```
docker compose restart rails sidekiq
```

Troubleshooting

Container Won't Start

Check logs:

docker compose logs [service_name]

# Common services to check:
docker compose logs rails
docker compose logs sidekiq
docker compose logs redis
docker compose logs elasticsearch

Check service health:

docker compose ps

# Look for (healthy) status on redis and elasticsearch
# Look for "Up X minutes" on rails and sidekiq

Rebuild from scratch:

# Stop all services
docker compose down

# Remove volumes (⚠️ deletes Redis and Elasticsearch data)
docker volume rm odapi_redis_data odapi_elasticsearch_data

# Rebuild and start
docker compose build --no-cache
docker compose up -d

# Setup databases (PostgreSQL is not in Docker)
docker compose exec rails bundle exec rails db:migrate
docker compose exec rails bundle exec rails db:seed

Database Connection Issues

Verify local PostgreSQL is running:

pg_isready
# or
psql -U your_username -c "SELECT 1"

Check database configuration:

docker compose exec rails rails db:version

Check Rails can connect to database:

docker compose exec rails rails runner "puts ActiveRecord::Base.connection.execute('SELECT 1').first"

Reset database connection:

docker compose restart rails sidekiq

Port Already in Use

If you get "port is already allocated" errors:

# Check what's using the port (e.g., 3000)
lsof -i :3000

# Kill the process or change the port in .env
RAILS_PORT=3001

Out of Memory

Increase Docker memory allocation:

Docker Desktop: Settings → Resources → Memory (increase to 4GB+)
Linux: No limit by default, but check system RAM

Permission Issues

If you encounter permission errors with volumes:

# Fix ownership (Linux/Mac)
sudo chown -R $USER:$USER .

# Or run containers with your user ID
docker compose run --rm -u $(id -u):$(id -g) rails [command]

Clean Everything

Remove all containers, volumes, and images:

docker compose down -v --rmi all
docker system prune -a --volumes

⚠️ Warning: This will delete ALL Docker data, not just ODAPI.

Performance Tips

Development Performance

Use Docker volume caching:
- Already configured in docker-compose.yml with bundle_cache volume

Disable unused services:

# Start only what you need
docker compose up postgres redis rails

Use Docker Compose profiles (coming soon)

Production Performance

Resource limits - Already configured in docker-compose.production.yml
Connection pooling - Configure in config/database.yml
Redis memory management - maxmemory-policy allkeys-lru enabled
Elasticsearch heap - Adjust ES_JAVA_OPTS based on server RAM

Docker Setup Verification Checklist

After running docker compose up -d, verify everything is working:

# 1. Check all services are running and healthy
docker compose ps
# Expected: redis, elasticsearch showing (healthy)
#          rails, sidekiq showing Up

# 2. Check local PostgreSQL is running
pg_isready
# Expected: accepting connections

# 3. Check Rails health endpoint
curl http://localhost:3000/up
# Expected: <body style="background-color: green"></body>

# 4. Check Elasticsearch
curl http://localhost:9200/_cluster/health
# Expected: "status":"green"

# 5. Check Redis
docker compose exec redis redis-cli ping
# Expected: PONG

# 6. Check Rails can connect to database
docker compose exec rails bundle exec rails runner "puts ActiveRecord::Base.connection.execute('SELECT 1').first"
# Expected: {"?column?"=>1}

# 7. Check Sidekiq is processing
docker compose logs sidekiq --tail 20
# Expected: No ERROR messages, should see job processing logs

# 8. Check Rails can connect to all services
docker compose exec rails bundle exec rails runner "puts 'Rails: OK, Redis: ' + Redis.new(url: ENV['BR_REDIS_HOST']).ping"
# Expected: Rails: OK, Redis: PONG

Configuration Files Reference

Files Modified for Docker Compatibility

docker-compose.yml
- Environment variables for Rails and Sidekiq
- Redis config: BR_REDIS_HOST and REDIS_URL
- No PostgreSQL service - uses local database
config/database.yml
- Uses Rails encrypted credentials for database configuration
- Rails.application.credentials.dig(:development, :database, ...)
db/seeds.rb
- Automatically creates sidekiq_cron_schedule setting
- Essential for Sidekiq startup
.env & .env.example
- No database credentials (stored in Rails credentials)
- Redis, Elasticsearch, and INPI API configuration

Environment Variables Used by Docker

Rails Container:

RAILS_ENV=development
BR_REDIS_HOST=redis://redis:6379/0    # For app code
REDIS_URL=redis://redis:6379/0        # For Sidekiq
ELASTICSEARCH_URL=http://elasticsearch:9200
RAILS_LOG_TO_STDOUT=true
INPI_API_URL=
INPI_API_USER=
INPI_API_PASS=

Sidekiq Container:

# Same as Rails, plus:
MALLOC_ARENA_MAX=2              # Memory optimization

Database Configuration: Database credentials are stored in Rails encrypted credentials, NOT environment variables. Edit with:

rails credentials:edit --environment development

Additional Resources

Development Notes

Volume Mounts

The development setup uses volume mounts for hot reloading:

volumes:
  - .:/app                        # Full app directory
  - bundle_cache:/usr/local/bundle  # Cached gems
  - ./log:/app/log                # Logs accessible on host
  - ./tmp:/app/tmp                # Temp files accessible on host

Code changes are reflected immediately without rebuilding. Only gem changes require bundle install or container rebuild.

Gem Installation

After updating Gemfile:

# Option 1: Install in running container
docker compose exec rails bundle install
docker compose restart rails sidekiq

# Option 2: Rebuild container
docker compose up -d --build rails sidekiq

Database Migrations

When creating new migrations:

# Generate migration
docker compose exec rails bundle exec rails generate migration AddFieldToTable

# Run primary database migrations
docker compose exec rails bundle exec rails db:migrate

# Run parser_jobs database migrations (if in db/parser_jobs_migrate/)
docker compose exec rails bundle exec rails db:migrate:parser_jobs

Running Tests in Docker

# Run all tests
docker compose exec rails bundle exec rspec

# Run specific test file
docker compose exec rails bundle exec rspec spec/models/company_spec.rb

# Run with coverage
docker compose exec rails bundle exec rspec --format documentation

# Run linter
docker compose exec rails bundle exec rubocop

Support

For issues specific to ODAPI Docker setup, check:

Known Issues & Solutions section above
GitHub Issues
Project documentation in /docs
CLAUDE.md for development guidelines

Common First-Time Setup Issues

If this is your first time running the Docker setup:

Local PostgreSQL required: Ensure your local PostgreSQL database is running before starting Docker services
Database credentials: Configure Rails credentials with your local database info (rails credentials:edit --environment development)
Fresh database required: Always run db:seed after db:migrate
Wait for health checks: Don't run migrations until Redis and Elasticsearch are healthy
Volume conflicts: If switching from old Docker setup, remove old volumes first
Port conflicts: Make sure ports 3000, 6379, 9200 are not already in use

Table of Contents​

Prerequisites​

Development Setup​

Quick Start​

Development Services​

Development Workflow​

Code Changes​

Debugging​

Architecture & Configuration​

Multi-Database Setup​

Database Configuration​

Redis Configuration​

Environment Variables​

PostgreSQL Version Notes​

Sidekiq Bootstrap Requirement​

Docker Compose Service Dependencies​

Production Setup​

Architecture​

Deployment​

Production Environment Variables​

Common Operations​

Database Operations​

Redis Operations​

Elasticsearch Operations​

Monitoring​

Known Issues & Solutions​

Issue: Sidekiq Crashes on Startup​

Issue: PostgreSQL Version Mismatch​

Issue: Redis Connection Refused​

Issue: Rails Can't Connect to Database​

Troubleshooting​

Container Won't Start​

Database Connection Issues​

Port Already in Use​

Out of Memory​

Permission Issues​

Clean Everything​

Performance Tips​

Development Performance​

Production Performance​

Docker Setup Verification Checklist​

Configuration Files Reference​

Files Modified for Docker Compatibility​

Environment Variables Used by Docker​

Additional Resources​

Development Notes​

Volume Mounts​

Gem Installation​

Database Migrations​

Running Tests in Docker​

Support​

Common First-Time Setup Issues​

Table of Contents

Prerequisites

Development Setup

Quick Start

Development Services

Development Workflow

Code Changes

Debugging

Architecture & Configuration

Multi-Database Setup

Database Configuration

Redis Configuration

Environment Variables

PostgreSQL Version Notes

Sidekiq Bootstrap Requirement

Docker Compose Service Dependencies

Production Setup

Architecture

Deployment

Production Environment Variables

Common Operations

Database Operations

Redis Operations

Elasticsearch Operations

Monitoring

Known Issues & Solutions

Issue: Sidekiq Crashes on Startup

Issue: PostgreSQL Version Mismatch

Issue: Redis Connection Refused

Issue: Rails Can't Connect to Database

Troubleshooting

Container Won't Start

Database Connection Issues

Port Already in Use

Out of Memory

Permission Issues

Clean Everything

Performance Tips

Development Performance

Production Performance

Docker Setup Verification Checklist

Configuration Files Reference

Files Modified for Docker Compatibility

Environment Variables Used by Docker

Additional Resources

Development Notes

Volume Mounts

Gem Installation

Database Migrations

Running Tests in Docker

Support

Common First-Time Setup Issues