154 lines
5 KiB
Markdown
154 lines
5 KiB
Markdown
# Domain and DNS Management System
|
|
|
|
A FastAPI application that allows users to upload CSV files containing domain and DNS records, storing them in a database and displaying them via an interactive UI and API interface.
|
|
|
|
## Features
|
|
|
|
- File upload system with detailed upload history, including ability to delete uploads
|
|
- Main SLD view with filtering by upload and time
|
|
- Comprehensive DNS Records view with multi-faceted filtering
|
|
- TinyDB database storage with precise timestamps for data persistence
|
|
- Deduplicated DNS records (by domain, class, and type) and domains (by FQDN) showing most recent entries
|
|
- Option to view only records from a specific upload
|
|
- Clean, simplified interface showing only the latest records
|
|
- Server-side filtering by upload, record type, class, TLD, and SLD (no JavaScript needed)
|
|
- Provides comprehensive RESTful API endpoints for programmatic access
|
|
- Responsive design that works on desktop and mobile
|
|
|
|
## Containerized Deployment
|
|
|
|
The application can be easily deployed in a secure rootless container using Podman or Docker.
|
|
|
|
For detailed container instructions, see [CONTAINER_INSTRUCTIONS.md](CONTAINER_INSTRUCTIONS.md).
|
|
|
|
Quick start:
|
|
```bash
|
|
# Make the run script executable
|
|
chmod +x run_container.sh
|
|
|
|
# Run the container (automatically uses podman if available, falls back to docker)
|
|
./run_container.sh
|
|
```
|
|
|
|
## Setup with Virtual Environment (Development)
|
|
|
|
1. Create a virtual environment:
|
|
```
|
|
python -m venv venv
|
|
```
|
|
|
|
2. Activate the virtual environment:
|
|
- On Windows:
|
|
```
|
|
venv\Scripts\activate
|
|
```
|
|
- On macOS/Linux:
|
|
```
|
|
source venv/bin/activate
|
|
```
|
|
|
|
3. Install dependencies:
|
|
```
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
4. Run the application:
|
|
```
|
|
python main.py
|
|
```
|
|
|
|
5. Access the application:
|
|
- Web interface: http://localhost:8000
|
|
- API endpoints:
|
|
- http://localhost:8000/api/slds
|
|
- http://localhost:8000/api/slds/{sld}
|
|
|
|
6. Deactivate the virtual environment when finished:
|
|
```
|
|
deactivate
|
|
```
|
|
|
|
## CSV File Format
|
|
|
|
The application accepts CSV files with the following format:
|
|
|
|
```
|
|
domain.example.com,3600,IN,A,192.168.1.1
|
|
domain.example.com,3600,IN,MX,10 mail.example.com.
|
|
subdomain.example.com,3600,IN,CNAME,domain.example.com.
|
|
```
|
|
|
|
Where columns are:
|
|
1. Domain name (fully qualified domain name)
|
|
2. TTL (Time To Live) in seconds
|
|
3. Record Class (usually IN for Internet)
|
|
4. Record Type (A, AAAA, MX, CNAME, TXT, etc.)
|
|
5. Record Data (IP address, hostname, or other data depending on record type)
|
|
|
|
## Domain Base Name Detection
|
|
|
|
The application includes functionality to identify base domains from fully qualified domain names, including handling of multi-part TLDs like ".co.uk" or ".com.au".
|
|
|
|
### Multi-Part TLD List
|
|
|
|
The application uses a hardcoded list of common multi-part TLDs to correctly extract base domains (e.g., "example.co.uk" from "mail.example.co.uk").
|
|
|
|
This list can be found in `main.py` as `MULTI_PART_TLDS`.
|
|
|
|
### Updating the TLD List
|
|
|
|
To ensure accurate domain parsing, you should periodically update the multi-part TLD list. The best sources for this information are:
|
|
|
|
1. **Public Suffix List (PSL)**: The most comprehensive and authoritative source
|
|
- Website: https://publicsuffix.org/list/
|
|
- GitHub: https://github.com/publicsuffix/list
|
|
- This list is maintained by Mozilla and used by browsers and DNS applications
|
|
|
|
2. **IANA's TLD Database**: The official registry of top-level domains
|
|
- Website: https://www.iana.org/domains/root/db
|
|
|
|
3. **Commercial Domain Registrars**: Often provide lists of available TLDs
|
|
- Examples: GoDaddy, Namecheap, etc.
|
|
|
|
For the most accurate and comprehensive implementation, consider implementing a parser for the Public Suffix List or using a library that maintains this list (e.g., `publicsuffix2` for Python).
|
|
|
|
## API Endpoints
|
|
|
|
- `/api/uploads` - Get all uploads
|
|
- `/api/domains` - Get all domains
|
|
- `/api/base-domains` - Get only unique base domains (e.g., example.com, example.co.uk) with simplified response format
|
|
- `/api/domains/{domain}` - Get domains by name
|
|
- `/api/dns` - Get all DNS records
|
|
- `/api/dns/types` - Get unique values for filters
|
|
|
|
### Query Parameters
|
|
|
|
You can filter the API results using the following query parameters:
|
|
|
|
- `upload_id` - Filter by specific upload
|
|
- `record_type` - Filter by DNS record type
|
|
- `record_class` - Filter by DNS record class
|
|
- `domain` - Search by domain name
|
|
- `base_domains_only` - Only show base domains (e.g., example.com not mail.example.com)
|
|
- `deduplicate` - For DNS records, control whether to show all records or deduplicate
|
|
|
|
Examples:
|
|
- `/api/domains?base_domains_only=true` - Show only base domains
|
|
- `/api/base-domains` - Get a simplified list of unique base domains
|
|
- `/api/dns?record_type=A&domain=example.com&deduplicate=false` - Show all A records for example.com without deduplication
|
|
|
|
### Response Format Examples
|
|
|
|
1. Base Domains Endpoint (`/api/base-domains`):
|
|
```json
|
|
[
|
|
{
|
|
"domain": "example.com",
|
|
"timestamp": "2025-04-08T12:00:00"
|
|
},
|
|
{
|
|
"domain": "example.co.uk",
|
|
"timestamp": "2025-04-08T12:00:00"
|
|
}
|
|
]
|
|
```
|