Building HTTP Servers
20 May 2024
Below are some of the common practices and features which you might want to think about and consider when building web servers running the HTTP protocol:
Routing and Middleware. HTTP servers receive requests from clients. The first step is to route the request to the appropriate handler, based on the request path and method. Sometimes, the request needs to be processed by middleware before it reaches the handler. Common middleware include logging, authentication, and session management.
Session Management. Requests from the same client are often related. A session is a way to keep track of such related requests. When a request is received, the server looks up the session associated with the request (and creates a new session if necessary). The identifier of the session is attached as a cookie in the response, so that the client can send it back in subsequent requests.
User Authentication. Web applications typically provide services that are available only to authenticated users. When a user logs in, a session is created which contains the user's identity. Thus, the server is able to determine whether a request is from an authenticated user by looking up the session, and checking if the user's identity is present.
User Authorisation. Even authenticated users may not have access to all services. Authorisation is the process of determining whether a user has permission to access a particular service. This is typically done by checking the user's role or permissions against the required role or permissions for the service, typically at the handler level.
Template Rendering. Web applications generate HTML pages to be displayed in the user's browser. Templates combine static HTML with dynamic data to produce the final HTML page. Templates may be structured and organised in a way that makes it easy to reuse and maintain them.
Form Processing. Users interact with web applications by submitting forms. When the server receives a form submission, it needs to decode the form data, and validate it. When there is an error, such as a missing or invalid field, the server needs to notify the user by sending back an error message. After the form data is validated, the handler can process the data, such as saving it to a database.
Database Storage. Web applications often need to store and retrieve data from a database. The server interacts with the database by sending SQL queries, through a database driver. Sometimes additional layers are added to abstract the database interaction, such as an ORM or a query builder.
Monitoring and Logging. Web applications need to be monitored for performance, errors, and other metrics. Logging is a way to record server activity, such as requests, responses, and errors. Monitoring and logging helps developers understand how the application is performing, and diagnose problems when they occur.
Error Handling. Web applications need to be able to detect and handle errors. When an error occurs, the server should respond with an appropriate error message, and log the error for further investigation.
Configuration. Web applications often have configuration values that need to be set, such as the database connection string, the server port, or the session store. Configuration values can be set through environment variables, command-line flags, or configuration files.
Testing. Web applications need to be tested to ensure that they work as expected. Tests can be written for individual functions, handlers, or the entire application. Tests can be run automatically to detect regressions and bugs.
Security Practices. Web applications need to be secure to protect user data and prevent attacks. Security practices include using TLS to encrypt data in transit, validating and sanitising user input, preventing SQL injection, XSS, and CSRF attacks, and keeping dependencies up-to-date.
Backup and Recovery. Web applications need to be backed up regularly to prevent data loss. Backups can be stored locally or in the cloud. Recovery procedures should be tested to ensure that data can be restored in case of a disaster.
Performance Optimisation. Web applications need to be optimised for speed and efficiency. Performance optimisation includes reducing latency, minimising resource usage, and caching data. Profiling and benchmarking can help identify bottlenecks and areas for improvement.
Scalability. Web applications need to be able to handle increased traffic and load. Scalability can be achieved by adding more servers, using a load balancer, or caching data. Horizontal scaling involves adding more servers, while vertical scaling involves upgrading the server hardware. At the application level, software architecture and design can be optimised for scalability, such as using well defined APIs and enforcing separation of concerns.
Continuous Integration and Deployment. Web applications need to be built, tested, and deployed automatically. Continuous integration involves automatically building and testing the application whenever changes are committed to the repository. Continuous deployment involves automatically deploying the application to production servers whenever changes are merged to the main branch. This helps ensure that the application is always up-to-date and working correctly.
Versioning. Web applications need to be versioned to maintain compatibility with clients and services. APIs and services should be versioned to allow for changes and updates without breaking existing clients. Versioning can be done through URL paths, headers, or query parameters.
Documentation. Web applications need to be documented for developers and users. Code should be well commented and documented to explain its purpose and usage. APIs and services should be documented to explain their endpoints, parameters, and responses. User documentation should explain how to use the application, and provide troubleshooting and support information.