High-level changes to Perspectives server / Network Notary code


Format:
+ new feature
* bug or behavior fix

We attempt to use meaningful version numbers, following semver.org:
	Given a version number MAJOR.MINOR.PATCH, increment the:
	MAJOR version when you make large architectural changes,
	MINOR version when you add functionality in a backwards-compatible manner
	PATCH version when you make backwards-compatible bug fixes.


3.5
-----
+ Add --logfile switch for scanner and server to handle logging to a file on disk.
	This lets us limit how much disk space is used for logging, so it doesn't run out of control.
* Change default admin scripts to use new --logfile switch
* Clean up admin scripts
* Fix multiple sources of logging, printing, and traceback to use logging module
* Fix print statements to use print function
* Fix threading bug with scanner: properly lock objects when recording scan results
* Organize unit tests: use one test module for each code module being tested
	This makes it much easier to see what is being tested.
* Convert old-style classes to new-style classes
* Add missing license header
* Determine whether sockets block based on OS (nonblocking sockets don't work on Windows)
* Remove some global variables and state from threaded_scanner
* Fix list_services code to be testable, and add minimal tests
* Minor code formatting and whitespace cleanup
* Add some docstrings and documentation


3.4.1
-----
* Fix HTML in static index page
* Move CSS to its own file, rather than inline, so Content Security Policies can be properly applied.
* Convert README and Amazon guide to markdown format
* Update Amazon guide
* Add document explaining the notary API
* Add basic guide on using nginx as a cache and proxy
* Fix 'upgrades' directory so steps are more clear
* Clean up unused function parameters


3.4
---
+ Add --cache-duration switch to allow control of cache expiry
* Lower default cache expiry to 12 hours to fit with notary defaults
+ Add switches to expose control of socket_queue_size and thread_pool, to allow performance tuning
* Convert scanner modules to use standard python logging (for better control)
+ Add --quiet mode for scans
* Add --screen-echo argument alias
+ Add --cache-only argument, to avoid reading from the database
* Add error message when connecting to the database fails with a blank password
* Fix formatting for creating database check constraints
* Organize and explain argument customization in admin scripts
+ Document performance tuning and cache duration behaviour
+ Update amazon guide:
	+ Add steps for verifying host ssh fingerprint
	+ Add requirements and notes on avoiding surprise costs
	+ Add section on using a Relational Database Service (RDS)


3.3.1
-----
* Properly catch and log known exceptions during scans
* Decipher and log SSL Alert records found during scans
* Log service name during scan errors, so we can see what needs fixing
* Log timeout values during scan timeout errors
* Add check for non-numeric scan port
* Fix exception type checing in threaded_scanner
* Always create log directory when starting notary
* Fix CSS parsing error
+ Document functions in built-in scanner module
+ Document intended version numbering scheme
+ Add document on how to prepare releases
+ Add document summarizing privacy and security steps (for easy review)
* Rename AUTHORS file to CREDITS, to reflect that not every contribution comes from writing code


3.3
-----
* Disable logging of request headers in newer versions of CherryPy
* Send all notary and scanner logs to the /logs directory (one central place)
* Fix error counts of socket errors while scanning
* Catch exceptions when reporting observations and committing database records
+ Add bash scripts to help administer notaries under unix environments
	(Dan's original shell scripts from the psv-admin depo updated for the current version)
	* Update admin scripts so they can be run from any location
		(they calculate their own path before running)
	* Update scripts to share common variables and functions
	* Calculate paths for crontab jobs automatically
		(then it doesn't matter where the notary software is located on the drive)
+ Add context manager for direct database connections
* Convert listing and counting queries to use raw db connections rather than ORM sessions (this is much faster when we don't need to work with database objects)
* Add thread locks for database connection counting
* Fix db2file.py to print records in a loop rather than joining them all together (uses far less RAM)
* Improve documentation:
	* Improve docs on upgrading notaries (both from 2->3.2 and 3.1->3.2)
	+ Add guide on running a notary hosted on Amazon Web Services (AWS)
		(Dan's original v2 guide updated for the current version)
	* Add sqlalchemy to the installation instructions
	* Explain all of the directories in this depo inside the README


3.2
---
+ Add support for caching data with local system memory only (--pycache).
	This makes it easy to cache data and improve notary performance even if you are unable to use a dedicated caching server.
+ Add default URL parameters so 'service_type' and 'port' are not required.
* Ignore requests with an empty 'host' string (raise HTTP 400 Bad Request)
* Ignore requests with additional, invalid parameters (raise HTTP 400 Bad Request. Thanks Angel!)
* Raise HTTP 400 for invalid service types rather than HTTP 404 (we'll never find any records, so don't give clients the impression that they should requery)
* Convert hardcoded SNI setting to command-line argument, so it's easier to toggle (Thanks Carl A.!)
* Turn off logging of request headers during errors and exceptions; we do not want to record any private or sensitive client information.
* Completely remove logging of access messages, even when log messages are sent to stdout.
* Fix bug with incorrect Primary Key on Observations.
	It's perfectly reasonable that we might see the same certificate used on the same site in non-contiguous blocks of time. Allow such changes to be stored in the database. (this has only been a bug since v3.0)
* Fix bug: place a cap on when we update an observation end time (GH 23).
* Similarly, for updating an observation with the same key: create a new observation instead of updating the existing one if a large period of time has passed.
* Remove ServiceScanKeyUpdated metric types (they're not needed)
+ Add data validation to database and code for Observation records
+ Add 'NOT NULL' to database fields that shouldn't allow null
+ Track the number of open database connections so we can see if we're leaking any
* Refactor database function report_observation() into the ndb class
+ Add some automated unit tests to help with refactoring and testing (more are always welcome!)
* Fix several 'raise' statements to properly re-raise exceptions
* Stop generating Traceback for socket exceptions when scanning; simply log the error and continue.
	This *significantly* improves speed and response times for scans that didn't work
	(and we're not losing any information).
* Change to use contextmanager for database sessions (i.e. 'with ndb.get_session():' blocks).
	This ensures session scope is obvious and properly cleaned up after use.
* Pass the existing db instance to on-demand scan threads. This way we don't waste time re-connecting, and connections are properly created and disposed.
* Use a semaphore and lock to rate-limit on-demand scans, to properly handle multithreading
	We now only scan a given site once at a time (i.e. do not launch multiple scans for the same site if one is already in progress).
+ Add documentation and script for upgrading from notary version 2.x to version 3.x.
* Add --verbose switch to scanner to generate less log data.


3.1
---
+ Add support for caching of service data with memcached, memcachier, or redis, to reduce database load.
+ Add metrics tables to allow tracking of useful performance statistics. See doc/metrics.txt for a detailed explanation.
+ Display whether a notary tracks performance metrics on the static index page
+ Add --dburl switch so db connection info can be easily passed all at once
* Fix bug where the notary server would use too many database connections (it now properly closes sessions and connections).
+ Attempt to fail gracefully in the face of catastrophic database errors
	- Attempt to fall back to the cache if we can't get data from the database
	(e.g. if there are too many connections or something went terribly wrong)
	- Fall back to printing and then ignoring metrics if they can't be written to the database
	- Serve a HTTP 503 code if we can't get data, rather than crashing
+ Allow notary public/private keys to be read from environment variables.
	Could be a security trade-off, but makes it easier to run notaries on some distributed systems.
+ Allow notary public/private keys to be exported as heroku config vars.
	Other systems could be supported as needed.
* Fix bug: actually use the webport argument (thanks mwgamera!)


3.0
---
Requires python 2.7.
+ Separate the database interface from the notary code.
    This lets us change the type of database much more easily,
    and gives us a central way to organize other modules that communicate with the database.
+ Use SQLAlchemy as a database ORM, to make database abstraction and maintenance even easier (and cleaner).
+ Refactor database tables to be 3rd Normal Form.
* Explicitly add a Primary Key of (service_id, key) for Observations.
	This was semi-implied before, as keys could not be null, and it doesn't make sense to have a null service_id.
	SQLAlchemy requires tables to have a Primary Key anyway.
* Fix bug when reporting observations that would attempt to update the previous key's end time when there was no previous key.
	(this was harmless, but saves us running an extra query that returns no results)
+ Clearly separate utility scripts and modules that do *not* rely on the notary database (in 'utils/') from those that do (in 'notary_utils/').
+ Separate client modules into client/
* Refactor standalone modules so they can be run by themselves or when imported by another module.
    Database functionality was removed from modules in util/ so they comply with the "Do One Thing Well" philosophy.
+ Implement clean argument parsing for all modules using argparse.
+ Add documentation and docstrings to many things that needed it
+ Explain the tasks a network notary server performs in the README
+ Add default arguments and actions wherever possible, to make it easier to set up and run a network notary server without having to read (or understand ;) anything.
+ Add --write-config-file and --read-config-file switches so all modules that connect to the database can share settings more easily.
+ Add a static index page to explain what a 'network notary' server is to visitors.
+ Add AUTHORS and CHANGELOG files


2.0
---
Implementation in python.
Requires python 2.5.
+ Calculate signatures for each service inside the webserver as needed. This makes scanning lighter weight, at the cost of making requests heavy-weight and subject to DoS.


1.0
---
Original implementation in C.
Signatures for each service's data are calculated by a separate tool outside of the webserver each time the data changes.
