Maintaining Referential Integrity - Good or Bad?

Question

We are planning on introducing simple Audit Trail in our database using triggers and separate history table for each table that requires auditing.

For example consider table StudentScore, it has few foreign keys (eg. StudentID, CourseID) linking it to corresponding parent tables (Student & Course).

Table StudentScore (
    StudentScoreID, -- PK
    StudentID ref Student(StudentID),  -- FK to Student
    CourseID ref Course(CourseID),   -- FK to Course
)

If StudentScore requires auditing, we are planning to create audit table StudentScoreHistory -

Table StudentScoreHistory (
    StudentScoreHistoryID, -- PK
    StudentScoreID,
    StudentID,
    CourseID,
    AuditActionCode,
    AuditDateTime,
    AuditActionUserID
)

If any row in StudentScore is modified we'll move old row to StudentScoreHistory.

One of the points raised during design discussion was to make StudentID and CourseID in StudentHistory table a FK, to maintain referential integrity. Argument made in favour of this was as we always mostly do a soft (logical Boolean flag) delete rather than hard delete, its good to maintain referential integrity to ensure we do not have any orphan ids in audit table.

Table StudentScoreHistory (
    StudentScoreHistoryID, -- PK
    StudentScoreID,
    StudentID ref Student(StudentID), -- FK to Student
    CourseID ref Course(CourseID), -- FK to Course
    AuditActionCode,
    AuditDateTime,
    AuditActionUserID
)

This seems to be a bit odd design to me. I do agree with @Jonathan Leffler's comment that audit record should not stop the deletion of parent data. Instead, if this is required, should be handled via foreign keys in main table and not in audit table. I want to get your opinion, to make sure I'm not missing some value in extending foreign keys to audit tables.

Now my question is: Is it a good design to have these foreign keys in History tables?

Any details on key arguments (e.x. performance, best practice, design flexibility etc) would be highly appreciated.

For benefit of anyone looking for specific purpose and our environment:

Purpose:

1. Maintain critical data history
2. Allow auditing of user activity with support to recreate scenario
3. To limited extent allow roll-back of user activity

Environment:

• Transactional database
• Not every table requires auditing
• Uses soft-delete to the extent possible, specifically for static/reference data
• Few highly transactional tables do use hard deletes

Answer 1

If you only plan to do soft deletes as you describe then i see no reason why you shouldn't use foreign keys.

Answer 2

I would suggest not extending foreign keys to audit tables. My recommendation is that the data in an audit be expanded to what the Foreign Key values are.

Instead of storing CourseID as "1", it will be "HTML4". That way if the Foreign Key value is removed, the audit table is still valid. This will also hold true if the Foreign Key value is changed from "HTML4" to "HTML5" anytime in the future. If you only stored the Foreign Key, then you would be telling the auditor that the previous students did "HTML5", which is not correct.

Another great benefit is the ability to ship off the audit trails to another server for data mining without any issues.

I have used the above setup for a while now and it works for me.

Answer 3

If you need to re-create the scenario, then I'd say yes you need the FKs, and having them I think would be an easier way to track through to the relevant related detail records. However, this makes deletions an issue, as well as info that may change in your primary key tables. In this case I'd say you don't want to delete records that have FKs in other tables, but rather use a soft delete as you've already indicated.

As far as info in the PK tables changing, caveat emptor. Setting up FKs would be a simple way to get some traceback ability, but it won't be perfect. There are trade-offs. To get an absolutely perfect history, you basically would need to create backup copies of all related records, any time an audit candidate record has something happen on it. You need to figure out the level of granularity that's appropriate and go with it, because a perfect record of events could be complicated to set up, and eat up a lot of space in the process.

Also, this may or may not be an option for you, but I would strongly consider a combination of tools like ApexSQL Audit + ApexSQL Log as opposed to a homegrown auditing solution. Based on your needs, those two tools combined with archiving your transaction logs periodically would cover what you need to do. The audit tool can store data in the same db or elsewhere, and the log tool can selectively recover data. Just a thought.

Answer 4

When discussing auditing, I would go back to the purpose behind it. It isn't really a backup but rather a history of what has been. For example, for StudentScore, you would want to be sure not to lose the fact that the student originally had a 65% when they now have a 95%. This audit trail would allow you to walk back through the changes to see what happened and who did it. From this, you could identify what a particular user did to abuse the system. In some ways this could be a type of backup since you could roll back these changes to their previous states without rolling back entire tables.

With this in mind (if my assumptions about what you are using this for are correct), the only place you would want a FK/PK relationship is between the history table and its "live" counterpart. Your audit (history) table should not refer to any other table because it is no longer a part of that system. Instead, it is simply a record of what has happened in one table. Period. The only referential integrity you might want to consider is between the history table and the live table (thus the possible FK/PK relationship). If you allow records to be deleted from the live table, don't include the FK in the history table. Then the history table could include deleted records (which is what you want if you allow deletions).

Don't get confused with relational integrity in the main database with this history table. The history tables are all stand-alone. They only serve as a history of one table (not a set of tables).

The relating of two history tables together is possible and even more advanced relations between the live and history tables together (Students and Courses with both live and history, for example) so you can deal with even the possibility that a student was deleted (shudder) since the record would still be in the history table. The only issue here would be if you don't keep the history for a particular table, in which case you are choosing to lose that data (if you allow deletes).

Answer 5

Your milage will obviously vary with the situation, but in my experience, keep referential integrity with the originating table's primary key and no more. This allows to avoid orphan IDs in the history able, while allowing fluid interactions with the related tables.

Suppose, for instance that you've something like this:

table scores (
 score_id,
 student_id ref students (student_id),
 course_id ref courses (course_id),
 score_date,
 score,
 pkey (score_id)
)

In that case, having an on delete cascade fkey referencing scores(score_id) on score_logs makes sense. It's the object; if it gets hard-deleted, might as well discard the history as well.

The foreign keys on student_id and course_id, by contrast, make less sense in my experience. They mean that you cannot do a (hard) delete on students and courses -- even when no live rows that reference them exist. This might be what you want to achieve, in which case ignore the tip. In my case, I find myself in need pruning users, comments, products, orders and so forth; foreign keys in the history logs make this inconvenient.

Also, note that there is a case where fkeys work against you. If you've an order line on an order, and the order line gets deleted, you still want the history on that order line. The correct pkey to use in this case is the order_id, not the order_line_id.

One last note, in case you end up opting to keep the fkeys: consider what they should be pointing to. With decoupled pieces of data (e.g. students and courses), it's reasonable to assume that the live row is fine. With strongly coupled pieces of data, however, (e.g. products and promos) what you'll really want is to be referencing both the fkey and its version.

Re the two previous points, you might find this related thread and answer interesting:

How do you create an audit trail for aggregate roots?

Answer 6

Being in the midsts of implementing very similar auditing system for the first time, I am currently facing that same concern. My opinion echoes that of BiggsTRC - your "live" table maintains the FK relation to the Course record and your history table only maintains relation to its "live" counterpart (StudentScore). This, I think, achieves not having orphans in the audit table.

Now, there is something else I did not see mentioned in the answers: in our current project, we saw value of maintaining a FK in the history table to the CourseHistory table, so that we know, what was the "state" of the Course record at time of the StudentScoreHistory audit entry. Of course that may or may not matter to you, depending on your system logic.

Our solution to your concern (in your answer to BiggsTRC), that you might have the same CourseId several times was to reference not the actual CourseId, but the PK column of CourseHistory table. We still don't have a firm decision how to accomplish this - whether we want to create audit entry of the Course record even if there was not a change, or try to introduce some logic to look-up the CourseHistory record that matches relevant Course state at time of StudentScoreHistory entry.