How can you provide field level access in the admin UI based on the context of the record being displayed?

Suppose you have a task assignment system where any user can create a task (the task owner) and assign it to any other user (the task assignee). A task collection might consist of title, description and isCompleted fields, in addition to the task owner and task assignee.

Using the field access hooks, how would one be able to limit editing of the task title description, and assignee to the task owner, while the task assignee can only update the isCompleted field?

It looks like field access hooks are called one time to determine whether or not a field is enabled in the admin UI, and you can only return true or false without any context of the record being displayed. The only solution I've been able to come up with is to return true in absence of any record context, rendering all fields are editable by any user, with an error only being generated when attempting to Save the record, since that is the only point at which the context of the record is provided to the access hook.

Thanks!