Contact the author.
OpenDLM Recovery and Hierarchical State Machine Copyright 2004 The OpenDLM Project Author: Ben Cahill (bc), email@example.com 1. Introduction ---------------- This document contains details of the OpenDLM cluster-wide lock state recovery mechanism, which is invoked any time there is a change in membership in the cluster. It includes a discussion of the Hierarchical State Machine (HSM) mechanism used for recovery. This document is aimed at developers, potential developers, students, and anyone who wants to know about the details of cluster recovery in OpenDLM. This document is not intended as a user guide to OpenDLM. Look in the OpenDLM WHATIS-odlm document for an overview of OpenDLM. Look in the OpenDLM HOWTO-install for details of configuring and setting up OpenDLM. This document may contain inaccurate statements, based on the author's limited understanding. Please contact the author (bc) if you see anything wrong or unclear. 1.1. Terminology ---------------- "Client" may mean the Recovery State Machine, which uses (is the only client of) the generic Heirarchical State Machine mechanism described herein. "Client- specific messages" and "client-defined handlers" apply to this definition. Alternatively, "client" may mean a user of OpenDLM (a client process, or a struct client that represents the client process), that is, an application or kernel code that requests locks. Also, see the ogfs-glossary document. 2. Recovery Overview --------------------- Since OpenDLM distributes the storage of lock information among all of the computers in a cluster, this distribution must be adjusted each time a computer enters or leaves the cluster, for whatever reason (boot, graceful shutdown, crash, etc.). Whenever membership changes, each and every active node runs the recovery mechanism. This mechanism re-distributes or deletes resource directory information and lock master information as the membership grows or shrinks. When a node joins the cluster, the recovery mechanism re-distributes any existing directory entries. This is vital for two reasons: -- so that the new node (actually, each node, new or not) carries its fair share of the directories, -- *and* (even more important) so that each directory entry can be found by a querying node. The distribution of the directory entries is based on a formula including a hash function of each resource name, the number of active nodes in the cluster, and the node ID of each computer. When a node needs directory information, it determines the node to query, based on that formula. The directory *must* reside on the proper node, and that node may change as nodes enter/exit the cluster. When a node leaves a cluster, the recovery mechanism re-distributes directory information among the surviving nodes, and may reconstruct directory entries that were lost on the exiting node (if the locks were mastered on surviving nodes), or will simply forget about directories that no surviving node knows or cares about. (is this accurate? -- discuss graceful exit vs crash?) (discuss redistribution of non-directory lock information) 3. Hierarchical State Machine (HSM) ------------------------------------ OpenDLM contains an implementation of a generic hierarchical state machine (HSM). The HSM works by sending "messages" to "state handlers". A state handler may do any or all of the following: -- perform an action, -- send another message to itself, -- request a transition to a new state, The messages may originate from inside the HSM itself (only when transitioning states), or from client-specific code. Client code includes the state handlers, as well as any other code than sends messages to stimulate the behavior of the state machine. A client (e.g. lock state recovery) of the HSM must define: 1) a state tree, 2) handler functions for each state within the tree, and 3) client-specific messages to drive the state machine from the outside Currently, within OpenDLM, the lock state recovery mechanism is the only client of the HSM. For examples while you read about the HSM, see the following: 1) state tree: dlm_recover.h, dlm_recov_states array. Each state definition includes a "parent" or "super" state, a state handler function name, and a state name. 2) state handlers: dlm_recover.c, functions beginning with "clm_st_" or "rc_" are state handler functions. "clm" relates to cluster management, and "rc" relates to lock state recovery. The two are tightly related. 3) messages: dlm_recover.h, an enum that starts with "CLM_MSG_NODE_INIT". For detailed discussion of the OpenDLM Recovery State Machine (the only client of the HSM) see sections 4 and 5 (and their subsections) within this document. 3.1 State Tree Behavior ------------------------ The HSM allows for a given state to have sub-states. The arrangement of states and substates creates a tree, with a single all-encompassing "top" state, and branches hanging down from there. The tree is defined separately by each client of the HSM. The HSM enters states by descending down the tree, and exits states by ascending up the tree. Consider the following tree: TOP / \ A B / | \ 1 2 3 To get from TOP to state 3, the state machine must: -- descend and enter state B, then -- (descend and) enter state 3. To get from state 3 to state 1, it must: -- ascend and exit state 3, then -- enter state 1 by way of the "least common ancestor" (LCA), state B. It never enters nor exits state B during this transition; it is always considered within state B, merely transitioning between two sub-states of state B. To get from state 1 to state A, it must: -- (ascend and) exit state 1, -- exit state B, then -- enter state A by way of the LCA state, TOP. 3.2 State Handlers and HSM Internal Messages --------------------------------------------- As the HSM traverses the tree, it sends "enter" or "exit" messages to the handler functions. These two messages are defined by the HSM, and are not client-specific. In our TOP -> state 3 example, above, the HSM would send "enter" messages to the handlers for states B and 3. In the 3 -> 1 example, it would send "exit" to state 3, and "enter" to state 1. Within many of the state handler functions defined by the recovery client, enter/exit messages are treated as no-ops. See dlm_recover.c, functions that begin with "clm_st_" and "rc_". When the HSM reaches a target/destination state, it sends a "start" message to the state handler. This message is also defined by the HSM, not the client. In the TOP -> 3 example, the HSM would send "start" to state 3 (but not state B, since that was not the target state). Within the recovery state handlers, "start" often results in an action and/or a transition to a new target state. The HSM responds immediately, resulting in more "enter" or "exit" messages while traversing, and a new "start" message to the new target's state handler. This can potentially iterate many times as the HSM and handlers autonomously sequence through a series of states. 3.3 Client-defined Messages ---------------------------- In addition to the HSM-internal enter/exit/start messages, the state machine responds to messages from sources outside the state machine. These are the client-defined messages that drive the HSM to start various autonomous sequences through the client-defined state tree. See section 4 for information on the client-defined messages for the OpenDLM Recovery State Machine. 3.3.1 Parents and Children --------------------------- The hierarchy allows a parent/super state to handle any messages that would be handled identically by some or all of its child states. For example, consider a situation in which 3 children would handle a message a certain way, but one other child would handle the message differently. The parent/super state could handle the message for the 3, each of which would simply ignore the message, causing it to be passed up to the parent, as described below. The one child with the different behavior would process the message in its unique way, without the parent seeing the message. To support the parent/super handling, the HSM executes the following procedure. The HSM attempts to deliver the message to the handler for the current state. However, if that handler doesn't process the message, the handler returns the message to the HSM. The HSM then tries the current state's parent state (super-state) handler, then its parent, iterating to the top of the tree until it finds a handler (the encompassing parent handler) that will process the message (on behalf of one of its children). 3.4 HSM Implementation ---------------------------- The implementation of the HSM is in src/kernel/dlmdk/hsm.c and hsm.h. The HSM engine waits for client-specific messages to be put on a message queue via the hsm_process_event() function. This is one of only two functions that are called by code outside of the HSM or the state handlers. The other public function is hsm_start() (see below). When pulling messages off of the queue, and delivering them to the state handlers, the dialog between the HSM engine and the state handlers is synchronous. This includes any processing done by, and any state changes that are requested by the current state state handler (these result in fairly immediate state changes). If a state handler sends a message to itself, that message gets placed on the incoming message queue by hsm_process_event(). This is an asynchronous operation; that message will get processed by the HSM only after all other messages in front of it have been processed. As an example, this means that if a handler sends "itself" a message, then requests a state change, that the state change will occur *before* the message gets processed. Important functions and macros are described in the next 3 sections: 3.4.1 Public HSM functions --------------------------- Only two calls are used for driving the HSM from the "outside", that is from code not within the HSM engine itself. hsm_process_event() may be called from any client-specific code, whether within state handlers or not. hsm_start() -- Starts the given state machine, putting it into TOP state. hsm_process_event() -- This is the main entry point for driving the state machine from the outside world or from within the state handlers. It adds a new message to the tail of the state machine's event queue, and processes the queue to deliver the messages to the state handlers. It will *not* execute the delivery loop if this or another process/CPU is already in the delivery loop driving the state machine. As an example, a given process will already be in the delivery loop when it receives a message from a state handler (being driven by the delivery loop). The message will eventually get executed, however, because the loop will keep delivering until the queue is empty. In some cases, then, processing of the message may be synchronous (within the hsm_process_event() call), while in other cases it will not be (messages from within the state handlers will always *not* be). Note that there is no indication to the caller as to whether the message was processed before returning to the caller. If the caller needs to know, it must look at global state variables to determine current state. 3.4.2 Private HSM functions ---------------------------- These (relatively) private functions support the dialog between the HSM and the state handlers. The first two of these functions are static within hsm.c. The third is called only from within state handlers (using STATE_TRAN macro, see next section). hsm_process_event_private() -- Sends the external message to the handler for the current state, and if that handler doesn't consume (process) the message, the function iterates up the tree as far as the top level, looking for a consumer of the message. Static within hsm.c hsm_execute_transition() -- Descends downward through the tree to a target state, sending "enter" messages to states it enters, and a "start" message to the target state. It *must* begin with the current state being an ancestor (parent/grandparent/etc) of the target state. Static within hsm.c hsm_exit_private() -- Ascends upward through the tree to a least common ancestor (LCA) of the current state and a target state, sending "exit" messages to the states it leaves. hsm_execute_transition() completes the transition, doing the descending/entering part of the journey. Invoked from within state handlers via STATE_TRAN macro (see next section). 3.4.3 Important macros include: ------------------------------- These macros are invoked from within state handlers to invoke a state transition. One or the other must be used, depending upon whether the path through the state hierarchy tree involves an upward transition (through a least-common-ancestor). STATE_START -- Sets 'next' (target) state. Used by state handlers to set new target state before returning to hsm_execute_transition(). Target must be a child/grandchild/descendent of current state, since hsm_execute_transition() knows only how to *descend* through the tree. STATE_TRAN -- Sets 'next' (target) state, and invokes hsm_exit_private() to *ascend* to the LCA of the current and the new target state, before returning to hsm_execute_transition(). Used for new targets that are *not* child/grandchild/descendent of current state. 4. OpenDLM Recovery State Machine Client-Defined Messages ---------------------------------------------------------- Messages get sent to the state machine via the hsm_process_event() function (see src/kernel/dlmdk/hsm.c), which is called from 4 different locations within OpenDLM (all also within src/kernel/dlmdk directory): -- clm_main.c: clm_master_loop(), based on input from the cluster manager or from the purge comm port -- clm_info.c: handle_new_topology(), when first initializing -- clm_msg.c: clm_sync(), when synchronizing recovery states for all nodes -- dlm_recovery.c: state handlers and other functions auto-sequence the state machine. Also, the PORT_RECOVERY comm callback listens for messages from other nodes. 4.1 Cluster Manager Messages ----------------------------- The clm_master_loop() function (clm_main.c) waits for events from the cluster manager. When one arrives, it calls the following functions to process the event, and eventually place the event on the tail of the dlm_transition_queue, implemented in dlm_sched.c: dlm_process_cm_message() (clm_msg.c) -> handle_new_topology() (clm_info.c) After calling these, clm_master_loop() then calls: new_change_request() (dlm_sched.c) to remove an event from the head of the dlm_transition_queue (may or may not be the one just placed on the queue), and then soon calls: hsm_process_event() (hsm.c) to send the next event message to the recovery state machine. (?? could this long process be simplified ??) The following messages result from cluster management events. Next to each message are shown the function calls that lead to placing the event onto the dlm_transition_queue: CLM_MSG_NODE_DELETE handle_new_topology() -> update_node_status() -> clms_node_delete() CLM_MSG_NODE_ADD handle_new_topology() -> clms_node_add() CLM_MSG_NODE_DOWN handle_new_topology() -> update_node_status() -> clms_node_down() The purge message CLM_MSG_NODE_PURGE also uses the dlm_transition_queue (see other section). Note that CLM_MSG_NODE_UP is not among the cluster event messages; it is generated by the recovery state machine. Another message is driven by handle_new_topology(), when the node is first initializing, without using the dlm_transition_queue. This tells our state machine to query other nodes for directory information: CLM_MSG_NODE_INIT 4.2 Sync Comm Port Message --------------------------- When the cluster is recovering from an exiting node, there are several steps at which all nodes in the cluster must be in sync with one another. The state machine waits for this, and moves along only when it receives: RC_MSG_GOT_BARRIER clm_sync() -> [handle_sync()] -> dlm_sync_notify_recovery() This barrier message is driven from clm_sync() in clm_msg.c. This function sends MSYNC messages, via the PORT_SYNC comm port, to all nodes to drive their state machines to a certain recovery state. When reached, each node lets each other know that the state has been reached, via the PORT_SYNCRET comm port. When our node sees that all nodes have reported the same state, it sends RC_MSG_GOT_BARRIER to our state machine, so it can progress to the next recovery state. 4.3 Purge Comm Port Message ---------------------------- The purge message is driven from pti_read_request(), which monitors the PORT_MSTR_CR comm port. A DLM_PURGELOCKS port message will result in a CLM_MSG_NODE_PURGE message to the recovery state machine: CLM_MSG_NODE_PURGE pti_read_request() -> master_purge() -> clms_node_purge() 4.3 Recovery Comm Port Messages -------------------------------- Many messages are received from the state machines in other nodes, via the PORT_RECOVERY comm port. The callback for that port is clmr_message_handler() (dlm_recovery.c). It forwards the following to our state machine: CLM_MSG_NODE_UP RC_MSG_DIR_RESTORE RC_MSG_DIR_END RC_MSG_DIR_NAK RC_MSG_DIR_NOT_UP RC_MSG_DIR_DATA /* directory query response */ RC_MSG_DIR_REQUEST /* initial directory query */ RC_MSG_RES_REQ_1ST /* new data to merge */ RC_MSG_RES_REQ /* new data to merge */ RC_MSG_RES_DATA /* master handle response */ RC_MSG_RES_RESTART /* new node starting */ 4.4 Recovery State Machine Messages ------------------------------------ Some messages originate within the state machine itself (dlm_recovery.c), and drive itself to perform various actions and sequence into different states. These include: RC_MSG_SEND_MORE RC_MSG_PURGE_CHECK RC_MSG_DIR_RETRY 5. Recovery State Machine Handlers ----------------------------------- The recovery state machine is implemented within src/kernel/dlmdk/dlm_recover.c. The recovery state hierarchy tree is only 3 states deep. Beneath the TOP level, there are 5 "CLM" states, 3 of which have children "RC" states. In addition to the state machine structure (which contains the current state, among other information), there are also a few global variables defined within dlm_recover.c, that describe state. Some simple methods are available in clm_main.c to allow querying these state variables or the current state machine state. The query methods are: -- clm_in_reconfig(): returns state of variable "dlm_recov_in_reconfig", which means we are in CLM_ST_RECONFIG (or one of its sub-states), trying to (re)build the cluster-wide distribution of lock information. -- clm_is_running(): returns TRUE if recovery state machine is in CLM_ST_RUN, which is the normal, non-recovery, operating mode. -- clm_in_delivery(): returns TRUE if recovery state machine is in RC_DIR_INIT, which means that we are gathering directory information from other nodes. -- clm_run_rcv(): returns state of variable "dlm_recov_can_transact", which indicates that we are in a state in which it is is legal to receive transactions (requests for lock operations). There is also a global variable which has no query method: -- grace_period: Indicates that we are in a particular range of sub-states of the CLM_ST_RECONFIG state. Entry into the RC_BARRIER2 sub-state sets this to 1. RC_BARRIER6 sub-state sets it to 0, just before entering CLM_ST_RUN, which also sets it to 0. Used in clm_process() to help determine if we can accept a lock request from a client at the moment. There are also a few static state variables used just within dlm_recovery.c: -- dlm_recov_in_purge: -- dlm_recov_in_add_delete: -- dlm_recov_updates_inflight: Indicates # of The states and messages supported by the Recovery State Machine handlers are described below: 5.1 CLM_ST_TOP --------------- The TOP state is a current state only for a brief instant, when the HSM is entered and started by hsm_start(), when initializing the OpenDLM system (kern_main() -> dlm_recov_init() -> hsm_start()). It transitions immediately to the CLM_ST_INIT state, never again to return to TOP as a current state. Therefore, TOP's main role is to be the highest level ancestor state for all of the other states. It serves as a default catch-all for messages that are not handled by states lower in the hierarchy (see section 3.3). It handles the following: START: Transitions immediately to CLM_ST_INIT. CLM_MSG_NODE_DOWN: Some node is leaving the cluster. All states (except for, currently, CLM_ST_RUN, which treats it the same way and is redundant) allow this message to bubble up to TOP, which (re)sets a few static recovery variables: dlm_recov_in_add_delete = FALSE; dlm_recov_in_purge = FALSE; dlm_recov_leaving_node = node that sent NODE_DOWN message. It then transitions to CLM_ST_DRAIN. In other words, no matter what state we're in, when we get a CLM_MSG_NODE_DOWN we stop what we're doing and go to CLM_ST_DRAIN. CLM_MSG_NODE_UP: Some node is joining the cluster. Almost all states allow this message to bubble up to TOP, which sends a RC_MSG_DIR_NAK back to the sending node, meaning that we're not in a state which can respond to a directory request. The sending node can wait before trying again. The only two states that handle this message directly are: CLM_ST_RUN (the only state that can successfully handle it) and RC_DIR_INIT (which sends a special RC_MSG_DIR_NOT_UP message back to the sending node). default: Any message that has no meaning to the current state or its ancestors causes a warning message to be sent to the system log. Operation of the DLM continues as if the message did not occur. 5.2 CLM_ST_INIT ---------------- CLM_ST_INIT is entered only from TOP, just after hsm_start(). It transitions immediately to its RC_INIT_WAIT sub-state, never to return to CLM_ST_INIT as a current state. This state has 2 leaf-level sub-states, but handles no messages on their behalf, and handles no messages for itself as a current state. It handles only: START: Transition immediately to RC_INIT_WAIT sub-state. 5.2.1 RC_INIT_WAIT ------------------- RC_INIT_WAIT is the first leaf level state to be current when initializing. It is entered only from CLM_ST_INIT, and once it exits, it never returns here again. In this state, the node is just waiting to be told to initialize its directory information. It handles the following messages: CLM_MSG_NODE_INIT: This message is sent by *this* node to itself, when handling the initial cluster topology configuration in handle_new_topology(). If this node is joining a cluster with other active members, it needs to get directory information from its neighbor, so we transition to the RC_DIR_INIT state. If this node is the only active member in the cluster, there's no neighbor from which to get directory info, so we transition directly to CLM_ST_RUN. EXIT: When exiting this state (to either RC_DIR_INIT or CLM_ST_RUN, or any other state such as CLM_ST_DRAIN), the handler initializes the cccp communications module via cccp_init(). 5.2.2 RC_DIR_INIT ------------------ This leaf state is a bit more complex than the ones we've discussed so far. In this state, we are trying to migrate directory information into our node from (one? all?) other nodes within the cluster that we just joined. Some of these nodes may not be ready to give us that information, so there is provision for retrying our requests, or skipping nodes that do not have useful directory information. It handles: ENTRY: It resets the purge logic, finds our closest neighbor in the cluster, and falls through to RC_MSG_DIR_RETRY, to try to get directory info from that neighbor. (?? Other state handlers seem to send a message instead of "falling through". In this case, the message would be RC_MSG_DIR_RETRY. Is there a reason that this state is implemented differently from others? Why is this done on ENTRY instead of START??). RC_MSG_DIR_RETRY: Sends CLM_MSG_NODE_UP to neighbor node, to request directory information migration from that node. This will be a *first*, not a *re*-try, if it fell through from the ENTRY: message. RC_MSG_DIR_RETRY always originates from within *this* node, and from within this state, from the handling of other messages (see below). RC_MSG_DIR_NAK: We receive this from the neighbor node if it is temporarily not in a state in which it can send directory data. We set a timer (3 seconds) and send ourselves RC_MSG_DIR_RETRY, to try the same node again. RC_MSG_DIR_NOT_UP: We receive this from the neighbor node if it is in the same state that we are, trying to get directory info from a neighbor node. Since it's just getting started, we know that it is gathering only the directory information for itself, and won't have any that belongs to us. We skip that node, find another node, and set the 3 second timer to send ourselves RC_MSG_DIR_RETRY. If we've already tried all the nodes(?? -- logic looks incomplete for this??), we give up and enter RUN state. RC_MSG_DIR_END: We receive this from the neighbor node if it has finished sending us directory information. We copy some migration control parameters from it, and enter the RUN state. Apparently, we need directory information from just one other node, since there is no provision to query other nodes once we receive this message(??? Shouldn't we gather dir info from all active nodes??). RC_MSG_DIR_RESTORE: This message contains directory information from another node! We insert the entry into our local directory. CLM_MSG_NODE_UP: We receive this from another node that is joining the cluster. Since we are still trying to initially get together our own directory information, we don't have any directory info that belongs on this new node. We send the RC_MSG_DIR_NOT_UP message, so it can look for another node to query. 5.3 CLM_ST_RUN --------------- This is the normal operating state. Since this is a "recovery" state machine, this state doesn't need to do much except wait for interesting things to happen in terms of cluster membership, then transition to the appropriate state. This is a "leaf" state, with no children. It handles: ENTRY: Just set a few global state variables to indicate that we're running: grace_period = 0; clm_ative = TRUE; dlm_recov_can_transact = TRUE; The call to purge_active() is a no-op for Linux. CLM_MSG_NODE_PURGE: Set a couple of global state variables to indicate that we're in purge mode: dlm_recov_in_add_delete = FALSE; dlm_recov_in_purge = TRUE; Then transition to CLM_ST_DRAIN. CLM_MSG_NODE_ADD: CLM_MSG_NODE_DELETE: Set a couple of global state variables to indicate that we're in add/delete mode: dlm_recov_in_add_delete = TRUE; dlm_recov_in_purge = FALSE; Then transition to CLM_ST_DRAIN. CLM_MSG_NODE_UP: Set the following state variable so we know which node is joining: dlm_recov_joining_node = node that sent NODE_UP message Transition to CLM_ST_SEND_DIR to send directory info to newly joining node. CLM_MSG_NODE_DOWN: Set a few global variables: dlm_recov_in_add_delete = FALSE; dlm_recov_in_purge = FALSE; dlm_recov_leaving_node = node that sent NODE_DOWN message. It then transitions to CLM_ST_DRAIN. This is identical and redundant with NODE_DOWN handling in TOP, and could be removed from the RUN handler. 5.4 CLM_ST_SEND_DIR -------------------- In this state, we're migrating some of our directory data to a node that is joining the cluster. This is a "leaf" state, with no children. It handles: ENTRY: We're getting ready to send our directory information to a newly joining node, so we reset the enumerator that will walk through our directory database, so it will start at the beginning of the database. We then send an RC_MSG_SEND_MORE to ourself, to effectively fall through into ... RC_MSG_SEND_MORE: We send directory entries to the newly joining node. Using rl_enum_counted() to walk through the directory hash table, we send the entries from up to 32 hash table slots, using clmr_send_dir_entry(). This function looks for those directory entries that belong in the newly joining node, based on the resource names and the new number of nodes, using clm_direct_dirsend() -> hashfunc(). See these functions to understand how directory entries are distributed throughout the cluster. ??? Once the 32 slots are done, how do we (or the other node) ask for more??? 5.5 CLM_ST_DRAIN ----------------- We transition to the DRAIN state when a node leaves the cluster. After a member node dies, this state (and its 2 leaf sub-states), clean out old, unusable information (about the dead node) that is still on this node. We can arrive here from several states, for several reasons. To define each case, the following global variables must be set appropriately in the previous state, before transitioning to DRAIN: dlm_recov_in_add_delete -- TRUE if here because of CLM_MSG_NODE_ADD or CLM_MSG_NODE_DELETE message (see RUN state handler). * dlm_recov_in_purge -- TRUE if here because of CLM_MSG_NODE_PURGE message (see RUN state handler). * * If here because of CLM_MSG_NODE_DOWN (see TOP state handler), both the dlm_recov_add_delete and dlm_recov_purge globals are FALSE. dlm_recov_leaving_node -- indicates which node is leaving the cluster. (??? it doesn't seem to get set for NODE_ADD or NODE_DELETE ???) This state handles the following messages: START: We transition immediately to RC_BARRIER1. EXIT: We set the global variable dlm_recov_can_transact to FALSE. This means that this node is not able to accept transaction requests from itself or other nodes; it is entering a reconfiguration state (RC_BARRIER1). 5.5.1 RC_BARRIER1 ------------------ This is the first of a series of "barrier" states. A barrier state is one that must be achieved by *all* nodes in the cluster, before we can move on to a next state. This state begins the process of recovery, after a node has died. This, like all of the other "RC_" states, is a leaf state. It handles the following messages: ENTRY: This does the real work of this state. As the first phase of a reconfiguration, it does the following on *this* node: -- reset_inflight() marks any "inflight" requests (requests that have been sent to the dead node, but not acknowledged by the dead node) to try them again after reconfiguration is complete (and the dead node is running again). -- clmr_purge_res() removes any locks, held by dead nodes, that are on any of the queues (grant/convert/wait) of the resources on our resource list. -- clmr_dir_purge_chk() (driven by rl_enum() iterator that goes through the directory entries on our node) removes any directory entries that point to the dead node(s). -- clm_sync() waits until *all* nodes finish the above 3 jobs. We'll receive RC_MSG_GOT_BARRIER when all nodes are done. RC_MSG_GOT_BARRIER: This message comes from within our own node (see dlm_sync_notify_recovery()), after our node determines that all active nodes have reached a common point in the recovery sequence. We simply transition to RC_PURGE_WAIT. 5.5.2 RC_PURGE_WAIT -------------------- This, like all of the other "RC_" states, is a leaf state. It handles the following messages: ENTRY: This state does the following on *this* node: -- reconfig_free_client_purge() (driven via client_apply() iterator on each OpenDLM client app on this node) continues any purge activity on this node that was blocked because a migration was in progress (??). Or, it starts a purge process that had not started because we were in recovery when it was requested. -- sends an RC_MSG_PURGE_CHECK message to ourself, to continue on. RC_MSG_PURGE_CHECK: Received from ourself. Does the following: -- reconfig_check_client_purge() (driven via client_apply() iterator on each OpenDLM client on this node) checks to make sure that all purging operations on this node are complete. -- If so, transition to RC_BARRIER2 (substate of CLM_ST_RECONFIG). ?? If not, there seems to be no way to continue on ???? There is no other source for RC_MSG_PURGE_CHECK to trigger another check!?! 5.6 CLM_ST_RECONFIG -------------------- This state has the largest number (7) of sub-states, and is used to (re-) distribute resource (including locks on the resource) and directory information across the cluster after cluster membership changes. If a node joins the cluster, then we arrive here via ?? If a node leaves the cluster, then we arrive here via the CLM_ST_DRAIN state (actually via its sub-states RC_BARRIER1 and RC_PURGE_WAIT), which cleans out unusable information about the dead node. In CLM_ST_RECONFIG (and substates), we then re-distribute information about the surviving nodes. CLM_ST_RECONFIG is never a target state (i.e. never a final "current" state). However, it may be entered or exited when the state machine is transitioning to or from its sub-states. It also serves as a "catch-all" handler for several messages, on behalf of its sub-states. It handles the following transitional messages from the HSM: ENTRY: Sets "dlm_recov_in_reconfig" global state variable to TRUE. EXIT: Sets "dlm_recov_in_reconfig" global state variable to FALSE. START: Makes sure that dlm_recov_in_reconfig = TRUE. Transitions immediately to RC_BARRIER2. These are the only places that the dlm_recov_in_reconfig variable is set or reset. It handles the following client-specific messages on behalf of its sub-states. In each case, this is the only place in which the message is handled. That is, each of the messages are valid for all of the sub-states of CLM_ST_RECONFIG, and are passed up to here. Conversely, they are invalid for any other states (e.g. CLM_ST_DRAIN) in the recovery state machine, and would be handled by TOP's default warning message. The following are handled here: RC_MSG_DIR_REQUEST: This message comes from another node's clmr_res() function, and is received on this node via the PORT_RECOVERY comm port. -- handle_dir_request(), dlm_recovery.c, makes sure that we have an active directory entry (i.e. we are the directory node for this resource), and sends a RC_MSG_DIR_DATA message back to the other node. This tells the other node which node is the (perhaps new) master node for the resource (see immediately below). RC_MSG_DIR_DATA: This message comes from another node's handle_dir_request() function, and is received on this node via the PORT_RECOVERY comm port. This message is a response from the (perhaps new) directory node back to us, confirming that it has (perhaps created) a valid directory entry for the resource, and telling us which node is the (perhaps new) master. We do the following for each resource structure on our node, depending on whether we are the (perhaps new) master of the resource: If master: -- clmr_clean(), dlm_recovery.c, scans through all locks on our (master) copy of the resource, and removes any locks that were held by dead nodes. It also invalidates (clears) the lock value block (LVB) for any resources that had exclusive (EX) or protected write (PW) locks held by a dead node. Such a node might have changed the LVB contents without us knowing about it. If not master: -- clmr_rebuild_send(), dlm_recovery.c, cleans out our (secondary) copy of the resource, removing any locks that are not held on this node; as a secondary copy, we don't need to know about any other nodes' locks. Then, it sends either a RC_MSG_RES_REQ or a RC_MSG_RES_REQ_1ST message, via PORT_RECOVERY, to the the master node, so the master node can know what we know about our own locks, and integrate it into the master copy of the resource. These messages are handled by either state RC_BARRIER4, or RC_RESOURCE_QUERY. The master node will respond with RC_MSG_RES_DATA (see immediately below). RC_MSG_RES_DATA: This message comes from another node's clmr_respond() function, called from their handle_resource_request() function after receiving an RC_MSG_RES_REQ(_1ST) (see immediately above) from us. It is received on this node via the PORT_RECOVERY comm port. We copy the master node's resource handle, and the node id of the master node, into our copy of the resource. 5.6.1 RC_BARRIER2 -------------------- This barrier state waits for all other nodes to get to barrier state 2 (meaning what??). This, like all of the other "RC_" states, is a leaf state. It handles the following messages: ENTRY: Sets two global variables: grace_period = 1; clm_active = TRUE; Then waits for all nodes to achieve barrier state 2. Once achieved, we receive the RC_MSG_GOT_BARRIER message, see below. RC_MSG_GOT_BARRIER: This message comes from within our own node (see dlm_sync_notify_recovery()), after our node determines that all active nodes have reached barrier state 2. We branch in two different directions, depending on whether we are in "dlm_recov_in_purge" mode: If in purge, we simply transition to RC_BARRIER6. If not in purge, we simply transition to RC_REBUILD_DIR. 5.6.2 RC_REBUILD_DIR -------------------- This state does a complete rebuild of directory information throughout the cluster, using information in each node's resource table. In this state, we use info from resource master copies, not non-master copies. They get used later, in the RC_RESOURCE_QUERY state. This state steps through all of the resource structures stored on this node, and sends the information about each resource to that resource's directory node. That directory node may be a new node, building a directory from scratch, or it may be a an old member node that is becoming the directory node for a new set of resources, due to the change in cluster membership. ENTRY: Several steps are executed here: -- reconfig_del_dead_client(), clm_client.c, removes any knowledge that we have on *this* node about clients on dead nodes, by removing those clients from our "cli_refs" list. Then, if we are in "dlm_recov_in_add_delete" mode: -- clm_change_topology(), clm_info.c, reorganizes our node list to contain only the new cluster membership. -- clmr_dir_move_check(), clm_recover.c, (called via rl_enum(), clm_rldb.c, for each directory entry on our node) deletes any active directory (but not cache) entries for resources for which we just were, but no longer are, the directory node. That is, it deletes directory entries that are migrating away from us due to the change in cluster membership. Then, regardless of mode: -- resource_table_iterator_create(), clm_resource.c, creates an iterator for going through the resource table on this node. -- send an RC_MSG_SEND_MORE message to ourself to do the next job. RC_MSG_SEND_MORE: We use the iterator to go through our node's list of resources, calling: -- clmr_dir(), clm_resource.c. If we are the current resource master, we have the most up-to-date resource information. We send the info to the resource's (perhaps new) directory node, via update_directory(). update_directory() sends the info within a rl_update_t structure, and delivers it via PORT_DIRUPDT. The info includes: resource name, name length, type (UNIX vs VMS) flags master node pointer to our copy of the resource (for use after ack) The port listener function for PORT_DIRUPDT is dir_read_update(), in clm_direct.c. On the directory node, it either modifies a pre-existing directory entry, or creates a new one, using the info sent above. We keep track of the number of updates "in flight" (sent to directory node, but not yet acknowledged by directory node) via global variable "dlm_recov_updates_inflight", which we increment each time we invoke update_directory(). The counter is decremented via dlm_recov_dir_msg_retire(), called after we get an acknowledgement. If we are not the master, we don't send info, but take the opportunity to make sure that our copy of the resource contains only locks held by this node in its queues (that's all that a local copy should contain). Then, we check "dlm_recov_updates_inflight" to see if all of our directory updates have been sent and retired. If so, we transistion to RC_BARRIER3. If not, we must get called again with RC_MSG_SEND_MORE, which is done by dlm_recov_dir_msg_retire() once the number of inflight updates is below a certain threshold (dlm_buff_limit = 500). ?? Something looks odd with this mechanism ... the iterator goes through all of the resources in one shot, so why would it need to be re-stimulated?? Then, we get hundreds of RC_MSG_SEND_MORE messages before the final one gets us to change state?? Even though the messages are to ourself, that's still a lot of pretty useless messages?? EXIT: Destroy the iterator created during ENTRY. 5.6.3 RC_BARRIER3 -------------------- This barrier state waits for all other nodes to get to barrier state 3 (meaning what??). This, like all of the other "RC_" states, is a leaf state. It handles the following messages: ENTRY: Waits for all nodes to achieve barrier state 3. Once achieved, we receive the RC_MSG_GOT_BARRIER message, see below. RC_MSG_GOT_BARRIER: This message comes from within our own node (see dlm_sync_notify_recovery()), after our node determines that all active nodes have reached barrier state 3. We simply transition to RC_RESOURCE_QUERY. 5.6.4 RC_RESOURCE_QUERY -------------------- ENTRY: Sets one global variable: dlm_recov_can_del_dir = TRUE; Then creates a resource table iterator, and sends an RC_MSG_SEND_MORE to fall through to do the job below. RC_MSG_SEND_MORE: Does the following: -- resend_updates(), clm_direct.c, handles any requests made to our node to delete directory entries, while we were unable to. We put them on a holding stack to be processed now. -- clmr_res(), dlm_recover.c, gets called for each resource known by our node, and does one of two things, depending on whether we are the master for the resource: master: clean out locks, held by dead nodes, from our resource's lock queues (grant/convert/wait) not master: send RC_MSG_DIR_REQUEST to the directory node. This tells the directory node to make sure that it has a directory entry. Also, the directory node sends back mastership info, so that we don't need to consult the directory again for this resource. As in RC_REBUILD_DIR state, we keep track of the number of updates "in flight" via global variable "dlm_recov_updates_inflight". After we're done iterating through the resources, we check for that to be 0, then transition to RC_BARRIER4. RC_MSG_RES_REQ: RC_MSG_RES_REQ_1ST: We are the master node for a certain resource. Another node is sending us (pushing it; we didn't request it, see CLM_ST_RECONFIG state, RC_MSG_DIR_DATA message) information about locks it holds on the resource. We call: -- handle_resource_request(), dlm_recover.c, integrates that info into our master copy of the resource structure, and sends the other node a copy of our resource handle (cluster-wide, smaller-than-the-resource-name-and-type resource identifier), using RC_MSG_RES_DATA. See CLM_ST_RECONFIG state, RC_MSG_RES_DATA message. Note that RC_MSG_RES_REQ(_1ST) is handled identically by the RC_BARRIER4 state. EXIT: Destroys the iterator created during ENTRY. 5.6.5 RC_BARRIER4 -------------------- This barrier state waits for all other nodes to get to barrier state 4 (meaning what??). This, like all of the other "RC_" states, is a leaf state. It handles the following messages: ENTRY: Waits for all nodes to achieve barrier state 4. Once achieved, we receive the RC_MSG_GOT_BARRIER message, see below. RC_MSG_RES_REQ: RC_MSG_RES_REQ_1ST: We are the master node for a certain resource. Another node is sending us (pushing it; we didn't request it, see CLM_ST_RECONFIG state, RC_MSG_DIR_DATA message) information about locks it holds on the resource. We call: -- handle_resource_request(), dlm_recover.c, integrates that info into our master copy of the resource structure, and sends the other node a copy of our resource handle (cluster-wide, smaller-than-the-resource-name-and-type resource identifier), using RC_MSG_RES_DATA. See CLM_ST_RECONFIG state, RC_MSG_RES_DATA message. Note that RC_MSG_RES_REQ(_1ST) is handled identically by RC_RESOURCE_QUERY state. RC_MSG_GOT_BARRIER: This message comes from within our own node (see dlm_sync_notify_recovery()), after our node determines that all active nodes have reached barrier state 4. We check resources that have been reassigned to this node as master, after their previous master node died, via: -- clmr_lkvlb(), dlm_recover.c, clears stale locks (marked by LOCK_STALE flag) from each queue (grant/convert/wait). If we are the new master, we have lost information from the dead node that previously was the master. We invalidate the lock value block (LVB) if there was a possibility that the dead master might have changed it, that is, if some dead node owned an exclusive (EX) or protected writed (PW) on the lock. We can't know that for sure, so we check for any locks that *we* know about that are >= concurrent write (CW). If there are any, they would have kept any dead nodes from owning a lock that allowed them to write the LVB. But if not, we invalidate the LVB, to be safe. We then transition to RC_BARRIER5. 5.6.6 RC_BARRIER5 -------------------- This barrier state waits for all other nodes to get to barrier state 5 (meaning what??). This, like all of the other "RC_" states, is a leaf state. It handles the following messages: ENTRY: Waits for all nodes to achieve barrier state 5. Once achieved, we receive the RC_MSG_GOT_BARRIER message, see below. RC_MSG_GOT_BARRIER: This message comes from within our own node (see dlm_sync_notify_recovery()), after our node determines that all active nodes have reached barrier state 5. We remove temporary directories that were created for locallocks during reconfiguration (when??), via: -- clmr_rm_locallock_dirs(), dlm_recover.c, which deletes a directory if the RLINFO_LOCAL flag is set We then transition to RC_BARRIER6. 5.6.7 RC_BARRIER6 -------------------- This barrier state waits for all other nodes to get to barrier state 6 (meaning what??). This, like all of the other "RC_" states, is a leaf state. It handles the following messages: ENTRY: Waits for all nodes to achieve barrier state 6. Once achieved, we receive the RC_MSG_GOT_BARRIER message, see below. RC_MSG_GOT_BARRIER: This message comes from within our own node (see dlm_sync_notify_recovery()), after our node determines that all active nodes have reached barrier state 6. This is the last stage of recovery, just before going into the RUN state. We do the following: -- wake_all_locks(), clm_clmlocks.c, looks through all resources for which we are (now perhaps new) master, and grant any locks that were blocked by dead nodes. -- pti_flush_resend(), clm_batch.c, resends messages that were destined for a dead node, but failed to transmit successfully. These messages were placed on our "resend" queue. If the original destination is alive again, we send it there, otherwise look for the appropriate new directory (or master?) node. -- resend_inflight(), clm_batch.c, resends messages that were destined for a dead node, reached that node successfully (just before it died), but we never received a response to the request in the message. These messages were placed on our "inflight" queue. As with resend queue, we try again sending to the original node if it is reachable, or look for the appropriate new directory node if not. -- reconfig_allow_client_purge(), clm_purge.c, does purges that were delayed by the reconfiguration. -- reset "grace_period" global state variable to 0. -- clm_change_topology(), clm_info.c, to make sure that we're set up for the latest cluster membership. We then transition to CLM_ST_RUN. 6 Purging ---------- "Purging" is the cluster-wide removal of now-useless lock information. There are several possible reasons for needing a purge: Client died -- A client app on a node died (PURGE_DEAD). PURGE_DEAD flag tells us to remove our knowledge of the client, in addition to our knowledge of the client's locks. Client request -- A client app requests (how?) a purge of its own locks, even though it is still alive (PURGE_ME). Purge orphan locks -- Orphan locks survive, even though the client or node, from which they were requested, dies. This is a special type of lock used during a client recovery process. Once recovery is complete, the orphan locks must be removed. Node died -- An entire node died A purge operation is requested via a "transaction" structure, just like other inter-node operations, that gets passed around within and between nodes. Within the transaction structure, there is a request-specific data area (a union). For purge requests, the area contains a struct "purgereq" (see clmstructs.h).