From Matthew Yglesias:
A few of us at work were talking about why it's adviser and protester but professor and and auditor and after bullshitting around for 10 minutes I thought "maybe I should ask a linguist." Have you ever blogged on this?
I don't think that we have, though you can find well-informed discussions elsewhere, e.g. here or here/here. The executive summary is that -er is (originally) Germanic while -or is (basically) Latin, often via French.
But this doesn't help much with the particular examples you cite, since all four words are from Latin via French. Like most things about English morphology and spelling, the full answer is complicated, and also more geological than logical. But the OED seems to have the whole story — lifted from the depths of the discussion, the key point is that
Many derivatives [formed with -er as an agentive suffix] existed already in Old English, and many more have been added in the later periods of the language. In modern English they may be formed on all vbs., excepting some of those which have [Latin- or French-derived] agent nouns ending in -or, and some others for which this function is served by ns. of different formation (e.g. correspond, correspondent). The distinction between -er and -or as the ending of agent nouns is purely historical and orthographical.
For a (much) longer treatment — you have been warned — press onward.
Let's start with the etymology given for -or, from an OED entry that was updated in 2004. Here's the background:
Partly < classical Latin -ōr-, -or (Old Latin -ōs), suffix of nouns of condition < the same Indo-European base as ancient Greek -ως, and partly < classical Latin -tōr-, -tor, suffix of agent nouns < the same Indo-European base as ancient Greek -τωρ.
The majority of Middle English words with this ending are borrowed via Anglo-Norman, Old French, or Middle French. Latin ō gives in early Old French a sound represented in writing by o or u , hence onor , onur for classical Latin honōr- . In Anglo-Norman this developed to ou , while in Central French eu ultimately became the usual form.
Then we get a bunch of -or subcases, which I'll present in slightly shortened form:
(i) Such nouns of condition now spelt with -or as existed in Middle English, such as error n., horror n., liquor n., pallor n., tenor n.1, were formerly frequently spelt with -our . […]
(ii) There are three varieties of agent noun formation:
(a) Those representing Latin agent nouns in which the agent suffix was not originally preceded by a vowel, as actor n., assessor n., author n., captor n., censor n., confessor n., doctor n., elector n., extensor n., factor n., flexor n., inventor n., lictor n., oppressor n., pastor n., possessor n., professor n., rector n., sculptor n., sponsor n., successor n., transgressor n., tutor n., victor n.1 These are of different ages, going back to Old French words in -or , -ur , Anglo-Norman words in -our , cognate with French –eur , or Latin words in -or . So far as they existed in Middle English, they were then spelt -our ; they are now all conformed to the Latin spelling in -or .
(b) Those representing agent nouns which in classical Latin ended in -ātor , -ētor , -itor , -ītor and which came down in living use into Old French. These terminations were regularly reduced from -ātōr- , etc., to (disyllabic) -eor , -eur , Anglo-Norman –eour , which became in French (monosyllabic) -eur and in Middle English -our , and thus fell together with those from simple -ōr- in (a). Examples are barrator n., conqueror n., donor n., emperor n., governor n., juror n. (To this group also belongs saviour , which has preserved the vowel before -our and is the main word of the group in which -our is kept in British spelling; […]) To these may be added agent nouns formed in French or Anglo-Norman on the verb stem, on the analogy of those in which -eor , -eur , -our , etc., represented classical Latin -ātōr- , etc., as grantor n., purveyor n., surveyor n., tailor n.1, warrior n. From lack of evidence it is sometimes uncertain whether the agent noun was already formed in post-classical Latin in -ator , -itor , or in French after these suffixes had been reduced to -eor and -eur , Anglo-Norman -our .
(c) Those representing agent nouns in -ātor , -ētor , -itor , -ītor , -ūtor , adopted in later times in French, or in English, which retain t , appearing in French as -ateur , -iteur , etc., and have now in English the same written form as in Latin, e.g. administrator n., agitator n., creator n., curator n., dictator n., equator n., gladiator n., imitator n., legislator n., navigator n., spectator n., translator n., vindicator n.; orator n., procurator n.1, senator n.; auditor n., creditor n., editor n., janitor n., monitor n., servitor n., executor n. These are of different ages: some from Old French or Anglo-Norman (in which case they formerly had -our , as creatour , creditour , dictatour , oratour , servitour ); some of later formation immediately from Latin, which have had the -or form from the first.
OK, now onward to -er. The OED warns that "This entry has not yet been fully updated (first published 1891)", but I don't think that the basic picture will have changed.
The basic source is
Middle English -er(e, -ar(e, Old English -ęre (Old Northumbrian often -are), forming ns.
You will probably want to pass over in silence the discussion of the earlier history, but here it it is how it starts:
… represents West Germanic -âri:—Germanic -ârjo-z, whence Old High German -âri (Middle High German -ære, modern German -er), and (with change of declension) ON. -ari (Old Icelandic -are, later -ari, Swedish -are, Danish -ere). The related and functionally equivalent West Germanic -ari (Old Saxon -eri, Dutch -er, Old High German -ari, -eri, Middle High German -ere) = Gothic -areis:—Germanic type -arjo-z (which by phonetic law would prob. have become in Old English -erge, in Old Norse -ri) has in Old English coalesced with this. […]
There are then a number of subcases:
1. In its original use the suffix -ārjo-z was added (like Latin -ārius) to ns., forming derivative ns. with the general sense ‘a man who has to do with (the thing denoted by the primary n.)’, and hence chiefly serving to designate persons according to their profession or occupation; e.g. Gothic dômareis, Old Norse dômari judge, < Germanic *dômo- judgement, doom n.; Gothic bôkareis, Old English bócere scribe, < Germanic *bôk- book n.; Old High German sangâri (modern German sänger), Old Norse sǫngare, Old English sangere (Middle English songere) singer, < Germanic *sangwo- song n. Of this type there are many specially English formations, e.g. hatter, slater, tinner. Where the primary n. ends in -w:—Middle English -ȝe:—Old English -ge, the suffix assumes the form -yer (in Middle English -iere, -yere), as in bowyer, lawyer, sawyer; and, either after the analogy of these or by assimilation to French derivatives in -ier (see -er suffix2), it appears as -ier suffix in certain other words of Middle English date, as brazier, clothier, collier, glazier, grazier, hosier. The English words of this formation not referring to profession or employment are comparatively few: examples are bencher, cottager, outsider, villager. With these may be compared a class of words chiefly belonging to mod. colloquial language, and denoting things or actions, as header, back-hander, fiver, out-and-outer, three-decker. A special use of the suffix, common to the modern Germanic langs. though scarcely to be found in their older stages, is its addition to names of places or countries to express the sense ‘a native of’, ‘a resident in’, e.g. Londoner, New Yorker, Icelander. With similar notion, derivatives in -er have been formed upon certain English adjs. indicating place of origin or residence, as foreigner, northerner, southerner.
Now comes the part that makes everything even more confused than it would otherwise have been, leading in particular to adviser and protester:
2. Most of the ns. which in early Teutonic gave rise to derivatives in -ārjo-z, also gave rise to wk. vbs. in -jan or -ôjan, to which the former stood related in sense as agent nouns; thus Gothic dômareis judge, served as the agent noun to dômjan to judge. Hence, by analogy, the suffix came to be regarded as a formative of agent nouns, and with this function it was added to verbal bases both of the weak and the strong conjugation. Many derivatives of this type existed already in Old English, and many more have been added in the later periods of the language. In modern English they may be formed on all vbs., excepting some of those which have agent nouns ending in -or, and some others for which this function is served by ns. of different formation (e.g. correspond, correspondent). The distinction between -er and -or as the ending of agent nouns is purely historical and orthographical; in the present spoken language they are alike pronounced /ə(r)/ , except that in law terms and in certain Latin words not fully naturalized, -or is still sounded /ɔː(r)/ . In received spelling, the choice between the two forms is often capricious, or determined by other than historical reasons. The agent nouns belonging to vbs. < Latin ppl. stems, and to those formed with -ate suffix1, usually end in -or, being partly adoptions from Latin, and partly assimilated to Latin analogies. But when the sense is purely agentive, without any added notion such as that of office, trade, or profession, function, etc., -er is often used; cf. inspector n., respecter n.; projector, rejecter. In a few instances both forms of the agent noun are still in current use, commonly without any corresponding distinction in sense, as asserter, assertor; sometimes with a distinction of technical and general sense (often however neglected) as accepter, acceptor. The Romanic -our, -or of agent nouns has been in most cases replaced by -er where the related vb. exists in English; exceptions are governor, conjuror (for which -er also occurs); in special sense we have saviour, but in purely agentive sense saver. In liar, beggar, the spelling -ar is a survival of the occasional Middle English variant -ar(e. The agent nouns in -er normally denote personal agents (originally, only male persons, though this restriction is now wholly obsolete); many of them, however, may be used to denote material agents, and hence also mere instruments; e.g. blotter, cutter, poker, roller, etc.
There's more, but I'll spare you.